Canberra's War on Duplicate Digital Images: How the Capital Stacks Up Against Global Peers

The National Archives of Australia has been quietly working through one of the more unglamorous problems in modern records management: tens of thousands of duplicate digital images clogging federal storage systems, driving up cloud costs and making it harder for researchers to find authoritative versions of government photographs and scanned documents. The problem is not unique to Canberra, but the capital's concentration of federal agencies on a single relatively compact footprint — from the Department of Finance in Barton to the Australian Bureau of Statistics headquarters in Belconnen — gives it both unusual exposure to the issue and an unusual opportunity to coordinate a response.

Why does this matter now? Several factors converged in the first half of 2026. The ACT government's ongoing digital transformation push, which includes the transition of multiple directorates to shared cloud platforms, has forced agencies to confront archive duplication that built up over more than a decade of siloed storage. At the same time, the Bureau of Meteorology's expanded imagery datasets — relevant after Sydney recorded its hottest June since 1859 — have put fresh pressure on federal data infrastructure. Storage is not free. Industry benchmarks suggest enterprise cloud image storage in Australia runs at roughly $23 to $35 per terabyte per month depending on the provider and retrieval tier, and agencies running unaudited archives can carry duplication rates of 30 percent or higher across unmanaged repositories.

What Canberra Is Actually Doing

The Australian National University's Digital Humanities Hub in Acton has been piloting a deduplication workflow since March 2026 using perceptual hashing — a technique that identifies visually identical or near-identical images even when file names and metadata differ. The program targets ANU's own research image collections, which span decades of fieldwork photography, but the methodology is being watched closely by the National Library of Australia on Parkes Place, which manages the PANDORA web archive and faces its own version of the same headache with captured screenshots and page renders that repeat across crawl cycles.

The ACT's approach sits somewhere between what Wellington and Ottawa have done. New Zealand's Department of Internal Affairs completed a whole-of-government digital asset deduplication audit in late 2024 and published its findings, reporting it recovered storage equivalent to roughly 4.2 petabytes across 23 agencies. Canada's Treasury Board Secretariat has mandated deduplication tooling as part of its GC Cloud framework since 2023. Canberra's effort remains more fragmented — agency by agency rather than mandated centrally — though the Digital Transformation Agency has flagged the issue in its 2025-26 work program documentation.

Edinburgh offers a cautionary tale. The National Records of Scotland ran a deduplication project in 2022 that surfaced an unexpected problem: automated tools deleted images flagged as duplicates that were, on closer inspection, distinct versions carrying different metadata or minor visual differences significant to archivists. The lesson landed hard in Australian library circles. The National Library's digital preservation team has pointed to the Scottish experience as a reason to keep human review loops in any automated pipeline, slowing throughput but protecting integrity.

What Researchers and Agencies Should Do Now

For ANU postgraduate researchers storing fieldwork imagery on university systems, the practical advice from the Digital Humanities Hub is to implement consistent file naming and embed EXIF metadata at the point of capture rather than trying to reconstruct it later. Gungahlin and Belconnen community organisations that have digitised local history collections and lodged them with ACT Heritage Library in Civic should request a holdings audit — the library's digitisation standards document, last updated in February 2025, includes guidance on file version control that many smaller depositors have not yet applied.

Federal public servants in the APS agencies clustered around London Circuit and Constitution Avenue face a more structural fix: the Digital Transformation Agency's cloud procurement panel, updated in January 2026, now includes provisions requiring agencies to document deduplication practices as part of cloud migration plans. Whether compliance is being actively checked is a different question. For now, the capital is moving — just not yet in lockstep.