Recently, a government contract in Australia highlighted big issues with how organizations are managing artificial intelligence. Deloitte Australia used AI to help prepare a lengthy report for the Department of Employment and Workplace Relations. But when the report was reviewed, it turned out to contain fake citations and references to non-existent court cases. This mistake forced Deloitte to refund part of its AU$440,000 contract. The incident shows that as companies adopt AI faster, their governance systems aren’t keeping up.

AI Mistakes in a Government Report

Deloitte used OpenAI’s GPT-4o to generate a 237-page independent review for the Australian government. The firm didn’t flag or disclose its AI use until after errors were found. The government confirmed that some footnotes and references in the report were incorrect, leading Deloitte to agree to refund the final installment of the contract. The government released a corrected version of the report and a statement of assurance. This case reveals how rapidly AI is being integrated into serious work without enough checks and balances.

How Experts Spot Fabrications

Dr. Christopher Rudge, a researcher at the University of Sydney, was able to identify the fabrications. He knew the authors cited in the report personally and recognized that the cited works did not exist. He noticed that the references seemed too perfect and tailored, which was a red flag. His expertise showed how domain knowledge remains critical in catching AI-generated errors, especially when AI tools can produce convincing but false information.

Lessons on Oversight and Responsibility

Industry analysts agree that this incident highlights the need for better governance when using AI. Sam Higgins from Forrester pointed out that the rapid adoption of generative AI is outpacing organizations’ ability to manage risks. Both vendors like Deloitte and their clients share responsibility for quality control. Sanchit Vir Gogia from Greyhound Research emphasized that clients can’t be surprised when AI tools they use internally appear in vendor work without proper checks. Transparency is key, but often companies only disclose AI use after mistakes happen, which undermines trust.

Improving Contracts and Checks

Experts say that organizations should require human experts to review AI-generated work before it’s finalized. Even with AI making things cheaper and faster, a human “sense check” can catch errors that AI might miss. Rudge notes that including subject-matter experts in the review process should become standard practice. Companies should ask clear questions about what AI tools are used, how hallucinations are detected, and whether humans are involved in validation. This helps prevent mistakes and ensures accountability.

Building Better AI Governance Frameworks

Beyond just vetting individual projects, organizations need comprehensive policies to manage AI risks. CIOs and procurement teams should include clauses in vendor contracts that require disclosure of AI use, quality checks, and liability for errors. Industry standards like the NIST AI Risk Management Framework or ISO/IEC 42001 can guide organizations in creating these policies. Experts recommend establishing joint review boards with both client and vendor representatives to examine AI-produced content before it’s approved. This collaborative approach is what maturity in AI governance looks like.

In summary, the Deloitte case serves as a wake-up call. As AI tools become more embedded in high-stakes work, organizations must develop stronger oversight, transparency, and accountability measures. Proper governance will be essential to prevent mistakes, protect reputation, and ensure AI’s benefits are realized responsibly.

Inspired by

• https://www.computerworld.com/article/4069521/deloittes-ai-governance-failure-exposes-critical-gap-in-enterprise-quality-controls.html

Sources

• gmpg.org
• gov.au
• anthropic.com
• podigee.com
• podigee.com