As AI-generated data becomes more common, organizations need to rethink how they trust and handle their information. A new report from Gartner highlights the growing risks associated with AI outputs and how traditional data management methods may no longer be enough. With many companies increasing their AI investments, the need for stronger data controls is more urgent than ever.

The Rise of AI and Its Risks

More businesses are adopting generative AI tools, with a recent survey showing that 84% plan to spend more on AI this year. However, this rapid growth comes with challenges. Large language models often train on outputs from earlier models, which can lead to a cycle of unverified and potentially faulty data. This phenomenon, known as model collapse, could undermine the accuracy and trustworthiness of AI systems.

To prevent this, Gartner recommends companies implement new governance strategies. This includes appointing a dedicated AI governance leader who works closely with data and analytics teams. Encouraging collaboration across departments—such as cybersecurity, data management, and AI teams—is also vital. Updating existing policies to address the unique risks posed by AI-generated data is another essential step.

Moving Toward a Zero-Trust Approach

Gartner predicts that by 2028, half of all organizations will need to adopt a zero-trust approach to data governance. This means companies can no longer assume that data is trustworthy just because it appears to be human-generated. Instead, they must verify and authenticate data at every step to protect business outcomes and financial stability.

Wan Fui Chan, a Gartner vice president, emphasizes the importance of this shift. He explains that AI-generated data is becoming so widespread and similar to human-created data that traditional trust models are no longer effective. Establishing strict verification measures will be crucial for safeguarding data and preventing errors or misuse.

The challenge is complicated by different government policies around AI. Some regions are pushing for tighter controls, requiring companies to implement stricter data verification processes. Others may adopt more flexible regulations, which can create a patchwork of standards that organizations must navigate carefully.

Real-World Examples of AI Data Issues

One notable example of AI causing data governance problems came from Deloitte Australia. The company had to refund part of a government contract after their AI-generated report contained errors, including nonexistent legal citations. This incident highlights the risks of relying too heavily on AI outputs without proper oversight.

Such cases underline the importance of verifying AI-produced data before it is used in decision-making or shared externally. As AI tools become more advanced and widespread, organizations need robust processes to catch errors and ensure data integrity at every stage.

In summary, as AI continues to evolve, so must the ways companies govern their data. Moving toward a zero-trust model can help organizations stay ahead of potential pitfalls and protect their operations from the risks of unverified AI outputs.

Inspired by

• https://www.infoworld.com/article/4122179/zero-trust-data-governance-needed-to-protect-ai-models-from-slop-2.html

Sources

• gartner.com
• gartner.com
• computerworld.com
• cio.com