More than 850 well-known figures have signed an open letter warning about the risks of developing superintelligent AI. They believe such AI could “break the operating system of human civilization.” The letter, released by the Future of Life Institute, describes superintelligence as AI that “significantly outperform all humans on essentially all cognitive tasks.” This means future systems could make strategic decisions on their own, rewrite their own code, and operate without human oversight.

The signatories include a wide range of influential people. They range from AI pioneers like Geoffrey Hinton and Yoshua Bengio to Nobel laureates, tech founders like Steve Wozniak, and former government officials such as Susan Rice. This diverse group suggests that concerns over AI are crossing political lines and becoming a broader policy issue.

Yuval Noah Harari, a well-known author and professor, added a personal note to the letter. He said superintelligence could “break the very operating system of human civilization” and noted that it’s not necessary. Instead, he advocates focusing on building controllable AI tools that help people today. This way, AI’s benefits can be harnessed safely and reliably.

Interestingly, the list of signatories doesn’t include current leaders of major AI firms like OpenAI, Google, Meta, or Microsoft. This highlights a divide in the industry. Some companies are investing heavily in AI development, even in superintelligence. For example, Meta CEO Mark Zuckerberg launched Meta Superintelligence Labs in June after investing $14.3 billion in Scale AI. Similarly, OpenAI’s Sam Altman announced a shift toward developing superintelligent systems.

This isn’t the first time the Future of Life Institute has called for caution. In March 2023, they urged a six-month pause on developing AI systems more powerful than GPT-4. That appeal gathered over 30,000 signatures, but AI companies largely ignored it.

For most businesses today, superintelligence remains a distant concern. Experts say it’s more of a theoretical risk than an immediate threat to enterprise IT. Sanchit Vir Gogia, CEO at Greyhound Research, explains that superintelligence is a long-term issue, not something to worry about within the next few years. He emphasizes that CIOs should focus on current AI challenges like data governance, model explainability, and validation.

The letter warns that superintelligence could do more than just disrupt jobs. It raises concerns about the “alignment problem”—making sure AI systems pursue goals compatible with human values. Current methods work for today’s AI, but they might fall short when systems become smarter than us, according to IBM research on superalignment.

Right now, AI is already transforming workplaces. A recent report from Indeed found that about 26% of recent job postings could be affected by generative AI. Fields like technology and finance face the highest risks. Goldman Sachs estimates that if AI is expanded across the economy, around 2.5% of US jobs could be displaced.

Meanwhile, companies are racing ahead with AI. Salesforce, for example, cut customer support roles from 9,000 to about 5,000 due to AI improvements. This shows how AI is already changing employment landscapes.

A potential ban on superintelligence development could shift the competitive balance among AI companies. If strict regulations slow down the creation of superintelligent systems, companies will likely turn to safer, more controllable AI models. Gogia points out that businesses prefer “adequate-but-verifiable” models, especially in regulated sectors. Smaller language models and enterprise-focused AI stacks with clear controls would become more popular.

Economically, a ban could slow down the booming AI market. Gartner predicts global AI spending will hit nearly $1.5 trillion in 2025 and surpass $2 trillion in 2026. AI is expected to boost global GDP by about 0.5% annually from 2025 to 2030. But if the US bans superintelligence, China might accelerate its own AI efforts. Despite US export restrictions, Chinese companies like DeepSeek and Alibaba are making big strides with open-source models that challenge Silicon Valley.

The absence of global rules on AI development places responsibility squarely on individual companies. Gogia advises CIOs to set up internal AI governance, including ethical guidelines, incident response protocols, and contracts that demand transparency and audit rights from vendors. These steps can help ensure AI development aligns with societal values and legal standards.

In the end, while superintelligence is still a future concern, today’s AI already influences workplaces and economies. Responsible development and regulation are key to harnessing AI’s benefits without risking unintended consequences. Companies that stay proactive and ethical in their AI strategies will be better prepared for what’s ahead.

Inspired by

• https://www.computerworld.com/article/4077074/call-to-ban-ai-superintelligence-could-redraw-the-global-tech-race-between-the-us-and-china.html

Sources

• gmpg.org
• futureoflife.org
• superintelligence-statement.org
• ibm.com
• hiringlab.org