Recent experiments with large language models (LLMs) show that while these tools are powerful, they are also designed with safety features that prevent them from helping with dangerous tasks. Researchers have tested models like GPT-5.2, GPT-5.3, Opus 4.6, and Sonnet 4.6 by asking them to assist in building a nuclear weapon. Unsurprisingly, all of these models refused to provide any helpful information. The knowledge to build a nuclear device is publicly available and well-documented, but the models are programmed to avoid unsafe topics, just like many other AI systems that restrict discussions on sensitive issues.

Can LLMs Be Bypassed or Compromised?

The main goal for some researchers isn’t to build weapons but to understand the limitations of these safety measures. They want to see if they can open up a sandbox environment where the models can be prompted to do more than they are supposed to. This includes tasks like writing files outside their containers, enumerating privileged access tokens, or assessing security vulnerabilities. The problem is that current safety features are designed to prevent these actions, but they can often be bypassed through complex prompt injections or other methods.

Many of the leading companies behind these models, including Anthropic and OpenAI, emphasize safety as a core feature. However, critics argue that this safety “theater” can sometimes hinder legitimate testing and research. They believe that the way safety is enforced may be more about liability and avoiding misuse than actually preventing all harmful use. When models refuse to answer questions about sensitive topics, it raises questions about whether they are truly safe or just heavily guarded against certain prompts.

The Dark Side of AI Safety and Abolition Techniques

Some researchers have looked into methods to bypass safety restrictions. For example, they found that certain models labeled as “abliterated” have had their refusal mechanisms removed. A technique called “ablation” involves using a model’s own activations to strip away safety layers. By doing this, the model can potentially answer questions it would normally refuse, including those related to dangerous activities.

In practice, this means that some “abliterated” models can provide information on sensitive topics or help with tasks that are typically off-limits. For instance, one researcher discovered a model called Qwen 3 Next Abliterated that, after removal of safety features, could offer tips on building nuclear weapons or hacking. While these models might not be perfect at performing complex tool calls, they demonstrate how safety measures can be bypassed with enough effort. This raises concerns about the effectiveness of current safety protocols in AI systems.

Overall, these experiments highlight a key issue: safety features are not foolproof. As models become more advanced and accessible, the risk of misuse grows. Developers and users need to understand that these tools, while helpful, can be manipulated if proper safeguards are not continuously improved. The debate continues on how to balance open research with responsible AI deployment to prevent dangerous misuse while allowing innovation.

Inspired by

• https://www.infoworld.com/article/4141328/an-llm-that-will-help-you-build-a-nuclear-weapon.html

Sources

• mit.edu
• huggingface.co
• dphn.ai
• huggingface.co
• tomshardware.com