Open-Weight Models, AI Benchmarks, and Disney’s AI Tool Shift

Supabase just open sourced a new tool called Supabase Evals. It tests AI coding agents like Claude Code, Codex, and OpenCode on real tasks. The tool scores their performance and powers a public leaderboard. It also runs daily checks inside Supabase’s systems.

Supabase Evals is free to use and runs locally with simple commands. It tests AI agents across different products, topics, and development stages. The tests split into two groups: benchmark scenarios and regression scenarios. Some agents, like Opus 5 and Kimi K3, scored 100% without any extra skills in certain tests.

This framework helps developers see how well AI can write code that works on actual platforms. One developer explained the challenge: AI must not only write code quickly but also make it fit the real system. The code has to use the right tools, match data formats, and produce something an engineer can review later.

Security Issues Persist in AI-Generated Code

Security remains a big problem for AI-generated code. A new report shows only 56% of AI code passes security checks. This rate hasn’t changed since last year. AI models usually write code with nearly perfect syntax, but close to half fail on security tasks without help.

OpenAI’s GPT-5.5 leads with a 68% security pass rate. Six other models score between 50% and 53%. Alibaba’s Qwen3.7-max scores 50%. Models built just for coding don’t perform better on security than general-purpose ones. Their average is around 51-52%.

Model size doesn’t affect security much. Large models average 53%, medium 51%, and small 51%. Models trained for reasoning do better, scoring 56%, versus 51% for others. Java code has the lowest security pass rate at 30%, though it is improving.

A security expert said AI code delivery raises new risks. The solution isn’t to block AI tools but to scan and fix all AI-generated code before use. He urged treating AI code like any unreviewed code.

New DataFlow-Harness Framework Boosts Workflow Accuracy

Researchers from Peking University and other Chinese institutes created DataFlow-Harness. It guides large language models to build clear, visual workflows for data processing. The system runs workflows as directed acyclic graphs (DAGs) and updates them incrementally.

On a 12-task benchmark, DataFlow-Harness passed 93.3% of tests. This is within one percentage point of the best known result. It cuts API costs by over 70% and cuts response times nearly in half compared to using Claude Code alone.

It was tested on real business tasks like question answering and schema normalization. The framework also helped build better data pipelines for training AI models, improving data quality. It scored 97.2% precision and 87.3% coverage on a task transforming textbooks into visual question-answer data.

Open-Weight Models Gain Ground Amid Industry Debate

Moonshot recently released the open weights for its Kimi K3 AI model. This means anyone with a powerful computer and internet can download and tweak the model. Open-weight models share their “recipe,” letting users customize or run them without fees beyond server costs.

Chinese AI company DeepSeek also made headlines for releasing open-weight models. Their approach allows low-cost development and strong performance. OpenRouter’s leaderboard favors open-weight models, with many top Chinese models listed.

Most U.S. companies still use proprietary models like OpenAI’s. These models are not downloadable and come with built-in guardrails. But big tech players like Nvidia, Meta, Palantir, and Microsoft support open weights. They oppose restrictions on sharing models.

AI leaders say open-weight models that don’t have dangerous features are a public good. Still, these models can be misused, as they lack kill switches and other controls. Researchers showed open weights could launch malware but also help contain threats.

Experts warn that restricting open models risks concentrating power in a few hands. They suggest auditing open models for security issues instead. The debate is ongoing, balancing openness with safety.

Disney’s AI Tool Shuffle Reflects Real-World Challenges

Disney plans to stop using Microsoft’s GitHub Copilot by August. Instead, it will switch to OpenAI’s Codex for coding help. Some Disney employees rarely used Copilot, citing its complexity and cost structure as issues.

Disney will keep access to Anthropic’s Claude Enterprise and Cursor tools. But it will drop Amazon’s Kiro and Q tools. Disney developers prefer Claude and Cursor, some using them tens of thousands of times daily.

One Disney AI staffer said the main goal is to speed up coding output. The company tracks AI tool use carefully and aims to raise coding velocity while managing tool quality.

Stay connected via Google News

Follow us for the latest AI News