AI Benchmarks Reveal Enterprise Java Migration Is Still a Mess

Enterprise Java modernization remains a brutal slog. Companies spend years migrating frameworks to improve maintainability, cloud readiness, and developer productivity. Yet, success is rare and expensive.

ScarfBench exposes how far AI agents fall short on these migrations. This open benchmark tests AI on cross-framework migration tasks using 34 applications, 102 framework versions, and 204 migration scenarios. It covers about 151,000 lines of code and more than 1,300 expert-written tests.

Current AI agents fail spectacularly. They achieve less than 10% behavioral success, meaning they rarely get the migrated application working as intended. Build success rates are higher, followed by deploy success, but passing behavioral validation is the real challenge. Framework semantics require translation—not just source code rewrites.

Jakarta EE migrations prove especially tough. The difficulty varies mainly by target framework. Agents also overestimate their success. Many report successful builds that actually fail when tested. Migration isn’t a straight line either. Changes ripple through configuration, web, database, and service layers in iterative cycles.

Java versions 8, 11, 17, and 21 have support windows ending between 2029 and 2032. Most organizations already wrestle with these timelines. Migration takes time—usually 32 to 44 weeks for just the initial roadmap phases. Security risks, compliance pressure, financial overhead, delivery bottlenecks, and talent shortages push companies to modernize. But wholesale rewrites fail more often than incremental moves.

Deep insight into existing codebases is critical. Manual discovery is slow and error-prone. AI-native tools promise a better way by ingesting code, building structured models, and revealing system intent. Still, failures mostly come from flawed harnesses—systems that manage context, workflows, and model calls. Poor orchestration causes missing context and ambiguous tasks.

GitHub’s Copilot harness stands out. It beats many vendor harnesses in task success and token efficiency. Public AI benchmarks like SWE-bench, TerminalBench, SkillsBench, and Win-Hill measure varied agent skills, but none nail enterprise migration challenges like ScarfBench.

Meanwhile, Legacy Squad, an open-source CLI tool, scans legacy Java/Spring Boot backends to generate structured modernization plans. Tested on production systems, it uncovered 20 findings, including authentication bypasses, non-expiring tokens, and hidden business rules. Legacy Squad produces detailed diagnostics: refactor specs, design documents, and modernization master plans. It runs locally with token-efficient context packs, never sending full repos to large language models.

Nassir Khan nails it: “Java estates of this scale don’t get modernized by moving fast. They get modernized by moving with full context and knowing what the systems actually do before deciding what to change.”

Legacy modernization is a marathon, not a sprint. Teams waste weeks chasing outdated plans and arguing priorities without real evidence, as one engineer put it. AI tools offer hope, but the journey remains treacherous. Framework migration demands deep comprehension, iterative fixes, and orchestration mastery—skills AI agents have yet to fully master.