Tsinghua University and CUHK researchers have open-sourced AKV, a serving system designed to manage KV cache in LLM agents. Its central idea is simple: instead of summarizing old history, an agent can drop selected cache entries and keep reusing the rest.
Across three models and four agent benchmarks, AKV reduced average cost by 27.7% while raising the average task score by 1.7 points. The system builds on SGLang, runs on NVIDIA GPUs, and its code uses the Apache-2.0 license. It includes a serving system and policy library, but no model weights.
A different way to manage long agent histories
LLM agents build long conversations as they work through research, coding, and tool-use tasks. A common approach, called context engineering, trims or summarizes the text history. That can reduce the amount of text the model processes, but editing the prefix invalidates every later KV state, so the retained suffix must be prefilled again.
AKV takes a different route. Cache engineering leaves the text trajectory untouched and changes only which KV entries remain visible to attention. Retained entries can be reused without recomputation, and they can still carry information from dropped messages. The research paper’s cost analysis found that KV cache reads made up 82.4% of total API cost, giving cache management a large role in an agent’s overall expense.
AKV defines three cache operations. Append handles new tokens as a standard update. Drop removes selected entries from future attention without changing their positions. Repos rewrites retained keys to positions 0 through r-1 using a RoPE rotation, while values stay unchanged, so hidden states do not need to be recomputed.
That last operation matters in the benchmark results. In ablations, adding Repos raised GPT-OSS-120B’s BrowseComp-Plus pass rate from 55% to 62%. The result shows why AKV does more than simply delete old cache entries: it can compact the remaining entries while preserving the information they contain.
The serving system behind the cache operations
The hard part is serving. Identical tokens can produce different cache states under different operation histories, so plain prefix matching could reuse the wrong states. AKV extends SGLang’s radix tree with drop and reposition nodes, then frees dropped KV only when no active request still needs it. Segment-batch attention restores evicted states in one prefill pass.
The interface is stateless and OpenAI-compatible. Clients send the full message history along with two optional fields: drop_message, which maps a trigger message index to messages that should be dropped, and reposition, which lists the indices after which positions should be compacted.
A CachePolicy builds those fields from a trigger, a selector, and a reposition rule. Built-in policies are turn-based, token-based, or model-based. A CacheTool also lets the model call drop_context itself, while the main policy uses a tool-response rolling drop that keeps the latest 12 tool responses.
The comparisons use Standard and Text Engineering as baselines. Standard summarizes at 96K tokens, or 192K for MiroThinker-1.7-Mini, while Text Engineering summarizes at 48K. AKV cuts cost by 21.9% compared with Text Engineering, and its cost figures are API-equivalent estimates based on published rates. Repositioned tokens are charged at the cache-hit rate, which gives the comparison a conservative cost estimate.
Lower cost, faster serving, stronger benchmark results
The evaluation covered BrowseComp-Plus, GAIA-Text, SWE-bench Verified, and xbench-DeepSearch-2510 across GPT-OSS-120B, MiniMax-M2.7, and MiroThinker-1.7-Mini. On BrowseComp-Plus at 96K tokens and above, GPT-OSS-120B scored 37% with AKV, compared with 24% for Standard and 32% for Text Engineering.
Across the broader results, average cost fell from 20.0 to 9.7 cents per task, while the average score rose from 40.0 to 43.2. AKV’s BrowseComp-Plus score reached 58.9, which was 3.4 points below Text Engineering’s 62.3 in that comparison. The overall results still showed a 27.7% lower average cost and a 1.7-point average score gain across the tested models and benchmarks.
Serving tests ran GPT-OSS-120B on 80 long-context SWE-bench Verified tasks. At each method’s peak setting, AKV delivered 1.18x output throughput and 1.53x input throughput, while P95 request latency was 1.51x faster. At concurrency 16, Standard’s P95 latency reached 229.90 seconds, compared with 30.63 seconds for AKV.
KV pressure pushed Standard into repeated re-prefill. With AKV, full-attention time per decode step fell from 7.082 milliseconds to 2.849 milliseconds. Under a GPU budget, AKV reached about a 47% pass rate on 100 SWE-bench Verified tasks, compared with 40.7% for Standard and 32.3% for Text Engineering.
MiniMax-M2.7 refined policies on 20 BrowseComp-Plus tasks before running 100 held-out tasks. Its iteration-6 policy cut cost by 17.8% against Standard while keeping baseline accuracy. Together, these results present AKV as a way to manage agent memory without changing the conversation itself: the system changes which cached information attention can see, then reuses the rest instead of paying to rebuild it.
Based on
- Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang — marktechpost.com
- Designing an AI serving harness built to last | MIT Technology Review — technologyreview.com
- Agentic AI Foundation Announces Four New Projects: Agent Skills, Open Responses, OpenEnv, and OpenSandbox | AP News — apnews.com
- Uncertainty-aware offline reinforcement learning for LLM inference serving with heterogeneous SLOs | Scientific Reports — nature.com
- Researchers made AI agents better by turning academic papers into working code — no retraining required | VentureBeat — venturebeat.com
