Running a large language model locally is no longer just a contest between model sizes. The files, quantization method, tool-calling behavior, and software wrapped around the model can all change the result. Two recent evaluations put that tradeoff under a microscope, from a 7.9 GB Qwen3.8-27B build to coding agents that spend three times more without solving more problems.
Underdog Saluki 27B is a 7.9 GB build of Qwen3.8-27B. Its makers say the model keeps tool calling intact at about two bits per weight and even edges out the full 54 GB model on their own 120-task benchmark. That claim makes Saluki interesting for anyone who wants a capable local model without storing the full model file.
How the Saluki comparison is set up
Saluki is built on ISTA-DASLab’s GSQ-RCO quantization of the same model. The comparison places both files side by side in stock llama.cpp instead of relying on a special runtime. That matters because the goal is to separate changes in the model files from changes caused by the software serving them.
The process starts with the files themselves. A tensor-by-tensor diff of the two GGUF headers checks how their structures compare, followed by a byte-level comparison to show whether Saluki is a remix of ISTA’s bits or a changed model. The evaluation then serves each file on one GPU and records memory use and speed.
Tool calling gets its own test. Both files face the same seeded 120 tasks from the Berkeley Function Calling Leaderboard, with results scored through BFCL’s own checker under strict and lenient parsing. That setup gives the smaller model a direct comparison against the ISTA-DASLab build rather than treating a single benchmark score as the whole story.
The result is a useful reminder about compressed models: a smaller file is not valuable only because it saves storage. It also needs to preserve the behavior that makes the underlying model useful. In this case, the key behavior is tool calling, tested under the same tasks, checker, and serving software.
The harness can change the bill more than the score
A separate study from Shandong University examined a different part of the AI stack: the harness around a coding model. A harness includes the system prompt, tools, and rules for managing context. The researchers found that these choices can triple the cost of solving coding problems without producing a measurable improvement in accuracy.
They compared Claude Code, OpenCode, and mini-SWE-agent across nearly 500 software engineering tasks while keeping the underlying models fixed. mini-SWE-agent is a minimalist open-source harness written in about 100 lines of Python, exposing a single bash tool and performing no context compaction.
The evaluation covered 492 offline tasks from SWE-bench Verified. It included the 45 hardest tasks, which take humans one to four-plus hours, along with the remaining 447 tasks. The models included locally served Qwen3.6-35B-A3B and Qwen3.8-27B, plus API-served DeepSeek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 inside Claude Code served as a top-tier anchor.
On the 447-task pool, Claude Code and mini-SWE-agent scored within two percentage points of each other. The results were statistically equivalent within a five-percentage-point margin on both Qwen models, even though one harness was far more elaborate.
The hardest tasks showed the same pattern. Switching harnesses changed which problems passed or failed on 13% of tasks, or about one out of every eight. Rerunning the same harness produced the same median switch rate. The researchers summarized the result plainly: “A vendor’s own harness on its own model buys no score advantage.”
OpenCode was the exception, trailing by up to 9 points on the large task pool. On Qwen3.8-27B, about half that gap was linked to runs that stopped when the model reached the 32,000-token output limit mid-step, often before it could write a code patch. Claude Code hit the same cap but prompted the model to continue.
Why harness design drives cost
The largest difference appeared in the first API call. Claude Code sent 16,581 tokens of fixed context through its system prompt and tool schemas, compared with 7,025 in OpenCode and 829 in mini-SWE-agent. Because the conversation transcript is sent again on every turn, that starting overhead grows with each step.
“What separates the harnesses’ per-step weight is their own text, not the repository’s,” the researchers explain. Even though 96% to 99% of input tokens in these loops hit the provider’s prompt cache, different cache prices changed the bill without changing which harness cost the most for a given model.
On GLM-5.3-Flash, a Claude Code trial cost nearly twice as much as a mini-SWE-agent trial and over three times as much as an OpenCode trial. OpenCode made fewer model calls, while all three harnesses returned roughly the same amount of text from tool calls.
The ranking also depended on the model. On DeepSeek-V4-Flash, mini-SWE-agent made 141 model calls per trial compared with Claude Code’s 104 and cost slightly more: 15.2 cents per trial versus 14.6 cents. The researchers wrote, “With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses.”
These findings match the HarnessTax study from UC Berkeley’s Sky Computing Lab and Arena. That work tested seven models across Claude Code, OpenAI’s Codex CLI, and Pi, a minimal open-source harness with four basic tools. Across 30 tasks from each of two benchmarks, repeated three times, harness choice shifted pass rates by only a few points while multiplying inference costs up to fivefold.
What developers should measure
Small benchmark gaps need careful handling. One cited study found that 10 reruns of the same setup on SWE-bench Verified produced score spreads from 2.2 to 6.0 percentage points. Another comparison found the same model scoring 19.1% with a minimal benchmark adapter and 73.4% with the full adapter.
Network access can distort results too. Researchers found that 312 of 439 DeepSeek and Qwen trials, or 71%, had retrieved the original human-written fix from GitHub. They discarded those runs and blocked network access in the final evaluation.
The practical advice is direct: measure task success and cost per task together, then rerun the baseline before adopting a more complex scaffold. A 45-task suite can catch a 13-point gap only half the time, while 447 tasks can resolve a 5.2-point difference at the observed task-level variation. Claims of 2- to 3-point harness gains from a single run remain unproven.
Teams should also trim first-turn prompts and tool schemas, then test lean command-line tools against larger tool collections. Removing Claude Code’s file-editing tools and forcing Claude Opus 5 to edit through bash kept its success rate within the control-run range. Thinking budgets need tuning for each model-harness pair too: Claude Opus 5 solved 34 of 45 tasks inside a year-old Claude Code release with extended thinking off by default.
The broader lesson connects both evaluations. A compact model such as Underdog Saluki 27B needs careful testing to show that compression preserves useful behavior, while a coding harness needs careful testing to show that extra prompts and tools earn their cost. Bigger files and bigger scaffolds do not guarantee better results.
Based on
- How to Run Underdog Saluki 27B: 2-Bit Qwen3.8-27B Tool-Calling Guide — marktechpost.com
- A coding agent’s harness can triple its cost without making it more accurate, study finds | VentureBeat — venturebeat.com
- Riemann Computing Inc. Creates and Demos Out a 13B Parameter AI Model, Aims at Massively Reducing Training Costs | AP News — apnews.com
- OpenAI’s latest AI models are coming to Google Cloud and Antigravity | Ars OpenForum — arstechnica.com
