Running a 753B-parameter language model on one workstation GPU sounds like a job for a data center. FreeToken, an Edge-Native MoE Serving Engine developed by researchers from UC Berkeley and UT Austin, takes a different approach: it treats the whole personal machine as an inference platform, not just a small GPU.
FreeToken can run 753B GLM-5.2 on a single workstation card. It can also deliver interactive speed for a 35B model on an 8 GB laptop GPU and run a 284B model on a gaming desktop. Those targets put large mixture-of-experts models within reach of solo developers, startups, SMB engineering teams, and certain enterprise applications.
Making a Large Model Fit on Local Hardware
The key idea is to avoid keeping every model parameter on the GPU for every token. Mixture-of-experts models contain many expert networks, but each token activates only a small part of that system. DeepSeek-V4-Flash, for example, activates six of 256 routed experts in each of its 43 layers. That means only 13B of its 284B parameters participate in any single token.
The full expert set still takes roughly 140 GB at FP4. FreeToken places inactive experts in host memory, then moves or computes the needed pieces based on the machine’s available resources. This allows a single workstation card to handle a model that cannot fit into the card’s memory as one complete package.
The research team argued that the missing piece is not hardware but a serving system. FreeToken treats a personal machine as a unified, elastic inference platform and maps computation and model state across the GPU, CPU, memory, and interconnect bandwidth available at that moment.
That design matters because moving data between system memory and GPU memory can become the main limit. A large model may fit across the whole computer, but serving it still requires careful choices about what to place on the GPU and what to process elsewhere.
Why Existing Approaches Struggle
FreeToken addresses failure modes seen in engines such as llama.cpp, KTransformers, Ollama, and MoE-Infinity. The problems include prefill destroying sparsity, static placement missing decode traffic, and consumer CPUs failing to provide enough computing power for the workload.
Prefill is the stage that processes the initial prompt, while decode produces the response token by token. Their memory and computing patterns differ, so a placement plan that works for one stage may waste resources during the other. FreeToken uses separate techniques to keep the model’s sparse structure useful throughout serving.
Its bandwidth-adaptive execution system splits cache misses according to PCIe bandwidth. The engine can fill experts into the GPU cache when the connection supports that transfer, or compute on the CPU when moving the data would cost too much time.
The available bandwidth differs across machines. The listed PCIe bandwidth profile, BP:BH, is 52.7:77.3 on an RTX 5090 and 11.8:47.5 on a 4060 laptop. FreeToken uses those hardware differences when deciding how to handle missing experts instead of applying one fixed placement strategy to every computer.
Caching, Checkpoints, and Measured Speed
FreeToken also uses semantic-aware caching. During prefill, it streams layer data and creates recurrent-state checkpoints at token boundaries. That gives the engine a way to reuse useful model state as generation continues, rather than treating every step as a fresh memory problem.
Its elastic memory management can rebuild the GPU expert cache without restarting the engine. For local users, that means changing which experts sit in GPU memory does not require shutting down the serving process and starting again.
The performance numbers show what this approach means in practice. On an RTX 5090, FreeToken sustains 77–83 tokens per second on Qwen3.6-35B-A3B in BF16. It reaches 22–25 tokens per second on DeepSeek-V4-Flash in MXFP4.
Those results do not mean every model will run at the same speed on every machine. They do show that FreeToken is built around the uneven memory and bandwidth limits found in workstation and consumer systems, including an 8 GB laptop GPU and a gaming desktop.
Open Access and a Changing Model Market
FreeToken is published on GitHub under the Apache-2.0 license, available on PyPI as freetoken v0.1.2, and shipped as a desktop app for Windows and Linux. Its target environment is Linux x86_64 with an NVIDIA GPU using driver r580+ and CUDA 13.
The engine exposes OpenAI- and Anthropic-compatible endpoints on port 1919. That gives developers a familiar way to connect applications to a local FreeToken server while using hardware outside a traditional hosted service.
The timing also lines up with a market where large open-weight models are competing on both capability and cost. Z.ai, the Chinese startup behind GLM-5.3, offers API access at $1.40 per million input tokens and $4.40 per million output tokens, with pricing unchanged from GLM-5.2.
That works out to $5.80 per million tokens, compared with $8 for Grok 4.6, $18 for Kimi K3, $30 for Claude Opus 5, and $35 for GPT-5.6 Sol. Artificial Analysis scores GLM-5.3 at 60 on its Intelligence Index, tying Kimi K3 as the top open weights model.
FreeToken does not remove the hardware limits of local inference. It changes how those limits are managed. By combining GPU caching, CPU computation, PCIe-aware execution, and state checkpoints, the engine aims to make models such as the 753B GLM-5.2 usable from a single workstation instead of requiring a larger cluster.
Based on
