What if a personal computer could serve an AI model built for far larger machines? Researchers from UC Berkeley and UT Austin developed FreeToken to make that idea practical, running a 753B GLM-5.2 model on a single workstation GPU.
FreeToken does not treat a personal machine as a small GPU. It treats the entire system as a unified, elastic inference platform that maps computation and model state across the available GPU, CPU, memory, and interconnect bandwidth.
FreeToken Turns Personal Hardware Into an AI Serving Platform
The results stretch from laptops to gaming desktops and workstation cards. FreeToken can run a 35B model at interactive speed on an 8 GB laptop GPU, run a 284B model on a gaming desktop, and run a 753B GLM-5.2 model on a single workstation card.
That range gives FreeToken a clear edge-native mission: keep large-model inference close to the user, even when the full model cannot fit inside GPU memory. Instead of depending on one fixed placement strategy, the engine continuously adapts the work to the hardware available inside the personal machine.
On an RTX 5090, FreeToken sustains 77–83 tokens per second on Qwen3.6-35B-A3B in BF16. The same card reaches 22–25 tokens per second on DeepSeek-V4-Flash in MXFP4, showing that the system can deliver useful generation rates across different model sizes and formats.
The project also aims to make that capability accessible beyond research demonstrations. FreeToken is Apache-2.0 licensed on GitHub, published on PyPI as freetoken v0.1.2, and shipped as a one-click desktop app for Windows and Linux. Its target setup is Linux x86_64 with an NVIDIA GPU using driver r580+ and CUDA 13.
Why Mixture-of-Experts Models Challenge Local Inference
FreeToken focuses on mixture-of-experts models, where each token activates only part of a much larger network. DeepSeek-V4-Flash activates six of 256 routed experts in each of its 43 layers, so only 13B of its 284B parameters participate in any single token.
That sparse computation does not remove the memory challenge. DeepSeek-V4-Flash’s full expert set occupies roughly 140 GB at FP4, with inactive experts sitting in host memory until execution needs them. The serving engine must move the right data at the right time without allowing memory transfers to erase the benefit of sparsity.
Existing engines such as llama.cpp, KTransformers, Ollama, and MoE-Infinity face failure modes that include prefill destroying sparsity, static placement missing decode traffic, and consumer CPUs proving insufficient. FreeToken addresses those problems with a group of systems designed around changing bandwidth and model demand.
- Bandwidth-adaptive execution: The q* policy splits each step’s cache misses according to DMA transfer bandwidths.
- Semantic-aware caching: The engine streams layer data during prefill and uses recurrent-state checkpoints at token boundaries.
- Elastic memory management: The GPU expert cache can be rebuilt without restarting the engine.
Measured B_P:B_H reaches 52.7:77.3 on an RTX 5090 server and 11.8:47.5 on a 4060 laptop. Those figures show how FreeToken accounts for different hardware conditions instead of assuming that every machine offers the same path between processing and memory.
The goal is not simply to shrink a large model until it fits. FreeToken coordinates the machine around the model, allowing its expert cache and execution plan to change as the workload moves from prefill to token generation.
Open Access, Local Deployment, and a New API Option
FreeToken exposes OpenAI- and Anthropic-compatible endpoints on port 1919, giving developers a familiar way to connect local inference to applications and tools. The project is positioned for solo developers, startups, SMB engineering teams, and industries such as healthcare, legal, defense, finance, and R&D.
That local serving option arrives as GLM-5.3 becomes available through an API. Z.ai offers GLM-5.3 at $1.40 per million input tokens and $4.40 per million output tokens, with pricing unchanged from GLM-5.2. Developers can build applications and agents on top of the model through its API.
GLM-5.3 also sits below several other listed frontier-model prices: Grok 4.6 costs $2.00 per million tokens at its lower context rate, Kimi K3 costs $3.00 per million tokens, and Claude Opus 5 and GPT-5.6 Sol each cost $5.00 per million tokens.
Artificial Analysis scores GLM-5.3 at 60 on its Intelligence Index, tying Kimi K3 as the top-performing open weights model. The score stands seven points above GLM-5.2, adding a performance signal to the model’s API pricing.
Z.ai plans to make GLM-5.3’s weights openly available, but the date and licensing have not been specified. That leaves FreeToken’s local-serving direction especially interesting: as more large models become available in forms developers can access, the ability to run them across personal hardware may become just as important as the models themselves.
FreeToken points toward a different scale for AI infrastructure. A laptop GPU, gaming desktop, or workstation card can become part of a flexible serving platform, and the next breakthrough may come from coordinating those resources better rather than simply adding more hardware.
Based on
