Running a huge language model usually means sending requests to a remote data center. FreeToken takes a different path: the edge-native mixture-of-experts serving engine is designed to run large models on personal hardware, including the 753B GLM-5.2 model on a single workstation GPU.
Researchers from UC Berkeley and UT Austin built FreeToken around a simple idea. The obstacle is not always the hardware itself, but the serving system that decides where computation and model data should live. FreeToken treats a personal machine as a unified inference platform, combining its GPU, CPU, memory, and interconnect bandwidth instead of treating the GPU as the entire computer.
That approach gives the project an unusually broad range. FreeToken can run a 35B model at interactive speed on an 8 GB laptop GPU, a 284B model on a gaming desktop, and the 753B GLM-5.2 model on one workstation card. Those targets matter because they bring models that normally require large infrastructure closer to solo developers, startups, small and medium-sized engineering teams, and enterprises with specific workloads.
Why mixture-of-experts models are difficult to run
Mixture-of-experts models are large, but they do not use every parameter for every token. DeepSeek-V4-Flash, for example, has 284B parameters but activates six of 256 routed experts in each of its 43 layers. Only 13B parameters participate in producing any single token.
That sparse design reduces the computation needed for each token, but it creates a memory problem. The full expert set in DeepSeek-V4-Flash occupies roughly 140 GB at FP4, so inactive experts need to remain in host memory while selected data moves to the GPU. The system must then predict which pieces will be needed and move them without slowing generation.
Existing engines such as llama.cpp, KTransformers, Ollama, and MoE-Infinity face failure modes tied to sparsity, decode traffic, and the limits of consumer CPUs. FreeToken addresses those problems with several parts that work together rather than relying on one fixed hardware layout.
FreeToken’s approach to memory and bandwidth
Its bandwidth-adaptive execution splits each step’s cache misses according to available PCIe bandwidth. That matters because a workstation and a laptop can have very different paths between the processor, memory, and GPU. The bandwidth profile B_P:B_H measures 52.7:77.3 on an RTX 5090 server, compared with 11.8:47.5 on a 4060 laptop.
FreeToken also uses semantic-aware caching. During prefill, it streams layer data, then uses a shared least-recently-used expert cache during decode. This lets the engine keep useful experts available while replacing data that no longer helps the current generation.
Its elastic memory management can rebuild the GPU expert cache without restarting the engine. That detail is easy to overlook, but it helps a personal machine adapt as model activity changes. Instead of locking the system into one memory plan, FreeToken can adjust the cache while the engine continues running.
The performance figures show why those design choices matter. On an RTX 5090, FreeToken sustains 77–83 tokens per second on Qwen3.6-35B-A3B in BF16 and 22–25 tokens per second on DeepSeek-V4-Flash in MXFP4. The reported results reach 1.5–2.3 times the strongest baseline, while decode stays within 12% of the single-turn rate across three agentic workloads.
Availability and what developers can do with it
FreeToken is available on GitHub under the Apache-2.0 license and on PyPI as freetoken v0.1.2. It also ships as a desktop app for Windows and Linux. The target setup is Linux x86_64 with an NVIDIA GPU running driver r580+ and CUDA 13, while the desktop app extends access to Windows and Linux users.
The engine exposes OpenAI-compatible and Anthropic-compatible endpoints on port 1919. That makes it easier for developers to connect existing tools, agents, and applications without rebuilding their integrations around a new interface.
FreeToken’s most striking demonstration is the 753B GLM-5.2 model on one workstation GPU. The project does not make that model small; it changes how the available hardware handles model state, cache data, and traffic between memory locations. That distinction is the heart of the system.
GLM-5.3, developed by Z.ai, adds a useful price comparison for developers deciding between local serving and an API. It is available through an API at $1.40 per million input tokens and $4.40 per million output tokens, with pricing unchanged from GLM-5.2. Based on those input and output costs, GLM-5.3 costs $5.80 per million tokens.
That price is higher than Muse Spark 1.2 Contributor, MiMo-V2.5 Flash, and DeepSeek-V4-Flash during off-peak use. It is lower than Grok 4.6, Kimi K3, Claude Opus 5, and GPT-5.6 Sol based on per-token costs. Artificial Analysis gives GLM-5.3 a score of 60 on its Intelligence Index, tying Kimi K3 as the top open weights model, though its cost per Intelligence Index task is about $0.68 compared with $0.44 for GLM-5.2.
That leaves developers with a clear tradeoff. The API offers a simple path for using GLM-5.3, while FreeToken aims to put massive models on hardware that a person or smaller team can control. For workloads where local access, flexible deployment, or avoiding remote infrastructure matters, FreeToken turns the workstation into more than a place to run a small model.
Based on
