The model fits nowhere. A team can fine-tune a model for three weeks, get the evaluation numbers it wanted, and then discover that serving the result requires a machine built around one inconvenient fact: the checkpoint alone is 140GB.
That storage problem becomes a memory problem at load time. A 70 billion parameter model stored in FP16 needs around 140GB of VRAM just to load, before accounting for the context window or the other memory demands of running inference.
This is where quantization stops being a research footnote and becomes basic engineering. Quantization stores model weights with fewer bits, trading numerical precision for a smaller memory footprint; for local users, that trade is often the difference between running a model and admiring its download progress.
Four Bits Changes the Hardware Equation
Compressing the same 70B model to 4-bit with AWQ or GPTQ reduces its size to around 35 to 40GB. That remains a substantial requirement, but it moves the model from “needs a large server” toward “might fit in a high-end local system.”
The reduction is not limited to one model family. Google’s Gemma 3 took its 27B model from 54GB down to roughly 14GB at 4-bit, while cutting the quality loss against plain post-training quantization roughly in half.
Gemma 4 pushed the smaller end of the range further, shipping quantization-aware checkpoints that bring its smallest 2B variant down to about 1GB. Quantization-aware training prepares the model for reduced precision during development instead of guessing at scales after the fact, which is a more deliberate approach than squeezing the model after training and hoping its answers survive.
Apple uses the same basic strategy for its on-device models on current iPhones, squeezing weights down to 2 bits through quantization-aware training. The goal is clear: keep capable models inside a device’s memory limits without sending every request elsewhere.
Qwen 3.8 27B uses about 18GB when loaded with Q4km weights, another example of how the selected format can determine whether a model fits a local machine. Most people doing local inference use smaller quantization at 4 to 6 bits, choosing a practical balance between memory use and model quality.
Local AI Still Has a Hardware Bill
Smaller files do not erase the rest of the system requirements. A 30B model with 8-bit quantization and a 128k context window would likely require a high-end GPU such as an RTX Blackwell 6000 or a Mac Studio Ultra with 96GB minimum.
Context matters because the model must handle more than its stored weights. A long 128k context window adds its own memory demand, so a model that fits under one configuration can become impractical when users ask it to retain much more information.
Systems with unified memory offer another path. Machines such as Mac or AMD Stirix Halo can accommodate larger models with less quantization because the memory system gives the processor and graphics hardware access to a shared pool.
That helps explain the appeal of local inference for home automation. “Most online models are overkill, and the whole point of self-hosted home automation is the never-ending challenge of keeping everything local.” The challenge is not only privacy or control; it is finding a model, quantization level, context window, and machine that can coexist without turning every request into a hardware negotiation.
There is also a philosophical argument lurking behind the memory discussion. Models are possibly conscious in the same sense that coffee machines are, but deep learning algorithms show no signs of consciousness whatsoever, by whatever definition one uses.
Questions about AI consciousness involve feelings, which have biological foundations. The fact that a model can generate language does not establish that it experiences anything, and shrinking its weights from 140GB to 35GB certainly does not create an inner life. It creates a smaller file.
Humans can perform calculations that reproduce LLM responses using pencil, paper, weights, lookup tables, and time. That observation separates computation from experience: a process can produce an answer without having feelings about the answer.
Quantization therefore solves a practical problem, not a metaphysical one. It makes large language models cheaper to store and easier to run, while leaving the harder questions of quality, context, and consciousness exactly where they were.
Based on
