Big models hit the wall at deployment. A team can fine-tune a model for three weeks, get the evaluation numbers it wanted, and then discover that serving the result is a separate problem. The checkpoint alone may weigh 140GB, which is an awkward surprise after all that optimization work.
A 70 billion parameter model stored in FP16 needs around 140GB of VRAM just to load. That requirement puts the full model beyond the reach of much consumer hardware before it answers a single prompt, turning model quality into a memory-budget problem.
The model does not need to ship at full precision with every parameter intact. It needs to ship as the leanest version that still does the job, and the two techniques that get it there—quantization and pruning—are neither exotic nor new.
Quantization Turns Memory Into a Manageable Constraint
Compressing a 70B model to 4-bit reduces its size to around 35 to 40GB. That remains a substantial file, but it changes the hardware conversation: a model that demanded about 140GB of VRAM now fits within a much smaller memory footprint.
Google’s Gemma 3 shows the same shift at a different scale. Its 27B model moves from 54GB down to roughly 14GB at 4-bit, shrinking the storage burden without requiring the model to abandon its core capability.
Gemma 4 pushes the numbers lower. Its quantization-aware checkpoints bring the smallest 2B variant down to about 1GB at 4-bit, a figure that makes local deployment look less like a data-center project and more like a practical hardware choice.
Apple uses quantization-aware training for its on-device models on current iPhones, squeezing weights down to 2 bits. The strategy is straightforward in outcome: fewer bits mean less memory pressure, allowing models to operate on hardware that was never going to hold the full-size version.
Qwen 3.8 27B uses about 18GB when loaded with Q4km weights at 8bit quantization. The figures are not perfectly uniform across models or configurations, but they make the central point clear—model size depends on how the weights are represented, not only on the parameter count printed on the model card.
Local Inference Means Choosing What Fits
Most people doing local inference use smaller quantization: 4-6 bits. That choice reflects available hardware memory, not a sudden rejection of larger models; most local inference is done with smaller quantization bits because the alternative is failing to fit the model at all.
A 30B model with a full 128k context window at 8bit quantization would require a large amount of memory, likely exceeding consumer GPU capacities. The context window matters here because loading the model is only part of the memory demand—serving a long context adds another burden to hardware already operating near its limit.
The practical benefits are clear: fewer GPUs to buy or rent, lower latency per request because less data has to move through memory, and the ability to put real capability on hardware that was never going to hold the full-size model. Efficiency is less glamorous than adding parameters, but it is what turns a checkpoint into usable software.
Pruning sits beside quantization in this effort to produce leaner models. The verified figures describe the memory gains from quantization, while the broader deployment goal remains the same: remove excess weight where possible and preserve the performance that matters.
That engineering discussion also brushes against a different question: whether increasingly capable deep learning systems deserve ethical consideration. The answer in the available argument is blunt: “Deep learning algorithms, even ones applied to language, show no signs of consciousness whatsoever, by whatever definition one uses.”
Another line makes the position even less subtle: “No, they shouldn’t. A rock doesn’t deserve ethical consideration, and neither does a coffee machine.” The hardware problem is concrete; the philosophical one remains a debate, although the sources offer no sign of consciousness in deep learning models.
For local users, the immediate decision is less cosmic. Choose the model, choose the quantization level, check the memory, and accept that a 30B model with a 128k context at 8bit may not fit on a consumer GPU. The future of practical LLM deployment may depend less on making models larger than on making them lean enough to leave the lab.
Based on
