NVIDIA has introduced a new method called Star Elastic, which allows multiple sizes of language models to be stored and used from a single checkpoint. Instead of training and maintaining separate models for different sizes, Star Elastic embeds smaller variants within a larger one. This approach can save time, storage, and compute costs, making it easier to deploy large language models at scale.

What Is Star Elastic and How Does It Work?

Star Elastic is a post-training technique that creates nested submodels within a larger language model. For example, instead of training separate models with 12 billion, 23 billion, and 30 billion parameters, Star Elastic trains one big model that contains these smaller versions as subsets. These smaller models reuse most of the weights from the larger one, which are selected based on their importance to the model’s accuracy.

The process involves scoring each component of the model—like attention heads, embedding channels, and expert layers—by how much they contribute to the model’s performance. The most important components form the smaller, nested models. This nested, weight-sharing setup enables quick extraction of different model sizes without additional training or fine-tuning.

How Does the Model Decide Which Parts to Use?

Star Elastic uses a special ranking system to decide which parts of the model are included in each size variant. This system considers multiple axes, such as the number of experts in a mixture of experts (MoE) layer or the number of attention heads. For MoE layers specifically, it employs a method called Router-Weighted Expert Activation Pruning (REAP). REAP ranks experts based on how much they are used during routing and their output strength, ensuring only the most relevant experts are kept for each submodel.

A key feature of Star Elastic is its learnable router. Unlike fixed compression methods, this router is trained along with the model. It receives a target size, like a 2.8 billion parameter model, and produces masks that select which parts of the model are active. These masks are differentiable, meaning they can be optimized during training using techniques like Gumbel-Softmax. This allows the model to adaptively determine the best subset of components for each size, all within a single training process.

Overall, Star Elastic offers a flexible way to create multiple model variants from one training run. This reduces the resources needed and simplifies deployment, especially for teams running inference at scale. The approach is demonstrated on a model called Nemotron Nano v3, a hybrid architecture with 30 billion total parameters, which can produce smaller variants with 23 billion and 12 billion parameters without extra fine-tuning.

This innovation could make large language models more accessible and cost-effective, enabling more organizations to deploy powerful AI without the heavy overhead traditionally involved in training and maintaining multiple models.

Inspired by

• https://fonts.googleapis.com