Researchers from MIT, the Improbable AI Lab, and ETH Zurich have developed a new fine-tuning technique to tackle a common problem in AI called “catastrophic forgetting.” This issue happens when models lose previously learned skills after being updated with new information. Their new method, called self-distillation fine-tuning (SDFT), aims to help models learn new tasks without sacrificing what they already know.

Addressing the Challenge of Continual Learning

Many organizations face a dilemma: they want their AI models to stay up-to-date with new data and skills, but traditional fine-tuning methods often cause models to forget earlier knowledge. To avoid this, some split their models into separate parts or adapters for each task. While this works, it increases costs and makes managing the systems more complex. It also requires constant retesting to ensure changes don’t cause regressions.

The new SDFT technique offers a different approach. It uses the model’s own ability to learn from context, essentially teaching itself how to update while keeping old skills intact. This is achieved by having the model generate training signals based on demonstrations, which helps it learn new tasks without losing previous capabilities.

How Self-Distillation Fine-Tuning Works

SDFT leverages in-context learning, a feature where models learn from examples provided during input. During training, the same model acts in two roles: a teacher and a student. The teacher is conditioned on the query and expert examples, while the student only sees the query. The teacher generates predictions based on the demonstration, and the student updates its parameters to match these predictions.

This process is called on-policy learning because the model learns from its own outputs, making it more effective at retaining older skills while acquiring new ones. The researchers say this method consistently outperforms traditional supervised fine-tuning across different tasks, achieving higher accuracy on new skills and reducing the risk of forgetting old ones.

Implications for Enterprise AI

The ability to sequentially add new skills without degrading existing performance could simplify how businesses update their AI systems. Instead of creating multiple separate models or adapters, a single model can continuously learn from new data. This makes updates faster, cheaper, and easier to manage.

Reinforcement learning, another approach for continual learning, also reduces forgetting but needs explicit reward functions, which aren’t always easy to define. SDFT offers an alternative by using the model’s demonstration-based signals, avoiding the need for these reward functions and making the process more straightforward.

In experiments, the researchers showed that models trained with SDFT could learn multiple skills over time without performance drops. This suggests that on-policy distillation — a process where models learn from their own outputs — could be a practical way to enable models to keep learning in real-world settings.

Inspired by

• https://www.computerworld.com/article/4131253/researchers-propose-a-self-distillation-fix-for-catastrophic-forgetting-in-llms-2.html

Sources

• infoworld.com
• arxiv.org
• infoworld.com
• informa.com
• greyhoundresearch.com