New fine-tuning method reduces catastrophic forgetting in LLMs
A new method, Sparse Memory Finetuning (SMF), shows promising results in reducing "catastrophic forgetting" during the fine-tuning of large language models, according to a study published on arXiv.
Vad har hänt
Researchers have introduced Sparse Memory Finetuning (SMF), a technique aimed at counteracting the phenomenon of "catastrophic forgetting" in pre-trained language models. The study, published on arXiv (ID: 2605.03229v1), demonstrates that SMF adds key-value memory layers to the model. During each training step, only a limited number of memory rows most relevant to the current data batch are updated.
Key facts
| Metod | Sparse Memory Finetuning (SMF) |
|---|---|
| Publiceringsdatum | 26 maj 2026 |
| Förbättring (MedMCQA) | 2,5 procentenheter |
| Glömska (generell kunskap) | Inom 1 poäng från basmodell |
”SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both.”
Varför det spelar roll
Catastrophic forgetting is a significant problem in AI, where models lose previously acquired knowledge when adapted to new tasks. The SMF method addresses this by selectively updating memory, preserving the model's general capacity while it learns new, specific tasks. This could lead to more robust and versatile AI systems.
Vem påverkas
Researchers and developers working on fine-tuning large language models are the primary stakeholders. Companies implementing LLMs in their products could also benefit from more stable models that retain broad knowledge.
EU-status
Ej relevant för EU-status.
Mer att veta
The study compared SMF with established methods such as LoRA and full fine-tuning. SMF improved performance on a medical question-answering task by 2.5 percentage points while keeping general knowledge loss (WikiText perplexity and TriviaQA accuracy) within one point of the base model. LoRA and full fine-tuning achieved greater improvements on the new task but with a distinct loss of general knowledge.
Quick answers about this story
Vad har hänt?
När hände det?
Varför spelar det roll?
Länken öppnar i nytt fönster och leder till utgivarens egen sida.
Källan har spårats automatiskt från utgivaren via Aheadlines signalkedja.