Google Proposes a Sleep Framework to Help Large Models Continuously Learn Without Forgetting
Google Research introduces a "sleeping framework" that adds an offline sleep phase to continual learning, enabling large language models to consolidate short-term context into long-term parameters and mitigate catastrophic forgetting. The model cycles between active interaction and offline consolidation, outperforming baselines like GPT-4 and SFT in tests involving 10-million-token-long contexts, continual translation, and mathematical reasoning.
Google Proposes a "Sleep Framework" to Help Large Models Continuously Learn Without Forgetting
Google Research has introduced the "Sleep Framework," which adds an offline sleep phase to continual learning. This allows large language models to consolidate short-term context into long-term parameters, mitigating catastrophic forgetting.
The model periodically switches between active interaction and offline consolidation. In tests involving 10-million-token-long contexts, continual translation, and mathematical reasoning, it outperformed baselines such as GPT-4 and SFT.
Source: Google Research
