Research · Preprint

Small Models, Long Memory: Learned Forgetting for Efficient Long-Context Recall

Compact networks trained to forget the right things recall as well as models ten times their size.

Ammar Alsiyabi, A. Collaborator, B. Advisor arXiv · 23 Sep 2026
Download PDF Code BibTeX
Figure 1. Recall accuracy as context length grows. Replace with your own figure.

Abstract

Long-context language models are expensive to train and to run. We show that a model with a tenth of the parameters can match much larger systems on long-context recall, if it is trained to decide what to keep and what to discard. Our method, learned forgetting, adds a small gating mechanism that scores each piece of context by its expected future usefulness.

Across four benchmarks, a 700M-parameter model with learned forgetting matches or exceeds 7B-parameter baselines on recall, while using 60% less memory at inference time.

Key findings

  • Forgetting is learnable: a lightweight gate predicts which tokens will be needed later.
  • Small models benefit most, closing up to 90% of the gap to models ten times their size.
  • Memory use at inference falls by 60% with no loss in accuracy.

Files

PDF Small Models, Long Memory — full paperPDF · 16 KB Download ↓

Cite this work

@article{alsiyabi2026small,
  title   = {Small Models, Long Memory: Learned Forgetting
             for Efficient Long-Context Recall},
  author  = {Alsiyabi, Ammar and Collaborator, A. and Advisor, B.},
  journal = {arXiv preprint},
  year    = {2026}
}