Attention to Mamba: A Recipe for Cross-Architecture Distillation

Apple Machine Learning Research Version 1 original current

Imported from official source

State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compared to their Attention-based counterparts. On the other hand, the community has built up a considerable body of knowledge on how to train Transformers, and many pretrained Transformer models are readily available. …

This version

Version
1 of 1
Recorded
November 10, 2026 20:01
Change
Initial
Content hash
b588624fbd48265793e03792ac7e0d6b
All versions
Revision history

Officially records where a publication came from, not whether it is true. Imported records are reproduced from an organization's own official source.