Mamba State Space Models: Fixed State, Linear Time

Fixed state instead of a growing cache

Albert Gu at Carnegie Mellon and Tri Dao at Princeton, the latter also the first author of FlashAttention, proposed an architecture with no attention and no feed-forward blocks. Where attention compares every token against every other, costing time quadratic in sequence length and a cache that grows as the context does, a state space model carries a fixed-size state forward and costs time linear in length, with constant memory per generated token. [1][5]

Earlier state space models were competitive on audio and genomics and lost badly on language, and the paper identifies why: they could not do content-based reasoning. Their recurrence applies the same transformation to every token regardless of what the token is. The fix is to make the relevant parameters functions of the input, so the model can decide what to keep and what to forget based on what it is reading. That is the selectivity in the name. [1]

Selectivity costs the property that made these models trainable. A fixed recurrence can be computed as a convolution; an input-dependent one cannot. The paper recovers trainability with a hardware-aware parallel scan built on the same reasoning as Dao's attention work: load parameters into fast on-chip memory, compute the recurrence there, never materialize the expanded state in high bandwidth memory, and recompute intermediates in the backward pass rather than storing them. The fused scan is reported at 20 to 40 times a standard implementation. [1]

What was measured, and at what size

The headline claims are four to five times the inference throughput of a comparable Transformer, with no cache to maintain, and a 3 billion parameter model matching Transformers twice its size. The synthetic results are the more striking ones: on an induction-heads task trained at 256 tokens, the model extrapolates to sequences of over a million, roughly four thousand times its training length, while attention baselines were tested only to 16,384, which is as far as memory allowed. [1]

Atlas interpretation: The scales deserve emphasis. The language sweep runs from 125 million to 1.3 billion parameters, and the largest trained model in the paper is 2.8 billion, which the abstract rounds to 3B. A 6.9 billion parameter version appears only untrained, in a throughput measurement. This is a December 2023 paper arguing about architecture against a frontier that was already serving GPT-4. Everything here is an argument by extrapolation from small models, which is the ordinary situation for academic architecture work and is worth stating rather than glossing. [1]

The limitation is the same property as the advantage

Two months later a paper from Harvard researchers proved that a two-layer Transformer can copy strings of exponential length while models of this family are fundamentally limited by their fixed-size state, and showed pretrained Transformers beating them at copying from and retrieving out of context. [3]

Atlas interpretation: This is not a bug that a later version fixes. A state of fixed size is exactly what makes inference cheap and exactly what caps how much of a long context can be recalled verbatim. Attention's growing cache is expensive because it keeps everything. The two designs are trading against each other on the same axis, which is why the resolution was not a winner. [3][1]

The same authors then argued the two families are more closely related than the rivalry suggested. Mamba-2, in May 2024, develops a duality between these models and variants of attention through decompositions of structured matrices, and reports a core layer 2 to 8 times faster than the original while staying competitive on language. [2]

What actually shipped was hybrids

The architecture spread, but almost never on its own. AI21's Jamba in March 2024 interleaves Transformer and state space layers with a mixture of experts. Microsoft's Samba pairs the design with sliding-window attention. Bamba, led by IBM with Princeton, Carnegie Mellon and Illinois, runs 29 state space layers to 3 attention layers. Nvidia's Nemotron-H replaces the majority of self-attention layers rather than all of them, and its Nemotron 3 Super in March 2026 repeats a published pattern that still places an attention layer in every cycle, retained specifically for precise recall. [4][6][7][8][9]

The largest deployment claimed is Tencent's Hunyuan-TurboS in May 2025, a hybrid mixture-of-experts model with 56 billion parameters active of 560 billion total, which its authors describe as the first industry-deployed large-scale Mamba model. An attention-free shipped model is Mistral's Codestral Mamba of July 2024, a 7 billion parameter code model released under Apache 2.0, in a niche where latency dominates. [10][11][12]

Atlas interpretation: The paper did not replace the Transformer, but it showed that a language model can be built without attention. Later production hybrids kept some attention layers for precise recall, while Codestral Mamba shipped as an attention-free code model. The result was a new design option rather than a single successor architecture. [1][9][7][3][12]