Mamba vs Transformers: The Growing Notebook vs the Fixed State (and Mamba 3)

Why does longer context still hurt latency and memory even when everyone says their model is “efficient”?

Two families answer that question differently. Transformers keep a growing notebook of keys and values so every new token can look anything up. Mamba compresses history into a fixed-size state so time stays linear and decode stays lean. Mamba 3 pushes that second idea harder for inference. In 2026, pure Transformers still run most production stacks (with better kernels and caching), while hybrids mix both on purpose.

Two ways to remember a long sequence

A Transformer’s core move is scaled dot-product self-attention : every position can attend to every other. Training parallelizes well, that is the Vaswani et al. (2017) design.

The catch is cost. Naive self-attention is O(T²) in sequence length T for both time and memory. At decode time you usually cache keys and values (the KV cache). That cache is the notebook, it grows with context, and long context attention often dominates latency.

An SSM(State space model) / Mamba model makes a different bet. History gets folded into a fixed-size state. One sticky note that gets rewritten as tokens arrive. You trade “look up anything in the notebook” for “update a compressed summary.” Cartesia’s framing of Mamba 3 puts it bluntly: fixed-state models compress history. Transformers keep a growing KV “database.”

What Transformers struggle with and how we patch them

Problem 1: Quadratic attention compute and memory as T grows (the FlashAttention paper restates this cost).

Problem 2: Autoregressive decode with a KV cache. Memory grows with context, and attention work grows with it too. The Mamba 3 authors restate the same shape which is quadratic compute plus linear KV memory at inference.

Production did not throw Transformers away. It patched them.

• FlashAttention (Dao et al.) is IO aware exact attention via tiling. Their paper reports ~3× speedup on GPT-2 at sequence length 1K, same math, smarter memory movement.

• Flash Decoding speeds long-context generation. The PyTorch blog cites up to ~8× faster generation in a CodeLlama-34B example.

• Prompt / prefix caching reuses already processed prefixes so you do not pay again (Anthropic and OpenAI both document this).

These patches help a lot. They do not erase the fundamental scaling shape.

Mamba’s bet: selective state spaces (1 → 2 → 3)

Earlier subquadratic sequence models often lagged attention on language. Gu & Dao (Mamba-1) argue the gap was weak content-based reasoning, the model could not “select” what to keep.

Mamba 1 makes the SSM parameters input-dependent (selective Δ, B, C) and uses a hardware-aware parallel scan. The abstract claims ~5× higher inference throughput than Transformers, linear scaling in length, and a Mamba-3B that matches roughly 2×size Transformers on language modeling.

Mamba 2 (SSD) links SSMs to structured attention / semiseparable matrices. It restricts A to scalar×identity and uses heads that play nicely with matmuls. The paper reports the core layer ~2–8× faster to train than Mamba-1’s selective SSM. On The Pile, their 2.7B claims match or beat prior Mamba and same data Pythia baselines.

The Mamba 3 paper still names the old SSM pains: quality–efficiency tradeoffs, weak state tracking, and linear decode that is still memory bound on real hardware.

Official code lives in state-spaces/mamba - Mamba, Mamba2, and Mamba3 modules. Mamba 3 is the inference-first chapter of the same family.

Mamba 3: three upgrades that matter at decode

Paper: Lahoti et al., arXiv:2603.15569 (submitted March 2026).

Three pillars:

1. Exp-trapezoidal discretization : generalizes Mamba-1/2’s exponential-Euler update. Set λₜ = 1 and you recover Euler; λₜ = 1/2 is classical trapezoid.

2. Complex-valued / data-dependent RoPE-like state : the paper reports near-perfect synthetic arithmetic and state-tracking where Mamba-2 and a Mamba 3 path without that RoPE-like state sit near random.

3. MIMO vs SISO : richer state updates. Authors claim MIMO with state size N = 64 matches Mamba-2 N = 128 perplexity (“half state > half latency”). MIMO can use up to ~4× decode FLOPs at fixed state size with similar wall-clock.

   SISO = Single Input Single Output.

   MIMO = Multiple Input Multiple Output.

H100, batch 128, sequence length 16,384. Model sizes differ.

Kernels: mamba_ssm/modules/mamba3.py (is_mimo, mimo_rank, d_state). Install from source for Mamba 3.

One more signal from the authors / Cartesia: they expect linear layers to mostly ship with global self-attention for language models going forward. That tees up hybrids.

2026 reality: Transformers, Mamba 3, and hybrids

Transformers remain the production default propped up by FlashAttention, Flash-Decoding, and prompt caching. If you need mature tooling and flexible retrieval, this is still the safe path.

Pure SSM / Mamba is strongest when you care about linear scaling and decode throughput. Mamba 3 SISO/MIMO around 1.5B is open via state-spaces/mamba, with Cartesia’s latency numbers as a public reference point. Hybrids are the practical middle, keep some global attention for retrieval, use SSM / linear layers for length and efficiency.

Named examples from the research:

• Jamba (AI21): Transformer + Mamba + MoE; roughly 1 attention : 7 Mamba; up to 256K context; their blog cites ~3× throughput vs Mixtral 8x7B on long contexts and ~140K context on one GPU.

• Zamba: Mamba backbone plus one shared global attention every six layers (7B).

• Hymba: parallel attention + SSM heads; Hymba-1.5B claims beat Llama-3.2-3B average accuracy with an 11.67× smaller cache and 3.49× throughput (paper-reported).

• Griffin (DeepMind): gated linear recurrence (Hawk / RG-LRU) plus local attention. Paper claims Llama-2-level quality with far fewer tokens, scaled to 14B.

How to choose (short):

• Need the notebook : flexible lookup, ecosystem, and “just ship it”? Stay on Transformers and use the patches.

• Need the sticky note : linear time and lean decode, and you will live with compressed state? Look at Mamba 3.

• Need both retrieval and length? Start with a hybrid (Jamba/Zamba/Hymba/Griffin-style), which is also where the Mamba 3 authors themselves point for language models.

The quadratic tax did not vanish. We either pay it smarter and compress harder or split the difference.

Sources

1. Attention Is All You Need

2. Mamba: Linear-Time Sequence Modeling with Selective State Spaces

3. Transformers are SSMs (Mamba-2 / SSD)

4. Mamba 3: Improved Sequence Modeling using State Space Principles

5. Mamba 3: An Inference-First State Space Model (Cartesia)

6. state-spaces/mamba