The KV Cache Bottleneck: Architectural Alternatives

In autoregressive decoding, each new token recomputes the keys and values for all preceding tokens. This cache grows linearly with context length, and in long-context serving it becomes the dominant memory and bandwidth cost - quite often more limiting than raw compute itself.
In this session, Abhay Kumar, co-founder of FrontiersMind, shall walk through the main architectural responses to this problem and what each one costs us. Grouped-query and multi-query attention bring down the number of KV heads. Cross-layer sharing Cross-Layer Attention and the shared KV cache in the recent Gemma…

Hosted by Team Lossfunk, Paras Chopra & Jay Joshi