KV Cache Size Calculator
Paste a model's config.json and get the exact KV cache
size per token, for your context length and batch, and the
longest context that fits in a memory budget. It reads
num_key_value_heads and the published
head_dim instead of guessing them, which is where
most hand calculations go wrong, and handles MLA, sliding-window
layers and quantized caches. Works on private fine-tunes:
nothing leaves your browser.
How is KV cache size calculated?
For standard and grouped-query attention, every layer stores a key and a value vector for each KV head and each token: 2 x layers x num_key_value_heads x head_dim x bytes per value x tokens x batch. The two most common mistakes are using num_attention_heads instead of num_key_value_heads, which overstates the cache by the GQA group size (4x for Llama 3.1 8B, 7x for Qwen2.5-7B), and deriving head_dim as hidden_size / num_attention_heads when the config publishes a different head_dim.
Does it handle DeepSeek MLA, sliding windows and hybrid models?
Yes. For multi-head latent attention (configs with kv_lora_rank, such as DeepSeek-V2/V3), each layer caches one compressed latent plus a RoPE key per token instead of separate K and V per head. When a config lists layer_types, sliding-attention layers are capped at the sliding window and linear-attention layers, which keep a fixed-size state, are left out. Multimodal configs are read from text_config.
What isn't included?
This is the KV cache only. Model weights, activations, CUDA graphs and engine overhead come on top, and serving engines also reserve memory in blocks or pages, so plan some headroom. Nothing you paste leaves your browser.