Tool
Fine-tuning memory calculator
Will this fine-tune fit? Pick a model size, the method and the batch, and see where the memory goes: weights, gradients, optimizer state, the LoRA adapter, activations and the fp32 logits, plus what evaluating the model at the same length needs. The formulas, their sources and the measured runs they were checked against are below the tool.
Calculator
Training peak
Fill in the form
| Consumer | GiB |
|---|
How to read the result
Allocated is what PyTorch's allocator hands out at the training peak, the number
torch.cuda.max_memory_allocated() reports. The training peak adds an
overhead line: the CUDA context, allocator fragmentation and, on unified memory, the Python process and
page cache that share the same pool. That is the number to compare with your GPU. GiB means
230 bytes; GPUs are sold in GB (109 bytes), so the "fits" line converts.
The evaluation line is the same model generating at the same length with batch 1: weights plus the KV cache. It is usually far below the training peak, which is why a model you can evaluate may still not train. For the bytes-per-parameter view of the same terms, see the memory table in Lesson 1.15 (GPU memory math); when a run does not fit, Lesson 1.16 (OOM and how to survive it) lists the levers in order.
The formulas
All terms are bytes. P is the total parameter count, T the trainable (adapter) parameters, n = micro-batch × tokens per row, L layers, h hidden size, q = heads × head size, kv = key/value heads × head size, f the MLP width (top-k × expert width for a mixture of experts), V the vocabulary.
| Term | Formula | Source |
|---|---|---|
| Weights | P × 4 (fp32), × 2 (bf16, fp16), × 6 (mixed precision: fp32 master + 16-bit copy)8-bit: 1 B, NF4: 4.127 bits with double quantization, 4.5 without, for the layers bitsandbytes quantizes (nn.Linear only)the rest (embeddings, an untied output head, norms, biases, the router and fused expert tensors) × 4 after prepare_model_for_kbit_training, × 2 without it | Lesson 1.15 memory table; QLoRA, Dettmers et al. 2023, arXiv:2305.14314, §3; the fp32 cast is peft 0.21's prepare_model_for_kbit_training, the recipe of Lesson 2.11. Counted on small models loaded in 4-bit on a CPU (bitsandbytes 0.50): within 0.03% |
| Full fine-tune: gradients, optimizer | gradients P × 4 (2 in pure 16-bit); AdamW P × 8 (P × 4 in pure 16-bit, where PyTorch keeps its moments in bf16); 8-bit AdamW P × 2mixed precision with AdamW: 6 + 4 + 8 = 18 B per parameter | The 18 B, with fp32 gradients, is Hugging Face's model memory anatomy count. ZeRO (Rajbhandari et al. 2019, arXiv:1910.02054, §3.1) counts fp16 gradients, 2 + 2 + 12 = 16 B; Lesson 1.15 gives both. 8-bit optimizers: Dettmers et al. 2021, arXiv:2110.02861 |
| LoRA adapter | T = L × Σ r × (d_in + d_out) over the adapted matricesfused experts (E experts, expert width f), picked by module name (target_modules with gate_proj, up_proj, down_proj, which peft 0.21 converts): gate and up together 2·E·r·(h + 2f), down E·r·(f + h), per layerpicking the expert tensors directly (target_parameters) gives about half the gate/up adapter, E·r·(h + 2f), so the figure here is then on the high sideadapter weights T × 4, gradients T × 4, AdamW T × 8 (8-bit: T × 2) | LoRA, Hu et al. 2021, arXiv:2106.09685; peft keeps adapters in fp32 |
| Activations, per layer and token (no checkpointing) | full: e(6h + 2q + 2kv + 4f) + 2(4 − e)h + 4·headsLoRA: e(2h + 2q + 2kv + 3f) + 2(4 − e)h + 4·heads + the adapters' inputse = 4 bytes in fp32, 2 in bf16/fp16, and 4 on a quantized base after prepare_model_for_kbit_training (its fp32 embeddings make the residual stream fp32); 2(4 − e)h is the RMSNorms computing in fp32mixture of experts: the routed tokens' expert MLP (top-k × expert width), routing indices and router probabilities, and q/k norms, from separate counts; on expert adapters, plus a full copy of the layer's expert tensors (e bytes each) whatever the batch | Counted, not assumed: bytes autograd keeps for backward (saved_tensors_hooks, the method of Lesson 1.15) on decoder layers with RMSNorm, a SwiGLU MLP and grouped-query attention with SDPA, for 8 shapes in fp32 and bf16, and on fused-expert layers for 8 more shapes, full and five LoRA target sets, with transformers 5.22, peft 0.21 and torch 2.15 on a CPU; the formulas reproduce every count. Counted without autocast: under bf16 autocast a LoRA run keeps 16-bit copies of the adapters' inputs, so the estimate is on the high side (by about 15–20% on small models). Korthikanti et al. 2022, arXiv:2205.05198, gives the classic analytic version; with SDPA or flash attention its sequence² term is gone |
| Adapter inputs (LoRA) | fp32 base: 4h if any of q, k, v; 4h if gate or up; 4f if down16-bit base: an fp32 copy of each adapted matrix's input (4h for q, k, v, gate, up; 4q for o; 4f for down); plus 4r per adapter | Same counts |
| With gradient checkpointing | L × e × h × n (each layer's input) + one layer rebuilt during backward | Same counts; Chen et al. 2016, arXiv:1604.06174 |
| Logits and loss | n × V × 12: fp32 log-probabilities kept for the loss (4) + the fp32 gradients of the log-probabilities and of the logits while the loss backpropagates (8)that is Hugging Face's loss path; it is 14 if your code still holds outputs.logits (16-bit) during backward, 16 if those logits are fp32 | Same counts for the 4 B; Wijmans et al. 2024, arXiv:2411.09009 |
| Overhead | 0.10 × allocated + 1 GiB (own GPU memory) or + 7 GiB (unified memory) | A rule of thumb calibrated on the one measured run below, not a law |
| Evaluation | (weights + KV cache) × 1.10 + the same constant; KV cache = 2 × L × kv × 2 B × tokens | PagedAttention, Kwon et al. 2023, arXiv:2309.06180, §2 |
Validation: the formulas against measured runs
Every row in the first two tables is a measurement made outside this calculator. The 135M rows are the model of the worked example in Lesson 1.15 (bytes autograd keeps for backward, fp32 on a CPU, batch 8 × 256 tokens). The 30B-A3B rows are one LoRA run on a 128 GB unified-memory GPU box: r = 16 on q, k, v and o, bf16 weights, paged 8-bit AdamW, micro-batch 2, rows up to about 1,504 tokens, gradient checkpointing on. The site's tests run the calculator's own code on these cases and fail when an estimate leaves its tolerance.
| Run and quantity | Measured | Calculator | Error | Tolerance |
|---|---|---|---|---|
| 135M, LoRA r=16 all linear, kept for backward | 2.89 GB | 2.89 GB | +0.0% | ±3% |
| … with gradient checkpointing | 0.55 GB | 0.55 GB | −0.2% | ±3% |
| … as a full fine-tune | 3.16 GB | 3.16 GB | −0.1% | ±3% |
| 135M, adapter parameters | 4,884,480 | 4,884,480 | exact | exact |
| 30B-A3B, adapter parameters | 13,369,344 | 13,369,344 | exact | exact |
| 30B-A3B, bf16 weights | ~57 GiB | 56.9 GiB | −0.2% | ±2% |
| 30B-A3B, allocated at the training peak | 63.6 GiB | 63.1 GiB | −0.7% | ±5% |
| 30B-A3B, whole pool at the training peak | ~77–78 GiB | 76.5 GiB | −1.4% | ±5% |
| 30B-A3B, evaluation at the same length | ~69.5 GiB | 69.7 GiB | +0.3% | ±5% |
The mixture-of-experts and quantized-base terms were counted on small models on a CPU (no GPU run behind them): saved-tensor bytes per layer and per token as in Lesson 1.15, and the bytes of the loaded weights including bitsandbytes' quantization state. Small shapes, so they test the formulas, not the overhead line.
| Run and quantity | Measured | Calculator | Error | Tolerance |
|---|---|---|---|---|
| Small fused-expert MoE, bf16 full fine-tune: bytes per token per layer | 12,180 B | 12,180 B | exact | exact |
| … LoRA r=8 on all linear, experts included | 13,588 B | 13,588 B | exact | exact |
| … bytes per layer whatever the batch (merged expert tensors) | 1,753,120 B | 1,753,120 B | exact | exact |
| … the same LoRA run in fp32: bytes per token per layer | 16,152 B | 16,152 B | exact | exact |
| … adapter parameters, 2 layers | 208,896 | 208,896 | exact | exact |
| Small dense model in NF4 after the kbit prep, LoRA r=8 on q, k, v, o: bytes per token per layer | 14,744 B | 14,736 B | −0.1% | ±1% |
| Dense model with biases in NF4 + double quantization after the kbit prep: weights | 285.5 MB | 285.5 MB | −0.01% | ±0.5% |
| Fused-expert MoE loaded in NF4 (experts stay 16-bit): weights | 68.2 MB | 68.1 MB | −0.03% | ±0.5% |
| … after the kbit prep (experts in fp32): weights | 135.0 MB | 134.9 MB | −0.01% | ±0.5% |
Consistency with Lesson 1.15's memory table
These rows are not measurements: they are the memory table's arithmetic, and the calculator is built on that table, so they check that it reproduces it, not that it is right.
| Quantity | Table | Calculator | Agreement | Tolerance |
|---|---|---|---|---|
| 135M, full fine-tune in fp32: weights, gradients, AdamW | 2.15 GB | 2.15 GB | 16 B × P | ±1% |
| Bytes per parameter, mixed precision + AdamW | 16–18 B | 18 B | the table's upper end (fp32 gradients) | exact |
| Bytes per parameter, plain fp32 + AdamW | 16 B | 16 B | exact | exact |
| Bytes per parameter, pure bf16 + PyTorch AdamW | 8 B | 8 B | exact | exact |
Two honest caveats. The overhead line is calibrated on that one 30B-A3B run, so the whole-pool rows are not an independent test of it; on another machine, driver or allocator setting the overhead can differ by a few GiB. And the 1.15 measurements count tensors kept after the forward pass, so the calculator's activation bar adds what they cannot see: with checkpointing on, the one layer that is rebuilt during backward (0.08 GB in the 135M case).
The unified-memory gotcha: loading
On a machine where the CPU and GPU share one memory pool, the training peak is not the only peak. Loading weights to the CPU and then copying them to the GPU briefly holds two copies, and the operating system's page cache can hold a third (the files just read). For the 30B-A3B model that is about 114 GiB plus up to 57 GiB of cache on a 128 GB box: an out-of-memory error before the first step, even though training itself peaks near 77 GiB. The fix is to create the CUDA context first and load the weights straight to the device. In the calculator, pick "unified memory" to see the load line.
What this ignores
- Attention without SDPA or flash attention ("eager"): it keeps a heads × tokens² matrix per layer, which this calculator does not add.
- Fused or chunked loss kernels, which shrink the logits term; and TRL casting a QLoRA adapter to bf16, which brings its states to about 8 bytes per parameter instead of 16.
- Other architectures: the activation formulas were counted on decoder layers with RMSNorm, a SwiGLU MLP and grouped-query attention, and on fused-expert mixture-of-experts layers with q/k norms (transformers 5). Other layouts, such as per-expert linear layers or no q/k norms on a mixture of experts, are approximated from the same terms.
- Quantized mixtures of experts on a GPU: that bitsandbytes leaves fused experts unquantized was checked by loading a small model on a CPU, not in a GPU training run.
- Multi-GPU sharding (ZeRO, FSDP), CPU offload, sequence packing, and DeepSpeed or framework buffers.
- Short-lived buffers: a quantized layer's dequantized copy, matmul workspace, the optimizer step's temporaries.
- Rows shorter than the longest one in their batch: real batches vary, so this is an upper bound for the length you enter.
- Speed. The calculator says whether a run fits, not how long it takes.
Questions
Why does a 1.5B model need over 20 GiB to train at 2,048 tokens?
Mostly the logits. Every token produces a score for every vocabulary entry, and the loss works on them in fp32: with a 152k vocabulary, a micro-batch of 4 × 2,048 tokens keeps about 4.6 GiB of fp32 log-probabilities and needs about twice that again while the loss backpropagates, about 14 GiB in all, against 2.9 GiB of bf16 weights. Halving the micro-batch or the row length halves it; a fused loss kernel avoids most of it.
Does gradient accumulation raise peak memory?
No. The peak is set by the micro-batch, the rows on the device in one step. Accumulation runs more micro-batches before each optimizer step and adds their gradients into buffers that exist anyway, so it raises the effective batch without raising the peak.
Why doesn't LoRA shrink activation memory?
LoRA freezes the base weights, so it removes their gradients and optimizer state. The backward pass still runs through every layer to reach the adapters, so the tensors each layer keeps for it are still there. Gradient checkpointing is the lever for activations: it keeps only each layer's input and rebuilds the rest during backward, at the cost of about one extra forward pass.
Why does a 30B mixture of experts need memory for 30B parameters when only 3B are active?
Every expert has to be in memory, because any token can be routed to any of them. The active count sets the compute per token and the size of the activations; the total count sets the weights. That is why the 30B-A3B run spent about 57 GiB on weights and only a few GiB on activations.
What does QLoRA save?
The frozen base: 4-bit NF4 weights take about a quarter of their bf16 size (4.127 bits per parameter with double quantization). Only the linear layers are quantized. The embeddings, an untied output head and the norms are not, and prepare_model_for_kbit_training, the recipe in Lesson 2.11, casts them to fp32, which also makes the activations fp32. So on a small model with a large vocabulary the saving is small: a 7B model with an untied 152k vocabulary keeps about 1.1 billion parameters in fp32, about 4 GiB. On a fused-expert mixture of experts (bitsandbytes 4-bit in transformers 5) the experts are not quantized at all. QLoRA helps most on large dense models with modest batches.
How accurate is this?
On the measured runs in the validation table it is within 1.5%, and the activation formula reproduces the per-layer counts it was fitted on exactly. Real runs differ by the overhead (driver, allocator, other processes), by architecture details this does not model, and by padding. Read it as a plan, then confirm with torch.cuda.max_memory_allocated() after the first steps.
Related
- Lesson 1.15 (GPU memory math): the four consumers and the memory table.
- Lesson 1.16 (OOM and how to survive it): what to change when it does not fit.
- Lesson 1.14 (LoRA knobs): rank, target modules and QLoRA.
- Lesson 2.11 (QLoRA hands-on): a 4-bit base with bitsandbytes, in code.
- Is the lift real? The companion tool for reading an evaluation.