LLM training and fine-tuning

Sizing the job

Full fine-tuning consumes roughly 16 bytes per parameter once you count weights, gradients and Adam optimiser state. A 7B model therefore needs around 112 GB before activations, which puts it beyond a single 80 GB card. Parameter-efficient methods change that maths completely: a QLoRA run on the same model fits inside 24 GB, which is why so many people start on an RTX 4090.

If you are pre-training rather than fine-tuning, the constraint moves from memory to interconnect. Gradient all-reduce across nodes will dominate step time on anything slower than InfiniBand, so the SXM nodes are the only sensible choice above one machine.

A typical setup

• Start an H100 SXM node with the CUDA 12.4 + PyTorch 2.4 image
• Attach a persistent volume for checkpoints so they survive teardown
• Pull your dataset over the 3.2 Tbps fabric, not over the public interface
• Run with torchrun and NCCL; the image ships a tuned NCCL config
• Destroy the instance when the run finishes - billing stops immediately

Cost example

A 7B LoRA fine-tune over 3 epochs on 50k samples takes roughly 4 hours on a single A100 80GB. At $1.49/hr that is under $6. The same job on an H100 SXM finishes in about 2.4 hours at $2.49/hr, so around $6 again - faster for the same money, which is usually the right trade.