Sorted cheapest to most expensive. The right answer is usually the smallest card whose VRAM fits your model — throughput rarely matters if you are swapping to host memory.
| GPU | Arch | VRAM | Memory | Bandwidth | FP16 | FP8 | Interconnect | $/hr | $/GB VRAM/hr |
|---|---|---|---|---|---|---|---|---|---|
| RTX 3090 | Ampere | 24 GB | GDDR6X | 936 GB/s | 142 TFLOPS | n/a | NVLink Bridge (pairs) | $0.29 | $0.012 |
| V100 | Volta | 32 GB | HBM2 | 900 GB/s | 125 TFLOPS | n/a | NVLink 300 GB/s | $0.35 | $0.011 |
| RTX 4090 | Ada Lovelace | 24 GB | GDDR6X | 1.01 TB/s | 330 TFLOPS | 660 TFLOPS | Not supported | $0.44 | $0.018 |
| RTX A6000 | Ampere | 48 GB | GDDR6 ECC | 768 GB/s | 155 TFLOPS | n/a | NVLink Bridge (pairs) | $0.79 | $0.016 |
| L40S | Ada Lovelace | 48 GB | GDDR6 ECC | 864 GB/s | 733 TFLOPS | 1,466 TFLOPS | Not supported | $0.99 | $0.021 |
| A100 40GB | Ampere | 40 GB | HBM2 | 1.56 TB/s | 312 TFLOPS | n/a | NVLink Bridge (pairs) | $1.09 | $0.027 |
| A100 80GB | Ampere | 80 GB | HBM2e | 2.0 TB/s | 624 TFLOPS | n/a | NVLink 600 GB/s | $1.49 | $0.019 |
| H100 PCIe | Hopper | 80 GB | HBM2e | 2.0 TB/s | 1,513 TFLOPS | 3,026 TFLOPS | NVLink Bridge (pairs) | $2.19 | $0.027 |
| H100 SXM | Hopper | 80 GB | HBM3 | 3.35 TB/s | 1,979 TFLOPS | 3,958 TFLOPS | NVLink 900 GB/s | $2.49 | $0.031 |
| H200 | Hopper | 141 GB | HBM3e | 4.8 TB/s | 1,979 TFLOPS | 3,958 TFLOPS | NVLink 900 GB/s | $3.29 | $0.023 |
A rough guide for inference in fp16: multiply parameter count in billions by 2 to get GB of weights, then add 20-30% for the KV cache and activations. A 70B model needs about 170 GB — two H100s or one H200.
Full fine-tuning with Adam needs roughly 16 bytes per parameter once optimiser state and gradients are counted. LoRA and QLoRA cut that by an order of magnitude, which is why a 7B LoRA run fits comfortably on a single 4090.
Inference at low batch size is memory-bandwidth bound, not compute bound. An H200 at 4.8 TB/s will beat an H100 at 3.35 TB/s on tokens per second even though their FP16 numbers are identical.
If a job spans more than one node, InfiniBand and NVLink dominate the result. An 8x H100 SXM node keeps all-reduce inside NVLink; PCIe cards fall back to the host bus and lose badly at scale.
The doubled FP8 numbers on Hopper and Ada require framework support and a quantisation step. Treat FP16 as the honest figure unless you have already validated FP8 accuracy for your model.
For prototyping, evaluation and coursework, a $0.29/hr RTX 3090 with 24 GB does the same job as a $2.49/hr H100 and leaves budget for more experiments.