Throughput-tuned nodes for serving models behind an API.
At batch size 1, token generation is bound by how fast weights can be streamed out of VRAM, not by tensor core throughput. That is why an H200 outruns an H100 on tokens per second despite identical FP16 figures, and why an L40S at 864 GB/s is a poor fit for latency-sensitive serving even though its compute numbers look strong.
Batching flips the picture. Once you are serving many concurrent requests, arithmetic intensity rises and the compute-heavy cards pull ahead. Decide which regime you are in before choosing hardware.
Cold starts cost more than most people budget for. Loading 70B of weights from NVMe takes 40-90 seconds. If your traffic is bursty, it is usually cheaper to keep one small instance permanently warm than to pay the latency penalty on every scale-up.