Skip to content

Independent marketplace for verified cloud accounts

GPU Instances for Training and Inference: What to Look For

Choosing GPU capacity in the cloud is less about peak performance numbers and more about memory, interconnect, availability and the billing model you commit to.

GPU Instances for Training and Inference: What to Look For

GPU selection discussions tend to start with performance comparisons and end with a shortage. In practice the sequence runs the other way. You first find out what your region actually has available, then what fits your memory requirement, then what the billing model does to your budget, and performance per chip turns out to be the least contested variable in the decision.

This guide is about the criteria that decide the outcome. It deliberately avoids quoting throughput figures or per hour rates, because those change frequently, vary by region and account, and the published number is rarely what your workload achieves anyway. Measure on the hardware you can actually get.

Training and inference are different purchases

The two workloads stress different things, and buying one kind of capacity for the other is a common and expensive mistake.

Training is throughput bound and tolerant of interruption if you checkpoint properly. It runs for hours or days, saturates the device, and benefits enormously from fast interconnect when it spans multiple GPUs. Nobody is waiting on an individual step, so batching is free and latency is irrelevant.

Inference is latency bound and intolerant of interruption. It runs continuously at variable load, often leaves a large part of the device idle between requests, and cares about how quickly a single request completes. Multi GPU interconnect usually does not matter unless the model does not fit on one device.

The practical consequence: training wants the largest, most tightly coupled machine you can get for the shortest time, and inference wants the smallest machine that holds the model with enough headroom, running for a long time. Those are almost opposite procurement strategies.

Memory is usually the binding constraint

For most teams the question is not how fast a GPU is but whether the work fits in its memory. If it does not fit, the fastest chip in the catalogue runs your job at zero steps per second.

For training, device memory has to hold the model parameters, the gradients, the optimiser state, and the activations for your batch size. Optimiser state is the part people forget, and depending on the optimiser it can be several times the size of the parameters themselves. Activations scale with batch size and sequence length, which is why reducing batch size is the first thing anyone tries when a job runs out of memory.

For inference, memory has to hold the parameters at whatever precision you serve them in, plus per request working state that grows with concurrency and context length. The second term is what catches people out: a model that loads comfortably can still exhaust memory under concurrent load.

Quantisation and lower precision formats reduce the parameter footprint substantially and are now routine, but they are a quality decision as well as a capacity one. Evaluate the quantised model on your own task rather than assuming the published quality holds for your data.

Interconnect and multi-GPU scaling

Once a job spans more than one GPU, how those GPUs talk to each other becomes a first order concern. Within a single host, high bandwidth direct links between devices are far faster than going over the host’s general purpose bus. Across hosts, the network fabric decides whether adding nodes helps or merely adds coordination overhead.

Scaling is rarely linear and it is important to measure rather than assume. Run your actual training step on one device, then two, then four, and look at the achieved speedup. If it falls off sharply, you are communication bound, and the answer is either a better interconnect tier, a different parallelism strategy, or accepting fewer devices.

Only commit to a large multi node setup once you have demonstrated the scaling. Reserved capacity that your job cannot use efficiently is the single most expensive mistake available in this category.

The rest of the machine matters

A GPU instance is a whole computer, and the non GPU parts starve the device surprisingly often. Three to check:

  • Host CPU and RAM. Data loading, augmentation and tokenisation run on the CPU. Too few vCPUs and the GPU waits for input, which looks exactly like a slow GPU in your metrics.
  • Local disk. Many instance families attach fast local storage that is ephemeral. It is excellent for dataset staging and it disappears when the instance stops, so nothing durable belongs there.
  • Network to storage. If training data streams from object storage every epoch, your effective throughput is bounded by that link. Understanding the figures in reading cloud storage performance specs helps here more than any GPU comparison.

Profile before you upgrade. A job that looks GPU bound is frequently input bound, and the fix is a bigger host or a better data layout rather than a bigger accelerator.

Availability, regions and queueing

GPU capacity is not uniformly available. The configuration you want may exist in some regions and not others, may be gated behind a quota request, and may be genuinely unavailable at busy times regardless of quota. Treat availability as a selection criterion in its own right.

Three things to do before you design around a particular instance family. Confirm it exists in a region you are allowed to use for your data, which matters if residency rules apply. Request the quota early, because approval is not instant and the default allocation for accelerators is frequently zero. And have a second choice, because designing a pipeline that only runs on one scarce configuration means your schedule depends on someone else’s capacity planning.

Region choice also interacts with where your data already sits. Moving a large training set across regions costs money and time, and the egress charges described in cloud egress fees explained apply to that transfer. Put the compute near the data, not the other way around. The mechanics of regions and zones are covered in how cloud regions and availability zones work.

Billing shapes: on-demand, reserved and preemptible

The same hardware is sold several ways, and the difference between them is larger than the difference between hardware generations for most budgets.

On demand is flexible and the most expensive per unit time. It is correct for exploratory work and for anything whose duration you cannot predict.

Committed or reserved capacity trades flexibility for a lower rate over a term. It only makes sense when you have evidence of sustained utilisation. Teams commit far too early, usually after one busy month, and then pay for idle accelerators through a quiet quarter.

Preemptible or spot capacity is cheaper and can be reclaimed with little notice. For training with reliable checkpointing it is often the right default, because losing ten minutes of progress is an acceptable price. For inference serving live traffic it is only viable with an on demand fallback behind it. The general tradeoffs are laid out in reserved, spot and on-demand pricing.

Trying hardware before you commit

Benchmark your own workload. Published numbers use reference implementations, reference batch sizes and reference data, and your pipeline is none of those. The measurement that matters is steps per hour on your model with your data loader, or requests per second at your latency target with your concurrency.

Promotional credits are a common way to fund that evaluation, and they are reasonable for it provided you treat them accurately. Credits are controlled by the provider, they carry an expiry date, and they often exclude particular services or instance families, which with accelerators is a restriction that bites specifically. Verify the balance and the terms inside the provider’s own console before you plan a schedule around them, and read running AI workloads on cloud credits for how to sequence experiments against an expiry date.

Accelerator availability and billing structure differ meaningfully between platforms. The hyperscalers carry the widest catalogue and the deepest quota machinery, while other platforms position themselves on simpler pricing and easier access. An AWS account and an Oracle Cloud account represent two quite different approaches to that market, and the AWS and Oracle Cloud comparison sets out where each tends to suit.

Questions to ask before you reserve capacity

Can you state, from a measurement rather than an estimate, the device memory your job needs at your target batch size? Have you profiled a single device run to confirm the GPU is the bottleneck? Do you know your scaling efficiency at the device count you plan to buy? Is the instance family available in the region where your data lives, and do you hold the quota? What is your utilisation over the last month, and does it justify a commitment term?

If any of those answers is a guess, rent before you commit. Accelerators are the one part of a cloud bill where a confident guess can cost more than the project.

Related comparisons

Hetzner vs Vultr
vs

Hetzner vs Vultr

This comparison comes down to one trade: Vultr puts compute in many more places around the world, while Hetzner delivers…

Google Cloud vs Oracle Cloud
vs

Google Cloud vs Oracle Cloud

Two enterprise platforms built around different centres of gravity. Oracle Cloud is organised around the database; Google Cloud is organised…

Google Cloud vs Vultr
vs

Google Cloud vs Vultr

Both platforms will rent you a GPU, and that is where the similarity ends. Vultr gives you the hardware; Google…

AWS vs UpCloud
vs

AWS vs UpCloud

UpCloud sells storage performance and operational simplicity as the product. This comparison tests that against AWS for latency sensitive workloads…

Keep reading

Need the account, not just the theory?

Browse documented configurations across every provider in the catalog, or ask us which one fits your workload.

Telegram