Skip to content

Independent marketplace for verified cloud accounts

Model Serving Architectures: Batch, Real-Time and Streaming

Batch, real-time and streaming serving solve different problems and cost different amounts. Picking the wrong shape is the most expensive mistake in model deployment.

Training gets the attention and serving gets the bill. A model that runs for a week during training and then serves predictions for two years will spend almost all of its lifetime cost on the second part, and the architecture you pick for serving decides how large that number is.

There are three broad shapes: batch, real time, and streaming. Most production systems use more than one. The decision is not about sophistication, it is about when the prediction is needed relative to when the input arrives.

Three shapes, one decision

Ask a single question about each prediction your system makes: how long after the input exists does the answer have to be available?

If the answer is “by tomorrow morning”, you want batch. If it is “before this HTTP request returns”, you want real time. If it is “within a few seconds of the event, and the events never stop”, you want streaming. Everything else in this guide follows from that answer.

Teams often default to real time because it sounds like the general case that covers the others. It does cover them, at considerably higher cost and complexity, and a real time endpoint used to compute predictions that nobody reads for six hours is the most common piece of over engineering in this field.

Batch serving

Batch scoring runs on a schedule, reads a set of inputs, computes predictions for all of them, and writes the results somewhere for later lookup. Recommendation lists computed overnight, risk scores refreshed daily, content classification over a backlog: all batch.

The economics are excellent. The job runs on a large machine for a short time and then the machine goes away, so you pay for exactly the compute you used. Throughput can be maximised freely because no single prediction is being waited on, which means large batch sizes and full device utilisation. Interruption is cheap if the job is restartable, so preemptible capacity is often a good fit.

The limitation is freshness. A prediction computed at two in the morning does not know about anything that happened since. For a user who signed up an hour ago there is no row in the results table at all, and you need a fallback. That cold start gap is the main reason batch systems grow a real time component later.

Real-time serving

A real time endpoint holds the model in memory and answers requests synchronously. Something is waiting, usually a user, so the latency budget is tight and the load is whatever your traffic happens to be.

This shape is where the operational difficulty lives. The model must be loaded before the first request arrives, which means scaling up is slow if the model is large. Load varies, so you either over provision for the peak or accept degradation during it. And the device sits partly idle between requests, which is to say you are paying for capacity you are not using, by design.

Request batching is the main lever. Collecting requests that arrive within a short window and running them through the model together raises utilisation substantially at the cost of a small latency increase. Dynamic batching with a short timeout is standard practice in serving frameworks, and tuning that timeout is one of the highest value things you can do for throughput.

Cold starts and scaling lag

Scaling a real time service means starting an instance, pulling a container image, loading model weights from storage into device memory, and warming up. That sequence takes long enough that reactive autoscaling on request volume is often too slow to be useful. Options are keeping a warm pool, scaling on a leading indicator rather than on the metric you care about, or accepting a degraded tier during the ramp. The health check and warm up details in blue-green and canary deployments apply directly to how you bring these instances into rotation.

Streaming serving

Streaming sits between the two. Events arrive continuously on a queue or log, a consumer scores them, and results are written downstream. Nothing is blocking on an individual result, but the acceptable delay is seconds rather than hours.

The useful property is backpressure. When the input rate exceeds what the consumers can handle, the queue absorbs the difference instead of returning errors to a caller. That makes streaming far more tolerant of load spikes than a synchronous endpoint, and it means you can size the consumers for something close to average load rather than peak.

The cost is operational. You now own a queue, consumer group management, delivery semantics, and the question of what happens to a message that fails repeatedly. Ordering and exactly once processing are harder than they appear, and a poison message with no dead letter path will stall a partition indefinitely.

Latency budgets you can actually hold

Write the budget down as a number at a percentile, and allocate it across the path before choosing hardware. Model compute is one term. The others are network to the client, feature lookup, any preprocessing, serialisation, and queuing delay at the server.

Feature lookup is the term that most often dominates and the one least often measured. If scoring requires fetching ten features from a remote store, and each lookup is a round trip, the model could be instant and the request would still be slow. Co locate the feature store with the serving tier, batch the lookups, and cache what is stable.

Budget at a high percentile, not the mean. Mean latency hides exactly the behaviour that causes complaints, and a serving tier tuned on averages will look fine in the dashboard while a meaningful share of users wait.

Hardware and autoscaling per shape

The three shapes want different machines and different scaling policies.

Shape Utilisation pattern Scaling signal Interruption tolerance
Batch High and sustained for a bounded window Queue of work to process High with checkpointing
Real time Low to moderate, spiky Concurrency or in flight requests Low
Streaming Moderate and fairly steady Consumer lag Moderate

Scale real time on concurrency rather than on device utilisation. Utilisation is a lagging and noisy signal for a workload where requests queue, and by the time it moves the latency has already degraded. For streaming, consumer lag is the correct signal and it is also the correct alert, since rising lag is the earliest honest sign that the system cannot keep up.

Not every model needs an accelerator for serving. Smaller models, and larger ones under light load, often serve acceptably on CPU, which removes an entire class of capacity and quota problems. Measure before assuming, and read what to look for in GPU instances for the memory and host side constraints that decide whether a device is a fit.

Versioning and rollout of models

A model version is a deployment artefact and deserves the same handling as application code. Give every version an immutable identifier, store the weights in versioned object storage, and record which data and code produced it.

Rollout differs from application rollout in one important way: a new model version can be entirely healthy from an infrastructure perspective and still be worse. Error rates and latency will not tell you. You need a prediction quality signal, which usually means shadow traffic, where the new version scores real requests and the results are compared offline without being served.

Keep the previous version loadable. Rolling back a model should be a configuration change, not a rebuild, and the only way to be sure is to have done it once deliberately.

Mixing shapes in one system

Most mature systems end up with a combination: batch precomputation for the common cases, a real time path for inputs the batch job has not seen, and streaming for event driven updates in between. This is a sensible destination, not a failure of design.

The trap is divergence. If the batch path and the real time path compute features differently, the two will disagree, and debugging that disagreement is miserable. Share the feature computation code between them. The training and serving consistency problem is covered in data pipelines for machine learning, and it is the same problem in a different place.

Where the cost actually lands

For batch, cost is roughly proportional to work done, which makes it predictable and easy to forecast. For real time, cost is proportional to provisioned capacity, which means it tracks your peak and your redundancy requirement rather than your traffic. For streaming it falls in between and includes the queue infrastructure itself.

Egress deserves a mention. A serving tier returning large payloads, or reading model weights across regions on every scale up event, generates transfer charges that are easy to miss when forecasting. The mechanics are in cloud egress fees explained.

Platform choice follows from which shape dominates. Batch heavy workloads reward raw compute value and are commonly run on providers with straightforward hourly pricing such as a Hetzner account or a UpCloud account. Real time serving with aggressive autoscaling and managed queues leans toward the larger platforms. The AWS and UpCloud comparison covers that split, and predictable pricing versus pay as you go covers which billing model suits which load profile.

Pick the shape from the latency requirement, not from the architecture diagram you would prefer to draw.

Related comparisons

Hetzner vs Linode
vs

Hetzner vs Linode

Both platforms were built by engineers for engineers, but they answer to different owners now. This compares operating philosophy and…

AWS vs OVH
vs

AWS vs OVH

When jurisdiction is the requirement rather than the preference, OVHcloud and AWS answer it differently. This comparison starts from data…

Kamatera vs UpCloud
vs

Kamatera vs UpCloud

Kamatera lets you specify a machine resource by resource with hands on support behind it. UpCloud offers a tighter product…

Digital Ocean vs Vultr
vs

Digital Ocean vs Vultr

Vultr gives you more places to run a server and more instance shapes to choose from. DigitalOcean gives you a…

Keep reading

How to Read a Cloud Account Listing
Cloud Accounts 7 min read

How to Read a Cloud Account Listing

Every line in an account listing is a claim that can be checked. This guide turns listing language into a set of…

Need the account, not just the theory?

Browse documented configurations across every provider in the catalog, or ask us which one fits your workload.

Telegram