Data Pipelines for Machine Learning in the Cloud
Machine learning pipelines fail differently from reporting pipelines. Reproducibility, feature consistency and file layout matter more than they do for analytics.
The pipeline is usually where a machine learning project actually lives or dies. Model code is a few hundred lines and changes rarely. The thing that moves, breaks, silently changes shape and quietly degrades your results is the data path feeding it. Most of the engineering effort on a mature ML system is spent there.
This guide covers how to build that path on cloud infrastructure, with attention to the places where machine learning requirements differ from the reporting pipelines most teams have built before.
Pipelines for ML differ from pipelines for reporting
An analytics pipeline is judged on whether today’s dashboard is correct. If a schema changed last month and the historical numbers shifted slightly, nobody usually notices or cares.
A machine learning pipeline is judged on something stricter. You need to be able to recreate the exact dataset a model was trained on, months later, to debug a regression or satisfy a question about how a decision was made. You also need the features computed at serving time to match the features computed at training time, precisely, because a small inconsistency there produces a model that looks excellent in evaluation and performs poorly in production.
Those two requirements, reproducibility and train/serve consistency, drive nearly every design decision below. They are the reason ML pipelines cost more to build than their analytics equivalents.
Ingestion and the raw landing zone
Land raw data first, unmodified, in object storage, partitioned by arrival date. Do not transform on the way in. The landing zone is your ability to rebuild everything downstream when you discover a bug in the transformation logic six weeks from now, and it only has that property if it is genuinely untouched.
Make the landing zone append only and immutable. Writes create new partitions, nothing overwrites, and lifecycle policies move older data to cheaper storage classes rather than deleting it. The tradeoffs between storage classes and retrieval behaviour are covered in cloud storage types explained, and they matter here because training data is read repeatedly but unpredictably.
Record the source and arrival time of every file as metadata, not as something inferred from a filename. Filename conventions break, and when a backfill produces files with yesterday’s naming scheme you will want a field you can trust.
Validation before transformation
Check incoming data before anything downstream consumes it. The checks that pay for themselves are not sophisticated:
- Schema. Expected columns present, types as declared. Fail loudly on an unexpected column rather than silently ignoring it, because a new column is frequently a sign that something upstream changed meaning.
- Volume. Row count within a plausible range for this source and this day. A feed delivering a tenth of its usual volume is broken, not quiet.
- Null rates. Per column, compared against recent history. A column that was two percent null and is now sixty percent null has changed upstream.
- Distribution. For important numeric fields, basic summary statistics against a recent baseline. This catches unit changes, which are otherwise invisible and devastating.
Decide per check whether a failure stops the pipeline or raises a warning. Stopping on everything means the pipeline halts weekly and people start skipping checks. Stopping on nothing means the checks are decoration. Schema and volume usually warrant a stop; distribution drift usually warrants a notification and a human look, routed with the same judgement described in monitoring and alerting that engineers do not ignore.
Feature computation and the training/serving gap
This is the single most common source of models that underperform in production without any obvious cause. Features are computed one way in the training pipeline, usually in a batch framework over historical data, and another way at serving time, usually in application code. The two drift apart, often over something trivial like how a missing value is filled or which timezone a timestamp is interpreted in.
Two approaches reduce the risk. Share the computation code so both paths call the same function, which works well when both run in the same language. Or precompute features in the pipeline and have the serving path read them rather than compute them, which is what a feature store does.
Whichever you pick, add a consistency test: take a sample of production requests, compute features through both paths, and compare. Run it on a schedule. A discrepancy found by a test is a bug; a discrepancy found by a model quality investigation three months later is a week of work.
Point in time correctness
Training on a feature value that would not have been known at prediction time leaks future information into the model. It produces excellent offline results and a model that fails in production. Any feature joined to a training label must be computed from data available at the label’s timestamp, which means your pipeline needs event time, not just processing time, and the join must respect it.
Storage layout and file sizes
How you lay files out determines how fast training reads them, and the effect is large enough to be worth deliberate attention.
Use a columnar format for tabular data, because training usually reads a subset of columns across all rows. Partition by the field you filter on most, usually date. And watch file sizes: thousands of tiny files make every read a separate request with its own latency, while a handful of enormous files prevents parallel reads and makes shuffling awkward. A compaction step that consolidates small files into moderately sized ones is routine maintenance.
Co locate the data with the compute. Reading a training set from object storage in another region on every epoch is slow and generates transfer charges, as set out in cloud egress fees explained. If the accelerator capacity you can get is in a different region from your data, move the data once and deliberately rather than streaming it repeatedly.
Orchestration and backfills
Pipeline steps have dependencies and an orchestrator is how you express them. The requirements are unglamorous: a dependency graph, retries with backoff, a schedule, visibility into what ran and what failed, and the ability to rerun a single step for a historical date.
That last capability is the one teams underestimate. Backfills are constant in machine learning work, because every change to feature logic implies recomputing history. Design each step to be idempotent and parameterised by date from the beginning. A pipeline where rerunning a step duplicates rows instead of replacing them will eventually corrupt a training set, and you will not notice until the model is worse.
Run the orchestrator somewhere with modest, predictable resource needs. It coordinates work rather than doing it, so it does not belong on expensive capacity. The deployment path for it should go through the same automation as everything else, following the infrastructure as code provider model.
Lineage, versioning and reproducibility
To recreate a training run you need four things pinned together: the data, the feature logic, the model code, and the configuration. Git handles the last two. The first two need explicit handling.
Version datasets by writing each materialised training set to an immutable path that includes a version or timestamp, and record that exact path in the training run’s metadata. Pointing a training job at a path that is overwritten by the next pipeline run means you can never reproduce anything.
Keep a run record for every training job: the dataset path, the code commit, the hyperparameters, the resulting metrics, and where the weights were written. This can be a managed experiment tracker or a table you write to yourself. The format matters much less than the discipline of writing it every time, including for the quick experiment that turns out to be the one that goes to production.
Access control on training data
Training data is frequently the most sensitive data a company holds, assembled in one place and read by jobs that run unattended. Grant pipeline identities read access to the specific prefixes they need, write access only to their own outputs, and nothing else. The patterns in least privilege IAM apply directly.
Pay attention to derived artefacts. Model weights, cached feature files, evaluation outputs and debugging samples all inherit the sensitivity of their source and all tend to be stored with weaker controls than the original. If personal data is involved, the obligations around retention and deletion follow it into every derived copy, which is covered in cloud compliance for GDPR, HIPAA and SOC 2. Residency requirements also constrain where you can place the pipeline, which may rule out the region with the accelerator availability you wanted.
Cost control in the data layer
Storage grows monotonically unless something stops it, and ML pipelines generate intermediate artefacts prolifically. Set lifecycle policies on every bucket at creation, expire intermediate outputs after a defined window, and clean up incomplete multipart uploads, which accumulate invisibly and bill like complete objects.
The processing side responds well to scheduling. Pipeline steps are batch work with no one waiting, so interruptible capacity suits them, and the pipeline clusters themselves should be torn down after each run rather than left warm. The lease based approach in automating environment teardown works as well for data infrastructure as for application environments.
Storage and egress behaviour varies considerably between platforms, and for a data heavy workload that difference often outweighs the compute price. Providers that include generous transfer allowances change the arithmetic for pipelines that move data frequently. The Hetzner and DigitalOcean comparison covers that ground, and a Scaleway account or a OVHcloud account are both worth evaluating when the data layer dominates the bill.
Build the pipeline so that any engineer can rerun last March and get last March’s numbers. Everything else is easier once that is true.