CI/CD Pipelines That Deploy to Cloud Infrastructure Safely
A practical look at how to structure a deployment pipeline that touches real cloud infrastructure, with particular attention to how the pipeline proves who it is.
A pipeline that builds a container image is a build system. A pipeline that creates load balancers, opens security groups and attaches IAM roles is something else: it is a piece of production infrastructure with standing permission to change your account. The second kind deserves a different level of care, and most of that care goes into two questions. What is the pipeline allowed to do, and how does it prove it is allowed to do it?
This guide works through the design of that second kind of pipeline. It assumes you already have a cloud account to deploy into, whether that is one you opened yourself or one you picked up from the cloud account catalog, and that you want the automation to be boring rather than clever.
What a cloud deployment pipeline actually has to protect
Before you pick tools, write down what would go wrong if the pipeline misbehaved. In most accounts the honest answer falls into three buckets. It could delete something stateful that nobody has a backup of. It could open a network path that was supposed to stay closed. It could spend money, quietly and continuously, by creating resources that nothing ever cleans up.
Those three outcomes drive almost every design choice that follows. A pipeline with narrow permissions cannot delete the database. A pipeline whose network changes are reviewed by a human cannot silently expose an admin port. A pipeline that tags everything it creates makes the spend question answerable. None of that requires an elaborate platform. It requires deciding, early, that the automation is a principal in your account and not a convenience.
Stages that earn their place
Long pipelines are not safer pipelines. Each stage should exist because it can fail in a way that the previous stage cannot catch. A workable shape for infrastructure work looks like this:
- Lint and format. Fast, no credentials, catches the syntax problems that would otherwise burn a slow plan run.
- Static policy checks. Rules about what the configuration is allowed to express, for example no publicly readable object storage, no security group rule open to the whole internet on a management port.
- Plan or dry run. Read-only credentials, produces a diff that a human can read.
- Approval. Only for environments where a mistake is expensive.
- Apply. Write credentials, scoped to one environment.
- Post-apply verification. A handful of assertions that the thing you deployed is actually reachable and healthy.
Resist the urge to add stages that only report. A stage that nobody reads is a stage that trains people to ignore the pipeline output, which is the same problem that produces alerts engineers learn to dismiss.
How the pipeline authenticates to the provider
This is the part people get wrong most often, usually by reaching for the fastest option. The fastest option is a long-lived access key pasted into the CI system’s secret store. It works immediately and it is the worst of the available choices, because that key is valid until someone revokes it, it is valid from anywhere, and it tends to accumulate permissions over time as new resources get added to the configuration.
Better options exist across the major platforms. Most now support some form of federated identity, where the CI system presents a short-lived token that the provider exchanges for temporary credentials scoped to a specific role. The mechanics differ between Amazon Web Services, Google Cloud and Microsoft Azure, and the exact feature names change, so read the current provider documentation rather than a blog post from two years ago. The pattern is the same everywhere: the pipeline holds no secret, it holds an identity, and the provider decides what that identity may do.
If you are on a smaller platform where federation is not offered, you are back to keys. In that case, make them narrow and make them rotatable. One key per environment, one role per key, and a documented rotation procedure that someone has actually run once. A rotation procedure that has never been executed is a plan, not a control.
Scoping the role
Write the deployment role’s policy from the resource list in your configuration, not from a template. If the configuration creates compute instances, a load balancer and DNS records, the role needs those three services and nothing else. Grant delete permission deliberately. There are resources you genuinely want the pipeline to be able to replace, and there are resources, usually the stateful ones, where you would rather the apply fail loudly than succeed destructively. The least privilege guide covers how to build these policies without ending up with a role that is theoretically minimal and practically unusable.
Keeping provider credentials out of the repository
Even with federation, pipelines accumulate secrets: database passwords, API tokens for third party services, signing keys. Keep them in a secret manager and have the pipeline fetch them at run time, rather than injecting them as environment variables defined in the CI configuration file. The difference matters because a secret fetched at run time has an access log, and a secret defined in the CI configuration is visible to everyone who can read the repository.
Add a scanner to the lint stage that fails the build on anything resembling a credential. It will produce false positives on test fixtures. Tolerate that. The alternative is finding out about a committed key from a provider abuse notification. The patterns in secrets management for cloud environments apply directly to pipeline secrets, and the storage backend you pick there should be the one your pipeline reads from.
Running plan and apply as separate decisions
The plan output is the only moment where a human can see what the automation intends to do while it is still cheap to stop. Treat it as a first class artifact. Save it, attach it to the change request, and apply exactly the saved plan rather than recomputing at apply time. Recomputing opens a window where the account changed between plan and apply, and what gets applied is not what was reviewed.
For low risk environments, skip the human and let plan flow straight into apply. The decision of where to put the gate is yours, but make it explicit per environment rather than per repository. Development can be fully automatic. Anything holding customer data should not be.
Pipelines across separate provider accounts
If you follow the advice in separating cloud accounts by environment, your pipeline needs to target several accounts, and the credential story multiplies. Keep one identity per target account. Do not build a single powerful identity that can assume into everything, because the blast radius of that identity is every environment at once.
Account boundaries also change how you think about parallelism. Deploying to staging and production from the same pipeline run means a transient provider error in one can leave the other half applied. Sequence them, and make the production step depend on staging having passed its verification checks rather than merely having applied.
Failure modes worth rehearsing
A pipeline you have never seen fail is a pipeline whose failure behaviour is unknown. Three rehearsals are worth the afternoon:
- Apply interrupted halfway. Kill the runner mid-apply. Find out whether your state is recoverable and whether the next run cleans up or compounds the mess.
- Credentials revoked. Disable the deployment identity and confirm the pipeline fails with a clear message rather than retrying forever or falling back to some other credential it found on the runner.
- Provider rate limiting. Large applies hit API limits. Confirm your tooling backs off rather than failing the whole run, and read how service limits and quotas work so you know which limits are raisable.
A short checklist before you wire the first pipeline
Run through this before the automation gets write access to anything that matters. Can you name every permission the deployment identity holds, without looking at a wildcard? Does the identity expire, and if not, is there a rotation date on someone’s calendar? Is there a plan artifact that a human could read? Can you revoke the pipeline’s access from one place, quickly, if something goes wrong at two in the morning? Does every resource the pipeline creates carry a tag that identifies the environment and the owner?
If you are still choosing where to run this, the tradeoff between a hyperscaler and a smaller platform shows up here in a specific way: federation support, fine grained IAM and policy tooling are much deeper on the large platforms, while smaller providers are faster to reason about because there is less surface area. The AWS and DigitalOcean comparison walks through that difference, and a DigitalOcean account is often enough for teams whose infrastructure is a handful of services rather than a hundred.
Build the pipeline so that the worst thing it can do on its worst day is something you can undo before lunch. Everything else is detail.