Skip to main content

GeekZilla.io

Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors

Why Terraform at Scale Breaks — And What Teams Get Wrong About Fixing It

There’s a moment most platform engineers recognize. It usually happens sometime around the third or fourth team onboarding to your infrastructure codebase. Suddenly, Terraform isn’t just a productivity tool anymore. It’s a coordination problem.

State files start conflicting. Someone applies a change from their laptop that hasn’t been peer-reviewed. A junior engineer accidentally destroys a production resource because nothing stopped them from running terraform destroy in the wrong workspace. Drift accumulates silently across environments. And the answer from most teams, at least initially, is more process: more Slack messages, more checklists, more “please don’t run that without telling someone first.”

That’s not a Terraform problem. It’s a scale problem. And process alone won’t solve it.

The Tool Was Built for One. The Team Has Grown to Twenty.

Terraform’s core design is remarkably elegant for what it was originally built to do: let a single engineer or a small team describe infrastructure declaratively, plan changes, and apply them with confidence. The HCL syntax is readable. The provider ecosystem is vast. The plan/apply workflow gives you a preview before you commit.

But that workflow was designed around individual agency. When you’re a team of two, “talk to each other before applying” scales fine. When you’re a team of twenty spread across time zones, working across dozens of modules and hundreds of resources, the informal coordination layer collapses under its own weight.

This is where IaC at scale reveals a structural gap that surprises a lot of teams: Terraform tells you what infrastructure will look like. It doesn’t tell you who’s allowed to change it, under what conditions, with whose approval, and in what order.

That second set of constraints (the governance layer) isn’t in the tool. It has to be built around it.

The Three Places Scale Actually Breaks IaC

Before you can fix the problem, it helps to name where things actually fall apart.

  1. State management becomes a liability

Terraform state is the source of truth for your infrastructure. It tracks resource IDs, dependencies, and metadata. At small scale, it’s easy to manage. At large scale, across multiple teams, environments, and dozens of modules, state files become a battleground. Concurrent applies corrupt them. Sensitive data leaks into them. Teams working in separate modules inadvertently step on each other’s resources because state boundaries weren’t designed to match team boundaries.

The fix isn’t just remote state backends (though that’s the starting point). It’s designing a state architecture that maps to how your teams actually work and enforcing that architecture consistently.

  1. Drift accumulates silently

One of the underappreciated risks in large IaC deployments is drift: the gap between what Terraform thinks exists and what actually exists in your cloud environment. Drift happens for completely mundane reasons. Someone made a manual change in the console to fix a production issue at 2am and never went back to update the code. A resource was modified by an automated process outside the IaC pipeline. A configuration change was applied without going through Terraform at all.

At small scale, drift is annoying but manageable. At large scale, it’s a quiet corruption of your infrastructure’s integrity. You can’t trust your plans, because the baseline is wrong.

  1. There’s no enforcement layer

This is the one that causes the most visible incidents. Terraform will happily let any authenticated user with state access and provider credentials apply any change to any resource. There’s nothing built-in that says “changes to production require two approvals,” or “this resource type can only be managed by the platform team,” or “you cannot remove this tag from any compute resource.”

Policy-as-code tooling (OPA, Sentinel, and others) exists precisely to fill this gap. But those tools need to be integrated into a pipeline that actually enforces them, not just recommended in a wiki doc.

Where Teams Go Wrong When They Try to Fix This

The instinct most teams have when Terraform starts to break at scale is to add tooling and process in parallel: a homegrown CI/CD wrapper here, a wiki page of conventions there, a Slack channel for “Terraform applies in progress.” This works up to a point, and then it creates a different problem: an increasingly brittle, undocumented, hard-to-onboard patchwork of automation that only two or three people fully understand.

The teams that navigate this well tend to share a common pattern: they treat the workflow as a first-class concern, not an afterthought. They ask not just whether the Terraform code works, but whether the Terraform process works: is it auditable, is it enforceable, does it scale with team growth, and does it reduce the blast radius of mistakes?

That shift in framing matters. If you’re primarily optimizing your Terraform modules and not your Terraform workflow, you’re tuning the engine while the steering wheel is broken.

A useful breakdown of specific pain points that surface as teams grow into Terraform can be found in this overview of common Terraform challenges, particularly the sections on state management and team collaboration, which map closely to the failure modes described above.

Spacelift is a workflow automation platform for Terraform and OpenTofu that adds policy enforcement, drift detection, and approval flows on top of standard IaC pipelines. Teams use it to bring governance and collaboration to infrastructure that has outgrown manual processes.

What a Mature IaC Workflow Actually Looks Like

There’s no universal answer, but mature teams tend to converge on a few consistent characteristics:

Automated, gated pipelines. No one applies Terraform from a local machine in production. Every apply goes through a pipeline that runs plan, enforces policy checks, requires approval where configured, and logs the outcome. The pipeline is the interface.

Policy as code, not policy as documentation. Governance rules are encoded and enforced programmatically, not communicated through Slack and hoped for. If a rule matters enough to write down, it matters enough to automate.

State architecture that matches org structure. State files are scoped to teams and domains, not arbitrarily. Access controls on state map to who should actually be able to modify those resources.

Drift detection on a schedule. Not just on apply but regularly, so teams know when reality has diverged from the codebase before it becomes an incident.

Clear ownership. Every resource, every module, every workspace has a named owner. When something breaks or needs to change, it’s obvious who handles it.

If you want to explore further, this primer on infrastructure as code concepts is a solid grounding in the underlying principles before layering workflow tooling on top.

The Uncomfortable Truth

Terraform at scale isn’t hard because Terraform is a bad tool. It’s hard because most engineering teams underestimate how much of infrastructure management is organizational, not technical. Modules and state files are the easy part. Getting twenty engineers to collaborate reliably on shared infrastructure, with auditability, with governance, with minimal blast radius, is an organizational design problem with a technical surface.

The teams that crack this aren’t necessarily using better tools. They’re asking better questions earlier: not just “how do we provision this?” but “how do we provision this safely, repeatedly, and at the scale we expect to need in eighteen months?”.

Picture of Johnathan Dale
Johnathan Dale

John is a cheerful and adventurous boy, loves exploring nature and discovering new things. Whether climbing trees or building model rockets, his curiosity knows no bounds.

Newsletter

Register now to get latest updates on promotions & coupons.