Remote Backends & Team Collaboration

Terraform / Infrastructure as Code

Chapter 6 · Remote Backends & Team Collaboration

Chapter 5 flagged two problems with local state: secrets sitting in a file tempting to accidentally commit, and no shared record for a team. Remote backends solve both.

Why Local State Breaks With a Team

A second teammate cloning the repository has zero record of what the first person's apply already created — their local terraform.tfstate simply doesn't exist yet. Worse, if two people do somehow share a state file and both run apply at nearly the same time, each reads the same starting state, computes a plan against it, and both try to write an updated state back afterward — whichever write happens last silently overwrites the other, potentially losing track of resources the first apply just created.

Remote Backends

A backend block inside terraform {} moves state storage off local disk entirely, into a shared, access-controlled location every team member and CI runner reads from and writes to.

terraform { backend "s3" { bucket = "my-company-tfstate" key = "projects/web-app/terraform.tfstate" region = "us-east-1" dynamodb_table = "terraform-locks" encrypt = true } }

The S3 + DynamoDB Backend

The classic AWS pattern: an S3 bucket holds the state file itself (with encrypt = true for encryption at rest — the same principle dbsec1-5/crypto1-5/6 already covered), and a DynamoDB table provides locking. Every teammate and CI job reads/writes the exact same object in S3, so there's genuinely one shared source of truth instead of N different local copies.

State Locking

Before writing to state, Terraform first acquires a lock — a conditional write to the DynamoDB table that only succeeds if no lock currently exists. If someone else's apply already holds the lock, Terraform blocks with a clear error rather than racing to write state at the same time:

Error: Error acquiring the state lock Lock Info: ID: 7c6f... Path: projects/web-app/terraform.tfstate Operation: OperationTypeApply Who: teammate@ci-runner-3

This is exactly what prevents the concurrent-write corruption described above — only one apply can ever be mid-flight against a given piece of state at a time.

Terraform Cloud / HCP Terraform

A managed alternative to hand-rolling S3+DynamoDB — built-in state storage, locking, a web UI, and run history, with no separate bucket or lock table to provision yourself. Genuinely the "batteries included" option, at the cost of depending on a hosted service rather than infrastructure the team fully owns.

Backend Migration

Switching from local state to a remote backend (or between two remote backends) isn't automatic — it requires an explicit migration:

$ terraform init -migrate-state Initializing the backend... Terraform detected that the backend type changed... Do you want to copy existing state to the new backend? Enter a value: yes

After migrating, every teammate needs to re-run terraform init themselves — their local Terraform still points at the old backend configuration until they do.

This is what actually protects the secrets in state
Chapter 4 and Chapter 5 both flagged that a secret's plaintext lives in state regardless of the sensitive flag. An encrypted S3 bucket with IAM access restricted to only the people/systems that genuinely need it is the real answer to that open question — protecting the storage location itself, since the file's contents can't be hidden from anyone who has already been granted read access.
force-unlock is a last resort
A crashed CI job can leave a lock stuck indefinitely, blocking every future apply. terraform force-unlock <lock-id> exists for exactly this — but confirm no apply is genuinely still in progress first. Force-unlocking while a real apply is actively running risks the exact concurrent-write corruption locking exists to prevent.

Hands-On Exercises

Exercise 1

Explain concretely what can go wrong if two team members run terraform apply at nearly the same time against shared local state with no locking mechanism.

📄 View solution
Exercise 2

In the S3 + DynamoDB backend pattern, explain DynamoDB's specific role, and what could still go wrong even with state safely stored remotely in S3 if DynamoDB locking weren't part of the setup.

📄 View solution
Exercise 3

A CI job crashes mid-apply and leaves a stuck lock, blocking the team. Explain why terraform force-unlock should be used cautiously, and what should be verified before running it.

📄 View solution

Chapter 6 Quick Reference

  • Remote backend — moves state to a shared, access-controlled location; solves both the secrets-in-git risk and team collaboration
  • S3 + DynamoDB — S3 stores state (encrypted at rest), DynamoDB provides locking
  • Lock — only one apply can hold it at a time, preventing concurrent state writes
  • Terraform Cloud/HCP — managed alternative with built-in storage, locking, and a web UI
  • terraform init -migrate-state — required for any backend change; every teammate must re-init afterward
  • terraform force-unlock — last resort only, after confirming no apply is genuinely still running