Terraform Drift & State Recovery

terraform apply -auto-approve
aws ec2 authorize-security-group-ingress --group-id <sg> \
  --protocol tcp --port 8080 --cidr 0.0.0.0/0      # the "incident fix"
 
terraform plan -detailed-exitcode
echo "exit: $?"        # 2 = drift

  ~ resource "aws_security_group" "demo" {
      ~ ingress { - from_port = 8080 ... }
    }

Two legitimate responses, and choosing wrongly causes the second incident:

• Revert — terraform apply removes the rule. Correct when the manual change was a mistake.
• Adopt — put the rule in the configuration. Correct when it was a real fix that must survive.

Never resolve it by running apply without reading the plan. That is what closed the port again at the worst moment.

-detailed-exitcode returns 2 for drift, which is designed for a scheduled job — finding drift on a Monday is much better than finding it mid-incident.

aws s3api create-bucket --bucket tfstate-recovery-demo-$(date +%s)   # by hand

Adding a matching resource block and applying fails: the bucket exists and Terraform tries to create it. Import instead:

resource "aws_s3_bucket" "adopted" {
  bucket = "tfstate-recovery-demo-1234567890"
}

terraform import aws_s3_bucket.adopted tfstate-recovery-demo-1234567890
terraform plan      # must report: No changes

That No changes is the test. If the plan wants to modify something, your configuration does not match reality yet — keep editing until it does, and resist the temptation to apply your way there.

Terraform 1.5+ can do this declaratively, which is reviewable:

import {
  to = aws_s3_bucket.adopted
  id = "tfstate-recovery-demo-1234567890"
}

aws s3 rm s3://<state-bucket>/demo/terraform.tfstate      # simulate it
terraform plan
# wants to create everything — it has lost all memory

Do not apply. That builds a second copy of everything you already have.

aws s3api list-object-versions --bucket <state-bucket> \
  --prefix demo/terraform.tfstate \
  --query 'Versions[].[VersionId,LastModified]' --output table
 
aws s3api get-object --bucket <state-bucket> \
  --key demo/terraform.tfstate --version-id <id> restored.tfstate
 
aws s3 cp restored.tfstate s3://<state-bucket>/demo/terraform.tfstate
terraform plan      # No changes

This is why the bootstrap stack enables versioning on the state bucket. A delete leaves a delete marker and the object is still there. Without versioning, the only recovery is importing every resource by hand.

Renaming a resource in your configuration makes Terraform plan a destroy and a create — same infrastructure, different address:

terraform state mv aws_instance.web aws_instance.frontend
terraform plan      # No changes

state mv updates the map only. No AWS API call touches the resource, so a rename costs nothing and no downtime.

terraform state list
terraform state show aws_instance.frontend
terraform state rm aws_instance.frontend   # forget it WITHOUT deleting it

state rm is the one to be careful with: Terraform forgets the resource and it keeps running and billing, invisible to your configuration. It is the right tool for handing a resource to another stack, and a good way to create an orphan by accident.

• Versioning and locking on the state bucket, always.
• -detailed-exitcode on a schedule, so drift finds you rather than the reverse.
• plan -out then apply that file, so what runs is what was reviewed.
• Before anything risky: terraform state pull > backup.tfstate.