Terraform State and My $220 AWS Bill

I learned Terraform state in the most annoying way possible: through an AWS bill.

When I was first learning Terraform, I provisioned some infrastructure in us-east-1. Later, I moved my experiment to ap-northeast-1, since I thought that will be a good latency improvement for my personal lab that nobody is using.

Somewhere in that process, I encounter a state lock from terrafrom, which is terrafrom protecting me from corrupting my infra, probably casued by me and my ai agent trying to operate terraform command at the same time. I deleted my local terraform.tfstate file because I did not yet understand remote backends, and I also misunderstood what the Terraform lock file was for.

In my head, the lock file felt like the thing that tracked the project. I thought Terraform should still be able to look at my configuration, look at AWS, and figure out what belonged to it.

That was terriably wrong.

The infrastructure in us-east-1 kept running quietly for about a month. I heard about plenty of horror story about AWS and Vercel serverless function, so I was actually examining the web console every time I destroy my lab infra to be extra careful. Unfortuantely, I was working in ap-northeast-1 the whole time, so I did not see it in the console during normal checks. Eventually the mistake showed up as a roughly $220 AWS bill.

This post is not a polished Terraform tutorial. It is a short learning log about the difference between configuration, state, and lock files — and why deleting state does not delete infrastructure.

What I wanted to understand

After the bill, the questions became painfully concrete:

• What does terraform.tfstate actually do?
• Why is *.tf configuration not enough for Terraform to manage existing infrastructure?
• What is the .terraform.lock.hcl file for if it does not track resources?
• Why did switching AWS regions make the mistake harder to notice?
• How should I avoid orphaning resources when using Terraform locally?

What I learned

My current mental model is:

• Terraform configuration describes intent. The *.tf files say what I want Terraform to manage.
• Terraform state records ownership and mapping. terraform.tfstate connects Terraform resources in code to real cloud resources.
• The lock file pins provider versions. .terraform.lock.hcl helps Terraform install consistent provider dependencies. It does not remember AWS resources.
• AWS resources keep existing even if Terraform forgets them. Deleting local state removes Terraform's memory, not the infrastructure itself.
• Regions are separate worlds for many AWS services. If I create resources in us-east-1 and then spend all my time looking at ap-northeast-1, I can easily miss what is still running elsewhere.

The durable primitive is:

Terraform config + Terraform state + provider credentials/region -> managed infrastructure

If the state disappears, Terraform still has configuration, but it has lost the map between that configuration and the real objects in AWS.

Terraform configuration is not Terraform state

The easy mistake is to treat Terraform like a script.

With a script, it feels natural to think:

I ran this file, so the file should describe what exists.

But Terraform is not only executing commands. It is maintaining a relationship between resource blocks and provider-side objects.

A resource block like this describes intent:

resource "aws_instance" "example" {
  ami           = "ami-..."
  instance_type = "t3.micro"
}

But after Terraform creates the instance, it needs to remember details such as:

• the AWS resource ID
• which Terraform address owns it
• which provider and region were used
• current known attributes
• dependencies between resources

That memory lives in terraform.tfstate.

Without state, Terraform cannot automatically know that aws_instance.example in my code is the same EC2 instance still running somewhere in AWS. It may see the configuration as something that needs to be created again, or it may simply have no relationship to the old resource at all.

The real infrastructure does not care that I deleted a local file. AWS keeps running whatever was already created.

What the lock file actually does

The file I misunderstood was .terraform.lock.hcl.

The name made it feel more important than it was in this context. It is important, but not because it tracks infrastructure.

The lock file records provider dependency selections. For example, it helps ensure that the same Terraform project keeps using the same version of the AWS provider unless I intentionally upgrade it.

A simplified mental model:

.terraform.lock.hcl -> provider versions and checksums
terraform.tfstate   -> real resource mapping and attributes
*.tf                -> desired configuration

So if I delete terraform.tfstate but keep .terraform.lock.hcl, Terraform may still know which provider version to install, but it no longer knows which AWS resources it created before.

That distinction cost me money because I treated dependency locking like infrastructure tracking.

The region switch made the orphaned resources quieter

The second mistake was switching regions.

I originally created infrastructure in:

us-east-1

Then I moved my work to:

ap-northeast-1

Many AWS console pages are region-scoped. If I am looking at ap-northeast-1, resources in us-east-1 may not appear in the place I am checking.

So the failure mode became quiet:

1. Terraform created resources in us-east-1.
2. I deleted the local state file.
3. Terraform no longer tracked those resources.
4. I switched my attention to ap-northeast-1.
5. The old resources kept running in us-east-1.
6. The bill eventually told me what the console did not.

The embarrassing part is that nothing especially complex happened. No exotic Terraform bug. No strange AWS behavior. Just a basic misunderstanding of where Terraform keeps its memory.

What I should have done instead

If I could redo that learning experiment, I would treat state as infrastructure data, not as disposable local clutter.

At minimum:

• Do not delete terraform.tfstate casually.
• Run terraform destroy before abandoning an experiment.
• Check all regions where I have recently created resources.
• Use AWS billing and cost alerts even for learning accounts.
• Prefer a remote backend once the project matters beyond a throwaway lab.

For Terraform projects that need to survive across machines or over time, local state is fragile. A remote backend, such as S3 with state locking, makes the state harder to lose and easier to share safely.

The important point is not only "use remote state." It is understanding why remote state matters: Terraform state is the ownership map. If I lose it, the cloud does not clean itself up for me.

Summary

The bill turned a vague concept into a concrete lesson.

Terraform has three different things that I had blurred together:

*.tf                  desired infrastructure
.terraform.lock.hcl   provider dependency lock
terraform.tfstate     mapping to real infrastructure

Deleting terraform.tfstate does not destroy AWS resources. It only makes Terraform forget them. If those resources are in another region, they can keep running quietly until billing catches them.

The next thing I want to understand better is how to recover cleanly from this kind of mistake: importing orphaned resources back into state, deciding when to destroy them manually, and setting up remote state so this failure mode becomes less likely in the first place.