S3 native state locking in Terraform

globart

globart

Posted on November 29, 2024

S3 native state locking in Terraform

Since the Terraform 0.5.0 release from May 2015th we've been able to store our state on S3 buckets.

But in order to ensure it's consistency, we've had to use state locking using DynamoDB table. Although the DynamoDB-associated costs are negligable, it is still nonetheless another resource, that you have create manually, which could prove to be pretty annoying if you are managing large infrastructure and/or many projects at once.

The DynamoDB was needed due to the fact that S3 only had eventual consistency, meaning that if you read a file some time after it was written, you would get its latest version. But if you attempted to read a file immediately after it was written, it wasn't guaranteed.
This was addressed when AWS introduced strong read-after-write consistency for S3 in December 2020

However, this didn't solve the issue entirely, because there was no build-in mechanism to check if an object exists, before creating it.
Fortunately, after another 4 years, Amazon introduced support for conditional writes in S3 in August 2024

These changes made it possible to start work on state locking without DynamoDB, which doesn't require any additional resources, apart from the bucket itself.
After a couple of months, S3 native state locking was introduced in Terraform 1.10.0 in November 2024
While similar discussion exists in OpenTofu repo since September 2023, at the time of writing this article, there was no equivalent solution developed. But I hope it will be created in the coming months.

So, while previously your S3 backend configuration looked like this:

terraform {
  backend "s3" {
    bucket         = "example-bucket"
    key            = "path/to/state"
    region         = "us-east-1"
    dynamodb_table = "example-table"
  }
}
Enter fullscreen mode Exit fullscreen mode

Now it can look like this:

terraform {
  backend "s3" {
    bucket       = "example-bucket"
    key          = "path/to/state"
    region       = "us-east-1"
    use_lockfile = true
  }
}
Enter fullscreen mode Exit fullscreen mode

Currently, while this feature is still experimental, use_lockfile argument is optional and defaults to false.
To support migration from older versions of Terraform which only support DynamoDB-based locking, it can be configured simultaneously with dynamodb_table argument. In a future minor version the DynamoDB locking mechanism will be removed.

💖 💪 🙅 🚩
globart
globart

Posted on November 29, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

What was your win this week?
weeklyretro What was your win this week?

November 29, 2024

Where GitOps Meets ClickOps
devops Where GitOps Meets ClickOps

November 29, 2024