S3 native state locking in Terraform
globart
Posted on November 29, 2024
Since the Terraform 0.5.0 release from May 2015th we've been able to store our state on S3 buckets.
But in order to ensure it's consistency, we've had to use state locking using DynamoDB table. Although the DynamoDB-associated costs are negligable, it is still nonetheless another resource, that you have create manually, which could prove to be pretty annoying if you are managing large infrastructure and/or many projects at once.
The DynamoDB was needed due to the fact that S3 only had eventual consistency, meaning that if you read a file some time after it was written, you would get its latest version. But if you attempted to read a file immediately after it was written, it wasn't guaranteed.
This was addressed when AWS introduced strong read-after-write consistency for S3 in December 2020
However, this didn't solve the issue entirely, because there was no build-in mechanism to check if an object exists, before creating it.
Fortunately, after another 4 years, Amazon introduced support for conditional writes in S3 in August 2024
These changes made it possible to start work on state locking without DynamoDB, which doesn't require any additional resources, apart from the bucket itself.
After a couple of months, S3 native state locking was introduced in Terraform 1.10.0 in November 2024
While similar discussion exists in OpenTofu repo since September 2023, at the time of writing this article, there was no equivalent solution developed. But I hope it will be created in the coming months.
So, while previously your S3 backend configuration looked like this:
terraform {
backend "s3" {
bucket = "example-bucket"
key = "path/to/state"
region = "us-east-1"
dynamodb_table = "example-table"
}
}
Now it can look like this:
terraform {
backend "s3" {
bucket = "example-bucket"
key = "path/to/state"
region = "us-east-1"
use_lockfile = true
}
}
Currently, while this feature is still experimental, use_lockfile
argument is optional and defaults to false
.
To support migration from older versions of Terraform which only support DynamoDB-based locking, it can be configured simultaneously with dynamodb_table
argument. In a future minor version the DynamoDB locking mechanism will be removed.
Posted on November 29, 2024
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.