If we deploy infrastructure in Amazon Web Services (AWS), one of the backend that we can use is S3. As referred to S3 State Locking documentation, state locking is an opt-in feature in S3 backend. Previously, state locking only can be enabled via DynamoDB. But since Terraform v1.11.0, Terraform S3 native state locking is generally available, so that we can enable state locking in S3 or DynamoDB. However, as stated in S3 State Locking documentation, DynamoDB based locking is deprecated and will be removed in a future minor version.

In this blog, I will explain how to migrate Terraform state locking from DynamoDB to S3 native state locking. In the example, I will run Terraform workflow both from CI/CD and local workstation, only for demo purpose. I use Amazon CodeCatalyst to run my Terraform workflow in CI/CD pipeline. But you may use other CI/CD tool or maybe running locally just to follow along how S3 state locking works with some adjustment on your side.

Disclaimer: All contents written in this blog are solely my personal opinion and doesn’t represent my employer opinion

Prerequisites

Here are the prerequisites that needed to follow configurations discussed in this blog:

1. AWS account

2. Preconfigured S3 bucket and DynamoDB table that will be used for Terraform backend migration

Configuration steps

Reconfigure and migrate Terraform state

1. As an example, I have existing Terraform backend configuration with DynamoDB state locking as referred with argument dynamodb_table

terraform {
  backend "s3" {
    bucket         = "my-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-tflocks"
    encrypt        = true
  }
}

2. Then I changed the structure of Terraform backend. I removed argument dynamodb_table and added argument use_lockfile and I set the value to true

terraform {
  backend "s3" {
    bucket       = "my-tfbucket"
    key          = "path/to/terraform.tfstate"
    region       = "ap-southeast-1"
    encrypt      = true
    use_lockfile = true
  }
}

3. Run terraform init to trigger backend changes after using S3 native state locking. In my case, I run it on my local with reason this just one time activity and I prefer to make it simple. But if you want to trigger the changes with different method, for example run in CI/CD, you may try it as well

4. First run of terraform init may be failed with error that stated that backend configuration is changed

✗ terraform init
Initializing the backend...
Initializing modules...
╷
│ Error: Backend configuration changed
│ 
│ A change in the backend configuration has been detected, which may require migrating existing state.
│ 
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".

5. Re-run terraform init, add args -reconfigure so that terraform will reconfigure and migrate your backend with new configuration that used S3 native state locking

✗ terraform init -reconfigure
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.47.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

6. For full context of code example, you can check this file inside this directory

Test Terraform S3 state locking

In this blog, I simulate Terraform S3 state locking by running multiple Terraform plan in the same period. One is ran in CI/CD, in my case is Amazon CodeCatalyst. And the other one is ran from my local to simplify testing process. But you may test with different test case and in real environment, more likely We will experience running multiple Terraform plan from multiple CI/CD pipelines.

1. Here is the example when I ran terraform plan in my CI/CD but in the same time I also ran terraform plan from my local workstation. Terraform plan will be failed

2. Terraform plan got error message: Error acquiring the state lock

Error: Error acquiring the state lock

Error message: operation error S3: PutObject, https response error
StatusCode: 412, RequestID: 12345ABCDE, HostID:
1a2b/3c4d/5e==,
api error PreconditionFailed: At least one of the pre-conditions you
specified did not hold
Lock Info:
ID:        1a2b3c4d5e-3ec1-1f71-9cdd-1a2b3c4d5e
Path:      path/to/terraform.tfstate
Operation: OperationTypePlan
Who:       lanandra@Luthfis-MacBook-Pro.local
Version:   1.11.2
Created:   2025-03-22 18:56:18.872219 +0000 UTC
Info:


Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

3. In this case, We can verify that Terraform state locking in S3 has been working as it will prevent multiple terraform plan run and overriding the state in the same period

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

1. Terraform state locking is a feature where Terraform will lock your state for all operations that could write state and prevent others from acquiring the lock and potentially corrupting your state

2. If we are using S3 as our Terraform backend, previously state locking only support DynamoDB but since Terraform v1.11.0, state locking also support S3 native state locking

3. DynamoDB based locking is deprecated and will be removed in a future minor version

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

Tagging enables you to group sets of resources by assigning metadata to cloud resources for a variety of purposes. These purposes include access control (such as ABAC), cost reporting, and automation (such as patching for select tagged instances). Tagging can also be used to create new resource constructs for visibility or control (such as grouping together resources that make up a microservice, application, or workload). Tagging is fundamental to providing enterprise-level visibility and control.

There are several use cases why we need tagging as explained in this document “Tagging use cases“, among them:

1. Tags for cost allocation and financial management

2. Tags for operations and support

3. Tags for data security, risk management, and access control

In this blog, I will explain how to shift left tag compliance check inside CI/CD. I leverage Cloud Custodian package called c7n-left to run policy directly against Infrastructure as Code which in this case is Terraform. If tag compliance check is passed then Terraform core workflow which consist plan and apply can be continued. Otherwise, if tag compliance check is failed then Terraform core workflow cannot be continued. CI/CD tool that I used in this blog is Amazon CodeCatalyst.

Disclaimer: All contents written in this blog are solely my personal opinion and doesn’t represent my employer opinion

Prerequisites

Here are the prerequisites that needed to follow configurations discussed in this blog:

1. AWS account

2. CodeCatalyst space and project that has been connected with AWS account and source repository. In this blog, I use GitHub as source repository. For configuration reference, please check my other blog "Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst"

3. S3 bucket and DynamoDB table that will be used for Terraform backend

Configuration steps

Directory structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sandbox-sbx0-ec2.yml
├── custodian/
│   └── tf-tag-compliance.yml
└── sandbox/
    └── sbx0/
        └── ec2/
            ├── backend.tf
            ├── main.tf
            ├── terraform.tfvars
            └── variables.tf

Custodian policy

We will start discussion with policy that will be used to evaluate Infrastructure as Code by Cloud Custodian

1. Under directory custodian, I created file called tf-tag-compliance.yml. Inside that file there is a policy named check-tag that will check all Terraform codes that related to AWS resources. Policy will check if tag called lanandra:environment is absent or not on all resources that taggable. I also added metadata to that policy

policies:
  - name: check-tag
    resource: "terraform.aws*"
    metadata:
      category: [tag]
      severity: low
    filters:
      - taggable
      - or:
        - type: value
          key: "tag:lanandra:environment"
          value: absent

Sample Terraform code

As an example, I created two EC2 instances, one that has tag lanandra:environment and the other doesn’t

1. Create Terraform files inside directory sandbox/sbx0/ec2 , main resources such as EC2 are defined inside main.tf file

2. EC2 that has tag lanandra:environment , this tag is got by merging local tags.

locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    "lanandra:environment" = "${var.env_name}"
    "lanandra:managedBy"   = "terraform"
  }
}

module "lab_ec2_0" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "5.6.1"

  name = "${var.env_name}-lab-0"

  ami                         = data.aws_ami.ubuntu_22_04.id
  instance_type               = "t3a.nano"
  availability_zone           = element(local.azs, 0)
  subnet_id                   = element(var.public_subnet_id, 0)
  vpc_security_group_ids      = [module.lab_sg.security_group_id]
  associate_public_ip_address = true
  key_name                    = var.env_name
  iam_instance_profile        = module.lab_instance_profile.iam_instance_profile_name

  enable_volume_tags = false
  root_block_device = [
    {
      encrypted   = true
      volume_type = "gp3"
      volume_size = 10
      tags = merge(
        local.tags,
        {
          Name = "${var.env_name}-lab-0"
        }
      )
    }
  ]

  tags = merge(
    local.tags,
    {
      Name = "${var.env_name}-lab-0"
    }
  )
}

2. EC2 that hasn’t got tag lanandra:environment

module "lab_ec2_1" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "5.6.1"

  name = "${var.env_name}-lab-1"

  ami                         = data.aws_ami.ubuntu_22_04.id
  instance_type               = "t3a.nano"
  availability_zone           = element(local.azs, 0)
  subnet_id                   = element(var.public_subnet_id, 0)
  vpc_security_group_ids      = [module.lab_sg.security_group_id]
  associate_public_ip_address = true
  key_name                    = var.env_name
  iam_instance_profile        = module.lab_instance_profile.iam_instance_profile_name

  enable_volume_tags = false
  root_block_device = [
    {
      encrypted   = true
      volume_type = "gp3"
      volume_size = 10
      tags = {
        Name = "${var.env_name}-lab-1"
      }
    }
  ]

  tags = {
    Name = "${var.env_name}-lab-1"
  }
}

3. For full context of Terraform codes/files, please refer to this directory

Run tag compliance in CI/CD pipeline

As mentioned in the beginning of this blog, I use Amazon CodeCatalyst as tool for CI/CD pipeline.

1. I created a workflow file called tf-sandbox-sbx0-ec2.yml and I put that inside .codecatalyst/workflow directory. Inside workflow/pipeline, I created action called tag-compliance. This action use container image python:3.12.8-bookworm that being pulled from Dockerhub. First, it will redirect working directory to ec2 directory (sandbox/sbx0/ec2). Then it will run several steps that needed to run tag compliance check using Cloud Custodian. In this case, I use specific package called c7n-left that designed to evaluate policies directly against infrastructure as code source assets (Please see Custodian policies for Infrastructure Code for more details). Then it will run command that will find custodian policy inside custodian directory with category value filtered to tag

  tag-compliance:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: python:3.12.8-bookworm
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: pip install c7n
      - Run: pip install c7n-left
      - Run: c7n-left run -p ../../../custodian/ -d . --filters="category=tag"
    Compute:
      Type: EC2

2. In workflow file, tag compliance action will be merged with other actions that used by Terraform core workflow. For full context, please see this file:

Name: tf-sandbox-sbx0-ec2
SchemaVersion: "1.0"

Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox\/sbx0\/ec2\/.*

Actions:
  tag-compliance:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: python:3.12.8-bookworm
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: pip install c7n
      - Run: pip install c7n-left
      - Run: c7n-left run -p ../../../custodian/ -d . --filters="category=tag"
    Compute:
      Type: EC2
  terraform-plan:
    Identifier: aws/build@v1
    DependsOn:
      - tag-compliance
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-for-approval:
    Identifier: aws/approval@v1
    DependsOn:
      - terraform-plan
    Configuration:
      ApprovalsRequired: 1
  terraform-apply:
    DependsOn:
      - wait-for-approval
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

3. We will simulate to provision EC2 with tag that needed by tag compliance first. Tag compliance action will run first as defined in workflow. Cause tag compliance is passed, then Terraform core workflow can be continued. Please see detailed logs of tag compliance to see the result

4. Here is the example result of successful tag compliance check using c7n-left. It said no failed result

5. As explained in point number 3, if tag compliance action is passed, then Terraform core workflow can be continued. So that it can run terraform apply to provision EC2

6. For next example, let’s see what if we want to provision EC2 that hasn’t got tag that needed by tag compliance action. In CI/CD, tag compliance action will be failed and the rest of Terraform core workflow cannot be continued. Please see detailed logs of tag compliance action for further details

7. Custodian will inform which file(s) and line(s) that has got policy violation

8. Custodian will also inform how many resources that failed and got policy violation and how many resources that passed in evaluation summary

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

1. We can shift left our tag compliance check inside CI/CD

   1. Cloud Custodian is one of the tool that can be leveraged to run tag compliance check

   2. Cloud Custodian has specific package called c7n-left to run policy check directly against Infrastructure code

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

In this blog, I will explain how to import existing AWS resource, specifically EC2 instance that provisioned using clickops way. Then, that EC2 instance will be imported into Terraform code via CI/CD pipeline. In this case, I am using Amazon CodeCatalyst as CI/CD tool. The goal is that EC2 instance later will be maintained from Terraform code.

In real world scenario, import is not only for EC2 but it could be any other resources that supported by AWS provider

Prerequisites

Here are the prerequisites that needed to follow configurations discussed in this blog:

1. AWS account

2. CodeCatalyst space and project that has been connected with AWS account and source repository. In this blog, I use GitHub as source repository. For configuration reference, please check my other blog "Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst"

3. S3 bucket and DynamoDB table that will used for Terraform backend

Configuration steps

Directory structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sandbox-sbx0-ec2.yml
└── sandbox/
    └── sbx0/
        └── ec2/
            ├── backend.tf
            ├── main.tf
            ├── terraform.tfvars
            └── variables.tf

Terraform import code

As an example, we will create terraform import code for EC2 instance:

1. EC2 instance configuration may vary, but we can adjust the configuration so that there is no drift when import. The main point we need is EC2 instance id, as an example, i have created EC2 instance called lab-import-tf and it has its own instance id

   

2. Create resource block for EC2. In main.tf, i created EC2 resource with reference from public EC2 module

    module "lab_import_tf" {
      source  = "terraform-aws-modules/ec2-instance/aws"
      version = "5.6.1"
   
      name = "lab-import-tf" # change with name that used by your ec2 instance
   
      ami                         = "ami-0695dfe470b88c986" # change with ami id that used by your ec2 instance
      instance_type               = "t3a.nano"              # change with instance type that used by your ec2 instance
      availability_zone           = element(local.azs, 0)
      subnet_id                   = element(var.public_subnet_id, 0)
      vpc_security_group_ids      = [module.lab_sg.security_group_id]
      associate_public_ip_address = true
      key_name                    = var.env_name
      iam_instance_profile        = module.lab_instance_profile.iam_instance_profile_name
   
      enable_volume_tags = false
      root_block_device = [
        {
          encrypted   = true
          volume_type = "gp3"
          volume_size = 10
          tags = merge(
            local.tags,
            {
              Name = "lab-import-tf"
            }
          )
        }
      ]
   
      tags = local.tags
    }
   

3. Create import block

   Note: import block is idempotent, meaning that applying an import action and running another plan will not generate another import action as long as that resource remains in your state. Refer to Terraform import configuration language

   Inside import block, in to argument, insert module name defined in point number 2 above. And in id argument, insert instance id as referred to point number 1

    # after import has been finished, please remove this import block
    import {
      to = module.lab_import_tf.aws_instance.this[0]
      id = "i-083e2e6033b02ae8a" # change with id of your ec2 instance
    }
   

4. For full context of Terraform codes/files, please refer to this directory

Run terraform import from CI/CD pipeline

In this blog, i use Amazon CodeCatalyst as tool for CI/CD pipeline.

1. Define workflow/pipeline for Terraform workflow. In this example, I create workflow that will run terraform plan and terraform apply inside directory sandbox/sbx0/ec2/

   For more details regading CodeCatalyst workflow, please see my other blog Gatekeep CodeCatalyst Workflow using Approval

    Name: tf-sandbox-sbx0-ec2
    SchemaVersion: "1.0"
   
    Triggers:
      - Type: PUSH
        Branches:
          - master
        FilesChanged:
          - sandbox\/sbx0\/ec2\/.*
   
    Actions:
      terraform-plan:
        Identifier: aws/build@v1
        Inputs:
          Sources:
            - WorkflowSource
        Environment:
          Name: sandbox
          Connections:
            - Name: lanandra-sandbox
              Role: tf-codecatalyst-admin-sandbox
        Configuration:
          Container:
            Registry: DockerHub
            Image: hashicorp/terraform:1.8.2
          Steps:
          - Run: cd sandbox/sbx0/ec2
          - Run: terraform fmt -check -no-color
          - Run: terraform init -no-color
          - Run: terraform validate -no-color
          - Run: terraform plan -no-color -input=false
        Compute:
          Type: EC2
      wait-for-approval:
        Identifier: aws/approval@v1
        DependsOn:
          - terraform-plan
        Configuration:
          ApprovalsRequired: 1
      terraform-apply:
        DependsOn:
          - wait-for-approval
        Identifier: aws/build@v1
        Inputs:
          Sources:
            - WorkflowSource
        Environment:
          Name: sandbox
          Connections:
            - Name: lanandra-sandbox
              Role: tf-codecatalyst-admin-sandbox
        Configuration:
          Container:
            Registry: DockerHub
            Image: hashicorp/terraform:1.8.2
          Steps:
          - Run: cd sandbox/sbx0/ec2
          - Run: terraform init -no-color
          - Run: terraform apply -auto-approve -no-color -input=false
        Compute:
          Type: EC2
   

2. As mentioned in section Terraform import code, I have created import block and resource block for EC2 that will be imported. For next action, I run CodeCatalyst workflow (tf-sandbox-sbx0-ec2) from CodeCatalyst web console. Workflow will run action terraform-plan, see Logs for detailed plan

   

3. In the logs, terraform-plan will perform import action. It will import instance id that has been defined in the code

   

4. Beside that, in terraform-plan summary, it said that it will import 1 item which instance id that mentioned in point number 3. But there is also 1 change. This change is related to adding tag "lanandra:managedBy" = "terraform"

   

5. Why this happened ? because when I provisioned EC2 instance for the first time from web console, I didn't add tag "lanandra:managedBy" = "terraform". But in my locals in main.tf, I declared to add that tag

    locals {
      azs = slice(data.aws_availability_zones.available.names, 0, 3)
   
      tags = {
        "lanandra:environment" = "${var.env_name}"
        "lanandra:managedBy"   = "terraform"
      }
    }
   

6. And inside my EC2 module, I have declared tags to use tags from locals

    module "lab_import_tf" {
      source  = "terraform-aws-modules/ec2-instance/aws"
      version = "5.6.1"
   
      name = "lab-import-tf" # change with name that used by your ec2 instance
   
    --redacted--
      root_block_device = [
        {
          encrypted   = true
          volume_type = "gp3"
          volume_size = 10
          tags = merge(
            local.tags,
            {
              Name = "lab-import-tf"
            }
          )
        }
      ]
   
      tags = local.tags
    }
   

7. After terraform-plan action has been finished and plan is expected, then workflow will continue to run action wait-for-approval. Please choose approve and click submit to continue

   

8. Click confirm button if asked for reconfirmation

   

9. Then workflow will continue to run terraform-apply, open Logs for more details

   

10. terraform-apply will continue to import and add tag like what have been previewed in terraform plan previously

    

11. Afterwards, we can verify resource has been imported to terraform state. One of the method is run terraform cli command inside directory where EC2 resources resided

    ➜  ec2 git:(master) terraform state list | grep instance
    module.lab_import_tf.aws_instance.this[0]
    

12. We also can trigger another workflow run from CodeCatalyst, terraform plan should state that there is no changes and infrastructure matches with current configuration

    

13. Clean up import block that have mentioned in point number 3

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

1. We can import existing AWS resources into Terraform codes using import block

2. Import block is idempotent

3. We can utilize CI/CD pipeline during import process

   1. Create history

   2. Act as single source of truth

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!