Declaratively import existing AWS resource into Terraform with CodeCatalyst workflow

Provisioning Amazon Web Services (AWS) resources came in many ways. One of them is create the resource from the console or some might say "clickops". But how if We want our resources that created in clickops way, later maintained in more codified way or in Infrastructure as Code (IaC)?

In this blog, I will explain how to import existing AWS resource, specifically EC2 instance that provisioned using clickops way. Then, that EC2 instance will be imported into Terraform code via CI/CD pipeline. In this case, I am using Amazon CodeCatalyst as CI/CD tool. The goal is that EC2 instance later will be maintained from Terraform code.

In real world scenario, import is not only for EC2 but it could be any other resources that supported by AWS provider

Prerequisites

Here are the prerequisites that needed to follow configurations discussed in this blog:

1. AWS account

2. CodeCatalyst space and project that has been connected with AWS account and source repository. In this blog, I use GitHub as source repository. For configuration reference, please check my other blog "Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst"

3. S3 bucket and DynamoDB table that will used for Terraform backend

Configuration steps

Directory structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sandbox-sbx0-ec2.yml
└── sandbox/
    └── sbx0/
        └── ec2/
            ├── backend.tf
            ├── main.tf
            ├── terraform.tfvars
            └── variables.tf

Terraform import code

As an example, we will create terraform import code for EC2 instance:

1. EC2 instance configuration may vary, but we can adjust the configuration so that there is no drift when import. The main point we need is EC2 instance id, as an example, i have created EC2 instance called lab-import-tf and it has its own instance id

   

2. Create resource block for EC2. In main.tf, i created EC2 resource with reference from public EC2 module

    module "lab_import_tf" {
      source  = "terraform-aws-modules/ec2-instance/aws"
      version = "5.6.1"
   
      name = "lab-import-tf" # change with name that used by your ec2 instance
   
      ami                         = "ami-0695dfe470b88c986" # change with ami id that used by your ec2 instance
      instance_type               = "t3a.nano"              # change with instance type that used by your ec2 instance
      availability_zone           = element(local.azs, 0)
      subnet_id                   = element(var.public_subnet_id, 0)
      vpc_security_group_ids      = [module.lab_sg.security_group_id]
      associate_public_ip_address = true
      key_name                    = var.env_name
      iam_instance_profile        = module.lab_instance_profile.iam_instance_profile_name
   
      enable_volume_tags = false
      root_block_device = [
        {
          encrypted   = true
          volume_type = "gp3"
          volume_size = 10
          tags = merge(
            local.tags,
            {
              Name = "lab-import-tf"
            }
          )
        }
      ]
   
      tags = local.tags
    }
   

3. Create import block

   Note: import block is idempotent, meaning that applying an import action and running another plan will not generate another import action as long as that resource remains in your state. Refer to Terraform import configuration language

   Inside import block, in to argument, insert module name defined in point number 2 above. And in id argument, insert instance id as referred to point number 1

    # after import has been finished, please remove this import block
    import {
      to = module.lab_import_tf.aws_instance.this[0]
      id = "i-083e2e6033b02ae8a" # change with id of your ec2 instance
    }
   

4. For full context of Terraform codes/files, please refer to this directory

Run terraform import from CI/CD pipeline

In this blog, i use Amazon CodeCatalyst as tool for CI/CD pipeline.

1. Define workflow/pipeline for Terraform workflow. In this example, I create workflow that will run terraform plan and terraform apply inside directory sandbox/sbx0/ec2/

   For more details regading CodeCatalyst workflow, please see my other blog Gatekeep CodeCatalyst Workflow using Approval

    Name: tf-sandbox-sbx0-ec2
    SchemaVersion: "1.0"
   
    Triggers:
      - Type: PUSH
        Branches:
          - master
        FilesChanged:
          - sandbox\/sbx0\/ec2\/.*
   
    Actions:
      terraform-plan:
        Identifier: aws/build@v1
        Inputs:
          Sources:
            - WorkflowSource
        Environment:
          Name: sandbox
          Connections:
            - Name: lanandra-sandbox
              Role: tf-codecatalyst-admin-sandbox
        Configuration:
          Container:
            Registry: DockerHub
            Image: hashicorp/terraform:1.8.2
          Steps:
          - Run: cd sandbox/sbx0/ec2
          - Run: terraform fmt -check -no-color
          - Run: terraform init -no-color
          - Run: terraform validate -no-color
          - Run: terraform plan -no-color -input=false
        Compute:
          Type: EC2
      wait-for-approval:
        Identifier: aws/approval@v1
        DependsOn:
          - terraform-plan
        Configuration:
          ApprovalsRequired: 1
      terraform-apply:
        DependsOn:
          - wait-for-approval
        Identifier: aws/build@v1
        Inputs:
          Sources:
            - WorkflowSource
        Environment:
          Name: sandbox
          Connections:
            - Name: lanandra-sandbox
              Role: tf-codecatalyst-admin-sandbox
        Configuration:
          Container:
            Registry: DockerHub
            Image: hashicorp/terraform:1.8.2
          Steps:
          - Run: cd sandbox/sbx0/ec2
          - Run: terraform init -no-color
          - Run: terraform apply -auto-approve -no-color -input=false
        Compute:
          Type: EC2
   

2. As mentioned in section Terraform import code, I have created import block and resource block for EC2 that will be imported. For next action, I run CodeCatalyst workflow (tf-sandbox-sbx0-ec2) from CodeCatalyst web console. Workflow will run action terraform-plan, see Logs for detailed plan

   

3. In the logs, terraform-plan will perform import action. It will import instance id that has been defined in the code

   

4. Beside that, in terraform-plan summary, it said that it will import 1 item which instance id that mentioned in point number 3. But there is also 1 change. This change is related to adding tag "lanandra:managedBy" = "terraform"

   

5. Why this happened ? because when I provisioned EC2 instance for the first time from web console, I didn't add tag "lanandra:managedBy" = "terraform". But in my locals in main.tf, I declared to add that tag

    locals {
      azs = slice(data.aws_availability_zones.available.names, 0, 3)
   
      tags = {
        "lanandra:environment" = "${var.env_name}"
        "lanandra:managedBy"   = "terraform"
      }
    }
   

6. And inside my EC2 module, I have declared tags to use tags from locals

    module "lab_import_tf" {
      source  = "terraform-aws-modules/ec2-instance/aws"
      version = "5.6.1"
   
      name = "lab-import-tf" # change with name that used by your ec2 instance
   
    --redacted--
      root_block_device = [
        {
          encrypted   = true
          volume_type = "gp3"
          volume_size = 10
          tags = merge(
            local.tags,
            {
              Name = "lab-import-tf"
            }
          )
        }
      ]
   
      tags = local.tags
    }
   

7. After terraform-plan action has been finished and plan is expected, then workflow will continue to run action wait-for-approval. Please choose approve and click submit to continue

   

8. Click confirm button if asked for reconfirmation

   

9. Then workflow will continue to run terraform-apply, open Logs for more details

   

10. terraform-apply will continue to import and add tag like what have been previewed in terraform plan previously

    

11. Afterwards, we can verify resource has been imported to terraform state. One of the method is run terraform cli command inside directory where EC2 resources resided

    ➜  ec2 git:(master) terraform state list | grep instance
    module.lab_import_tf.aws_instance.this[0]
    

12. We also can trigger another workflow run from CodeCatalyst, terraform plan should state that there is no changes and infrastructure matches with current configuration

    

13. Clean up import block that have mentioned in point number 3

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

1. We can import existing AWS resources into Terraform codes using import block

2. Import block is idempotent

3. We can utilize CI/CD pipeline during import process

   1. Create history

   2. Act as single source of truth

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!