Declaratively import existing AWS resource into Terraform with CodeCatalyst workflow

Provisioning Amazon Web Services (AWS) resources came in many ways. One of them is create the resource from the console or some might say "clickops". But how if We want our resources that created in clickops way, later maintained in more codified way or in Infrastructure as Code (IaC)?

In this blog, I will explain how to import existing AWS resource, specifically EC2 instance that provisioned using clickops way. Then, that EC2 instance will be imported into Terraform code via CI/CD pipeline. In this case, I am using Amazon CodeCatalyst as CI/CD tool. The goal is that EC2 instance later will be maintained from Terraform code.

workflow

In real world scenario, import is not only for EC2 but it could be any other resources that supported by AWS provider

Prerequisites

Here are the prerequisites that needed to follow configurations discussed in this blog:

AWS account
CodeCatalyst space and project that has been connected with AWS account and source repository. In this blog, I use GitHub as source repository. For configuration reference, please check my other blog "Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst"
S3 bucket and DynamoDB table that will used for Terraform backend

Configuration steps

Directory structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sandbox-sbx0-ec2.yml
└── sandbox/
    └── sbx0/
        └── ec2/
            ├── backend.tf
            ├── main.tf
            ├── terraform.tfvars
            └── variables.tf

Terraform import code

As an example, we will create terraform import code for EC2 instance:

EC2 instance configuration may vary, but we can adjust the configuration so that there is no drift when import. The main point we need is EC2 instance id, as an example, i have created EC2 instance called lab-import-tf and it has its own instance id

Create resource block for EC2. In main.tf, i created EC2 resource with reference from public EC2 module

 module "lab_import_tf" {
   source  = "terraform-aws-modules/ec2-instance/aws"
   version = "5.6.1"

   name = "lab-import-tf" # change with name that used by your ec2 instance

   ami                         = "ami-0695dfe470b88c986" # change with ami id that used by your ec2 instance
   instance_type               = "t3a.nano"              # change with instance type that used by your ec2 instance
   availability_zone           = element(local.azs, 0)
   subnet_id                   = element(var.public_subnet_id, 0)
   vpc_security_group_ids      = [module.lab_sg.security_group_id]
   associate_public_ip_address = true
   key_name                    = var.env_name
   iam_instance_profile        = module.lab_instance_profile.iam_instance_profile_name

   enable_volume_tags = false
   root_block_device = [
     {
       encrypted   = true
       volume_type = "gp3"
       volume_size = 10
       tags = merge(
         local.tags,
         {
           Name = "lab-import-tf"
         }
       )
     }
   ]

   tags = local.tags
 }

Create import block

Note: import block is idempotent, meaning that applying an import action and running another plan will not generate another import action as long as that resource remains in your state. Refer to Terraform import configuration language

Inside import block, in to argument, insert module name defined in point number 2 above. And in id argument, insert instance id as referred to point number 1
```
 # after import has been finished, please remove this import block
 import {
   to = module.lab_import_tf.aws_instance.this[0]
   id = "i-083e2e6033b02ae8a" # change with id of your ec2 instance
 }
```
For full context of Terraform codes/files, please refer to this directory

Run terraform import from CI/CD pipeline

In this blog, i use Amazon CodeCatalyst as tool for CI/CD pipeline.

Define workflow/pipeline for Terraform workflow. In this example, I create workflow that will run terraform plan and terraform apply inside directory sandbox/sbx0/ec2/

For more details regading CodeCatalyst workflow, please see my other blog Gatekeep CodeCatalyst Workflow using Approval

 Name: tf-sandbox-sbx0-ec2
 SchemaVersion: "1.0"

 Triggers:
   - Type: PUSH
     Branches:
       - master
     FilesChanged:
       - sandbox\/sbx0\/ec2\/.*

 Actions:
   terraform-plan:
     Identifier: aws/build@v1
     Inputs:
       Sources:
         - WorkflowSource
     Environment:
       Name: sandbox
       Connections:
         - Name: lanandra-sandbox
           Role: tf-codecatalyst-admin-sandbox
     Configuration:
       Container:
         Registry: DockerHub
         Image: hashicorp/terraform:1.8.2
       Steps:
       - Run: cd sandbox/sbx0/ec2
       - Run: terraform fmt -check -no-color
       - Run: terraform init -no-color
       - Run: terraform validate -no-color
       - Run: terraform plan -no-color -input=false
     Compute:
       Type: EC2
   wait-for-approval:
     Identifier: aws/approval@v1
     DependsOn:
       - terraform-plan
     Configuration:
       ApprovalsRequired: 1
   terraform-apply:
     DependsOn:
       - wait-for-approval
     Identifier: aws/build@v1
     Inputs:
       Sources:
         - WorkflowSource
     Environment:
       Name: sandbox
       Connections:
         - Name: lanandra-sandbox
           Role: tf-codecatalyst-admin-sandbox
     Configuration:
       Container:
         Registry: DockerHub
         Image: hashicorp/terraform:1.8.2
       Steps:
       - Run: cd sandbox/sbx0/ec2
       - Run: terraform init -no-color
       - Run: terraform apply -auto-approve -no-color -input=false
     Compute:
       Type: EC2

As mentioned in section Terraform import code, I have created import block and resource block for EC2 that will be imported. For next action, I run CodeCatalyst workflow (tf-sandbox-sbx0-ec2) from CodeCatalyst web console. Workflow will run action terraform-plan, see Logs for detailed plan
In the logs, terraform-plan will perform import action. It will import instance id that has been defined in the code
Beside that, in terraform-plan summary, it said that it will import 1 item which instance id that mentioned in point number 3. But there is also 1 change. This change is related to adding tag "lanandra:managedBy" = "terraform"

Why this happened ? because when I provisioned EC2 instance for the first time from web console, I didn't add tag "lanandra:managedBy" = "terraform". But in my locals in main.tf, I declared to add that tag

 locals {
   azs = slice(data.aws_availability_zones.available.names, 0, 3)

   tags = {
     "lanandra:environment" = "${var.env_name}"
     "lanandra:managedBy"   = "terraform"
   }
 }

And inside my EC2 module, I have declared tags to use tags from locals

 module "lab_import_tf" {
   source  = "terraform-aws-modules/ec2-instance/aws"
   version = "5.6.1"

   name = "lab-import-tf" # change with name that used by your ec2 instance

 --redacted--
   root_block_device = [
     {
       encrypted   = true
       volume_type = "gp3"
       volume_size = 10
       tags = merge(
         local.tags,
         {
           Name = "lab-import-tf"
         }
       )
     }
   ]

   tags = local.tags
 }

After terraform-plan action has been finished and plan is expected, then workflow will continue to run action wait-for-approval. Please choose approve and click submit to continue
Click confirm button if asked for reconfirmation
Then workflow will continue to run terraform-apply, open Logs for more details
terraform-apply will continue to import and add tag like what have been previewed in terraform plan previously
Afterwards, we can verify resource has been imported to terraform state. One of the method is run terraform cli command inside directory where EC2 resources resided
```
➜  ec2 git:(master) terraform state list | grep instance
module.lab_import_tf.aws_instance.this[0]
```
We also can trigger another workflow run from CodeCatalyst, terraform plan should state that there is no changes and infrastructure matches with current configuration
Clean up import block that have mentioned in point number 3

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

We can import existing AWS resources into Terraform codes using import block
Import block is idempotent
We can utilize CI/CD pipeline during import process
1. Create history
2. Act as single source of truth

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

Declaratively import existing AWS resource into Terraform with CodeCatalyst workflow

Comments

More from this blog

Migrate Terraform state locking from DynamoDB to S3

Leveraging Cloud Custodian to shift left tag compliance

Prerequisites

Configuration steps

Directory structure

Terraform import code

Run terraform import from CI/CD pipeline

Summary

Command Palette

Comments

More from this blog

Prerequisites

Configuration steps

Directory structure

Terraform import code

Run terraform import from CI/CD pipeline

Summary