Leveraging Cloud Custodian to shift left tag compliance

Tagging is one of the capability that included in Establishing Your Cloud Foundation on AWS whitepaper under category Governance, Risk Management, and Compliance.

Tagging enables you to group sets of resources by assigning metadata to cloud resources for a variety of purposes. These purposes include access control (such as ABAC), cost reporting, and automation (such as patching for select tagged instances). Tagging can also be used to create new resource constructs for visibility or control (such as grouping together resources that make up a microservice, application, or workload). Tagging is fundamental to providing enterprise-level visibility and control.

There are several use cases why we need tagging as explained in this document “Tagging use cases“, among them:

1. Tags for cost allocation and financial management

2. Tags for operations and support

3. Tags for data security, risk management, and access control

In this blog, I will explain how to shift left tag compliance check inside CI/CD. I leverage Cloud Custodian package called c7n-left to run policy directly against Infrastructure as Code which in this case is Terraform. If tag compliance check is passed then Terraform core workflow which consist plan and apply can be continued. Otherwise, if tag compliance check is failed then Terraform core workflow cannot be continued. CI/CD tool that I used in this blog is Amazon CodeCatalyst.

Disclaimer: All contents written in this blog are solely my personal opinion and doesn’t represent my employer opinion

Prerequisites

Here are the prerequisites that needed to follow configurations discussed in this blog:

1. AWS account

2. CodeCatalyst space and project that has been connected with AWS account and source repository. In this blog, I use GitHub as source repository. For configuration reference, please check my other blog "Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst"

3. S3 bucket and DynamoDB table that will be used for Terraform backend

Configuration steps

Directory structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sandbox-sbx0-ec2.yml
├── custodian/
│   └── tf-tag-compliance.yml
└── sandbox/
    └── sbx0/
        └── ec2/
            ├── backend.tf
            ├── main.tf
            ├── terraform.tfvars
            └── variables.tf

Custodian policy

We will start discussion with policy that will be used to evaluate Infrastructure as Code by Cloud Custodian

1. Under directory custodian, I created file called tf-tag-compliance.yml. Inside that file there is a policy named check-tag that will check all Terraform codes that related to AWS resources. Policy will check if tag called lanandra:environment is absent or not on all resources that taggable. I also added metadata to that policy

policies:
  - name: check-tag
    resource: "terraform.aws*"
    metadata:
      category: [tag]
      severity: low
    filters:
      - taggable
      - or:
        - type: value
          key: "tag:lanandra:environment"
          value: absent

Sample Terraform code

As an example, I created two EC2 instances, one that has tag lanandra:environment and the other doesn’t

1. Create Terraform files inside directory sandbox/sbx0/ec2 , main resources such as EC2 are defined inside main.tf file

2. EC2 that has tag lanandra:environment , this tag is got by merging local tags.

locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    "lanandra:environment" = "${var.env_name}"
    "lanandra:managedBy"   = "terraform"
  }
}

module "lab_ec2_0" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "5.6.1"

  name = "${var.env_name}-lab-0"

  ami                         = data.aws_ami.ubuntu_22_04.id
  instance_type               = "t3a.nano"
  availability_zone           = element(local.azs, 0)
  subnet_id                   = element(var.public_subnet_id, 0)
  vpc_security_group_ids      = [module.lab_sg.security_group_id]
  associate_public_ip_address = true
  key_name                    = var.env_name
  iam_instance_profile        = module.lab_instance_profile.iam_instance_profile_name

  enable_volume_tags = false
  root_block_device = [
    {
      encrypted   = true
      volume_type = "gp3"
      volume_size = 10
      tags = merge(
        local.tags,
        {
          Name = "${var.env_name}-lab-0"
        }
      )
    }
  ]

  tags = merge(
    local.tags,
    {
      Name = "${var.env_name}-lab-0"
    }
  )
}

2. EC2 that hasn’t got tag lanandra:environment

module "lab_ec2_1" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "5.6.1"

  name = "${var.env_name}-lab-1"

  ami                         = data.aws_ami.ubuntu_22_04.id
  instance_type               = "t3a.nano"
  availability_zone           = element(local.azs, 0)
  subnet_id                   = element(var.public_subnet_id, 0)
  vpc_security_group_ids      = [module.lab_sg.security_group_id]
  associate_public_ip_address = true
  key_name                    = var.env_name
  iam_instance_profile        = module.lab_instance_profile.iam_instance_profile_name

  enable_volume_tags = false
  root_block_device = [
    {
      encrypted   = true
      volume_type = "gp3"
      volume_size = 10
      tags = {
        Name = "${var.env_name}-lab-1"
      }
    }
  ]

  tags = {
    Name = "${var.env_name}-lab-1"
  }
}

3. For full context of Terraform codes/files, please refer to this directory

Run tag compliance in CI/CD pipeline

As mentioned in the beginning of this blog, I use Amazon CodeCatalyst as tool for CI/CD pipeline.

1. I created a workflow file called tf-sandbox-sbx0-ec2.yml and I put that inside .codecatalyst/workflow directory. Inside workflow/pipeline, I created action called tag-compliance. This action use container image python:3.12.8-bookworm that being pulled from Dockerhub. First, it will redirect working directory to ec2 directory (sandbox/sbx0/ec2). Then it will run several steps that needed to run tag compliance check using Cloud Custodian. In this case, I use specific package called c7n-left that designed to evaluate policies directly against infrastructure as code source assets (Please see Custodian policies for Infrastructure Code for more details). Then it will run command that will find custodian policy inside custodian directory with category value filtered to tag

  tag-compliance:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: python:3.12.8-bookworm
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: pip install c7n
      - Run: pip install c7n-left
      - Run: c7n-left run -p ../../../custodian/ -d . --filters="category=tag"
    Compute:
      Type: EC2

2. In workflow file, tag compliance action will be merged with other actions that used by Terraform core workflow. For full context, please see this file:

Name: tf-sandbox-sbx0-ec2
SchemaVersion: "1.0"

Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox\/sbx0\/ec2\/.*

Actions:
  tag-compliance:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: python:3.12.8-bookworm
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: pip install c7n
      - Run: pip install c7n-left
      - Run: c7n-left run -p ../../../custodian/ -d . --filters="category=tag"
    Compute:
      Type: EC2
  terraform-plan:
    Identifier: aws/build@v1
    DependsOn:
      - tag-compliance
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-for-approval:
    Identifier: aws/approval@v1
    DependsOn:
      - terraform-plan
    Configuration:
      ApprovalsRequired: 1
  terraform-apply:
    DependsOn:
      - wait-for-approval
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: tf-codecatalyst-admin-sandbox
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.8.2
      Steps:
      - Run: cd sandbox/sbx0/ec2
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

3. We will simulate to provision EC2 with tag that needed by tag compliance first. Tag compliance action will run first as defined in workflow. Cause tag compliance is passed, then Terraform core workflow can be continued. Please see detailed logs of tag compliance to see the result

4. Here is the example result of successful tag compliance check using c7n-left. It said no failed result

5. As explained in point number 3, if tag compliance action is passed, then Terraform core workflow can be continued. So that it can run terraform apply to provision EC2

6. For next example, let’s see what if we want to provision EC2 that hasn’t got tag that needed by tag compliance action. In CI/CD, tag compliance action will be failed and the rest of Terraform core workflow cannot be continued. Please see detailed logs of tag compliance action for further details

7. Custodian will inform which file(s) and line(s) that has got policy violation

8. Custodian will also inform how many resources that failed and got policy violation and how many resources that passed in evaluation summary

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

1. We can shift left our tag compliance check inside CI/CD

   1. Cloud Custodian is one of the tool that can be leveraged to run tag compliance check

   2. Cloud Custodian has specific package called c7n-left to run policy check directly against Infrastructure code

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!