Create automated pull requests for Terraform-managed AWS infrastructure by using GitHub Actions - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsRelated resources

Create automated pull requests for Terraform-managed AWS infrastructure by using GitHub Actions

Matt Padgett, Ashish Bhatt, Ashwin Divakaran, Sandip Gangapadhyay, and Prafful Gupta, Amazon Web Services

Summary

This pattern presents an automation utility that’s designed to eliminate the manual, repetitive work involved in managing changes across multiple Terraform repositories. Many organizations use Terraform repositories to manage their infrastructure as code (IaC), often with hundreds of separate repositories representing different environments, services, or teams. Managing these repositories at scale presents a significant operational challenge. Routine tasks such as updating a parameter, upgrading module versions, or applying configuration changes often require creating and managing pull requests (PRs) across many repositories multiple times a day.

Even for simple changes, this repetitive and manual process is time-consuming and error prone. Engineers must consistently apply the same change across all targeted repositories and craft meaningful PR titles and descriptions. In addition, they often must interact with external tools like Jira to fetch or include issue tracking references. These tasks, while necessary, are undifferentiated heavy lifting that consume valuable engineering time and reduce overall efficiency. The lack of automation in this workflow creates friction, slows down delivery, and increases the cognitive burden on teams tasked with maintaining large-scale Terraform infrastructures.

Solution overview

To address this challenge, this pattern offers a utility that’s entirely configuration-driven, allowing users to define their desired changes in a structured configuration file. This file specifies the target repositories, modules, parameters, and values using a clearly defined schema.

Once configured, the utility performs the following automated steps:

  1. Reads the user-defined configuration to determine the scope and nature of changes

  2. Creates a new branch in each target repository with the required updates applied

  3. Generates a PR for each change, ensuring consistency across all repositories

  4. Sends Slack notifications (optional) to alert stakeholders with direct links to the created PRs

By automating these repetitive tasks, the utility significantly reduces the time, effort, and risk associated with managing large-scale infrastructure updates. It enables teams to focus on higher-value engineering work while helping to ensure that changes are applied consistently and can be traced across all repositories.

Prerequisites and limitations

Prerequisites

  • An active AWS account.

  • Python version 3.8 or later.

  • A GitHub personal access token (PAT). For more information, see Creating a personal access token (classic) in the GitHub documentation.

  • The GitHub PAT can access your target repositories so that the utility can perform operations like creating branches and pull requests. For more information, see this pattern’s GitHub code repository.

Limitations

  • Configuration complexity presents the primary challenge. The automation's effectiveness is constrained by the capabilities of its configuration file. Although the system handles standard changes efficiently, complex infrastructure modifications might require manual intervention and certain edge cases remain beyond the scope of automated handling.

  • Security and access presents significant considerations particularly in managing GitHub access tokens and API rate limits. Organizations must carefully balance the need for automation with secure credential storage and management, ensuring proper access controls while maintaining operational efficiency.

  • Validation constraints pose another notable limitation because the automated system has limited ability to validate business logic and environment-specific requirements. Complex dependencies and cross-service interactions often necessitate human oversight because automated validation can’t fully capture all contextual nuances and business rules.

  • Scale and performance issues emerge when dealing with large-scale infrastructure changes. The system must operate within GitHub API limits while managing numerous repositories simultaneously. Resource-intensive operations across extensive infrastructure can create performance bottlenecks that require careful management.

  • Integration boundaries restrict the system's flexibility because it's primarily designed to work with specific tools like GitHub and Slack. Organizations that use different tools might need custom solutions and the workflow customization options of this pattern are limited to supported integration points.

Architecture

The following diagram shows the workflow and components for this solution.

Workflow to create automated pull requests using GitHub Actions.

The workflow consists of the following steps:

  1. The developer triggers GitHub Actions by specifying the Terraform repository.

  2. The automation utility reads the defined configurations.

  3. The automation utility also pulls the provided Terraform repository.

  4. The automation utility creates a new branch and makes updates to Terraform templates locally.

  5. The automation utility pushes the new branch to the repository and creates a new PR.

  6. The automation utility uses Slack notifications that include PR links to notify developers and enables Terraform templates for AWS Cloud deployment.

Tools

  • GitHub is a developer platform that developers can use to create, store, manage, and share their code.

  • GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that’s tightly integrated with GitHub repositories. You can use GitHub Actions to automate your build, test, and deployment pipeline.

  • HashiCorp Terraform is an infrastructure as code (IaC) tool that helps you create and manage cloud and on-premises resources.

  • Slack, a Salesforce offering, is an AI-powered conversational platform that provides chat and video collaboration, automates processes with no code, and supports information sharing.

Code repository

The code for this pattern is available in the GitHub Automated Terraform Infrastructure Update Workflow using GitHub Actions repository.

Best practices

  • Effective change management is crucial for successful implementation. Organizations should adopt a gradual rollout strategy for large-scale changes. Maintain consistent branch naming conventions and PR descriptions and ensure comprehensive documentation of all changes.

  • Security controls must be rigorously implemented, focusing on least-privilege access principles and secure credential management. Enable branch protection rules to prevent unauthorized changes. Conduct regular security audits to maintain system integrity.

  • A robust testing protocol should include automated terraform plan execution in continuous integration and continuous deployment (CI/CD) pipelines. The protocol should also include pre-commit validation checks, and dedicated review environments for critical changes. This multi-layered testing approach helps catch issues early and ensures infrastructure stability.

  • Monitoring strategy needs to encompass comprehensive alerting mechanisms, detailed success/failure metrics tracking, and automated retry mechanisms for failed operations. This strategy helps to ensure operational visibility and enables quick response to any issues that arise.

  • Configuration standards should emphasize version control for all configurations, maintaining modularity for reusability and scalability. Clear documentation of schema and examples helps teams understand and use the automation system effectively.

Epics

TaskDescriptionSkills required

Set up the repository.

To set up the repository, run the following commands:

# Clone the automation tool repository
git clone https://github.com/aws-samples/sample-terraform-pr-automation-utility
cd sample-terraform-pr-automation-utility

# Copy example configuration
cp config.example.yaml config.yaml
AWS DevOps

Install dependencies.

To install and verify the Python dependencies, run the following commands:

# Install Python dependencies
pip3 install -r requirements.txt

# Verify installation
python3 -c "import github; import hcl2; import yaml; import requests; print('All packages installed successfully')"
AWS DevOps

Configure the GitHub token.

To configure the GitHub token and then verify that it works, run the following commands:

# Set GitHub token environment variable
export GITHUB_TOKEN="your_github_token_here"

# Verify token works
curl -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user
AWS DevOps
TaskDescriptionSkills required

Set up the config.yaml file.

To define your target repositories and desired changes, edit the config.yaml file as follows:

repositories:
  - owner: "your-org"
    repo: "your-terraform-repo"
    files:
      - path: "variables.tf"
        changes:
          variables:
            - app_version:
                default:
                  update:
                    - from: ["1.0.0"]
                      to: "1.1.0"

settings:
  pr_title_template: "Infrastructure Update - {{timestamp}}"
  slack:
    username: "Terraform Bot"
    icon_emoji: ":terraform:"
    notify_on_success: true
    notify_on_error: true
    notify_batch_summary: true
AWS DevOps
TaskDescriptionSkills required

Do pre-flight testing.

Always test your configuration before running it on production repositories. Use the following commands:

# 1. Test configuration syntax
python3 -c "from main import get_config_content; get_config_content()"

# 2. Run in dry-run mode first
DRY_RUN=true python3 main.py

# 3. Test with minimal configuration
# Use a simple config.yaml with just one repository and one change
AWS DevOps

Verify repository access.

To verify that the GitHub token can access the repository, run the following command:

# Test GitHub token access
curl -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/repos/owner/repo-name

# Should return repository information, not 404
AWS DevOps
TaskDescriptionSkills required

Run the automation utility by using the GitHub Actions UI.

To run the automation utility using the GitHub Actions UI, do the following:

  1. Navigate to your repository on GitHub.

  2. Choose the Actions tab.

  3. Choose the Terraform Infrastructure Update Automation workflow.

  4. Choose Run workflow.

  5. Configure the following workflow inputs:

    • Source configuration:

      • Select the target branch for automation.

      • Specify the path to the configuration file (config.yaml) that contains the desired changes.

    • Preview controls:

      • Choose the Preview option to review changes without applying them.

    • Branch management:

      • Enter the base branch for creating new feature branches.

      • Enter the branch prefix configuration.

      • Choose the Automatically close obsolete pull requests checkbox.

    • Notifications setup:

      • Enter your URL for Slack webhook URL for rich notifications (overrides repository secret).

      • Choose the Test Slack integration before processing (dry run only) checkbox.

      • Enter your information for an Override default Slack channel (e. g., #infrastructure-test).

    • Advanced settings:

      • Choose the Enable debug logging for troubleshooting checkbox.

AWS DevOps

(Alternative) Run the automation utility from the command line.

If you prefer, you can run the automation utility from the command line instead of by using the GitHub Actions UI. Use the following command:

# Run actual automation
python3 main.py
AWS DevOps
TaskDescriptionSkills required

Review the created PRs and changes.

To monitor the results of the GitHub workflow execution, do the following:

  • Check the workflow run logs for processing status.

  • Review the PRs that the automation utility created.

  • Monitor Slack notifications (if configured).

AWS DevOps
TaskDescriptionSkills required

(Optional) Clean up PRs.

Close abandoned or unnecessary PRs.

AWS DevOps

Related resources

AWS Prescriptive Guidance

GitHub documentation