Launch a CodeBuild project across AWS accounts using Step Functions and a Lambda proxy function - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshooting

Launch a CodeBuild project across AWS accounts using Step Functions and a Lambda proxy function

Created by Richard Milner-Watts (AWS) and Amit Anjarlekar (AWS)

Code repository: Cross-Account CodeBuild Proxy

Environment: Production

Technologies: DevOps; Management & governance; Operations; Serverless

AWS services: AWS CodeBuild; AWS Lambda; AWS Step Functions; AWS X-Ray; AWS CloudFormation

Summary

This pattern demonstrates how to asynchronously launch an AWS CodeBuild project across multiple AWS accounts by using AWS Step Functions and an AWS Lambda proxy function. You can use the pattern’s sample Step Functions state machine to test the success of your CodeBuild project.

CodeBuild helps you launch operational tasks using the AWS Command Line Interface (AWS CLI) from a fully-managed runtime environment. You can change the behavior of your CodeBuild project at runtime by overriding environment variables. Additionally, you can use CodeBuild to manage workflows. For more information, see Service Catalog Tools on the AWS Workshop website and Schedule jobs in Amazon RDS for PostgreSQL using AWS CodeBuild and Amazon EventBridge on the AWS Database Blog.

Prerequisites and limitations

Prerequisites

  • Two active AWS accounts: a source account for invoking a Lambda proxy function with Step Functions and a target account for building a remote CodeBuild sample project

Limitations

  • This pattern cannot be used to copy artifacts between accounts.

Architecture

The following diagram shows the architecture that this pattern builds.

Architecture diagram of launching a CodeBuild project across multiple AWS accounts

The diagram shows the following workflow:

  1. The Step Functions state machine parses the supplied input map and invokes the Lambda proxy function (codebuild-proxy-lambda) for each account, Region, and project you defined.

  2. The Lambda proxy function uses AWS Security Token Service (AWS STS) to assume an IAM proxy role (codebuild-proxy-role), which is associated with an IAM policy (codebuild-proxy-policy) in the target account.

  3. Using the assumed role, the Lambda function launches the CodeBuild project and returns the CodeBuild job ID. The Step Functions state machine loops and polls the CodeBuild job until receiving a success or failure status.

The state machine logic is shown in the following image.

Workflow of Step Functions state machine

Technology stack

  • AWS CloudFormation

  • CodeBuild

  • IAM

  • Lambda

  • Step Functions

  • X-Ray

Tools

  • AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.

  • AWS CloudFormation Designer provides an integrated JSON and YAML editor that helps you view and edit CloudFormation templates.

  • AWS CodeBuild is a fully managed build service that helps you compile source code, run unit tests, and produce artifacts that are ready to deploy.

  • AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.

  • AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

  • AWS Step Functions is a serverless orchestration service that helps you combine AWS Lambda functions and other AWS services to build business-critical applications.

  • AWS X-Ray helps you collect data about the requests that your application serves, and provides tools that you can use to view, filter, and gain insights into that data to identify issues and opportunities for optimization.

Code

The sample code for this pattern is available in the GitHub Cross Account CodeBuild Proxy repository. This pattern uses the AWS Lambda Powertools for Python library to provide logging and tracing functionality. For more information on this library and its utilities, see Powertools for AWS Lambda (Python).

Best practices

  1. Adjust the wait time values in the Step Function state machine to minimize polling requests for job status. Use the expected execution time for the CodeBuild project.

  2. Adjust the MaxConcurrency property of the map in Step Functions to control how many CodeBuild projects can run in parallel.

  3. If required, review the sample code for production readiness. Consider what data might be logged by the solution and whether the default Amazon CloudWatch encryption is sufficient.

Epics

TaskDescriptionSkills required

Record the AWS account IDs.

AWS account IDs are required to set up access across accounts.

Record the AWS account ID for your source and target accounts. For more information, see Finding your AWS account ID in the IAM documentation.

AWS DevOps

Download the AWS CloudFormation templates.

  1. Download the sample_target_codebuild_template.yaml AWS CloudFormation template from the GitHub repository for this pattern.

  2. Download the codebuild_lambda_proxy_template.yaml AWS CloudFormation template from the GitHub repository for this pattern.

Note: In the AWS CloudFormation templates, <SourceAccountId> is the AWS account ID for the source account, and <TargetAccountId> is the AWS account ID for the target account.

AWS DevOps

Create and deploy the AWS CloudFormation stack.

  1. Sign in to the AWS Management Console for your source account, open the AWS CloudFormation console, and then choose Stacks.

  2. Choose Create Stack and then choose With new resources (standard).

  3. For Template source, choose Upload a template file.

  4. For Upload a template file, choose file, and then choose your downloaded codebuild_lambda_proxy_template.yaml file. Choose Next.

  5. For Stack name, enter a name for the stack (for example, codebuild-lambda-proxy).

  6. Replace the crossAccountTargetRoleArn parameter with your <TargetAccountId> (for example, <arn:aws:iam::123456789012:role/proxy-lambda-codebuild-role>). Note: You aren’t required to update the default value for the targetCodeBuildProject parameter.

  7. Choose Next, accept the default stack creation options, and then chose Next.

  8. Choose the I acknowledge that AWS CloudFormation might create IAM resources with custom names check box, and then choose Create stack.

Note: You must create the AWS CloudFormation stack for the proxy Lambda function before creating any resources in target accounts. When you create a trust policy in a target account, the IAM role is translated from the role name to an internal identifier. This is why the IAM role must already exist.

AWS DevOps

Confirm the creation of the proxy function and state machine.

  1. Wait for the AWS CloudFormation stack to reach CREATE_COMPLETE status. This should take less than one minute.

  2. Open the AWS Lambda console, choose Functions, and then find the lambda-proxy-ProxyLambda-<GUID> function.

  3. Open the AWS Step Functions console, choose state machines, and then find the sample-crossaccount-codebuild-state-machine state machine.

AWS DevOps
TaskDescriptionSkills required

Create and deploy the AWS CloudFormation stack.

  1. Sign in to the AWS Management Console for your target account, open the AWS CloudFormation console, and then choose Stacks.

  2. Choose Create Stack and then choose With new resources (standard).

  3. For Template source, choose Upload a template file.

  4. For Upload a template file, choose Choose file, and then choose the sample_target_codebuild_template.yaml file. Choose Next.

  5. For Stack name, enter a name for the stack (for example: sample-codebuild-stack).

  6. Replace the crossAccountSourceRoleArn parameter with your <SourceAccountId> (for example, <arn:aws:iam::123456789012:role/codebuild-proxy-lambda-role>).

  7. Choose Next, accept the default stack creation options, and then choose Next.

  8. Choose the I acknowledge that AWS CloudFormation might create IAM resources with custom names check box, and then choose Create stack.

AWS DevOps

Verify the creation of the sample CodeBuild project.

  1. Wait for the AWS CloudFormation stack to reach CREATE_COMPLETE status. This should take less than one minute.

  2. Open the AWS CodeBuild console and then find the sample-codebuild-project project.

AWS DevOps
TaskDescriptionSkills required

Launch the state machine.

  1. Sign in to the AWS Management Console for your source account, open the AWS Step Functions console, and then choose State machines.

  2. Choose the sample-crossaccount-codebuild-state-machine state machine and then choose Start execution.

  3. In the Input editor, enter the following JSON, and replace <TargetAccountID> with the AWS account ID of the account that contains the CodeBuild project.

    {
  "crossAccountTargetRoleArns": [
    {
      "arn": "arn:aws:iam::<TargetAccountID>:role/proxy-lambda-codebuild-role",
      "region": "eu-west-1",
      "codeBuildProject": "sample-codebuild-project",
      "SampleValue1": "Value1",
      "SampleValue2": "Value2"
    }
  ]
}

    Note: The key-value pairs are passed as environment variables from the function in the source account to the CodeBuild project in the target account.

  4. Choose Start execution.

  5. On the Details tab of the state machine page, check if Execution Status is set to Succeeded. This confirms that your state machine is running. Note: It can take around 30 seconds for the state machine to reach Succeeded status.

  6. To see the output and input of a step in the state machine, expand that step in the Execution event history section. For example, expand the Lambda - CodeBuild Proxy – Start step. The output includes details on the overridden environment variables, the original payload, and the CodeBuild job ID.

AWS DevOps

Validate the environment variables.

  1. Sign in to the AWS Management Console for your target account.

  2. Open the AWS CodeBuild console, expand Build, and then choose Build projects.

  3. Choose the sample-codebuild-project project, and then choose View details.

  4. On the Build history tab, choose the most recent build of the project, and then choose View logs.

  5. In the log output, verify that the environment variables printed to STDOUT match the environment variables from the Step Functions sample state machine.

AWS DevOps

Troubleshooting

IssueSolution

Step Functions execution is taking longer than expected.

Adjust the MaxConcurrency property of the map in the Step Function state machine to control how many CodeBuild projects can run in parallel.

The execution of the CodeBuild jobs is taking longer than expected.

  1. Adjust the wait time values in the Step Functions state machine to minimize polling requests for job status. Use the expected execution time for the CodeBuild project.

  2. Consider whether CodeBuild is the appropriate tool to be using. For example, the time required to initialize a CodeBuild job can be significantly longer than AWS Lambda. If high throughput and fast completion times are a requirement, consider migrating the business logic to AWS Lambda and using a fan-out architecture.