Menu
AWS CodePipeline
User Guide (API Version 2015-07-09)

Tutorial: Create a Four-Stage Pipeline

Now that you've created your first pipeline in Tutorial: Create a Simple Pipeline (Amazon S3 Bucket) or Tutorial: Create a Simple Pipeline (AWS CodeCommit Repository), you can start creating more complex pipelines. This tutorial will walk you through the creation of a four-stage pipeline that uses a GitHub repository for your source, a Jenkins build server to build the project, and an AWS CodeDeploy application to deploy the built code to a staging server. After the pipeline is created, you will edit it to add a stage with a test action to test the code, also using Jenkins.

Before you can create this pipeline, you must configure the required resources. For example, if you want to use a GitHub repository for your source code, you must create the repository before you can add it to a pipeline. As part of setting up, this tutorial walks you through setting up Jenkins on an Amazon EC2 instance for demonstration purposes.

Before you begin this tutorial, you should have already completed the general prerequisites in Getting Started.

Step 1: Set Up Prerequisites

To integrate with Jenkins, AWS CodePipeline requires you to install the AWS CodePipeline Plugin for Jenkins on any instance of Jenkins you want to use with AWS CodePipeline. You should also configure a dedicated IAM user to use for permissions between your Jenkins project and AWS CodePipeline. The easiest way to integrate Jenkins and AWS CodePipeline is to install Jenkins on an Amazon EC2 instance that uses an IAM instance role that you create for Jenkins integration. In order for links in the pipeline for Jenkins actions to successfully connect, you must configure proxy and firewall settings on the server or Amazon EC2 instance to allow inbound connections to the port used by your Jenkins project. Make sure you have configured Jenkins to authenticate users and enforce access control before you allow connections on those ports (for example, 443 and 8443 if you have secured Jenkins to only use HTTPS connections, or 80 and 8080 if you allow HTTP connections). For more information, see Securing Jenkins.

Note

This tutorial uses a code sample and configures build steps that convert the sample from Haml to HTML. You can download the open-source sample code from the GitHub repository by following the steps in this topic. You will need the entire sample in your GitHub repository, not just the .zip file.

This tutorial also assumes that:

  • You are familiar with installing and administering Jenkins and creating Jenkins projects.

  • You have installed Rake and the Haml gem for Ruby on the same computer or instance that hosts your Jenkins project.

  • You have set the required system environment variables so that Rake commands can be run from the terminal or command line (for example, on Windows systems, modifying the PATH variable to include the directory where you installed Rake).

Copy or Clone the Sample into a GitHub Repository

Clone the sample and push to a GitHub repository

  1. Download the sample code from the GitHub repository, or clone the repositories to your local computer. There are two sample packages:

  2. From the repository, choose Fork to clone the sample repo into a repo in your Github account. For more information, see the GitHub documentation.

Create an IAM Role to Use for Jenkins Integration

As a best practice, consider launching an Amazon EC2 instance to host your Jenkins server and using an IAM role to grant the instance the required permissions for interacting with AWS CodePipeline.

  1. Sign in to the IAM console at https://console.aws.amazon.com/iam/.

  2. In the IAM console, in the navigation pane, choose Roles, and then choose Create New Role.

  3. On the Set Role Name page, type the name of the role you will create specifically for Jenkins integration (for example JenkinsAccess), and then choose Next Step. On the Select Role Type page, choose the Select button next to Amazon EC2, and then choose Next Step. On the Attach Policy page, select the AWSCodePipelineCustomActionAccess managed policy, and then choose Next Step. Review the role information, and then choose Create Role.

When you create the Amazon EC2 instance where you will install Jenkins, in Step 3: Configure Instance Details, make sure you choose the instance role (for example, JenkinsAccess).

For more information about instance roles and Amazon EC2, see IAM Roles for Amazon EC2, Using IAM Roles to Grant Permissions to Applications Running on Amazon EC2 Instances, and Creating a Role to Delegate Permissions to an AWS Service.

Install and Configure Jenkins and the AWS CodePipeline Plugin for Jenkins

To install Jenkins and the AWS CodePipeline Plugin for Jenkins

  1. Create an Amazon EC2 instance where you will install Jenkins, and in Step 3: Configure Instance Details, make sure you choose the instance role you created (for example, JenkinsAccess). For more information about creating Amazon EC2 instances, see Launch an Amazon EC2 Instance.

    Note

    If you already have Jenkins resources you want to use, you can do so, but you must create a special IAM user, apply the AWSCodePipelineCustomActionAccess managed policy to that user, and then configure and use the access credentials for that user on your Jenkins resource. If you want to use the Jenkins UI to supply the credentials, configure Jenkins to only allow HTTPS. For more information, see Troubleshooting.

  2. Install Jenkins on the Amazon EC2 instance. For more information, see the Jenkins documentation for installing Jenkins and starting and accessing Jenkins, as well as details of integration with Jenkins in Product and Service Integrations.

  3. Launch Jenkins, and on the home page, choose Manage Jenkins.

  4. On the Manage Jenkins page, choose Manage Plugins.

  5. Choose the Available tab, and in the Filter search box, type AWS CodePipeline. Choose AWS CodePipeline Plugin for Jenkins from the list and choose Download now and install after restart.

  6. On the Installing Plugins/Upgrades page, select Restart Jenkins when installation is complete and no jobs are running.

  7. Choose Back to Dashboard.

  8. On the main page, choose New Item.

  9. In Item Name, type a name for the Jenkins project (for example, MyDemoProject). Choose Freestyle project, and then choose OK.

    Note

    Make sure the name for your project meets the requirements for AWS CodePipeline. For more information, see Limits.

  10. On the configuration page for the project, select the Execute concurrent builds if necessary check box. In Source Code Management, choose AWS CodePipeline. If you have installed Jenkins on an Amazon EC2 instance and configured the AWS CLI with the profile for the IAM user you created for integration between AWS CodePipeline and Jenkins, leave all of the other fields empty.

  11. Choose Advanced, and in Provider, type a name for the provider of the action as it will appear in AWS CodePipeline (for example, MyJenkinsProviderName). Make sure this name is unique and easy to remember. You will use it when you add a build action to your pipeline later in this tutorial, and again when you add a test action.

    Note

    This action name must meet the naming requirements for actions in AWS CodePipeline. For more information, see Limits.

  12. In Build Triggers, clear any check boxes, and then select Poll SCM. In Schedule, type five asterisks separated by spaces, as follows:

    Copy
    * * * * *

    This polls AWS CodePipeline every minute.

  13. In Build, choose Add build step. Choose Execute shell (Amazon Linux, RHEL, or Ubuntu Server) Execute batch command (Windows Server), and then type the following:

    Copy
    rake

    Note

    Make sure your environment is configured with the variables and settings required to run rake; otherwise, the build will fail.

  14. Choose Add post-build action, and then choose AWS CodePipeline Publisher. Choose Add, and in Build Output Locations, leave the location blank. This configuration is the default. It will create a compressed file at the end of the build process.

  15. Choose Save to save your Jenkins project.

Step 2: Create a Pipeline in AWS CodePipeline

In this part of the tutorial, you will create the pipeline using the Create Pipeline wizard.

To create an AWS CodePipeline automated release process

  1. Sign in to the AWS Management Console and open the AWS CodePipeline console at http://console.aws.amazon.com/codepipeline.

  2. If necessary, use the region selector to change the region to the same region where your pipeline resources are located. For example, if you created resources for the previous tutorial in us-east-2, make sure the region selector is set to US East (Ohio).

    For more information about the regions and endpoints available for AWS CodePipeline, see Regions and Endpoints.

  3. On the introductory page, choose Get started.

    If you see the Welcome page, choose Create pipeline.

  4. In Step 1: Name, in Pipeline name, type MySecondPipeline, and then choose Next step.

    Note

    If you choose another name for your pipeline, be sure to use it instead of MySecondPipeline for the rest of this tutorial. After you create a pipeline, you cannot change its name. Pipeline names are subject to some limitations. For more information, see Limits.

  5. In Step 2: Source, in Source provider, choose GitHub, and then choose Connect to GitHub. This will open a new browser window that will connect you to GitHub. If prompted to sign in, provide your GitHub credentials.

    Important

    Do not provide your AWS credentials on the GitHub website.

    After you have connected to GitHub, choose the repository and branch where you pushed the sample you want to use for this tutorial (aws-codepipeline-jenkins-aws-codedeploy_linux.zip or AWSCodePipeline-Jenkins-AWSCodeDeploy_Windows.zip), and then choose Next step.

    Note

    There is a limit to the number of OAuth tokens you can use in GitHub for a particular application, such as AWS CodePipeline. Within a single AWS account, AWS CodePipeline will automatically update existing equivalent OAuth tokens to in an attempt to avoid exceeding this limit. If you exceed this limit as a result of connecting many AWS accounts with the same GitHub user account, you can use personal tokens. For more information, see To configure a pipeline to use a personal access token from GitHub.

  6. In Step 3: Build, choose Add Jenkins. In Provider name, type the name of the action you provided in the AWS CodePipeline Plugin for Jenkins (for example MyJenkinsProviderName). This name must exactly match the name in the AWS CodePipeline Plugin for Jenkins. In Server URL, type the URL of the Amazon EC2 instance where Jenkins is installed. In Project name, type the name of the project you created in Jenkins, such as MyDemoProject, and then choose Next step.

  7. In Step 4: Deploy, reuse the AWS CodeDeploy application and deployment group you created in Tutorial: Create a Simple Pipeline (Amazon S3 Bucket). In Deployment provider, choose AWS CodeDeploy. In Application name, type CodePipelineDemoApplication, or choose the refresh button, and then choose the application name from the list. In Deployment group, type CodePipelineDemoFleet, or choose it from the list, and then choose Next step.

    Note

    You can use your own AWS CodeDeploy resources or create new ones, but you might incur additional costs.

  8. In Step 5: Service Role, from Role name, choose the service role you created for AWS CodePipeline (for example, AWS-CodePipeline-Service), and then choose Next step.

    Note

    Service role creation is only required the first time you create a pipeline in AWS CodePipeline. If you followed the steps in one of the simple pipeline tutorials, you already created this service role and will be able to choose it from the drop-down list. Because the drop-down list will display all IAM service roles associated with your account, if you chose a name different from the default, choose that name instead. If you have not yet created a service role, choose Create role.

    If you are using an AWS CodeCommit repository instead of a GitHub repository, and are using a service role that was created before April 18, 2016, make sure it includes the permissions required to access AWS CodeCommit. For more information, see Add Permissions for Other AWS Services.

  9. In Step 6: Review, review the information, and then choose Create pipeline.

  10. The pipeline automatically starts and runs the sample through the pipeline. You can view progress and success and failure messages as the pipeline builds the Haml sample to HTML and deploys it a web page to each of the Amazon EC2 instances in the AWS CodeDeploy deployment.

Step 3: Add Another Stage to Your Pipeline

Now you will add a test stage and then a test action to that stage that uses the Jenkins test included in the sample to determine whether the web page has any content. This test is for demonstration purposes only.

Note

If you did not want to add another stage to your pipeline, you could add a test action to the Staging stage of the pipeline, before or after the deployment action.

Add a Test Stage to Your Pipeline

Look Up the IP Address of an Instance

To verify the IP address of an instance where you deployed your code

  1. After Succeeded is displayed for the pipeline status, in the status area for the Staging stage, choose Details.

  2. In the Deployment Details section, in Instance ID, choose the instance ID of one of the successfully deployed instances.

  3. Copy the IP address of the instance (for example, 192.168.0.4). You will use this IP address in your Jenkins test.

Create a Jenkins project for Testing the Deployment

To create the Jenkins project

  1. On the instance where you installed Jenkins, open Jenkins and from the main page, choose New Item.

  2. In Item Name, type a name for the Jenkins project (for example, MyTestProject). Choose Freestyle project, and then choose OK.

    Note

    Make sure the name for your project meets the AWS CodePipeline requirements. For more information, see Limits.

  3. On the configuration page for the project, select the Execute concurrent builds if necessary check box. In Source Code Management, choose AWS CodePipeline. If you have installed Jenkins on an Amazon EC2 instance and configured the AWS CLI with the profile for the IAM user you created for integration between AWS CodePipeline and Jenkins, leave all the other fields empty.

    Important

    If you are configuring a Jenkins project and it is not installed on an Amazon EC2 instance, or it is installed on an Amazon EC2 instance that is running a Windows operating system, complete the fields as required by your proxy host and port settings, and provide the credentials of the IAM user you configured for integration between Jenkins and AWS CodePipeline.

  4. Choose Advanced, and in Category, choose Test.

  5. In Provider, type the same name you used for the build project (for example, MyJenkinsProviderName). You will use this name when you add the test action to your pipeline later in this tutorial.

    Note

    This name must meet the AWS CodePipeline naming requirements for actions. For more information, see Limits.

  6. In Build Triggers, clear any check boxes, and then select Poll SCM. In Schedule, type five asterisks separated by spaces, as follows:

    Copy
    * * * * *

    This polls AWS CodePipeline every minute.

  7. In Build, choose Add build step. If you are deploying to Amazon Linux, RHEL, or Ubuntu Server instances, choose Execute shell , and then type the following, where the IP address is the address of the Amazon EC2 instance you copied earlier:

    Copy
    TEST_IP_ADDRESS=192.168.0.4 rake test

    If you are deploying to Windows Server instances, choose Execute batch command, and then type the following, where the IP address is the address of the Amazon EC2 instance you copied earlier:

    Copy
    set TEST_IP_ADDRESS=192.168.0.4 rake test

    Note

    The test assumes a default port of 80. If you want to specify a different port, add a test port statement, as follows:

    Copy
    TEST_IP_ADDRESS=192.168.0.4 TEST_PORT=8000 rake test
  8. Choose Add post-build action, and then choose AWS CodePipeline Publisher. Do not choose Add.

  9. Choose Save to save your Jenkins project.

Create a Fourth Stage

To add a stage to your pipeline that includes the Jenkins test action

  1. Sign in to the AWS Management Console and open the AWS CodePipeline console at http://console.aws.amazon.com/codepipeline.

  2. In Name, choose the name of the pipeline you created, MySecondPipeline.

  3. On the pipeline details page, choose Edit.

  4. On the Edit page, choose + Stage to add a stage immediately after the Staging stage.

  5. In the name field for the new stage, type a name (for example, Testing), and then choose + Action.

  6. In the Action category drop-down list, choose Test. In Action name, type MyJenkinsTest-Action. In Test provider, choose the provider name you specified in Jenkins (for example, MyJenkinsProviderName). In Project name, type the name of the project you created in Jenkins (for example, MyTestProject). In Input artifacts, choose the artifact from the Jenkins build whose default name is MyBuiltApp, and then choose Add action.

    For more information about input and output artifacts and the structure of pipelines, see Pipeline Structure Reference.

  7. On the Edit page, choose Save pipeline changes. In the Save pipeline changes dialog box, choose Save and continue.

  8. Although the new stage has been added to your pipeline, a status of No executions yet is displayed for that stage because no changes have triggered another run of the pipeline. To run the sample through the revised pipeline, on the pipeline details page, choose Release change.

    The pipeline view shows the stages and actions in your pipeline and the state of the revision running through those four stages. The time it takes for the pipeline to run through all stages will depend on the size of the artifacts, the complexity of your build and test actions, and other factors.

Step 4: Clean Up Resources

After you complete this tutorial, you should delete the pipeline and the resources it uses so you will not be charged for continued use of those resources. If you do not intend to keep using AWS CodePipeline, delete the pipeline, then the AWS CodeDeploy application and its associated Amazon EC2 instances, and finally, the Amazon S3 bucket used to store artifacts. You should also consider whether to delete other resources, such as the GitHub repository, if you do not intend to keep using them.

To clean up the resources used in this tutorial

  1. Open a terminal session on your local Linux, macOS, or Unix machine, or a command prompt on your local Windows machine, and run the delete-pipeline command to delete the pipeline you created. For MySecondPipeline, you would type the following command:

    Copy
    aws codepipeline delete-pipeline --name "MySecondPipeline"

    This command returns nothing.

  2. To clean up your AWS CodeDeploy resources, follow the instructions in Cleaning Up.

  3. To clean up your instance resources, delete the Amazon EC2 instance where you installed Jenkins. For more information, see Clean Up Your Instance and Volume.

  4. If you do not intend to create more pipelines or use AWS CodePipeline again, delete the Amazon S3 bucket used to store artifacts for your pipeline. To delete the bucket, follow the instructions in Deleting a Bucket.

  5. If you do not intend to use the other resources for this pipeline again, consider deleting them by following the guidance for that particular resource. For example, if you want to delete the GitHub repository, follow the instructions in Deleting a repository on the GitHub website.