Setting up for AWS Data Pipeline - AWS Data Pipeline

Setting up for AWS Data Pipeline

Before you use AWS Data Pipeline for the first time, complete the following tasks.

After you complete these tasks, you can start using AWS Data Pipeline. For a basic tutorial, see Getting Started with AWS Data Pipeline.

Sign up for AWS

When you sign up for Amazon Web Services (AWS), your AWS account is automatically signed up for all services in AWS, including AWS Data Pipeline. You are charged only for the services that you use. For more information about AWS Data Pipeline usage rates, see AWS Data Pipeline.

If you have an AWS account already, skip to the next task. If you don't have an AWS account, use the following procedure to create one.

To create an AWS account

  1. Open https://portal.aws.amazon.com/billing/signup.

  2. Follow the online instructions.

    Part of the sign-up procedure involves receiving a phone call and entering a verification code on the phone keypad.

Create IAM Roles for AWS Data Pipeline and Pipeline Resources

AWS Data Pipeline requires IAM roles that determine the permissions to perform actions and access AWS resources. The pipeline role determines the permissions that AWS Data Pipeline has, and a resource role determines the permissions that applications running on pipeline resources, such as EC2 instances, have. You specify these roles when you create a pipeline. Even if you do not specify a custom role and use the default roles DataPipelineDefaultRole and DataPipelineDefaultResourceRole, you must first create the roles and attach permissions policies. For more information, see IAM Roles for AWS Data Pipeline.

Allow IAM Principals (Users and Groups) to Perform Necessary Actions

To work with a pipeline, an IAM principal (a user or group) in your account must be allowed to perform required AWS Data Pipeline actions and actions for other services as defined by your pipeline.

To simplify permissions, the AWSDataPipeline_FullAccess managed policy is available for you to attach to IAM principals. This managed policy allows the principal to perform all actions that a user requires and the iam:PassRole action on the default roles used with AWS Data Pipeline when a custom role is not specified.

We highly recommend that you carefully evaluate this managed policy and restrict permissions only to those that your users require. If necessary, use this policy as a starting point, and then remove permissions to create a more restrictive inline permissions policy that you can attach to IAM principals. For more information and example permissions policies, see

A policy statement similar to the following example must be included in a policy attached to any IAM principal that uses the pipeline. This statement allows the IAM principal to perform the PassRole action on the roles that a pipeline uses. If you do not use default roles, replace MyPipelineRole and MyResourceRole with the custom roles that you create.

{ "Version": "2012-10-17", "Statement": [ { "Action": "iam:PassRole", "Effect": "Allow", "Resource": [ "arn:aws:iam::*:role/MyPipelineRole", "arn:aws:iam::*:role/MyResourceRole" ] } ] }

The following procedure demonstrates how to create an IAM group, attach the AWSDataPipeline_FullAccess managed policy to the group, and then add users to the group. You can use this procedure for any inline policy

To create a user group DataPipelineDevelopers and attach the AWSDataPipeline_FullAccess policy

  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Groups, Create New Group.

  3. Enter a Group Name, for example, DataPipelineDevelopers, and then choose Next Step.

  4. Enter AWSDataPipeline_FullAccess for Filter and then select it from the list.

  5. Choose Next Step and then choose Create Group.

  6. To add users to the group:

    1. Select the group you created from the list of groups.

    2. Choose Group Actions, Add Users to Group.

    3. Select the users you want to add from the list and then choose Add Users to Group.