Automatically rotate IAM user access keys at scale with AWS Organizations and AWS Secrets Manager

Created by Tracy Hickey (AWS), Gaurav Verma (AWS), Laura Seletos (AWS), Michael Davie (AWS), and Arvind Patel (AWS)

Summary

Important As a best practice, AWS recommends that you use AWS Identity and Access Management (IAM) roles instead of IAM users with long-term credentials such as access keys. The approach documented in this pattern is intended only for legacy implementations that require long-lived AWS API credentials. For these implementations, we still recommend that you consider options for using short-term credentials, such as using Amazon Elastic Compute Cloud (Amazon EC2) instance profiles or IAM Roles Anywhere. The approach in this article is only for cases where you are unable to change to using short-term credentials immediately, and you require long-term credentials to be rotated on a schedule. With this approach, you are responsible for periodically updating your legacy application code or configuration to use the rotated API credentials.

Access keys are long-term credentials for an IAM user. Regularly rotating your IAM credentials helps prevent a compromised set of IAM access keys from accessing components in your AWS account. Rotating IAM credentials is also an important part of security best practices in IAM.

This pattern helps you automatically rotate IAM access keys by using AWS CloudFormation templates, which are provided in the GitHub IAM key rotation repository.

The pattern supports deployment in a single account or multiple accounts. If you’re using AWS Organizations, this solution identifies all AWS account IDs within your organization and dynamically scales as accounts are removed or new accounts are created. The centralized AWS Lambda function uses an assumed IAM role to locally run the rotation functions across multiple accounts that you select.

• New IAM access keys are generated when existing access keys are 90 days old. 

• The new access keys are stored as a secret in AWS Secrets Manager. A resource-based policy allows only the specified IAM principal to access and retrieve the secret. If you choose to store keys in the management account, the keys for all accounts are stored in the management account.

• The email address assigned to the owner of the AWS account where the new access keys were created receives a notification.

• The previous access keys are deactivated at 100 days old, and then deleted at 110 days old.

• A centralized email notification is sent to the AWS account owner.

Lambda functions and Amazon CloudWatch automatically perform these actions. You can then retrieve the new access key pair and replace them in your code or applications. The rotation, deletion, and deactivation periods can be customized.

Prerequisites and limitations

• At least one active AWS account.

• AWS Organizations, configured and set up (see tutorial).

• Permissions to query AWS Organizations from your management account. For more information, see AWS Organizations and service-linked roles in the AWS Organizations documentation.

• An IAM principal that has permissions to launch the AWS CloudFormation template and associated resources. For more information, see Grant self-managed permissions in the AWS CloudFormation documentation.

• An existing Amazon Simple Storage Service (Amazon S3) bucket to deploy the resources.

• Amazon Simple Email Service (Amazon SES) moved out of the sandbox. For more information, see Moving out of the Amazon SES sandbox in the Amazon SES documentation.

• If you choose to run Lambda in a virtual private cloud (VPC), the following resources, which should be created before you run the main CloudFormation template:

  • A VPC.

  • A subnet.

  • Endpoints for Amazon SES, AWS Systems Manager, AWS Security Token Service (AWS STS), Amazon S3, and AWS Secrets Manager. (You can run the endpoint template that's provided in the GitHub IAM key rotation repository to create these endpoints.)

• The Simple Mail Transfer Protocol (SMTP) user and password stored in AWS Systems Manager parameters (SSM parameters). Parameters must match the main CloudFormation template parameters.

Architecture

Technology stack  

• Amazon CloudWatch

• Amazon EventBridge

• IAM

• AWS Lambda

• AWS Organizations 

• Amazon S3

Architecture

The following diagrams show the components and workflows for this pattern. The solution supports two scenarios for storing the credentials: in a member account and in the management account.

Option 1: Store the credentials in a member account

Option 2: Store the credentials in the management account

The diagrams show the following workflow:

1. An EventBridge event initiates an account_inventory Lambda function every 24 hours.

2. This Lambda function queries AWS Organizations for a list of all AWS account IDs, account names, and account emails. 

3. The account_inventory Lambda function initiates an access_key_auto_rotation Lambda function for each AWS account ID and passes the metadata to it for additional processing.

4. The access_key_auto_rotation Lambda function uses an assumed IAM role to access the AWS account ID. The Lambda script runs an audit against all users and their IAM access keys in the account. 

5. If the IAM access key's age hasn’t exceeded the best practice threshold, the Lambda function takes no further action.

6. If the IAM access key's age has exceeded the best practice threshold, the access_key_auto_rotation Lambda function determines which rotation action to perform.

7. When action is required, the access_key_auto_rotation Lambda function creates and updates a secret in AWS Secrets Manager if a new key is generated. A resource-based policy is also created that allows only the specified IAM principal to access and retrieve the secret. In the case of option 1, the credentials are stored in Secrets Manager in the respective account. In the case of option 2 (if the StoreSecretsInCentralAccount flag is set to True), the credentials are stored in Secrets Manager in the management account. 

8. A notifier Lambda function is initiated to notify the account's owner of the rotation activity. This function receives the AWS account ID, account name, account email, and the rotation actions that were performed. 

9. The notifier Lambda function queries the deployment S3 bucket for an email template and dynamically updates it with the relevant activity metadata. The email is then sent to the account owner's email address.

Notes:

• This solution supports resiliency in multiple Availability Zones. However, it doesn’t support resiliency in multiple AWS Regions. For support in multiple Regions, you can deploy the solution in the second Region and keep the key rotation EventBridge rule disabled. You can then enable the rule when you want to run the solution in the second Region.

• You can run this solution in audit mode. In audit mode, IAM access keys aren’t modified, but an email is sent to notify users. To run the solution in audit mode, set the DryRunFlag flag to True when you run the key rotation template or in the environment variable for the access_key_auto_rotation Lambda function.

Automation and scale

The CloudFormation templates that automate this solution are provided in the GitHub IAM key rotation repository and listed in the Code section. In AWS Organizations, you can use CloudFormation StackSets to deploy the ASA-iam-key-auto-rotation-iam-assumed-roles.yaml CloudFormation template in multiple accounts instead of deploying the solution individually to each member account. 

Tools

AWS services

• Amazon CloudWatch helps you monitor the metrics of your AWS resources and the applications you run on AWS in real time.

• AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.

• AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

• AWS Organizations is an account management service that helps you consolidate multiple AWS accounts into an organization that you create and centrally manage.

• AWS Secrets Manager helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.

• Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

• Amazon Simple Email Service (Amazon SES) helps you send and receive emails by using your own email addresses and domains.

• Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.

• Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS. 

• Amazon VPC endpoints provide an interface to connect to services powered by AWS PrivateLink, including many AWS services. For each subnet that you specify from your VPC, an endpoint network interface is created in the subnet and assigned a private IP address from the subnet address range. 

Code

The required AWS CloudFormation templates, Python scripts, and runbook documentation are available in the GitHub IAM key rotation repository. The templates are deployed as follows.

Template	Deploy in	Notes
ASA-iam-key-auto-rotation-and-notifier-solution.yaml	Deployment account	This is the main template for the solution.
ASA-iam-key-auto-rotation-iam-assumed-roles.yaml	Single or multiple member accounts where you want to rotate the credentials	You can use CloudFormation stack sets to deploy this template in multiple accounts.
ASA-iam-key-auto-rotation-list-accounts-role.yaml	Central/management account	Use this template to keep an inventory of accounts in AWS Organizations.
ASA-iam-key-auto-rotation-vpc-endpoints.yaml	Deployment account	Use this template to automate the creation of endpoints only if you want to run the Lambda functions in a VPC (set the RunLambdaInVPC parameter to True in the main template).

Epics

Set up the solution

Task	Description	Skills required
Choose your deployment S3 bucket.	Sign in to the AWS Management Console for your account, open the Amazon S3 console, and then choose the S3 bucket for your deployment. If you want to implement the solution for multiple accounts in AWS Organizations, sign in to the management account for your organization.	Cloud architect
Clone the repository.	Clone the GitHub IAM key rotation repository to your local desktop.	Cloud architect
Upload the files to the S3 bucket.	Upload the cloned files to your S3 bucket. Use the following default folder structure to copy and paste all cloned files and directories: asa/asa-iam-rotation Note You can customize this folder structure in the CloudFormation templates.	Cloud architect
Modify the email template.	Modify the iam-auto-key-rotation-enforcement.html email template (located in the template folder) according to your requirements. Replace [Department Name Here] at the end of the template with your department's name.	Cloud architect

Deploy the solution

Task	Description	Skills required
Launch the CloudFormation template for key rotation.	Launch the ASA-iam-key-auto-rotation-and-notifier-solution.yaml template in the deployment account. For more information, see Selecting a stack template in the CloudFormation documentation. Specify values for parameters, including: CloudFormation S3 Bucket Name (S3BucketName) – The name of the deployment S3 bucket that contains your Lambda code. CloudFormation S3 Bucket Prefix (S3BucketPrefix) – The S3 bucket’s prefix. Assumed IAM Role Name (IAMRoleName) – The role name that the key-rotation Lambda function will assume to rotate the keys. IAM Execution Role Name (ExecutionRoleName) – The name of the IAM execution role that’s used by the key-rotation Lambda function. Inventory Execution Role Name (InventoryExecutionRoleName) – The name of the IAM execution role that’s used by the account_inventory Lambda function. Dry Run Flag (Audit Mode) (DryRunFlag) – Set to True to turn on audit mode (default). Set to False to turn on enforcement mode. Account to List Organization Accounts (OrgListAccount) – The account ID of the central/management account that will be used to list the accounts in the organization. List Accounts Role Name (OrgListRole) – The role name that will be used to list the accounts in the organization. Secrets Store flag for central account (StoreSecretsInCentralAccount) – Set to True to store secrets in the central account. Set to False to store secrets in the respective account. Regions to replicate the credentials (CredentialReplicationRegions) – The AWS Regions where you want to replicate the credentials (Secrets Manager), separated by commas; for example, us-east-2,us-west-1,us-west-2. Skip the Region where you are creating the stack. Run Lambda in VPC (RunLambdaInVpc) – Set to True to run Lambda functions in a specified VPC. You must have VPC endpoints created and attach a NAT gateway to the subnet that contains the Lambda function. For more information, see the re:Post article that covers this option. VPC Id for Lambda functions (VpcId), VPC CIDR for Security Group Rule (VpcCidr), and Subnet Id for Lambda functions (SubnetId) – Provide information about the VPC, CIDR, and subnet if you set RunLambdaInVpc to True. Admin Email Address (AdminEmailAddress) – A valid email address to send notifications to. AWS Organization ID (AWSOrgID) – The unique ID of your organization. This ID begins with o- and is followed by 10-32 lowercase letters or digits. Email Template File name [Audit Mode] (EmailTemplateAudit) and [Enforce Mode] (EmailTemplateEnforce) – The filename of the email HTML template to be sent out by the notifier module for audit mode and enforce mode. SMTP User SSM Parameter Name (SMTPUserParamName) and SMTP Password SSM Parameter Name (SMTPPasswordParamName) – User and password information for Simple Mail Transfer Protocol (SMTP).	Cloud architect
Launch the CloudFormation template for assumed roles.	In the AWS CloudFormation console, launch the ASA-iam-key-auto-rotation-iam-assumed-roles.yaml template for each account where you want to rotate keys. If you have more than one account, you can deploy the main CloudFormation template in your management account as a stack and deploy the ASA-iam-key-auto-rotation-iam-assumed-roles.yaml template with CloudFormation stack sets to all required accounts. For more information, see Working with AWS CloudFormation StackSets in the CloudFormation documentation. Specify values for the following parameters: Assumed IAM Role Name (IAMRoleName) – IAM role name that will be assumed by the Lambda access_key_auto_rotation function. You can keep the default value. IAM Execution Role Name (ExecutionRoleName) – The IAM role that will assume the sub-account role to run the Lambda function. Primary AWS Account ID (PrimaryAccountID) – The AWS account ID where the main template will be deployed.  IAM Exemption Group (IAMExemptionGroup) – The IAM group name being used to facilitate IAM accounts that you want to exclude from automatic key rotation.	Cloud architect
Launch the CloudFormation template for account inventory.	Launch the ASA-iam-key-auto-rotation-list-accounts-role.yaml template in the management/central account Specify values for the following parameters: Assumed IAM Role Name (IAMRoleName) – IAM role name that the Lambda access_key_auto_rotation function will assume. IAM Execution Role Name for Account Lambda (AccountExecutionRoleName) – The name of the IAM role that the Lambda notifier function will assume. IAM Execution Role Name for rotation Lambda (RotationExecutionRoleName) – The name of the IAM role that the Lambda access_key_auto_rotation function will assume. Primary AWS Account ID (PrimaryAccountID) – The AWS account ID where the main template will be deployed.	Cloud architect
Launch the CloudFormation template for VPC endpoints.	This task is optional. Launch the ASA-iam-key-auto-rotation-vpc-endpoints.yaml template in the deployment account.  Specify values for the following parameters: VPC ID (pVpcId), Subnet Id (pSubnetId), and CIDR range for VPC (pVPCCidr) – Provide information about the VPC, CIDR, and subnet. Set the parameter for each VPC endpoint to True. If you already have endpoints, you can choose False.	Cloud architect

Related resources

• Security best practices in IAM (IAM documentation)

• AWS Organizations and service-linked roles (AWS Organizations documentation)

• Selecting a stack template (CloudFormation documentation)

• Working with AWS CloudFormation StackSets (CloudFormation documentation)