Access a bastion host by using Session Manager and Amazon EC2 Instance Connect

Created by Piotr Chotkowski (AWS) and Witold Kowalik (AWS)

Summary

A bastion host, sometimes called a jump box, is a server that provides a single point of access from an external network to the resources located in a private network. A server exposed to an external public network, such as the internet, poses a potential security risk for unauthorized access. It’s important to secure and control access to these servers.

This pattern describes how you can use Session Manager and Amazon EC2 Instance Connect to securely connect to an Amazon Elastic Compute Cloud (Amazon EC2) bastion host deployed in your AWS account. Session Manager is a capability of AWS Systems Manager. The benefits of this pattern include:

• The deployed bastion host doesn’t have any open, inbound ports exposed to the public internet. This reduces the potential attack surface.

• You don’t need to store and maintain long-term Secure Shell (SSH) keys in your AWS account. Instead, each user generates a new SSH key pair each time they connect to the bastion host. AWS Identity and Access Management (IAM) policies that are attached to the user’s AWS credentials control access to the bastion host.

Intended audience

This pattern is intended for readers who have experience with basic understanding of Amazon EC2, Amazon Virtual Private Cloud (VPC), and Hashicorp Terraform.

Prerequisites and limitations

Prerequisites

• An active AWS account

• AWS Command Line Interface (AWS CLI) version 2, installed and configured

• Session Manager plugin for the AWS CLI, installed

• Terraform CLI, installed

• Storage for the Terraform state, such as an Amazon Simple Storage Service (Amazon S3) bucket and an Amazon DynamoDB table that serve as a remote backend to store the Terraform state. For more information on using remote backends for the Terraform state, see S3 Backends (Terraform documentation). For a code sample that sets up remote state management with an S3 backend, see remote-state-s3-backend (Terraform Registry). Note the following requirements:

  • The S3 bucket and DynamoDB table must be in the same AWS Region.

  • When creating the DynamoDB table, the partition key must be LockID (case-sensitive), and the partition key type must be String. All other table settings must be at their default values. For more information, see About primary keys and Create a table in the DynamoDB documentation.

• An SSH client, installed

Limitations

• This pattern is intended as a proof of concept (PoC) or as a basis for further development. It should not be used in its current form in production environments. Before deployment, adjust the sample code in the repository to meet your requirements and use case.

• This pattern assumes that the target bastion host uses Amazon Linux 2 as its operating system. While it is possible to use other Amazon Machine Images (AMIs), other operating systems are out of scope for this pattern.

  Note

  Amazon Linux 2 is nearing end of support. For more information, see the Amazon Linux 2 FAQs.

• In this pattern, the bastion host is located in a private subnet without an NAT gateway and internet gateway. This design isolates the EC2 instance from the public internet. You can add a specific network configuration that allows it to communicate with the internet. For more information, see Connect your virtual private cloud (VPC) to other networks in the Amazon VPC documentation. Similarly, following the principle of least privilege, the bastion host doesn’t have access to any other resources in your AWS account unless you explicitly grant permissions. For more information, see Resource-based policies in the IAM documentation.

Product versions

• AWS CLI version 2

• Terraform version 1.3.9

Architecture

Target technology stack

• A VPC with a single private subnet

• The following interface VPC endpoints:

  • amazonaws.<region>.ssm – The endpoint for the Systems Manager service.

  • amazonaws.<region>.ec2messages – Systems Manager uses this endpoint to make calls from SSM Agent to the Systems Manager service.

  • amazonaws.<region>.ssmmessages – Session Manager uses this endpoint to connect to your EC2 instance through a secure data channel.

• A t3.nano EC2 instance running Amazon Linux 2

• IAM role and instance profile

• Amazon VPC security groups and security group rules for the endpoints and EC2 instance

Target architecture

The diagram shows the following process:

1. The user assumes an IAM role that has permissions to do the following:

   • Authenticate, authorize, and connect to the EC2 instance

   • Start a session with Session Manager

2. The user initiates an SSH session through Session Manager.

3. Session Manager authenticates the user, verifies the permissions in the associated IAM policies, checks the configuration settings, and sends a message to SSM Agent to open a two-way connection.

4. The user pushes the SSH public key to the bastion host through Amazon EC2 metadata. This must be done before each connection. The SSH public key remains available for 60 seconds.

5. The bastion host communicates with the interface VPC endpoints for Systems Manager and Amazon EC2.

6. The user accesses the bastion host through Session Manager by using a TLS 1.2 encrypted bidirectional communication channel.

Automation and scale

The following options are available to automate deployment or to scale this architecture:

• You can deploy the architecture through a continuous integration and continuous delivery (CI/CD) pipeline.

• You can modify the code to change the instance type of the bastion host.

• You can modify the code to deploy multiple bastion hosts. In the bastion-host/main.tf file, in the aws_instance resource block, add the count meta-argument. For more information, see the Terraform documentation.

Tools

AWS services

• AWS Command Line Interface (AWS CLI) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.

• Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.

• AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.

• AWS Systems Manager helps you manage your applications and infrastructure running in the AWS Cloud. It simplifies application and resource management, shortens the time to detect and resolve operational problems, and helps you manage your AWS resources securely at scale. This pattern uses Session Manger, a capability of Systems Manager.

• Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

Other tools

• HashiCorp Terraform is an open-source infrastructure as code (IaC) tool that helps you use code to provision and manage cloud infrastructure and resources. This pattern uses Terraform CLI.

Code repository

The code for this pattern is available in the GitHub Access a bastion host by using Session Manager and Amazon EC2 Instance Connect repository.

Best practices

• We recommend using automated code-scanning tools to improve the security and quality of the code. This pattern was scanned by using Checkov, a static code-analysis tool for IaC. At a minimum, we recommend that you perform basic validation and formatting checks by using the terraform validate and terraform fmt -check -recursive Terraform commands.

• It’s a good practice to add automated tests for IaC. For more information about the different approaches for testing Terraform code, see Testing HashiCorp Terraform (Terraform blog post).

• During deployment, Terraform uses the replaces the EC2 instance each time a new version of the Amazon Linux 2 AMI is detected. This deploys the new version of the operating system, including patches and upgrades. If the deployment schedule is infrequent, this can pose a security risk because the instance doesn’t have the latest patches. It is important to frequently update and apply security patches to deployed EC2 instances. For more information, see Update management in Amazon EC2.

• Because this pattern is a proof of concept, it uses AWS managed policies, such as AmazonSSMManagedInstanceCore. AWS managed policies cover common use cases but don't grant least-privilege permissions. As needed for your use case, we recommend that you create custom policies that grant least-privilege permissions for the resources deployed in this architecture. For more information, see Get started with AWS managed policies and move toward least-privilege permissions.

• Use a password to protect access to SSH keys and store keys in a secure location.

• Set up logging and monitoring for the bastion host. Logging and monitoring are important parts of maintaining systems, from both an operational and security perspective. There are multiple ways to monitor connections and activity in your bastion host. For more information, see the following topics in the Systems Manager documentation:

  • Monitoring AWS Systems Manager

  • Logging and monitoring in AWS Systems Manager

  • Auditing session activity

  • Logging session activity

Epics

Deploy the resources

Task	Description	Skills required
Clone the code repository.	In a command-line interface, change your working directory to the location where you want to store the sample files. Enter the following command. git clone https://github.com/aws-samples/secured-bastion-host-terraform.git	DevOps engineer, Developer
Initialize the Terraform working directory.	This step is necessary for only the first deployment. If you are redeploying the pattern, skip to the next step. In the root directory of the cloned repository, enter the following command, where: $S3_STATE_BUCKET is the name of S3 bucket that contains the Terraform state $PATH_TO_STATE_FILE is the key to the Terraform state file, such as infra/bastion-host/tetfstate $AWS_REGION is the Region where the S3 bucket is deployed terraform init \ -backend-config="bucket=$S3_STATE_BUCKET" \ -backend-config="key=$PATH_TO_STATE_FILE" \ -backend-config="region=$AWS_REGION Note Alternatively, you can open the config.tf file and, in the terraform section, manually provide these values.	DevOps engineer, Developer, Terraform
Deploy the resources.	In the root directory of the cloned repository, enter the following command. terraform apply -var-file="dev.tfvars" Review the list of all changes that will be applied to your AWS account, and then confirm the deployment. Wait until all resources are deployed.	DevOps engineer, Developer, Terraform

Set up the local environment

Task	Description	Skills required
Configure the SSH connection.	Update the SSH configuration file to allow SSH connections through Session Manager. For instructions, see Allowing SSH connections for Session Manager. This allows authorized users to enter a proxy command that starts a Session Manager session and transfers all data through a two-way connection.	DevOps engineer
Generate the SSH keys.	Enter the following command to generate a local private and public SSH key pair. You use this key pair to connect to the bastion host. ssh-keygen -t rsa -f my_key	DevOps engineer, Developer

Connect to the bastion host by using Session Manager

Task	Description	Skills required
Get the instance ID.	In order to connect to the deployed bastion host, you need the ID of the EC2 instance. Do one of the following to locate the ID: Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/. In the navigation pane, choose Instances. Locate the bastion host instance. In the AWS CLI, enter the following command. aws ec2 describe-instances To filter the results, enter the following command, where $BASTION_HOST_TAG is the tag you assigned to the bastion host. The default value for this tag is sandbox-dev-bastion-host. aws ec2 describe-instances \ --filters "Name=tag:Name,Values=$BASTION_HOST_TAG" \ --output text \ --query 'Reservations[*].Instances[*].InstanceId' \ --output text Copy the ID of the EC2 instance. You use this ID later.	General AWS
Send the SSH public key.	Note In this section, you upload the public key to the instance metadata of the bastion host. After the key is uploaded, you have 60 seconds to start a connection with the bastion host. After 60 seconds, the public key is removed. For more information, see the Troubleshooting section of this pattern. Complete the next steps quickly to prevent the key from being removed before you connect to the bastion host. Send the SSH key to the bastion host by using EC2 Instance Connect. Enter the following command, where: $INSTANCE_ID is the ID of the EC2 instance $PUBLIC_KEY_FILE is the path to your public key file, such as my_key.pub Important Be sure to use the public key and not the private key. aws ec2-instance-connect send-ssh-public-key \ --instance-id $INSTANCE_ID \ --instance-os-user ec2-user \ --ssh-public-key file://$PUBLIC_KEY_FILE Wait until you receive a message indicating that the key has been successfully uploaded. Continue to the next step immediately.	General AWS
Connect to the bastion host.	Enter the following command to connect to the bastion host through Session Manager, where: $PRIVATE_KEY_FILE is the path to your private key, such as my_key $INSTANCE_ID is the ID of the EC2 instance ssh -i $PRIVATE_KEY_FILE ec2-user@$INSTANCE_ID Confirm the connection by entering yes. This opens an SSH connection by using Session Manager. Note There are other options for opening an SSH connection with the bastion host. For more information, see Alternative approaches to establish an SSH connection with the bastion host in the Additional information section of this pattern.	General AWS

(Optional) Clean up

Task	Description	Skills required
Remove the deployed resources.	In order to remove all deployed resources, run the following command from the root directory of the cloned repository. terraform destroy -var-file="dev.tfvars" Confirm the removal of the resources.	DevOps engineer, Developer, Terraform

Troubleshooting

Issue	Solution
TargetNotConnected error when trying to connect to the bastion host	Reboot the bastion host according to the instructions in Reboot your instance in the Amazon EC2 documentation. After the instance has successfully rebooted, resend the public key to the bastion host and reattempt the connection.
Permission denied error when trying to connect to the bastion host	After the public key is uploaded to the bastion host, you have only 60 seconds to start the connection. After 60 seconds, the key is automatically removed, and you can’t use it to connect to the instance. If this occurs, you can repeat the step to resend the key to the instance.

Related resources

AWS documentation

• AWS Systems Manager Session Manager (Systems Manager documentation)

• Install the Session Manager plugin for the AWS CLI (Systems Manager documentation)

• Allowing SSH connections for Session Manager (Systems Manager documentation)

• About using EC2 Instance Connect (Amazon EC2 documentation)

• Connect using EC2 Instance Connect (Amazon EC2 documentation)

• Identity and access management for Amazon EC2 (Amazon EC2 documentation)

• Using an IAM role to grant permissions to applications running on Amazon EC2 instances (IAM documentation)

• Security best practices in IAM (IAM documentation)

• Control traffic to resources using security groups (Amazon VPC documentation)

Other resources

• Terraform Developer webpage

• Command: validate (Terraform documentation)

• Command: fmt (Terraform documentation)

• Testing HashiCorp Terraform (HashiCorp blog post)

• Checkov webpage

Additional information

Alternative approaches to establish an SSH connection with the bastion host

Port forwarding

You can use the -D 8888 option to open an SSH connection with dynamic port forwarding. For more information, see these instructions at explainshell.com. The following is an example of a command to open an SSH connection by using port forwarding.

ssh -i $PRIVATE_KEY_FILE -D 8888 ec2-user@$INSTANCE_ID

This is kind of connection opens a SOCKS proxy that can forward traffic from your local browser through the bastion host. If you are using Linux or MacOS, to see all options, enter man ssh. This displays the SSH reference manual.

Using the provided script

Instead of manually running the steps described in Connect to the bastion host by using Session Manager in the Epics section, you can use the connect.sh script included in the code repository. This script generates the SSH key pair, pushes the public key to the EC2 instance, and initiates a connection with the bastion host. When you run the script, you pass the tag and key name as arguments. The following is an example of the command to run the script.

./connect.sh sandbox-dev-bastion-host my_key