Building a Scalable and Secure Multi-VPC AWS Network Infrastructure - Building a Scalable and Secure Multi-VPC AWS Network Infrastructure

Building a Scalable and Secure Multi-VPC AWS Network Infrastructure

Publication date: April 17, 2024 (Document history)

Amazon Web Services (AWS) customers often rely on hundreds of accounts and virtual private clouds (VPCs) to segment their workloads and expand their footprint.This level of scale often creates challenges around resource sharing, inter-VPC connectivity, and on-premises facilities to VPC connectivity.

This whitepaper describes best practices for creating scalable and secure network architectures in a large network using AWS services such as Amazon Virtual Private Cloud (Amazon VPC), AWS Transit Gateway, AWS PrivateLink, AWS Direct Connect, Gateway Load Balancer, AWS Network Firewall, and Amazon Route 53. It demonstrates solutions for managing growing infrastructure—ensuring scalability, high availability, and security while keeping overhead costs low.

Introduction

AWS customers begin by building resources in a single AWS account that represents a management boundary which segments permissions, costs, and services. However, as the customer’s organization grows, greater segmentation of services becomes necessary to monitor costs, control access, and provide easier environmental management. A multi-account solution solves these issues by providing specific accounts for IT services and users within an organization. AWS provides several tools to manage and configure this infrastructure, including AWS Control Tower

A diagram depicting AWS Control Tower initial deployment

AWS Control Tower initial deployment

When you set up your multi-account environment using AWS Control Tower, it creates two Organizational Units (OUs):

  • Security OU – Within this OU, AWS Control Tower creates two accounts:

  • Log Archive

  • Audit (This account corresponds to the security Tooling account discussed previously in the guidance.)

  • Sandbox OU – This OU is the default destination for accounts created within AWS Control Tower. It contains accounts in which your builders can explore and experiment with AWS services, and other tools and services, subject to your team’s acceptable use policies.

AWS Control Tower allows you to create, register, and manage additional OUs to expand the initial environment to implement the guidance.

The following diagram shows the OUs initially deployed by AWS Control Tower. You can expand your AWS environment to implement any of the recommended OUs included in the diagram, to meet your requirements.

A diagram depicting AWS organizational OUs.

AWS organizational OUs

For further details on multi-account environment using AWS Control Tower, refer to Appendix E in the Organizing Your AWS Environment Using Multiple Accounts whitepaper.

Note

In this whitepaper, “Control Tower” is a broad term for the scalable, secure, and performant multi-account/multi-VPC setup where you deploy your workloads. This setup can be built using different tools. You can find more information about best practices, design principles and benefits of multi-account cloud foundation in the Organizing Your AWS Environment Using Multiple Accounts whitepaper.

Most customers begin with a few VPCs to deploy their infrastructure. The number of VPCs a customer creates is usually related to their number of accounts, users, and staged environments (production, development, test, and so on). As cloud usage grows, the number of users, business units, applications, and Regions that a customer interacts with also grow, leading to the creation of new VPCs.

As the number of VPCs grows, cross-VPC management becomes essential for the operation of the customer’s cloud network. This whitepaper covers best practices for three specific areas in cross-VPC and hybrid connectivity:

IP address planning and management

In order to build a scalable multi-account multi-VPC network design, IP address planning and management is imperative. A good IP addressing scheme needs to consider your current and future networking needs. Your IP address scheme IP needs to cover your on-prem workloads, your cloud workloads, and should also allow for future expansion (for example, addition of new AWS Regions, business units, and mergers or acquisitions). It should also prevent your teams from inadvertently creating overlapping IP CIDRs. If overlapping IP CIDR is desired such as for isolated or disconnected workloads, this decision needs to be conscious and should account for implications on routing, security and cost. You might also need to consider creating necessary approval processes for such exceptions. A good IP addressing scheme also helps with simplifying your network design and routing configuration.

Key considerations:

  • Plan your IP addressing scheme (both public and private IPs) up front and select an IP address management tool to allocate, manage, and track IP address usage across all of your workloads.

  • Use hierarchical and summarized IP addressing schemes.

  • Plan for consistent IP assignment based on environment, AWS Region, organization, or business unit.

  • Designate distinct IP CIDRs (both IPv4 and IPv6) for on-premises and cloud networks.

  • Proactively prevent and track overlapping IP CIDRs.

  • Size your IP CIDRs appropriately to enable scaling and future growth.

  • Enable your workloads for IPv6 or dual-stack compatibility to reduce IP conflicts and address IPv4 space depletion.

You can use Amazon VPC IP Address Manager (IPAM) to simplify planning, tracking, and monitoring both public and private IP addresses for your AWS workloads. IPAM allows you to organize, allocate, monitor, and share IP address space across multiple AWS Regions and AWS accounts. It also helps with automatic allocation of CIDRs to VPCs using specific business rules.

Refer to the Amazon VPC IP Address Manager Best Practices, Managing IP pools across VPCs and Regions using Amazon VPC IP Address Manager, and IP Address Management for AWS Control Tower blog posts to learn IP addressing best practices and how to use IPAM to manage IP pools across VPCs, AWS Regions, and AWS Control Tower.

Are you Well-Architected?

The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.

For more expert guidance and best practices for your cloud architecture—reference architecture deployments, diagrams, and whitepapers—refer to the AWS Architecture Center.