AWS Direct Connect  - Building a Scalable and Secure Multi-VPC AWS Network Infrastructure

AWS Direct Connect 

While VPN over internet is a great option to get started, internet connectivity may not be reliable for production traffic. Because of this unreliability, many customers choose AWS Direct Connect. AWS Direct Connect is a networking service that provides an alternative to using the internet to connect to AWS. Using AWS Direct Connect, data that would have previously been transported over the internet is delivered through a private network connection between your facilities and AWS. In many circumstances, private network connections can reduce costs, increase bandwidth, and provide a more consistent network experience than internet-based connections. There are five ways to use AWS Direct Connect to connect to VPCs:

        A diagram depicting ways to connect your on-premises data centers using AWS Direct Connect

Ways to connect your on-premises data centers using AWS Direct Connect

  1. Create a private virtual interface (VIF) to a VGW attached to a VPC — You can create 50 VIFs per Direct Connect connection, allowing you to connect to a maximum of 50 VPCs (one VIF provides connectivity to one VPC). There is one BGP peering per VPC. Connectivity in this setup is restricted to the AWS Region that the Direct Connect location is homed to. The one-to-one mapping of VIF to VPC (and lack of global access) makes this the least preferred way to access VPCs in the Landing Zone.

  2. Create a private VIF to a Direct Connect gateway associated with multiple VGWs (each VGW is attached to a VPC) — A Direct Connect gateway is a globally available resource. You can create the Direct Connect gateway in any Region and access it from all other Regions (except China). A Direct Connect Gateway can connect to up to 10 VPCs (via VGWs) globally in any AWS account over a single private VIF. This is a great option if a Landing Zone consists of a small number of VPCs (ten or fewer VPCs) and/or you need global access. There is one BGP peering session per Direct Connect Gateway per Direct Connect connection. Direct Connect gateway is only for north/south traffic flow and does not permit VPC-to-VPC connectivity. Refer to Virtual private gateway associations in the AWS Direct Connect documentation for more details. With this option, the connectivity is not restricted to the AWS Region where the Direct Connect location is homed to.

  3. Create a transit VIF to a Direct Connect gateway associated with Transit Gateway – You can associate a Transit Gateway instance to a Direct Connect gateway via a Transit VIF, which is only available over dedicated or hosted connections with speeds of one Gbps or greater. This option allows you to connect your on-premises data center to up to three Transit Gateway instances per Direct Connect Gateway (which can connect to thousands of VPCs) across different AWS Regions and AWS accounts over single transit VIF and BGP peering. This is the simplest setup among the four options for connecting multiple VPCs at scale, but you should be mindful of the Transit Gateway quotas. One key limit to note is that you can advertise only 20 prefixes from a Transit Gateway to on-premises router over the transit VIF. With the previous two options, you pay for Direct Connect pricing. For this option, you also pay for the Transit Gateway attachment and data processing charges. For more information, refer to the Transit Gateway Associations on Direct Connect documentation.

  4. Create a VPN connection to Transit Gateway over Direct Connect public VIF – A public VIF allows you to access all AWS public services and endpoints using the public IP addresses. When you create a VPN attachment on a Transit Gateway, you get two public IP addresses for VPN endpoints at the AWS side. These public IPs are reachable over the public VIF. You can create as many VPN connections to as many Transit Gateway instances as you want over Public VIF. When you create a BGP peering over the public VIF, AWS advertises the entire AWS public IP range to your router. To ensure that you only permit certain traffic (for example, allowing traffic only to the VPN termination endpoints) you are advised to use a firewall on-premises facilities. This option can be used to encrypt your Direct Connect at the network layer.

  5. Create GRE tunnels to Transit Gateway over a transit VIF – The Transit Gateway Connect attachment type supports GRE. With Transit Gateway Connect, SD-WAN infrastructure can be natively connected to AWS without having to set up IPsec VPNs between SD-WAN network virtual appliances and Transit Gateway. The GRE tunnels can be established over a transit VIF, having Transit Gateway Connect as the attachment type, providing higher bandwidth performance compared to a VPN connection. For more information, refer to the Simplify SD-WAN connectivity with AWS Transit Gateway Connect blog post.

While the “transit VIF to Direct Connect gateway” option might appear to be the best option because it lets you consolidate all your on-premises connectivity for a given AWS Region at a single point, (Transit Gateway) using a single BGP session per Direct Connect connection. Given some of the limits and considerations around this option, AWS expects customers to use both the “Create a private VIF” option and the “Transit VIF to Direct Connect gateway” option for their Landing Zone connectivity requirements. The following figure illustrates a sample setup where Transit VIF is used as a default method for connecting to VPCs, and a private VIF is used for an edge use case where huge amount of data must be transferred from an on-premises Data Center to the media VPC. Private VIF is used to avoid Transit Gateway data processing charges. As a best practice, you should have at least two connections at two different Direct Connect locations for maximum redundancy—a total of four connections. You create one VIF per connection for a total of four private VIFs and four transit VIFs. You can also create a VPN as a backup connectivity to AWS Direct Connect connections.

With the “Create GRE tunnels to Transit Gateway over a transit VIF” option, you get the capability to natively connect your SD-WAN infrastructure with AWS. It eliminates the need to setup IPsec VPNs between SD-WAN network virtual appliances and Transit Gateway.

        A diagram depicting a sample reference architecture for hybrid connectivity

Sample reference architecture for hybrid connectivity

Use the Network Services account for creating Direct Connect resources enabling demarcation of network administrative boundaries. Direct Connect connection, Direct Connect gateway and Transit Gateway can all reside in a Network Services account. To share the AWS Direct Connect connectivity with your Landing Zone, simply share the Transit Gateway through AWS RAM with other accounts.

MACsec security on Direct Connect connections

Customers can use MAC Security Standard (MACsec) encryption (IEEE 802.1AE) with their Direct Connect connections for 10 Gbps and 100 Gbps dedicated connections at select locations. With this capability, customers can secure their data on the layer 2 level, and Direct Connect delivers point-to-point encryption as well. To enable the Direct Connect MACsec feature, ensure that the MACsec pre-requisites are met. Because MACsec protects links on a hop-by-hop basis, you must have a dedicated connection that is transparent to layer 2 traffic and therefore MACsec compatible. Your device must have a direct layer 2 adjacency with our Direct Connect device. Your last-mile provider can help you verify that your connection will work with MACsec. For more information refer to Adding MACsec security to AWS Direct Connect connections.

AWS Direct Connect resiliency recommendations

With AWS Direct Connect, customers can achieve highly resilient connectivity into their Amazon VPCs and AWS resources from their on-premises networks. It is best practice that customers connect from multiple data centers, to eliminate any single point physical location failures. It is also recommended that, depending on the type of workloads, customers utilize more than one Direct Connect circuit for redundancy.

AWS also offers the AWS Direct Connect Resiliency Toolkit, which provides customers with a connection wizard with multiple redundancy models; to help them determine which model works best for their service level agreement (SLA) requirements and design their hybrid connectivity using Direct Connect connections accordingly. For more information, refer to AWS Direct Connect Resiliency Recommendations.