AWS Outposts rack network troubleshooting checklist

Use this checklist to help troubleshoot a service link that has a status of DOWN.

Connectivity with Outpost network devices

If your service link is down, check the BGP peering status on the customer local network devices that are connected to the Outpost network devices.

If the BGP peering status is DOWN, follow these steps:

1. Ping the remote peer IP on the Outpost network devices from the customer devices. You can find the peer IP address in the BGP configuration of your device. You can also refer to the Network readiness checklist provided to you at the time of installation.

2. If pinging is unsuccessful, check the physical connection and ensure that connectivity status is UP.

   1. Confirm the LACP status of the customer local network devices.

   2. Check the interface status on the device. If DOWN, complete steps c and d. If UP, skip to step 3.

   3. Check the customer local network devices and confirm that the optical module is working.

   4. Replace faulty fibers and ensure the lights (Tx/Rx) are within acceptable range.

3. If pinging is successful, check the customer local network devices and ensure that the following BGP configurations are correct by confirming:

   1. That the local Autonomous System Number (Customer ASN) is correctly configured.

   2. That the remote Autonomous System Number (Outpost ASN) is correctly configured.

   3. That the Interface IP and remote peer IP addresses are correctly configured.

   4. That the advertised and received routes are correct.

4. If your BGP session is flapping between active and connect states, verify that TCP port 179 and other relevant ephemeral ports are not blocked on the customer local network devices.

5. If you need to troubleshoot further, check the following items:

   1. BGP and TCP debugs on the customer local network devices

   2. BGP logs on the customer local network devices

   3. Packet capture for the customer local network devices

6. If the issue persists, perform MTR / traceroute / packet captures from your Outpost connected router to the Outpost network device peer IPs. Use the Enterprise support plan from the AWS Support console to share the test results with AWS Support.

If BGP peering status is UP between the customer local network devices and the Outpost network devices, but the service link is still DOWN, you can troubleshoot further by checking the following devices on your customer local network devices. Use one of the following checklists, depending on how your service link connectivity is provisioned.

• Edge routers connected with AWS Direct Connect – Public virtual interface in use for service link connectivity. For more information, see AWS Direct Connect public virtual interface connectivity to AWS Region.

• Edge routers connected with AWS Direct Connect – Private virtual interface in use for service link connectivity. For more information, see AWS Direct Connect private virtual interface connectivity to AWS Region.

• Edge routers connected with Internet Service Providers (ISPs) – Public internet in use for service link connectivity. For more information, see ISP public internet connectivity to AWS Region.

AWS Direct Connect public virtual interface connectivity to AWS Region

Use the following checklist to troubleshoot edge routers connected with AWS Direct Connect when a public virtual interface is in use for service link connectivity.

1. Confirm that the devices connecting directly with the Outpost network devices are receiving the service link /26 IP ranges through BGP.

   1. Confirm the routes that are being received through BGP from your device.

   2. Check the route table of the service link Virtual Routing and Forwarding instance (VRF). It should show that it is using the /26 IP range.

2. To ensure Region connectivity, check the route table for the service link VRF. It should include the AWS Public IP ranges or default route.

3. If you are not receiving the AWS Public IP ranges in the service link VRF, check the following items.

   1. Check the AWS Direct Connect link status from the edge router or AWS Management Console.

   2. If the physical link is UP, check the BGP peering status from the edge router.

   3. If the BGP peering status is DOWN, ping the peer AWS IP and check the BGP configuration in the edge router. For more information, see Troubleshooting AWS Direct Connect in the AWS Direct Connect User Guide and My virtual interface BGP status is down in the AWS console. What should I do?.

   4. If BGP is established and you are not seeing the default route or AWS public ranges in the VRF, contact AWS Support, using the Enterprise support plan.

4. If you have an on-premises firewall, check the following items.

   1. Confirm that the required ports for service link connectivity are allowed in the network firewalls. Use traceroute on port 443 or any other network troubleshooting tool to confirm the connectivity through the firewalls and your network devices. The following ports are required to be configured in the firewall policies for the service link connectivity.

      • TCP protocol – Source port: TCP 1025-65535, Destination port: 443.

      • UDP protocol – Source port: TCP 1025-65535, Destination port: 443.

   2. If the firewall is stateful, ensure that the outbound rules allow the Outpost’s service link IP range (/26 – provided by the customer) to the AWS Public IP ranges. For more information, see Outpost connectivity to AWS Regions.

   3. If the firewall is not stateful, make sure to allow the inbound flow also (from the AWS Public IP ranges to the service link /26 IPs).

   4. If you have configured a virtual router in the firewalls, ensure that the appropriate routing is configured for traffic to and from the Outpost and AWS Region.

5. If you have configured NAT in the on-premises network to translate the Outpost’s service link /26 IP ranges to your owned public IPs, check the following items.

   1. Confirm that the NAT device is not overloaded and has free ports to allocate for new session.

   2. Confirm that the NAT device is correctly configured to perform the address translation.

6. If the issue persists, perform MTR / traceroute / packet captures from your edge router to the AWS Direct Connect peer IPs. Use the Enterprise support plan from the AWS Support console to share the test results with AWS Support.

AWS Direct Connect private virtual interface connectivity to AWS Region

Use the following checklist to troubleshoot edge routers connected with AWS Direct Connect when a private virtual interface is in use for service link connectivity.

1. If connectivity between the Outpost rack and AWS Region is using the AWS Outposts private connectivity feature, check the following items.

   1. Ping the remote peering AWS IP from the edge router and confirm the BGP peering status.

   2. Ensure that BGP peering over the AWS Direct Connect private virtual interface between your service link endpoint VPC and the Outpost installed on your premises is UP. For more information, see Troubleshooting AWS Direct Connect in the AWS Direct Connect User Guide, My virtual interface BGP status is down in the AWS console. What should I do?, and How can I troubleshoot BGP connection issues over Direct Connect?.

   3. The AWS Direct Connect private virtual interface is a private connection to your edge router in your chosen AWS Direct Connect location and uses BGP to exchange routes. Your private Amazon VPC CIDR range is advertised through this BGP session to your edge router. Similarly, the /26 IP address range for the Outpost service link is advertised to the region through BGP from your edge router.

   4. Confirm that the Network ACLs associated with the service link private endpoint in your VPC allow the relevant traffic. For more information, see Network readiness checklist.

   5. If you have an on-premises firewall, ensure that the firewall has outbound rules allowing the service link IP ranges and the Outpost service endpoints (the network interface IP addresses) located in the VPC or the VPC CIDR. Ensure that the TCP 1025-65535 and UDP 443 ports are not blocked. For more information, see Introducing AWS Outposts private connectivity.

   6. If the firewall is not stateful, ensure that the firewall has rules and policies to allow inbound traffic to the Outpost from the Outpost service endpoints in the VPC.

2. If you have more than 100 networks in your on-premises network, you can advertise a default route over the BGP session to AWS on your private virtual interface. If you don't want to advertise a default route, summarize the routes so that the number of advertised routes is less than 100.

3. If the issue persists, perform MTR / traceroute / packet captures from your edge router to the AWS Direct Connect peer IPs. Use the Enterprise support plan from AWS Support to share the test results with AWS Support.

ISP public internet connectivity to AWS Region

Use the following checklist to troubleshoot edge routers connected with an ISP when public internet is in use for service link connectivity.

• Confirm that the internet link is up.

• Confirm that the public servers are accessible from your edge devices connecting with the ISP.

If the internet or public servers are not accessible through the ISP links, complete the following steps.

1. Confirm whether BGP peering status with ISP routers is established.

   1. Confirm that the BGP is not flapping.

   2. Confirm that the BGP is receiving and advertising the required routes from the ISP.

2. In case of static route configuration, check that the default route is properly configured on the edge device.

3. Confirm whether you can reach the internet using another ISP connection.

4. If the issue persists, perform MTR / traceroute / packet captures on your edge router. Share the results with your ISP's technical support team for further troubleshooting.

If the internet and public servers are accessible via the ISP links, complete the following steps.

1. Confirm whether any of your publicly accessible EC2 instances or load balancers in the Outpost home Region are accessible from your edge device. You can ping or use telnet to confirm the connectivity, and use traceroute to confirm the network path.

2. If you use VRFs to separate traffic in your network, confirm that the service link VRF has routes or policies directing traffic to and from the ISP (internet) and VRF. See the following checkpoints.

   1. Edge routers connecting with the ISP. Check the edge router’s ISP VRF route table to confirm that the service link /26 IP range is present.

   2. Customer local network devices connecting with the Outpost. Check the configurations of the VRFs and ensure that the routing and policies required for routing between the service link VRF and the ISP VRF are configured properly. Usually, a default route is sent from the ISP VRF into the service link VRF for traffic to the internet.

   3. If you configured source-based routing in the routers connected to your Outpost, confirm that the configuration is correct.

3. Ensure that the on-premises firewalls are configured to allow outbound connectivity (TCP 1025-65535 and UDP 443 ports) from the Outpost service link IP ranges to the public AWS IP ranges. If the firewalls are not stateful, ensure that inbound connectivity to the Outpost is also configured.

4. Ensure that NAT is configured in the on-premises network to translate the Outpost’s service link /26 IP ranges to Public IPs. In addition, confirm the following items.

   1. The NAT device is not overloaded and has free ports to allocate for new sessions.

   2. The NAT device is correctly configured to perform the address translation.

If the issue persists, perform MTR / traceroute / packet captures.

• If the traceroute results show that packets are dropping or blocked at the on-premises network, check with your network or technical team for further guidance.

• If the traceroute results show that the packets are dropping or blocked at the ISP’s network, contact the ISP’s technical support team.

• If the traceroute results do not show any issues, collect the results from all tests (such as MTR, telnet, traceroute, packet captures, and BGP logs) and use the Enterprise support plan from the AWS Support console to contact AWS Support.