# Amazon Elastic VMware Service User Guide

-----
*****Copyright &copy; Amazon Web Services, Inc. and/or its affiliates. All rights reserved.*****

-----
Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon.

-----
## Contents
+ [What is Amazon EVS?](#what-is-evs)
   + [Get started with Amazon Elastic VMware Service](#getting-started)
   + [Differences between deployment modes](#deployment-mode-differences)
   + [Concepts and components of Amazon EVS](#concepts)
   + [Amazon EVS architecture](#architecture)
+ [Plan](#plan-overview)
   + [Choose versions, hosts, and size](#plan-hosts)
      + [Choose VCF and ESX versions](#plan-versions)
      + [Choose instance types](#plan-instance-types)
      + [Size your environment](#plan-sizing)
      + [Reserve capacity for hosts](#plan-capacity)
   + [Choose storage](#plan-storage)
      + [External storage options for diskless hosts](#external-storage-diskless)
   + [Plan your network](#plan-network)
      + [Plan your VPC and VLAN subnets](#plan-network-sizing)
      + [Supported network configurations](#plan-network-supported)
      + [Plan NSX Edge and VPC Route Server design](#plan-nsx-design)
      + [Plan HCX connectivity](#plan-hcx-connectivity)
   + [Plan costs and licenses](#plan-cost)
      + [VCF licensing](#vcf-license-mgmt)
   + [Plan security, support, and recovery](#plan-security-support)
      + [Shared responsibility and support](#plan-shared-responsibility)
      + [Account permissions and guardrails](#plan-account-guardrails)
      + [Plan for resilience and recovery](#plan-resilience)
   + [Deployment prerequisite checklist](#evs-deployment-prereq-checklist)
+ [Deploy](#deploy-overview)
   + [Automate your deployment](#deploy-automate)
      + [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo)
      + [Create Amazon EVS resources with AWS CloudFormation](#creating-resources-with-cloudformation)
   + [Prepare your AWS account](#setting-up)
      + [Create an IAM role for Amazon EVS administrators](#setup-iam-role)
   + [Build the network foundation](#network-foundation)
      + [Create a VPC with subnets and route tables](#network-create-vpc)
      + [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp)
      + [Set up VPC Route Server and on-premises connectivity](#network-route-server)
      + [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong)
      + [Associate VLAN subnets with a route table](#evs-env-config-custom-rt)
      + [Clean up network resources](#cleanup)
   + [Deploy in Self-deployed mode](#getting-started-self-deployed)
      + [Step 1: Create the environment](#self-deployed-create-env)
      + [Step 2: Create DNS records](#self-deployed-host-dns-records)
      + [Step 3: Add hosts](#self-deployed-add-hosts)
      + [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf)
      + [Step 4b (local storage): Run VCF bringup](#sd-install-bringup)
      + [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless)
         + [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore)
      + [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec)
      + [Step 5: Configure NSX networking](#self-deployed-nsx-edge)
      + [Step 6: Create connectors](#self-deployed-create-connectors)
      + [Step 7: Verify the environment](#self-deployed-verify)
      + [Clean up a Self-deployed environment](#self-deployed-cleanup)
   + [Deploy in EVS-deployed mode](#evs-deployed-overview)
      + [Create an environment](#getting-started-create-env)
      + [Verify environment creation](#verify-env-creation)
      + [Retrieve VCF credentials and access appliances](#access-vcf)
      + [Delete an EVS-deployed environment](#getting-started-cleanup-env-hosts)
   + [Connect external storage](#external-storage-connect)
      + [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap)
      + [Configure FSx for ONTAP as an NFS datastore](#config-fsx-nfs-datastore)
      + [Configure FSx for ONTAP as an iSCSI datastore](#config-fsx-iscsi-datastore)
      + [Configure FSx for ONTAP as an NVMe/TCP datastore](#config-fsx-nvme-datastore)
+ [Operate](#operate-overview)
   + [Monitor your environment](#monitor-overview)
      + [Monitor environment status](#evs-env-status-check)
      + [Get notified about host events](#monitor-notifications)
      + [Monitor hosts and VCF](#monitor-observability)
      + [Log API calls with CloudTrail](#logging-using-cloudtrail)
   + [Manage hosts](#host-management)
      + [Host maintenance and retirement events](#evs-host-maintenance)
      + [Add a host](#host-add-overview)
      + [Add a host (VCF 9.x)](#host-add)
      + [Replace a failed host (Self-deployed)](#host-replace)
      + [Delete a host](#host-delete-overview)
      + [Delete a host (VCF 9.x)](#host-delete)
      + [Add a host (VCF 5.x)](#evs-env-create-host)
      + [Replace a host (EVS-deployed)](#host-replace-evs-deployed)
      + [Delete a host (VCF 5.x)](#evs-env-delete-host)
   + [Update and upgrade](#evs-lifecycle-mgmt)
      + [Use the Amazon EVS custom add-on depot](#addon-depot)
   + [Manage connectors and credentials](#connectors-overview)
      + [Create a connector](#evs-env-create-connector)
      + [Update a connector](#evs-env-update-connector)
      + [Delete a connector](#evs-env-delete-connector)
      + [Rotate vCenter connector credentials](#rotate-vcenter-credentials)
      + [Manage secrets](#evs-env-secret-rotation)
   + [Windows Server entitlements](#entitlements)
      + [Create Windows Server entitlements](#create-entitlements)
      + [Activate Windows Server on entitled VMs](#activate-windows-server)
      + [Delete Windows Server entitlements](#delete-entitlements)
   + [Migrate workloads with HCX](#migrate-evs-hcx)
      + [Prepare for HCX migration](#hcx-prepare)
      + [Configure HCX public internet connectivity](#evs-env-hcx-internet-access)
      + [Optimize and verify HCX migration](#hcx-optimize-verify)
   + [Use VCF features on Amazon EVS](#operate-vcf-features)
      + [Set up NSX Federation](#nsx-federation)
+ [Security in Amazon Elastic VMware Service](#security)
   + [Data protection for Amazon EVS](#data-protection)
   + [Identity and access management for Amazon Elastic VMware Service](#security-iam)
      + [How Amazon EVS works with IAM](#security_iam_service-with-iam)
      + [Amazon EVS identity-based policy examples](#security-iam-id-based-policy-examples)
      + [AWS managed policies for Amazon EVS](#security-iam-awsmanpol)
      + [Using service-linked roles for Amazon EVS](#using-service-linked-roles)
   + [Resilience for Amazon EVS](#disaster-recovery-resiliency)
   + [Compliance validation for Amazon EVS](#compliance-validation)
   + [Infrastructure security for Amazon EVS](#infrastructure-security)
+ [Troubleshooting](#troubleshooting)
   + [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed)
   + [Troubleshoot environment status checks](#troubleshoot-status-checks)
   + [Troubleshoot hosts](#troubleshoot-hosts)
   + [Troubleshoot networking](#troubleshoot-networking)
   + [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements)
   + [Troubleshoot Amazon EVS identity and access](#security-iam-troubleshoot)
   + [Get help from AWS Support and Broadcom](#get-help)
+ [Reference](#reference-overview)
   + [Supported VCF versions and instance types](#versions-provided)
   + [VCF feature support on Amazon EVS](#vcf-feature-support)
   + [Service quotas for Amazon EVS](#service-quotas-evs)
   + [AWS Regions for Amazon EVS](#regions-reference)
   + [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference)
   + [VCF Installer settings for Amazon EVS](#vcf-installer-settings)
   + [Use the AWS CLI, SDKs, and API with Amazon EVS](#evs-api-cli-reference)
+ [Document history for the Amazon Elastic VMware Service User Guide](#doc-history)

-----



# What is Amazon EVS?
<a name="what-is-evs"></a>

Amazon Elastic VMware Service (Amazon EVS) allows you to run VMware Cloud Foundation (VCF) workloads directly on AWS. Amazon EVS runs VCF on bare-metal Amazon EC2 instances inside your own Amazon Virtual Private Cloud (VPC). Your teams keep the same VCF tools, skills, and operating procedures that they use on premises. You don’t need to re-platform, refactor applications, or retrain staff, and you can bring your existing VCF subscription with its portability entitlement.

You can use Amazon EVS to accelerate cloud migrations, to extend on-premises environments for hybrid scaling, and for disaster recovery.

To start, see [Get started with Amazon Elastic VMware Service](#getting-started).

## Features of Amazon EVS
<a name="servicename-feature-overview"></a>

The following are key features of Amazon EVS:
+  **Run VCF in your own VPC.** VCF runs inside your own Amazon VPC, next to the other AWS resources that your applications use.
+  **Accelerate your migration** by keeping your existing skills, processes, and technology solutions.
+  **Keep full control** of your virtualization stack, including add-ons and third-party solutions.
+  **Extend with AWS services.** Expand your VMware-based workloads with direct access to a broad range of AWS services, such as managed databases, analytics, serverless, containers, and generative AI.
+  **Bring your own VCF subscription** and use its portability entitlement on AWS.
+  **Scale your environment.** Add hosts to your environment as your workloads grow.

## Who manages what
<a name="what-is-evs-responsibilities"></a>

 AWS provides the infrastructure, and you’re responsible for the virtualization platform and your workloads. This split follows the [AWS shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/). Responsibilities for each area are as follows.


| Area |  AWS  | You | 
| --- | --- | --- | 
| Infrastructure | The AWS networking and VPC connectivity that Amazon EVS provisions, the bare metal hosts and their firmware. | Your VPC design, route tables, network access control lists, and connectivity to your other networks. | 
| VCF software | Provides the Amazon Machine Images (AMIs) for the VMware ESX hypervisor. | The VCF components and the workloads that run on them. You upgrade and patch the VCF stack. | 
| VCF subscriptions | Not applicable. | Bring your own VCF subscription, and keep valid license keys in your VCF management appliance. | 
| Workloads | Not applicable. | Your virtual machines, their guest operating systems and your data. | 

You can also work with AWS Partners to manage and operate the environment for you. For the full model, see [Shared responsibility and support](#plan-shared-responsibility) and [Security in Amazon Elastic VMware Service](#security).

## How you work with Amazon EVS
<a name="get-started-evs"></a>

You work with Amazon EVS in three stages: Plan, Deploy, and Operate.

1.  [Plan](#plan-overview). Successfully deploying Amazon EVS starts with a clear plan. Plan your AWS resources, such as VPCs, Route Servers, and DNS, along with your VCF resources and Amazon EC2 instance types.

1.  [Deploy](#deploy-overview). To deploy, you prepare your AWS account and network, create an Amazon EVS environment, add hosts, and deploy VCF.

1.  [Operate](#operate-overview). Configure VCF in the vSphere Client to meet your needs, such as logins, policies, and monitoring. Connect your environment to your on-premises data center, and migrate your workloads ([Migrate workloads with HCX](#migrate-evs-hcx)). Monitor the environment, manage hosts and connectors, and apply updates.

## Tools
<a name="what-is-evs-tools"></a>

The [Solutions for Amazon EVS](https://github.com/aws/solutions-for-amazon-evs) repository on GitHub is where Amazon EVS provides tooling and solutions that you can use. The tooling for deployment is in the `Deploy` folder.

The EVS Deployment Orchestrator (EDO) is one of the tools in the repository, in the `Deploy` folder. You can use EDO to deploy VCF 9.x environments from one AWS CloudFormation stack. For more information, see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo).

## Accessing Amazon EVS
<a name="accessing-evs"></a>

You can create and manage Amazon EVS resources with the interfaces in the following table.


| Interface | What it is | More information | 
| --- | --- | --- | 
| Amazon EVS console | A web interface to create and manage Amazon EVS environments. | None | 
|  AWS CLI  | Commands to create and manage Amazon EVS resources, supported on Windows, macOS, and Linux. |  [AWS Command Line Interface](https://aws.amazon.com/cli)  | 
|  AWS CloudFormation | Templates that provision Amazon EVS resources, such as the resource type `AWS::EVS::Environment`. |  [Amazon Elastic VMware Service resource type reference](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/AWS_EVS.html) in the * AWS CloudFormation Template Reference*  | 

For the API, SDKs, and CLI reference, see [Use the AWS CLI, SDKs, and API with Amazon EVS](#evs-api-cli-reference).

## Pricing
<a name="what-is-evs-pricing"></a>

You pay for the Amazon EC2 bare metal instances that run your hosts and for the other AWS resources that you use. For prices, see [Amazon EVS pricing](https://aws.amazon.com/evs/pricing). To plan costs and VCF subscriptions, see [Plan costs and licenses](#plan-cost).

## Regions
<a name="what-is-evs-regions"></a>

For the AWS Regions where Amazon EVS is available, see [AWS Regions for Amazon EVS](#regions-reference).

**Topics**

## Get started with Amazon Elastic VMware Service
<a name="getting-started"></a>

Choose how you deploy VMware Cloud Foundation (VCF) on Amazon Elastic VMware Service (Amazon EVS), and find the pages for each stage of the journey.

The procedures in [Deploy in Self-deployed mode](#getting-started-self-deployed) deploy VCF 9.x. Pages in [Deploy in EVS-deployed mode](#evs-deployed-overview) cover VCF 5.2.x. A page with no line and outside those sections covers both.

### Choose your path
<a name="getting-started-choose-path"></a>

Find the row that matches what you want to do, and go to the page in the second column.


| If you want to | Go to | 
| --- | --- | 
| Deploy VCF 9.x in Self-deployed mode with one AWS CloudFormation stack |  [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo)  | 
| Deploy VCF 9.x in Self-deployed mode, step by step |  [Deploy in Self-deployed mode](#getting-started-self-deployed)  | 
| Use diskless hosts, for example r7i instance types (Self-deployed mode, VCF 9.x) |  [Choose storage](#plan-storage), then [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless)  | 
| Deploy VCF 5.2.x in EVS-deployed mode, where Amazon EVS installs VCF for you |  [Deploy in EVS-deployed mode](#evs-deployed-overview)  | 
| Run or change an existing environment |  [Operate](#operate-overview)  | 
| Compare the deployment modes first |  [Differences between deployment modes](#deployment-mode-differences)  | 
| Size, price, or plan first |  [Plan](#plan-overview)  | 
| Check whether an instance type, VCF version, or VCF feature is supported |  [Supported VCF versions and instance types](#versions-provided) and [VCF feature support on Amazon EVS](#vcf-feature-support)  | 

### From plan to operations
<a name="getting-started-journey"></a>

Every deployment follows the same three stages. Complete them in order.

1.  **Plan.** Choose the VCF version, instance types, storage and network design, and estimate costs and VCF subscriptions. Start at [Plan](#plan-overview), and use the [Deployment prerequisite checklist](#evs-deployment-prereq-checklist) to record your values.

1.  **Deploy.** Prepare your AWS account, then deploy the environment in the way that you chose.

   1.  [Prepare your AWS account](#setting-up) 

   1. For a manual deployment, build the network foundation ([Build the network foundation](#network-foundation)). If you use EDO, skip this step because EDO builds the network for you.

   1. For VCF 9.x in Self-deployed mode, use [Deploy in Self-deployed mode](#getting-started-self-deployed) or [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo). For VCF 5.2.x in EVS-deployed mode, use [Deploy in EVS-deployed mode](#evs-deployed-overview).

   1. (Optional) [Connect external storage](#external-storage-connect) 

1.  **Operate.** Monitor the environment, manage hosts, connectors and entitlements, apply updates, and migrate workloads. Start at [Operate](#operate-overview).

**Important**  
You incur AWS charges for the Amazon EC2 bare metal instances from the time you add hosts to an environment, even if you haven’t installed VCF on them. For prices, see [Amazon EVS pricing](https://aws.amazon.com/evs/pricing). For costs and VCF subscriptions, see [Plan costs and licenses](#plan-cost).

### Next steps
<a name="getting-started-next-steps"></a>

After your environment is running, migrate your workloads to Amazon EVS with VMware Hybrid Cloud Extension (HCX). For more information, see [Operate](#operate-overview) and [Migrate workloads with HCX](#migrate-evs-hcx).

## Differences between deployment modes
<a name="deployment-mode-differences"></a>

Amazon Elastic VMware Service (Amazon EVS) offers two deployment modes for VMware Cloud Foundation (VCF):
+ In **Self-deployed** mode, Amazon EVS creates the VLAN subnets when you create the environment. You add the hosts and install VCF yourself. You can deploy VCF 5.x or VCF 9.x in this mode.
+ In **EVS-deployed** mode, Amazon EVS creates the VLAN subnets and four hosts, and installs VCF 5.2.x for you. EVS-deployed mode doesn’t support VCF 9.x deployments. To automate a VCF 9.x deployment, use the [EVS Deployment Orchestrator (EDO)](https://github.com/aws/solutions-for-amazon-evs) in the Solutions for Amazon EVS repository on GitHub.

Choose Self-deployed mode to install VCF yourself, to deploy VCF 9.x, or to use diskless hosts. Choose EVS-deployed mode to have Amazon EVS install VCF 5.2.x for you.

You choose the mode when you create the environment. In the console, the version list includes a **Self-deployed** option. In the API, the values are `SELF_DEPLOYED`, `VCF-5.2.2`, and `VCF-5.2.1`.

### Comparison table
<a name="deployment-mode-comparison"></a>

Each row covers one topic. The second and third columns give the behavior in each mode. The value `Not available` means that the mode doesn’t support the item.


| Topic | Self-deployed | EVS-deployed | 
| --- | --- | --- | 
| Who installs VCF | You install VCF yourself. For VCF 9.x, you can use the VCF Installer, or the EVS Deployment Orchestrator (EDO) from the [Solutions for Amazon EVS](https://github.com/aws/solutions-for-amazon-evs) repository on GitHub. For instructions, see [Deploy in Self-deployed mode](#getting-started-self-deployed). | Amazon EVS installs VCF during environment creation. For instructions, see [Deploy in EVS-deployed mode](#evs-deployed-overview). | 
| VCF versions and API value | VCF 5.x or VCF 9.x. Amazon EVS provides VCF 9.0 and VCF 9.1 as the VCF 9.x versions. The API value is `SELF_DEPLOYED`. | VCF 5.2.2 (`VCF-5.2.2`) and VCF 5.2.1 (`VCF-5.2.1`). VCF 5.2.1 is restricted, and you request access through AWS Support. | 
| ESX version | You can choose an ESX version when you add a host. If you don’t choose one, Amazon EVS uses the default that `get-versions` reports as `defaultEsxVersion`. | Each VCF release has a default ESX version. For the values, see [Supported VCF versions and instance types](#versions-provided). | 
| VCF license keys | You don’t provide license keys when you create the environment. You add them in your VCF management appliance after you install VCF. For more information, see [VCF licensing](#vcf-license-mgmt). | You provide a VCF solution key and a vSAN license key when you create the environment. You must keep valid keys in SDDC Manager. | 
| Hosts at environment creation | None. Environment creation provisions the VLAN subnets only. You add hosts after the environment is created. | Four hosts. Amazon EVS environments in this mode require four hosts for initial deployment. | 
| Route server peers | You don’t select peers when you create the environment. You configure BGP peering between the NSX Tier-0 gateway and Amazon VPC Route Server after you install VCF. | You choose two route server peer IDs when you create the environment. The route server, its endpoints, and its peers must exist before you create the environment. | 
| NSX setup | You configure the NSX Edge cluster, the Tier-0 gateway, and routing to your VPC after VCF bringup. For instructions, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge). | Amazon EVS deploys an Active/Standby NSX Edge cluster with two NSX Edge nodes and configures the NSX routing and uplinks for you. | 
| Management connector | VCF Operations connector (`OPERATIONS_MANAGER`). You can also create a vCenter connector (`VCENTER`). | SDDC Manager connector (`SDDC_MANAGER`). You can also create a vCenter connector (`VCENTER`). | 
| Principal storage | vSAN on hosts with local NVMe instance store volumes, or external storage on diskless hosts. For the choice, see [Choose storage](#plan-storage). | vSAN on hosts with local NVMe instance store volumes. | 
| Diskless hosts (for example, r7i instance types) | Available with VCF 9.x. For instructions, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless). | Not available | 
| Supplemental storage with AWS services | Amazon FSx for NetApp ONTAP as an NFS, iSCSI, or NVMe/TCP datastore. For more information, see [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap). | Amazon FSx for NetApp ONTAP as an NFS, iSCSI, or NVMe/TCP datastore. For more information, see [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap). | 
| Tools in the Solutions for Amazon EVS repository |  [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo) (EDO) in the [Solutions for Amazon EVS GitHub repository](https://github.com/aws/solutions-for-amazon-evs). | Not available | 
| Who installs, upgrades, and patches VCF | You install, upgrade, and patch the VCF stack. | Amazon EVS does the initial VCF installation. You upgrade and patch the VCF stack. | 
| Support |  AWS Support covers Amazon EVS and its associated infrastructure services, including VCF. For more information, see [Get help from AWS Support and Broadcom](#get-help). |  AWS Support covers Amazon EVS and its associated infrastructure services, including VCF. For more information, see [Get help from AWS Support and Broadcom](#get-help). | 

In both modes, AWS manages the AWS resources that Amazon EVS provisions, the bare metal hosts and their firmware. For the full split of responsibilities, see [Shared responsibility and support](#plan-shared-responsibility).

### Related topics
<a name="deployment-mode-related"></a>

For more information about the modes, see the following topics.
+ For the VCF and ESX versions that each mode supports, see [Supported VCF versions and instance types](#versions-provided).
+ For the VCF features that you can use in each mode, see [VCF feature support on Amazon EVS](#vcf-feature-support).

## Concepts and components of Amazon EVS
<a name="concepts"></a>

Learn the Amazon Elastic VMware Service resources and VMware components before you plan an environment.

### Amazon EVS environment
<a name="concepts-evs-virt-env"></a>

An Amazon EVS *environment* is a logical container for VMware Cloud Foundation (VCF) resources, such as ESX hosts, VMware vSAN, VMware NSX, and VMware vCenter Server. When you create an environment, Amazon EVS creates the VLAN subnets in your VPC that VCF uses. Each Amazon EVS environment contains its own dedicated set of VLAN subnets. You then add hosts and connect the environment to your VCF management appliance with connectors.

In EVS-deployed environments (VCF 5.2.x), the environment contains a consolidated VCF domain, and each environment maps directly to an SDDC Manager appliance. For how the two deployment modes differ, see [Differences between deployment modes](#deployment-mode-differences). For the components and how they connect, see [Amazon EVS architecture](#architecture).

### Amazon EVS host
<a name="concepts-evs-host"></a>

An Amazon EVS *host* is a VMware ESX host that runs on an Amazon EC2 bare metal instance. Hosts with local storage use their local NVMe instance store volumes for vSAN datastores, which store your management and workload virtual machines. Diskless hosts, which need VCF 9.x, have no instance store and use external storage instead. For the storage choices, see [Choose storage](#plan-storage).

**Warning**  
Instance store volumes are ephemeral. Data on these volumes doesn’t persist if you stop or terminate the underlying Amazon EC2 instance. If you stop or terminate an Amazon EC2 instance that Amazon EVS uses before you decommission its host in VCF, you can lose data.  
For more information about host maintenance, see [Host maintenance and retirement events](#evs-host-maintenance).

### Service access subnet
<a name="concepts-service-access-subnet"></a>

The *service access subnet* is a standard VPC subnet that allows Amazon EVS to access the VCF deployment. When you create an Amazon EVS environment, you specify the VPC and the subnet for Amazon EVS to use for service access.

Amazon EVS provisions elastic network interfaces into the service access subnet to provide management connectivity to VCF appliances and ESX hosts. Amazon EVS needs this connectivity to deploy, manage, and monitor the VCF deployment.

### Amazon EVS VLAN subnet
<a name="concepts-evs-network"></a>

An *Amazon EVS VLAN subnet* is an Amazon VPC subnet that Amazon EVS manages. VLAN subnets provide VPC connectivity for Amazon EVS hosts and for VCF appliances such as NSX, VMware HCX, and vCenter Server. Each VLAN subnet has a VLAN tag, so that VLAN network traffic is segmented logically.

Amazon EVS creates all of the VLAN subnets that the service uses when you create the environment. You provide the CIDR blocks that the VLAN subnets use. Each CIDR block must be between a `/28` and a `/24` netmask. It must be inside a CIDR block that’s associated with the VPC, and it must not overlap with any other subnet in the VPC, including the other VLAN subnets.

Separate VLAN subnets carry host management, vMotion, vSAN, NSX overlay, NSX Edge, management appliance, HCX uplink, and NSX uplink traffic. Expansion VLAN subnets support other VCF functions, such as NSX Federation. For the list of VLAN subnet types, their VLAN IDs, and their IP address rules, see [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference).

You can’t add VLAN subnets or change their CIDR blocks after you create the environment. Size the VLAN subnet CIDR blocks for the number of hosts that you plan to run, including future growth, before you create the environment. For sizing guidance, see [Plan your VPC and VLAN subnets](#plan-network-sizing).

When Amazon EVS creates the VLAN subnets, it associates them with the main route table of your VPC. After deployment, you must explicitly associate each VLAN subnet with a route table. We recommend a custom route table. For the steps, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt). For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).

**Important**  
 Amazon EC2 security group rules are not enforced on Amazon EVS elastic network interfaces that are attached to VLAN subnets. To control traffic to and from VLAN subnets, you must use a network access control list. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

### VMware NSX
<a name="concepts-nsx"></a>

NSX is a software-defined networking platform that provides network virtualization. Amazon EVS environments use NSX to create and manage the overlay network where VCF appliances and workloads run. NSX Edge nodes connect the overlay network to your VPC through the NSX uplink VLAN subnet and Amazon VPC Route Server.

Who configures the NSX Edge cluster and NSX routing depends on the deployment mode. For the comparison, see [Differences between deployment modes](#deployment-mode-differences).

For more information about common NSX concepts, see [Key Concepts](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide/overview-of-nsx/key-concepts-nsxt.html) in the *VMware NSX Installation Guide* in the Broadcom documentation.

### Amazon EVS connectors
<a name="concepts-connector"></a>

An Amazon EVS *connector* lets Amazon EVS communicate with one VCF management appliance in your environment. Each connector uses the fully qualified domain name (FQDN) of the appliance and credentials that you store in an AWS Secrets Manager secret. Amazon EVS uses the management connector to stay in sync with your VCF deployment and to monitor vSAN capacity. It uses the optional vCenter connector to monitor virtual machines for Windows Server entitlements. For more information about the vSAN capacity notification, see [vSAN capacity warning received in AWS Health](#troubleshoot-vsan-capacity).

The management connector type depends on the deployment mode. Self-deployed environments use a VCF Operations connector (`OPERATIONS_MANAGER`). EVS-deployed environments use an SDDC Manager connector (`SDDC_MANAGER`). For connector types, reachability checks, and procedures, see [Manage connectors and credentials](#connectors-overview).

### Windows Server license entitlement
<a name="concepts-windows-server-license-entitlement"></a>

A *Windows Server license entitlement* lets virtual machines in your Amazon EVS environment use AWS-provided Windows Server licenses. An entitlement depends on a vCenter connector. For requirements and procedures, see [Windows Server entitlements](#entitlements).

### VMware HCX
<a name="concepts-hcx"></a>

VMware Hybrid Cloud Extension (HCX) is an application mobility platform for application migration, workload rebalancing, and disaster recovery across data centers and clouds. You can use HCX to migrate your VMware-based workloads to Amazon EVS. To choose how HCX connects to your on-premises site, see [Plan HCX connectivity](#plan-hcx-connectivity). To migrate workloads, see [Migrate workloads with HCX](#migrate-evs-hcx).

## Amazon EVS architecture
<a name="architecture"></a>

An Amazon Elastic VMware Service environment runs VMware Cloud Foundation (VCF) on Amazon EC2 bare metal instances inside your own Amazon VPC.

### VCF components
<a name="architecture-components"></a>

Amazon EVS implements a VCF consolidated architecture model. In this model, VCF management components and customer workloads run together on a consolidated domain. The environment is managed from a single vCenter Server, with vSphere resource pools that isolate management workloads from customer workloads.

The consolidated domain contains the following VCF management components:
+ ESX hosts
+ A vCenter Server instance
+ The VCF management appliance (SDDC Manager for VCF 5.2.x)
+ A vSAN datastore, on hosts with local storage
+ An NSX Manager cluster (three nodes in the standard topology, one node in a simple deployment)
+ A vSphere cluster
+ An NSX Edge cluster

The following diagram shows an example Amazon EVS architecture and how the components in the environment are connected. The area labeled **Consolidated domain** is the Amazon EVS environment with a consolidated domain architecture. The area labeled **Amazon EVS networks** is the underlying Amazon EVS network topology.

![NSX Edge nodes connect NSX overlay networks to VPC VLAN subnets through VPC Route Server in a consolidated VCF domain.](https://docs.aws.amazon.com/evs/latest/userguide/images/amazon-evs-architecture.png)


### Network topology
<a name="network-architecture"></a>

An Amazon EVS environment has two network layers, as follows.

 ** Amazon VPC underlay**   
The VPC and the Amazon EVS VLAN subnets that Amazon EVS creates in it form the underlay network for your VCF deployment. The underlay provides connectivity for NSX overlay networks, host management, vMotion, and vSAN. Amazon VPC Route Server provides dynamic routing between the underlay network and the overlay networks. For more information, see [Amazon EVS VLAN subnet](#concepts-evs-network).

 **VMware NSX overlay**   
The NSX overlay network carries the VCF management appliances and your workloads. You can configure additional NSX overlay networks to isolate workloads or applications from each other within your environment. For VCF 5.2.x, see [Overlay Design for VMware Cloud Foundation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vcf-design-5-2/nsx-t-design-for-the-management-domain/overlay-design-for-nsx-t-data-center-for-the-management-domain.html#GUID-025420AC-D915-439A-BF2A-D3C9F2FAB433-en) in the Broadcom documentation.

An Amazon EVS environment runs in one Availability Zone. All VPC subnets that Amazon EVS uses must be in that Availability Zone. For more information, see [VPC and subnets](#plan-network-supported-vpc).

#### Where workloads run
<a name="architecture-workload-placement"></a>

**Important**  
Deploy the guest virtual machines that run your workloads on NSX overlay networks. Amazon EVS VLAN subnets carry VCF underlay traffic only, and Amazon EVS doesn’t support guest virtual machines on them.

#### How the layers connect
<a name="architecture-edge-routing"></a>

An Active/Standby NSX Edge cluster with two NSX Edge nodes connects the two network layers. The NSX Edge nodes carry traffic between the overlay networks and the VPC. They also carry internet traffic, and traffic for private connectivity through Direct Connect or AWS Site-to-Site VPN with a transit gateway.

Who sets up the NSX Edge cluster and its routing depends on the deployment mode. For the comparison, see [Differences between deployment modes](#deployment-mode-differences).

#### NSX Edge cluster
<a name="evs-nsx-edge"></a>

The NSX Edge cluster has two NSX Edge nodes that run in Active/Standby mode. It provides the platform on which the Tier-0 and Tier-1 gateways run, along with IPsec VPN connections and their Border Gateway Protocol (BGP) routing.

For design guidance for the NSX Edge cluster and VPC Route Server, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design).

#### Tier-0 gateway
<a name="evs-t0-gateway"></a>

The Tier-0 gateway handles all north-south traffic between the logical and physical networks, and is created on the NSX overlay network. It runs on the NSX Edge cluster and peers with VPC Route Server over BGP.

Amazon EVS supports only one Tier-0 gateway for an Active/Standby NSX Edge cluster with two NSX Edge nodes. This Tier-0 gateway connects to and advertises all overlay networks that you configure for use with Amazon EVS.

#### Tier-1 gateway
<a name="evs-t1-gateway"></a>

The Tier-1 gateway handles east-west traffic between routed network segments within an environment, and is created on the NSX overlay network. The Tier-1 gateway has downlink connections to segments and uplink connections to the Tier-0 gateway. You can create and configure additional Tier-1 gateways if you need them.

#### Before you begin
<a name="evs-subnets"></a>

Before you create an environment, you need the following:
+ A VPC
+ A service access subnet
+ DNS and NTP settings
+ CIDR blocks for the VLAN subnets
+ A VPC Route Server instance

For the requirements, the sizes, and the settings that you can’t change after creation, see [Plan your VPC and VLAN subnets](#plan-network-sizing). For instructions, see [Build the network foundation](#network-foundation).

### Amazon EVS resources
<a name="evs-resources"></a>

Amazon EVS provisions the following AWS resources in the VPC that you allow Amazon EVS to access. You can see them in the AWS Management Console and the AWS CLI after they are created.
+ Amazon EVS elastic network interfaces that provide connectivity to your VCF appliances and hosts
+ Amazon EVS ESX hosts that run on Amazon EC2 bare metal instances (see [Amazon EVS host](#concepts-evs-host))
+ Amazon EVS VLAN subnets that connect your VPC to VCF appliances (see [Amazon EVS VLAN subnet](#concepts-evs-network))

**Important**  
Changing these resources outside of the Amazon EVS console and API can affect the availability and stability of your Amazon EVS environment.

# Plan
<a name="plan-overview"></a>

Some choices, such as the instance type and the storage model, decide which deployment path you follow. Work through the topics in this chapter in order, and record your values in the [deployment prerequisite checklist](#evs-deployment-prereq-checklist).

The procedures in [Deploy in Self-deployed mode](#getting-started-self-deployed) deploy VCF 9.x. Pages in [Deploy in EVS-deployed mode](#evs-deployed-overview) cover VCF 5.2.x. A page with no line and outside those sections covers both.

## Decisions to make before you deploy
<a name="plan-overview-decisions"></a>

The following table lists the decisions to make and the topic for each one.


| Decision | What you decide | Topic | 
| --- | --- | --- | 
| Versions and hosts | The VMware Cloud Foundation (VCF) version and deployment mode, the instance type, the number of hosts, and how you reserve capacity for them. |  [Choose versions, hosts, and size](#plan-hosts)  | 
| Storage | Whether your principal storage is vSAN ESA on local NVMe or external storage for diskless hosts, which need VCF 9.x, and what supplemental storage you add. |  [Choose storage](#plan-storage)  | 
| Network | The VPC CIDR block, the VLAN subnet sizes, the connections to your other networks, the NSX Edge design, and HCX connectivity. |  [Plan your network](#plan-network)  | 
| Costs and licenses | Which charges apply, how to lower the cost of hosts, and how you bring your VCF subscription. |  [Plan costs and licenses](#plan-cost)  | 
| Security, support, and recovery | Who manages each part of the environment, which account guardrails apply, and how you recover from a failure. |  [Plan security, support, and recovery](#plan-security-support)  | 

When you have made these decisions, see [Deploy](#deploy-overview) to choose a deployment path. To deploy VCF 9.x with the EVS Deployment Orchestrator (EDO), a tool in the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs), see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo).

**Topics**

## Choose versions, hosts, and size
<a name="plan-hosts"></a>

The hosts in an Amazon Elastic VMware Service environment are Amazon Elastic Compute Cloud (Amazon EC2) bare metal instances that run ESX. Make the host decisions in the following order, because each choice limits the next one. The VMware Cloud Foundation (VCF) version decides which instance types you can use, and the instance type affects how many hosts you need. The number of hosts then decides the capacity and the quotas that you request.

1.  [Choose VCF and ESX versions](#plan-versions). Choose the VCF version and deployment mode, and check which ESX builds are available.

1.  [Choose instance types](#plan-instance-types). Choose local NVMe hosts, or diskless hosts for VCF 9.x, and then an instance size.

1.  [Size your environment](#plan-sizing). Decide how many hosts you need, and which quotas to raise before you start.

1.  [Reserve capacity for hosts](#plan-capacity). Reserve EC2 capacity for your hosts, including a spare host.

**Topics**
+ [Choose VCF and ESX versions](#plan-versions)
+ [Choose instance types](#plan-instance-types)
+ [Size your environment](#plan-sizing)
+ [Reserve capacity for hosts](#plan-capacity)

### Choose VCF and ESX versions
<a name="plan-versions"></a>

When you create an Amazon Elastic VMware Service environment, the value that you choose sets the deployment mode and, in EVS-deployed mode, the VMware Cloud Foundation (VCF) version. Choose the mode and the version first, because they decide who installs VCF, which instance types you can use, and which ESX builds your hosts can run.

For the supported versions, instance types, and ESX builds in one table, see [Supported VCF versions by instance type](#current-vcf-versions).

#### Choose a VCF version and deployment mode
<a name="plan-versions-mode"></a>

When you create an environment, the value that you choose sets the deployment mode, as the following table shows.


| Deployment mode | Value that you choose | Who installs VCF | 
| --- | --- | --- | 
| Self-deployed |  `SELF_DEPLOYED`  | You do. Amazon EVS creates the VLAN subnets. You add the hosts and install VCF yourself. You can deploy VCF 5.x or VCF 9.x in this mode. Amazon EVS provides VCF 9.0 and 9.1 as the VCF 9.x versions. | 
| EVS-deployed |  `VCF-5.2.2` or `VCF-5.2.1`  | Amazon EVS installs VCF 5.2.x when it creates the environment. EVS-deployed mode doesn’t support VCF 9.x deployments. | 

For a side-by-side comparison of the two modes, see [Differences between deployment modes](#deployment-mode-differences). For the steps in each mode, see [Deploy in Self-deployed mode](#getting-started-self-deployed) and [Deploy in EVS-deployed mode](#evs-deployed-overview).

Some instance types are available only with some VCF versions. For example, diskless hosts are available only with VCF 9.x. Check the table before you choose an instance type. For more information, see [Choose instance types](#plan-instance-types).

Some VCF versions have the status `RESTRICTED`. To use a restricted version, you must request access first. For more information, see [Request access to restricted VCF versions](#version-restrictions).

Amazon EVS doesn’t provide every VCF and ESX version that Broadcom releases. Before you choose a version, check that the other VMware products that you plan to use support it in the [Broadcom Interoperability Matrix](https://interopmatrix.broadcom.com/Interoperability?col=1).

If you plan to deploy VCF 9.x with the EVS Deployment Orchestrator (EDO), check which VCF versions EDO supports before you choose. EDO is a tool in the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs). For more information, see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo).

#### Choose an ESX version
<a name="plan-versions-esx"></a>

Each VCF version has a default ESX version that is based on the Broadcom VCF Bill of Materials (BOM). How you choose the ESX version depends on when you choose it.

 **When you create an environment**   
You can’t choose a specific ESX version. Amazon EVS applies the default ESX version for the VCF version that you select.

 **When you add a host**   
You can choose any ESX version that Amazon EVS provides for the instance type of the host. If you don’t choose one, Amazon EVS uses the default ESX version. For the `SELF_DEPLOYED` value, `get-versions` reports the default as `defaultEsxVersion`.

 **After you add a host**   
You can upgrade its ESX version only with vCenter Lifecycle Manager.

Confirm that the ESX version that you choose is compatible with your VCF release in the Broadcom Interoperability Matrix. For more information, see [ESX version considerations](#esx-versions).

#### Check which versions are available to you
<a name="plan-versions-check"></a>

The versions and builds available to your account can differ from the table, for example when a version is restricted. You can check them in two ways.
+ In the Amazon EVS console, the create environment wizard shows the VCF versions, and the add host wizard shows the ESX versions.
+ With the AWS CLI, run the `get-versions` command for the Region where you plan to deploy.

In the following command, replace `<region-code>` with the Region where you plan to deploy.

```bash
aws evs get-versions --region <region-code>
```

The response lists each VCF version with its status, default ESX version, and instance types. It also lists the ESX versions that you can choose for each instance type. For an example response, see [Check what is available to you](#check-provided-versions).

Next, choose an instance type. See [Choose instance types](#plan-instance-types).

### Choose instance types
<a name="plan-instance-types"></a>

Each Amazon Elastic VMware Service host is an Amazon Elastic Compute Cloud (Amazon EC2) bare metal instance. The instance type sets the storage, vCPUs, and memory of the host. Choose the storage model first, and then the size. The storage model decides your deployment path and which VMware Cloud Foundation (VCF) versions you can use.

For the instance types, their storage model, vCPUs and memory, and the VCF versions and ESX builds that each supports, see [Supported VCF versions by instance type](#current-vcf-versions).

#### Step 1. Choose a storage model
<a name="plan-instance-types-storage"></a>

Amazon EVS offers two storage models for hosts. The model that you choose decides where your VCF management appliances and workload virtual machines are stored.


| Storage model | Principal storage | VCF version | Choose it when | 
| --- | --- | --- | --- | 
| Local NVMe | vSAN ESA on the NVMe instance store volumes of the hosts. You can add external storage as supplemental storage. | All | You want storage that grows with the number of hosts, and that VCF configures during bringup. | 
| Diskless | External storage that you provide, such as an NFS datastore. The hosts have no local disks for data. | 9.x | You want to keep all datastores on external storage that you manage, and size storage separately from compute. | 

Diskless hosts are available only in Self-deployed mode. For the VCF versions and instance types that each mode supports, see [Supported VCF versions and instance types](#versions-provided).

For the full comparison of storage options, see [Choose storage](#plan-storage).

#### Step 2. Choose a size
<a name="plan-instance-types-size"></a>

Within a storage model, the instance types differ in vCPUs, memory and, for local NVMe, instance store capacity. Choose the size from the resources that your workloads need, and from the number of hosts that this size gives you.
+ For the vCPUs and memory of each instance type, see [Supported VCF versions by instance type](#current-vcf-versions).
+ For network bandwidth and other hardware details, see [Amazon EC2 instance types](https://aws.amazon.com/ec2/instance-types).
+ For the number of hosts and the quotas to raise, see [Size your environment](#plan-sizing).
+ For how instance size affects the physical cores that your VCF subscription must cover, see [VCF licensing](#vcf-license-mgmt).

The `get-versions` command returns the instance types for one AWS Region. Run it for the Region where you plan to deploy before you decide. For more information, see [Check which versions are available to you](#plan-versions-check).

#### Step 3. Plan for one instance type per cluster
<a name="plan-instance-types-mixing"></a>

All hosts in a VCF cluster must use the same instance type. Plan the instance type for each cluster before you add hosts.

When you install VCF, you set the Enhanced vMotion Compatibility (EVC) mode of the cluster to match the instance type of its hosts. For the EVC values, see [Management cluster settings](#installer-settings-cluster). Diskless hosts need VCF 9.x, and you set their EVC mode after deployment. For more information, see [Verify the result](#getting-started-diskless-verify).

Next, decide how many hosts you need. See [Size your environment](#plan-sizing).

### Size your environment
<a name="plan-sizing"></a>

The number of hosts in your Amazon Elastic VMware Service environment sets the capacity for the VMware Cloud Foundation (VCF) management appliances and your workloads. It also sets the quotas, the Amazon Elastic Compute Cloud (Amazon EC2) capacity, the IP addresses, and the VCF subscription cores that you need. Size the environment before you request any of them.

#### Minimum and maximum hosts
<a name="plan-sizing-hosts"></a>

The minimum number of hosts depends on how VCF is deployed and on the VCF topology that you choose.


| VCF version | Minimum hosts | 
| --- | --- | 
| 9.x | The number that your VCF topology needs. Amazon EVS doesn’t set a minimum of its own. You add the hosts after you create the environment. For the minimum host counts, including the vSAN requirements, see the [Broadcom documentation for VCF 9.0](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html). | 
| 5.2.x | Four. Amazon EVS creates four hosts when it creates the environment, because VCF requires them for vSAN quorum. You can add hosts after deployment. | 

Broadcom recommends at least four hosts for production workloads. For hosts with local storage, four hosts keep vSAN quorum and keep the cluster available during maintenance and host failures. For more information, see [Plan for resilience and recovery](#plan-resilience).

If your environment has fewer hosts than your VCF topology needs, the host count status check fails. For more information, see [Troubleshoot environment status checks](#troubleshoot-status-checks).

The maximum number of hosts in an environment is limited by the **Host count per EVS environment** quota. For the default value and how to raise it, see [Service quotas for Amazon EVS](#service-quotas-evs).

#### Management appliance sizes
<a name="plan-sizing-appliances"></a>

The VCF management appliances run on the hosts of the management domain, so they use part of its compute and storage capacity. The appliances include vCenter, NSX Manager, NSX Edge nodes, and SDDC Manager or VCF Operations. Size the management domain for the appliances and for the workloads that share its hosts.

For the appliance sizes and the resources that each size needs, see the Broadcom documentation for your VCF version. For VCF 5.2.x, see the [Planning and Preparation Workbook](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/planning-and-preparation-workbook-5-2.html) in the Broadcom documentation.

For VCF 9.x, see the [Planning and Preparation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/planning-and-preparation.html) page in the Broadcom documentation.

If you use diskless hosts, which need VCF 9.x, the management appliances run on your external principal datastore. For datastore sizing, see [External storage options for diskless hosts](#external-storage-diskless).

#### Quotas to raise before you start
<a name="plan-sizing-quotas"></a>

Amazon EVS and Amazon EC2 quotas apply per AWS Region. Check these quotas before you create the environment and before you add hosts, because a request that exceeds a quota fails.


| Quota | What it must cover | 
| --- | --- | 
| Amazon EVS: **Host count per EVS environment**  | Every host that you plan to create in one environment, including temporary hosts that you add when you replace a host. Environment creation fails if this quota is lower than your VCF topology needs. | 
| Amazon EVS: **Environment count per AWS account**  | Every environment that you plan to create in the Region. | 
| Amazon EC2: **Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances**  | The vCPUs of all hosts, plus the vCPUs of every other Standard instance that runs in the Region. For the vCPUs of each instance type, see [Supported VCF versions by instance type](#current-vcf-versions). | 
| Amazon EC2: **Dedicated Hosts** for your instance family | The number of Dedicated Hosts that you plan to use, if you use Dedicated Hosts. | 
| Amazon VPC IPAM: **Amazon-provided contiguous public IPv4 CIDR block netmask length**  | The default value of /28 is enough if you plan to use HCX internet connectivity. You don’t need to request an increase. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity). | 

For the default values and the steps to request an increase, see [Service quotas for Amazon EVS](#service-quotas-evs). A quota increase can take time to approve, so request it early.

The number of hosts also decides other values that you plan next.

 **IP addresses**   
Each host uses addresses in the VLAN subnets. You can’t add VLAN subnets or change their CIDR blocks after you create the environment. For more information, see [Plan your VPC and VLAN subnets](#plan-network-sizing).

 **EC2 capacity**   
For more information, see [Reserve capacity for hosts](#plan-capacity).

 **VCF subscription cores**   
For more information, see [VCF licensing](#vcf-license-mgmt).

Next, reserve capacity for your hosts. See [Reserve capacity for hosts](#plan-capacity).

### Reserve capacity for hosts
<a name="plan-capacity"></a>

Amazon Elastic VMware Service launches Amazon Elastic Compute Cloud (Amazon EC2) bare metal instances as the ESX hosts of your environment. To make sure that capacity is available when you add hosts, we recommend that you request an Amazon EC2 Capacity Reservation before you deploy.

#### How Capacity Reservations work
<a name="plan-capacity-reservations"></a>

A Capacity Reservation reserves EC2 capacity for an instance type in one Availability Zone, for any duration. Create the reservation for the instance type of your hosts, in the Availability Zone of your Amazon EVS environment. An Amazon EVS environment runs in one Availability Zone. For more information, see [VPC and subnets](#plan-network-supported-vpc).

You can create a Capacity Reservation at any time, and you can choose when it starts.

 **For immediate use**   
The reservation starts when you create it.

 **For a future date**   
You can request the reservation 5 to 120 days ahead. We recommend that you request it at least 8 weeks ahead. The minimum commitment is 14 days for most instance types.

**Important**  
Billing for a Capacity Reservation starts when the reservation becomes active, whether or not instances use it.

Each Capacity Reservation has instance matching criteria.

 **Open**   
An Open reservation is used automatically by any instance with matching attributes. Any matching instance in your account can use it, not only your Amazon EVS hosts.

 **Targeted**   
A Targeted reservation is used only by instances that explicitly target it.

You can create a future-dated reservation only as Targeted. After it becomes active, you can change it to Open.

For more information, see [Reserve compute capacity with EC2 On-Demand Capacity Reservations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html) in the *Amazon EC2 User Guide*. For the steps, see [Create a Capacity Reservation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservations-create.html) in the *Amazon EC2 User Guide*.

#### Capacity and quotas
<a name="plan-capacity-quotas"></a>

A Capacity Reservation reserves EC2 capacity. Your account also needs Amazon EVS and Amazon EC2 quotas that cover every host. For the quotas to check, see [Quotas to raise before you start](#plan-sizing-quotas).

#### Keep a spare host
<a name="plan-capacity-spare-host"></a>

You’re responsible for monitoring host health and replacing failed hosts. When you replace a host, you add the new host before you remove the old one, so your environment needs capacity for one more host during the replacement.

If you use Capacity Reservations or EC2 Dedicated Hosts, we recommend that you keep a spare host for mission-critical workloads.
+ A Capacity Reservation gives you access to EC2 capacity in an Availability Zone. A spare host adds redundancy on top of that capacity.
+ With Dedicated Hosts, a spare host keeps your environment running for mission-critical workloads if a primary host needs maintenance or has an issue.

Include the spare host and any temporary host in the **Host count per EVS environment** quota and in your Capacity Reservation. For the quota, see [Service quotas for Amazon EVS](#service-quotas-evs). For the replacement steps, see [Replace a failed host (Self-deployed)](#host-replace).

#### Release capacity that you no longer need
<a name="plan-capacity-release"></a>

When you delete hosts or an environment, cancel the Capacity Reservations that you no longer need, because they’re billed until you cancel them. For more information, see [Cancel a Capacity Reservation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservations-release.html) in the *Amazon EC2 User Guide*.

Next, choose your storage. See [Choose storage](#plan-storage).

## Choose storage
<a name="plan-storage"></a>

Principal storage is the first shared storage that VMware Cloud Foundation (VCF) uses when it creates a domain or cluster. It holds the VCF management appliances. Supplemental storage is storage that you add after VCF is deployed. The storage model of your hosts decides which principal storage you can use, so choose storage together with the instance type.

### Storage options
<a name="plan-storage-options"></a>

The following table compares the storage options for Amazon Elastic VMware Service hosts.


| Option | Principal storage | Instance types | VCF version | Next step | 
| --- | --- | --- | --- | --- | 
| vSAN Express Storage Architecture (ESA) on local NVMe | vSAN ESA on the NVMe instance store volumes of the hosts. You configure vSAN ESA when you install VCF. | Hosts with local NVMe | All | For the support conditions, see [vSAN ESA](#feat-vsan-esa). For the vSAN settings, see [Step 4b (local storage): Run VCF bringup](#sd-install-bringup). | 
| vSAN ESA with supplemental external storage | vSAN ESA, as in the first row. You add Amazon FSx for NetApp ONTAP datastores (NFS, iSCSI, or NVMe/TCP) after VCF is deployed. | Hosts with local NVMe | All |  [Connect external storage](#external-storage-connect)  | 
| External principal storage | An NFS v3 datastore on external storage that you create before you install VCF, such as Amazon FSx for OpenZFS or Amazon FSx for NetApp ONTAP. You can add supplemental storage after VCF is deployed. | Diskless hosts | 9.x |  [External storage options for diskless hosts](#external-storage-diskless)  | 

For the vSAN storage architectures that Amazon EVS supports, see the [vSAN ESA](#feat-vsan-esa) and [vSAN OSA](#feat-vsan-osa) rows in [VCF feature support on Amazon EVS](#vcf-feature-support).

### Hosts with local NVMe
<a name="plan-storage-local"></a>

vSAN ESA uses the NVMe instance store volumes of the hosts. Instance store volumes are ephemeral. Data on them doesn’t persist if the underlying EC2 instance is stopped or terminated.

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API.

For more information, see [Host maintenance and retirement events](#evs-host-maintenance).

vSAN capacity grows when you add hosts. To add capacity without adding hosts, add supplemental datastores on Amazon FSx for NetApp ONTAP. For more information, see [Connect external storage](#external-storage-connect).

### Diskless hosts
<a name="plan-storage-diskless"></a>

This section applies to Self-deployed environments that run VCF 9.x.

Diskless hosts have no local disks for data, so vSAN isn’t available on them. Every datastore, including the one that holds the VCF management appliances, must be on external storage that all hosts in the cluster can reach. You create and manage that storage, and Amazon EVS doesn’t create, configure, or monitor it.

For a comparison of Amazon FSx for NetApp ONTAP, Amazon FSx for OpenZFS, and Everpure Cloud Dedicated, and the requirements that apply to all of them, see [External storage options for diskless hosts](#external-storage-diskless). For the deployment steps, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

**Topics**

### External storage options for diskless hosts
<a name="external-storage-diskless"></a>

This page applies to Self-deployed environments that run VCF 9.x on diskless hosts. For how EVS-deployed environments (VCF 5.2.x) differ, see [Differences between deployment modes](#deployment-mode-differences).

Diskless Amazon Elastic VMware Service hosts boot from an Amazon EBS volume and have no NVMe instance store volumes to use as vSAN capacity. Every datastore that holds your virtual machines must be on external storage that all hosts in the cluster can reach over the network. This includes the datastore for the VMware Cloud Foundation (VCF) management appliances.

Each diskless host has one EBS boot volume. ESX shows this volume as a local NVMe disk that holds only the ESX system partitions. The host has no VMFS data datastore on local disks, and vSAN isn’t available. For the diskless instance types and the VCF versions that they support, see [Supported VCF versions by instance type](#current-vcf-versions).

#### Requirements for diskless hosts
<a name="external-storage-diskless-instance-types"></a>

Diskless instance types have the following requirements.
+ You can use them only with VCF 9.x. They aren’t available in EVS-deployed mode, where Amazon EVS deploys VCF 5.2.x for you.
+ You must provide and manage the external storage. Amazon EVS doesn’t create, configure, or monitor it.
+ You can’t use vSAN on local disks.
+ The external storage must be in the same VPC and Availability Zone as your hosts.

For the procedure to deploy an environment with diskless hosts, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

#### Storage options
<a name="external-storage-diskless-options"></a>

The following storage services are options for diskless hosts.


| Characteristic | Amazon FSx for NetApp ONTAP | Amazon FSx for OpenZFS | Everpure Cloud Dedicated | 
| --- | --- | --- | --- | 
| Service type | Fully managed AWS file system service. | Fully managed AWS file system service. | Software-defined block storage array that you deploy from AWS Marketplace into your AWS account, and that you manage. | 
| Protocols for Amazon EVS | NFS v3, NFS v4.1, iSCSI, and NVMe/TCP. | NFS. The deployment in this guide uses NFS v3. | iSCSI, with VMFS datastores. | 
| Principal storage for the management domain | NFS v3 through the VCF Installer, according to Broadcom and NetApp documentation for ONTAP. Other protocols need the Broadcom Converge workflow, which you run outside Amazon EVS. For more information, see [Principal and supplemental storage](#external-storage-diskless-principal). | NFS v3 through the VCF Installer. For the procedure, see [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore). | Broadcom Converge workflow. | 
| Supplemental storage | Yes. See [Connect external storage](#external-storage-connect). | Not covered in this guide. | VMFS datastores on iSCSI that you add after VCF is deployed. | 
| Amazon EVS procedures |  [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap)  |  [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore)  | Everpure documentation. See [Everpure Cloud Dedicated](#external-storage-diskless-everpure). | 

The deployment procedure in this guide uses FSx for OpenZFS with NFS v3. For FSx for NetApp ONTAP as principal storage and for Everpure Cloud Dedicated, follow the Broadcom, NetApp, and Everpure documentation that applies to your option.

#### Principal and supplemental storage
<a name="external-storage-diskless-principal"></a>

For the definitions of principal and supplemental storage, see [Choose storage](#plan-storage). For the management domain, the VCF Installer places the management appliances on the principal datastore.

The VCF Installer deploys the management domain directly onto an NFS v3 datastore. To use another protocol as principal storage, such as iSCSI, NFS v4.1, or NVMe over TCP, you first build the datastore and a vCenter instance yourself. Then you convert the environment to VCF. For more information, see [article 416270](https://knowledge.broadcom.com/external/article/416270) in the Broadcom knowledge base.

#### Requirements for all storage options
<a name="external-storage-diskless-requirements"></a>

Whichever option you choose, plan for the following.

Network placement  
Create the storage in the same VPC and Availability Zone as your Amazon EVS environment. Amazon EVS is a single Availability Zone service. Place the storage endpoints in the Amazon EVS service access subnet or in another subnet of the same VPC. Don’t place storage in an Amazon EVS VLAN subnet, because Amazon EVS manages those subnets. For more information about the service access subnet, see [Concepts and components of Amazon EVS](#concepts).

Network path  
ESX hosts reach the storage through the VPC. Traffic from the Amazon EVS VLAN subnets uses the VPC local route. If you associate the VLAN subnets with a custom route table, make sure that the route table keeps the local route to the subnet that contains the storage. If you use a network ACL for the VLAN subnets, allow the storage protocol ports in both directions. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

Access rules  
Allow the ESX host IP addresses to reach the storage. For NFS, add the host management VLAN subnet CIDR to the export or access list. In the security group of the storage, allow the protocol ports from the host management VLAN subnet CIDR. EC2 security group rules aren’t enforced on the Amazon EVS VLAN subnet network interfaces, but they’re enforced on the storage network interfaces.

DNS and IP addressing  
Use the IP address of the storage endpoint when you configure the datastore in the VCF Installer specification. Whether ESX hosts can resolve the DNS name of the storage depends on your DNS configuration.

MTU  
The host management network in Amazon EVS uses an MTU of 1500, and the NFS datastores in this guide mount through that network. For more information, see [MTU by network function](#vlan-subnet-mtu).

Capacity  
The VCF Installer checks the capacity of the principal datastore before it deploys. Size the datastore for the management domain that you plan to deploy, and leave room for growth. The datastore size that the ESX host reports depends on the storage. For example, a datastore on an Amazon FSx for OpenZFS volume without a quota shows the capacity of the file system. For a sizing example, see [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore).

Availability  
The management appliances run on the external datastore. If the storage is unavailable, the management plane and your workloads are unavailable. Choose a storage deployment type that matches your availability requirements, and test how your environment behaves during a storage failover. For more information, see [Plan for resilience and recovery](#plan-resilience).

#### Amazon FSx for NetApp ONTAP
<a name="external-storage-diskless-fsx-ontap"></a>

Amazon FSx for NetApp ONTAP provides NFS and block storage that you can use as external datastores with Amazon EVS. The FSx for ONTAP datastore topics in [Connect external storage](#external-storage-connect) assume that your environment is already deployed and that you use the vSphere Client. For diskless hosts, the principal datastore must exist before VCF is deployed. Use the VMware Host Client and the VCF Installer instead, as described in [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

For NFS v3 principal storage, the VCF Installer needs the following.
+ The datastore is mounted on the first host before you validate the specification.
+ The export allows the ESX host management addresses.
+ The specification declares an NFS network.

For more information, see [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec).

For NetApp guidance on ONTAP as principal NFS storage for the VCF management domain, see [VCF management domain with NFS principal storage](https://docs.netapp.com/us-en/netapp-solutions-virtualization/vmware/vmw-vcf-mgmt-principal-nfs.html) in the NetApp documentation. For NetApp guidance on FSx for ONTAP with Amazon EVS, see [Amazon FSx for NetApp ONTAP with Amazon EVS](https://docs.netapp.com/us-en/netapp-solutions-cloud/vmware/evs-mount-fsxn-intro.html) in the NetApp documentation.

For file system limits and availability options, see [Quotas and limits](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/limits.html) and [Availability and durability](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/high-availability-AZ.html) in the *Amazon FSx for NetApp ONTAP User Guide*.

#### Amazon FSx for OpenZFS
<a name="external-storage-diskless-fsx-openzfs"></a>

Amazon FSx for OpenZFS provides NFS file storage that is built on the OpenZFS file system. The VCF Installer deploys the management domain onto an NFS v3 datastore on an FSx for OpenZFS volume. For the procedure, see [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore).

For more information, see the [Amazon FSx for OpenZFS User Guide](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/what-is-fsx.html).

#### Everpure Cloud Dedicated
<a name="external-storage-diskless-everpure"></a>

Everpure Cloud Dedicated is a block storage array from Everpure (formerly Pure Storage) that runs on Amazon EC2 instances in your AWS account. You subscribe to it and deploy it from AWS Marketplace, and you operate it. According to Everpure, Everpure Cloud Dedicated connects to Amazon EVS over iSCSI and presents VMFS datastores.

**Note**  
Everpure Cloud Dedicated was previously named Pure Cloud Block Store and Pure Storage Cloud Dedicated. Everpure and AWS Marketplace documentation might use any of these names.

The VCF Installer doesn’t accept an iSCSI VMFS datastore as principal storage. Use Everpure Cloud Dedicated in one of the following ways.
+ As supplemental storage that you add after VCF is deployed on a different principal datastore, such as an FSx for OpenZFS or FSx for ONTAP NFS datastore.
+ As principal storage through the Broadcom Converge workflow, which you run outside of Amazon EVS. Everpure doesn’t document this configuration for Amazon EVS. For more information, see [article 416270](https://knowledge.broadcom.com/external/article/416270) in the Broadcom knowledge base.

For the following tasks, use the Everpure documentation.
+ Subscribe to the product and deploy the array into your VPC.
+ Plan the subnets, security groups, IAM role, and quotas that the array needs.
+ Configure the iSCSI network, multipathing, and the Everpure vSphere plug-in.
+ Create volumes and VMFS datastores.

Place the array in the same Availability Zone as your hosts, and use separate VPC subnets for the array interfaces. Don’t use an Amazon EVS VLAN subnet for the array.

For more information, see the following resources.
+  [Everpure Cloud Dedicated: Support for Amazon Elastic VMware Service](https://www.everpuredata.com/content/dam/pdf/en/solution-briefs/sb-pure-storage-cloud-dedicated-support-for-amazon-evs.pdf) solution brief
+  [Everpure Cloud Dedicated (subscription)](https://aws.amazon.com/marketplace/pp/prodview-j2mtejfozqlto) in AWS Marketplace 
+  [Everpure Community posts about Amazon EVS](https://purecommunity.purestorage.com/tag/evs) 

#### Broadcom references for external principal storage
<a name="external-storage-diskless-broadcom"></a>

The following Broadcom articles describe the requirements that the VCF Installer enforces for NFS principal storage.
+  [Article 414088](https://knowledge.broadcom.com/external/article/414088) in the Broadcom knowledge base describes the requirement that the NFS datastore is mounted on the first ESX host before validation.
+  [Article 446573](https://knowledge.broadcom.com/external/article/446573) in the Broadcom knowledge base describes the causes of NFS pre-check failures, including port group names, MTU, and export lists.
+  [Article 416270](https://knowledge.broadcom.com/external/article/416270) in the Broadcom knowledge base describes which storage types can be principal storage and how to use the Converge workflow.

## Plan your network
<a name="plan-network"></a>

Amazon Elastic VMware Service deploys your environment into a VPC that you provide, and creates the VLAN subnets for VMware Cloud Foundation (VCF) in that VPC. You can’t change the VPC CIDR block, add VLAN subnets, or change the VLAN subnet CIDR blocks after you create the environment. Make the following decisions before you create the environment, in this order.

1. Size the VPC CIDR block, the service access subnet, and the 10 VLAN subnets. For more information, see [Plan your VPC and VLAN subnets](#plan-network-sizing).

1. Check which connections to other VPCs, to on-premises networks, and to the internet Amazon EVS supports. For more information, see [Supported network configurations](#plan-network-supported).

1. Plan the NSX Edge cluster, the Tier-0 gateway, and the BGP peering with Amazon VPC Route Server. For more information, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design).

1. If you plan to migrate workloads with VMware HCX, choose private or internet connectivity. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity).

Record your values in the [Deployment prerequisite checklist](#evs-deployment-prereq-checklist). When you are ready to build the network, see [Build the network foundation](#network-foundation).

**Topics**
+ [Plan your VPC and VLAN subnets](#plan-network-sizing)
+ [Supported network configurations](#plan-network-supported)
+ [Plan NSX Edge and VPC Route Server design](#plan-nsx-design)
+ [Plan HCX connectivity](#plan-hcx-connectivity)

### Plan your VPC and VLAN subnets
<a name="plan-network-sizing"></a>

Your Amazon Elastic VMware Service environment uses address space from one VPC CIDR block for the service access subnet and the 10 VLAN subnets that Amazon EVS creates. Size this space for the hosts and VMware Cloud Foundation (VCF) management appliances that you plan to run over the lifetime of the environment, because you can’t add space later.

You can’t change these settings after you create the environment.
+ The VPC CIDR block
+ The CIDR blocks of the VLAN subnets
+ The number of VLAN subnets

#### VPC CIDR block
<a name="plan-network-sizing-vpc"></a>

Amazon EVS deploys your environment into a VPC that you provide. The VPC CIDR block must meet these requirements:
+ It is an IPv4 CIDR block. Amazon EVS doesn’t support IPv6.
+ It is /22 or larger.
+ It has room for the service access subnet and all 10 VLAN subnets. It also needs room for any other subnets that you create in the VPC, for example a public subnet for a NAT gateway.
+ It belongs to one environment only. A VPC can’t be used by more than one deployed Amazon EVS environment.

Use a CIDR block that is unique across your environments.

The /22 minimum doesn’t leave room for /24 VLAN subnets. A /22 holds four /24 blocks. If you use /24 for the service access subnet and for each of the 10 VLAN subnets, you need 11 /24 blocks, so choose a /20 or larger. The example values in the [Deployment prerequisite checklist](#evs-deployment-prereq-checklist) use this layout.

An Amazon EVS environment runs in one Availability Zone. All VPC subnets that Amazon EVS uses must be in that Availability Zone, in a [Region where Amazon EVS is available](#regions-reference). For more information, see [VPC and subnets](#plan-network-supported-vpc).

For HCX internet connectivity, you also add a public IPv4 CIDR block to the VPC as an additional CIDR. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity).

#### Service access subnet
<a name="plan-network-sizing-service-access"></a>

The service access subnet is a standard VPC subnet that you create before you create the environment. Amazon EVS uses it for service access to your environment, and the endpoints of Amazon VPC Route Server run in it.

Follow these guidelines for the service access subnet.
+ Create it in the same Availability Zone as the rest of the environment.
+ We recommend a /24 netmask, so that the subnet has room to grow.
+ Leave room in it for the two route server endpoints.

For more information, see [Concepts and components of Amazon EVS](#concepts).

#### VLAN subnets
<a name="plan-network-sizing-vlans"></a>

Amazon EVS creates 10 VLAN subnets in your VPC when you create the environment. You enter a CIDR block for each one. Every VLAN subnet CIDR block must meet these requirements:
+ Its size is between /28 and /24.
+ It is inside a CIDR block that is associated with the VPC. The block can be the primary CIDR block or a secondary CIDR block.
+ It doesn’t overlap with any other subnet in the VPC, including the other VLAN subnets and the service access subnet.

The following table lists the VLAN subnets. The API name column shows the name that you use in the `initialVlans` parameter.


| VLAN subnet | API name | Carries | Size rule | Sizing guidance | 
| --- | --- | --- | --- | --- | 
| Host management |  `vmkManagement`  | ESX host management traffic. | /28 to /24. | One IP address per host. Plan for the largest number of hosts that you expect in the environment. | 
| vMotion |  `vMotion`  | vMotion traffic between hosts. | Same size as host management. | One or more IP addresses per host, depending on your VCF configuration. | 
| vSAN |  `vSan`  | vSAN traffic between hosts. | Same size as host management. | One or more IP addresses per host, depending on your VCF configuration. | 
| VTEP |  `vTep`  | Host overlay (Geneve) traffic. | Same size as host management. | Two IP addresses per host. | 
| Edge VTEP |  `edgeVTep`  | NSX Edge overlay traffic. | /28 to /24. | IP addresses for the NSX Edge nodes. | 
| Management VM |  `vmManagement`  | VCF management appliances. | /28 to /24. | IP addresses for the appliances that you plan to deploy, for example vCenter, the NSX Manager cluster, the NSX Edge nodes, and SDDC Manager or VCF Operations. | 
| HCX uplink |  `hcx`  | VMware HCX appliances. | /28 to /24. Exactly /28 for a public HCX VLAN. | IP addresses for the HCX appliances, if you use HCX. For the public HCX VLAN rules, see [Plan HCX connectivity](#plan-hcx-connectivity). | 
| NSX uplink |  `nsxUplink`  | NSX Edge uplinks to the VPC. | /28 to /24. | IP addresses for the NSX Edge uplink interfaces. The route server peers use addresses from this subnet. For more information, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design). | 
| Expansion VLAN 1 |  `expansionVlan1`  | Not assigned. You use it to extend your VCF deployment. | /28 to /24. | Reserve space to extend your deployment, for example for NSX Federation. | 
| Expansion VLAN 2 |  `expansionVlan2`  | Not assigned. You use it to extend your VCF deployment. | /28 to /24. | Reserve space to extend your deployment, for example for NSX Federation. | 

As a starting point, use /24 for each VLAN subnet unless you have a specific reason to choose otherwise. Because you can’t add or resize VLAN subnets later, extra address space after creation can come only from the expansion VLANs.

The VLAN subnets use these MTU values:
+ 1500 for host management, Management VM and NSX uplink.
+ 8500 for vMotion, vSAN, VTEP, and Edge VTEP.

#### Reserved IP addresses
<a name="plan-network-sizing-reserved"></a>

Amazon EVS VLAN subnets are VPC subnets, so the standard VPC reserved addresses apply. In each VLAN subnet and in the service access subnet, AWS reserves the first four IP addresses and the last IP address of the CIDR block. You can’t assign these addresses to hosts, appliances, or route server peers. For example, in `10.1.1.0/24` the reserved addresses are `10.1.1.0` to `10.1.1.3` and `10.1.1.255`. For more information, see [Subnet CIDR blocks](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-sizing.html) in the *Amazon VPC User Guide*.

Count the reserved addresses when you size each subnet. A /28 has 16 addresses, of which 11 are usable.

The public HCX VLAN has its own reserved addresses. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity).

#### VLAN IDs
<a name="plan-network-sizing-vlan-ids"></a>

Amazon EVS assigns a VLAN ID to each VLAN subnet. You use these IDs when you configure VCF networking, for example the Edge VTEP VLAN ID in the NSX uplink profile. For more information, see [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference).

After you create the environment, you can look up the IDs in the Amazon EVS console. In the navigation pane, choose **Environments**, choose your environment, and then choose the **Networks and connectivity** tab. You can also run `aws evs list-environment-vlans`.

When you have sized your subnets, record the values in the [Deployment prerequisite checklist](#evs-deployment-prereq-checklist). To create the VPC and the service access subnet, see [Create a VPC with subnets and route tables](#network-create-vpc).

### Supported network configurations
<a name="plan-network-supported"></a>

Amazon Elastic VMware Service supports the following network connections and VPC configurations in both deployment modes. The tables also list what Amazon EVS doesn’t support. In each table, the **Supported** column shows `Yes`, `No`, or `Required`, and the **Notes** column gives the condition or the alternative.

#### Connections to on-premises networks and other VPCs
<a name="plan-network-supported-external"></a>

The following table lists the supported and unsupported connections to on-premises networks and other VPCs.


| Configuration | Supported | Notes | 
| --- | --- | --- | 
|  AWS Transit Gateway | Required | You must use a transit gateway to connect an Amazon EVS environment to your on-premises networks. The transit gateway does not learn NSX routes from Amazon VPC Route Server. After you create the environment, add static routes for the NSX overlay networks to the transit gateway route tables. | 
|  Direct Connect with a transit gateway | Yes | Associate the transit gateway with a Direct Connect gateway. For more information, see [Direct Connect gateways and transit gateway associations](https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-connect-transit-gateways.html). | 
|  AWS Site-to-Site VPN attached to a transit gateway | Yes | For more information, see [AWS Site-to-Site VPN attachments](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpn-attachments.html) in the * Amazon VPC Transit Gateway User Guide*. You can combine Site-to-Site VPN with Direct Connect. For more information, see [Private IP AWS Site-to-Site VPN with Direct Connect](https://docs.aws.amazon.com/vpn/latest/s2svpn/private-ip-dx.html). | 
|  AWS Cloud WAN | Yes | Like a transit gateway, Cloud WAN does not learn NSX routes from VPC Route Server. Add static routes for the NSX overlay networks. | 
|  Direct Connect private virtual interface (VIF) to the Amazon EVS VPC | No | Use a transit gateway with Direct Connect. | 
|  AWS Site-to-Site VPN that terminates directly in the Amazon EVS VPC | No | Attach the Site-to-Site VPN connection to a transit gateway. | 
| VPC peering | No | Use a transit gateway or Cloud WAN to connect to other VPCs. | 

For steps to connect on-premises networks, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

#### Internet access
<a name="plan-network-supported-internet"></a>

The following table lists the supported options for internet access.


| Configuration | Supported | Notes | 
| --- | --- | --- | 
| NAT gateway for outbound internet access | Yes | Amazon EVS requires a NAT gateway for outbound internet connectivity. In Self-deployed mode, the VMware Cloud Foundation (VCF) Installer reaches the Broadcom online depot through it. For the subnet and route rules, see [Gateway route requirements](#vpc-main-rt-reqs). | 
| Elastic IP addresses on the HCX VLAN subnet | Yes | Amazon EVS supports associating Elastic IP addresses only with the HCX VLAN subnet, for HCX internet connectivity. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity). | 
| Inspection of north-south traffic | Yes | You can place AWS Network Firewall or partner firewalls behind a Gateway Load Balancer at the internet ingress and egress points of the VPC. | 

#### VPC and subnets
<a name="plan-network-supported-vpc"></a>

The following table lists the supported and unsupported VPC and subnet configurations.


| Configuration | Supported | Notes | 
| --- | --- | --- | 
| IPv4 VPC CIDR block of /22 or larger | Required | For sizing, see [Plan your VPC and VLAN subnets](#plan-network-sizing). | 
| More than one Amazon EVS environment in the same VPC | No | Each Amazon EVS environment needs its own VPC. A VPC can’t be used by another deployed Amazon EVS environment. | 
| IPv6 | No | Amazon EVS supports IPv4 only. Amazon EVS also does not support private IPv6 Global Unicast Address (GUA) CIDRs. | 
| Subnets in more than one Availability Zone | No | An Amazon EVS environment runs in one Availability Zone. All VPC subnets that Amazon EVS uses must be in that Availability Zone. | 
| VPC, subnets, and environment in different accounts | No | You must create the VPC, the subnets, and the Amazon EVS environment in the same account. Amazon EVS does not support cross-account sharing of VPC subnets or of environments. | 
| Secondary VPC CIDR block for the public HCX CIDR | Required | Required for HCX internet connectivity. You add the public IPv4 CIDR block from your IPAM pool to the VPC as an additional CIDR. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity). | 
| Custom route table during environment creation | No | During creation, the VLAN subnets are implicitly associated with the VPC main route table. Configure the main route table with routes to your DNS servers and on-premises systems. Use a custom route table only after the environment is created. | 
| Explicit route table association of the VLAN subnets after creation | Required | After the environment is created, associate the VLAN subnets explicitly with a route table. Without this, NSX connectivity fails. For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt). | 
| Gateway VPC endpoint for Amazon S3  | No | Use an interface VPC endpoint with AWS PrivateLink for Amazon S3. For more information, see [AWS PrivateLink for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html) in the *Amazon Simple Storage Service User Guide*. | 

#### Traffic control and workload placement
<a name="plan-network-supported-traffic"></a>

The following table lists the supported and unsupported options for traffic control and workload placement.


| Configuration | Supported | Notes | 
| --- | --- | --- | 
| Security groups on the network interfaces in VLAN subnets | No | Security groups do not apply to traffic on the elastic network interfaces in Amazon EVS VLAN subnets. Use a network ACL to control traffic to and from the VLAN subnets. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong). | 
| Network ACLs on VLAN subnets | Yes | You can use the default network ACL of the VPC or a custom network ACL. For HCX internet connectivity, we strongly recommend a custom network ACL dedicated to the HCX VLAN subnet. | 
| VCF management appliances on VLAN subnets | Yes | vCenter, NSX Manager, the NSX Edge nodes, SDDC Manager, and HCX appliances run on the VLAN subnets. | 
| Guest workload VMs on VLAN subnets | No | The VLAN subnets carry VCF underlay traffic only. Deploy workload VMs on NSX overlay networks. | 
| NSX Federation | Yes | Use the expansion VLAN subnets for Federation traffic. For more information, see [Set up NSX Federation](#nsx-federation). | 

For the NSX Edge and routing limits, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design).

### Plan NSX Edge and VPC Route Server design
<a name="plan-nsx-design"></a>

An NSX Edge cluster connects the NSX overlay networks of your environment to the VPC underlay. The Tier-0 gateway on that cluster exchanges routes with Amazon VPC Route Server over BGP, so plan the ASNs and peer addresses before you create the environment.

#### NSX components
<a name="plan-nsx-design-components"></a>

 **Tier-0 gateway**   
The Tier-0 gateway handles all north-south traffic between the NSX overlay networks and the VPC. Amazon Elastic VMware Service supports only one Tier-0 gateway, on an Active/Standby NSX Edge cluster with two NSX Edge nodes. This Tier-0 gateway connects to and advertises all overlay networks that you configure.

 **Tier-1 gateways**   
A Tier-1 gateway handles east-west traffic between routed segments. It has downlinks to segments and an uplink to the Tier-0 gateway. You can create additional Tier-1 gateways if you need them.

 **NSX Edge cluster**   
The Edge cluster has two NSX Edge nodes in Active/Standby mode. It runs the Tier-0 and Tier-1 gateways and their BGP routing. The active Edge node carries all north-south traffic, so an undersized Edge node can become a bottleneck. We recommend the Large form factor for the Edge nodes.

 **Amazon VPC Route Server**   
VPC Route Server is a VPC service that Amazon EVS uses for BGP-based dynamic routing between the NSX Edge nodes and your VPC route tables. You create the route server, its endpoints and its peers yourself.

#### How the deployment modes differ
<a name="plan-nsx-design-modes"></a>

The route server requirements are the same in both modes. Who deploys the Edge cluster and configures BGP differs.


| Item | VCF 9.x (Self-deployed) | VCF 5.2.x (EVS-deployed) | 
| --- | --- | --- | 
| Edge cluster and Tier-0 gateway | You deploy them in NSX Manager after you install VMware Cloud Foundation (VCF). | Amazon EVS deploys them when it creates the environment. | 
| Route server peers at environment creation | You don’t enter peers when you create the environment. | You enter two route server peers when you create the environment. Amazon EVS assigns their IP addresses to the two NSX Edge nodes. | 
| BGP on the Tier-0 gateway | You configure BGP neighbors, route redistribution, and route filters yourself. | Amazon EVS configures BGP peering with the route server peers that you entered. | 
| Peering topology | We recommend a full mesh of four peers, so that each Edge node peers with both route server endpoints. A full mesh keeps routes propagating if a route server endpoint goes into maintenance. You must create at least two peers. | Two peers, one for each NSX Edge node. | 

For the Self-deployed steps, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge).

#### Route server requirements
<a name="plan-nsx-design-route-server"></a>

Your route server must meet these requirements, or the environment deployment fails:
+ The route server is associated with your VPC.
+ The route server has at least two endpoints, both in the service access subnet.
+ The peer ASN on each route server peer matches the ASN of the NSX Edge Tier-0 gateway.
+ Each route server peer uses a unique, unused IP address from the NSX uplink VLAN subnet. The NSX uplink VLAN subnet doesn’t exist yet when you create the peers, so choose the addresses from the CIDR block that you plan to use for it.
+ Route server propagation is enabled on the route table associated with the service access subnet. Every propagated route table has at least one explicit subnet association, otherwise BGP route advertisement fails.

Plan the following values for the route server and the Tier-0 gateway.


| Value | Rule | Example | 
| --- | --- | --- | 
| Amazon-side ASN (route server) | In the range 1 to 4294967295. AWS recommends a private ASN, 64512 to 65534 (16-bit) or 4200000000 to 4294967294 (32-bit). The Tier-0 gateway uses it as the remote ASN. |  `65022`  | 
| NSX Edge Tier-0 ASN | A private ASN. It must match the peer ASN on every route server peer. The NSX Edge default is 65000. |  `65000`  | 
| Route server peer IP addresses | Unique, unused addresses inside the CIDR block that you choose for the NSX uplink VLAN subnet. Don’t use the reserved addresses of the subnet. |  `10.1.8.10`, `10.1.8.200`  | 

Also plan for these behaviors:
+ Amazon EVS supports only the default BGP keepalive mechanism for peer liveness detection. Amazon EVS doesn’t support multi-hop Bidirectional Forwarding Detection (BFD).
+ We recommend that you enable persistent routes on the route server, with a persist duration of 1 to 5 minutes. Routes then stay in the route server’s routing database even if all BGP sessions end.
+ For HCX internet connectivity, configure route server propagation for both the service access subnet and the public subnet of the VPC.
+ BGP connectivity status stays down until the environment is deployed and operational.

#### How routes reach the rest of your network
<a name="plan-nsx-design-routes"></a>

The Tier-0 gateway advertises the NSX overlay prefixes to the route server, and route server propagation installs them in your VPC route tables. A transit gateway or AWS Cloud WAN doesn’t learn these routes. Add static routes for your overlay networks to the transit gateway route tables after you create the environment.

In a Self-deployed environment, you choose what the Tier-0 gateway advertises. We recommend that you redistribute only overlay network routes, so that the NSX uplink subnet and the route server endpoint routes aren’t advertised back to AWS. The Edge uplink interfaces and the route server endpoints are in different subnets, so the BGP sessions are multihop sessions with a hop limit of at least 2.

The NSX uplink network uses MTU 1500. The overlay (TEP) networks use MTU 8500.

#### Limits
<a name="plan-nsx-design-limits"></a>

Amazon EVS has the following limits.
+ One Tier-0 gateway for each environment
+ One NSX Edge cluster with two Edge nodes in Active/Standby mode for the Tier-0 gateway
+ At least two route server endpoints, in the service access subnet

#### Availability Zone failure
<a name="plan-nsx-design-az"></a>

An Amazon EVS environment runs in one Availability Zone. For more information, see [VPC and subnets](#plan-network-supported-vpc). For recovery designs across Availability Zones or Regions, see [Plan for resilience and recovery](#plan-resilience).

To create the route server, endpoints and peers, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

### Plan HCX connectivity
<a name="plan-hcx-connectivity"></a>

You can use VMware Hybrid Cloud Extension (HCX) to migrate VMware workloads to Amazon Elastic VMware Service over private connectivity or over the public internet. Choose the option before you create the environment, because internet connectivity needs a public CIDR block in your VPC and a matching HCX VLAN subnet.

**Note**  
HCX isn’t installed in an Amazon EVS environment by default.

#### Choose a connectivity option
<a name="plan-hcx-connectivity-choose"></a>

Choose one connectivity option for your environment.

 **Private connectivity**   
Uses Direct Connect or AWS Site-to-Site VPN with a transit gateway. It gives more consistent performance and reliability.

 **Internet connectivity**   
Uses the public internet. It is quicker to set up and needs Amazon VPC IP Address Manager (IPAM) and Elastic IP addresses.

Some sites have private connectivity with consistent performance but lower throughput, because of VPN encryption or limited link speed. Others have high-throughput internet connectivity with more variable performance. Choose the option that works best for your sites.

We don’t recommend HCX internet connectivity for applications that are sensitive to network jitter or latency, time-critical vMotion operations, or large-scale migrations with strict performance requirements. For these cases, use HCX private connectivity, because a private dedicated connection gives more reliable performance.

The following table compares the two options.


| Aspect | Private connectivity | Internet connectivity | 
| --- | --- | --- | 
| Overview | Uses only private connections within the VPC. You can use Direct Connect or Site-to-Site VPN with a transit gateway for external connectivity. | Uses public internet connectivity with Elastic IP addresses. You can migrate without a dedicated private connection. | 
| Best suited for |  +  Time-sensitive vMotion operations. <br />+  Large-scale migrations. <br />+  Applications sensitive to latency or jitter. <br />+  High-volume data transfers. <br />+  Organizations that already use Direct Connect or Site-to-Site VPN.   |  +  Locations without Direct Connect or Site-to-Site VPN. <br />+  Cost-sensitive projects.   | 
| Key benefits |  +  Consistent low-latency connectivity. <br />+  Dedicated bandwidth. <br />+  More reliable network performance. <br />+  You can turn off the default HCX encryption on private networks to improve performance. <br />+  No public IP address management.   |  +  Faster setup than private connectivity. <br />+  Cost-effective for smaller migrations.   | 
| Key considerations |  +  More complex initial setup. <br />+  Higher upfront infrastructure costs. <br />+  Longer implementation timeline. <br />+  No direct internet connectivity for any HCX component.   |  +  More variable network performance. <br />+  Possible bandwidth limits. <br />+  Higher latency than private connectivity. <br />+  Each HCX component needs its own Elastic IP address from the public IPAM pool. <br />+  Each Elastic IP address association gives that HCX component direct internet connectivity.   | 

#### Private connectivity architecture
<a name="plan-hcx-connectivity-private"></a>

HCX private connectivity uses these components:
+  **Amazon EVS network components** 
  + Private VLAN subnets only, including a private HCX VLAN subnet.
  + Network ACLs for traffic control.
  + Dynamic Border Gateway Protocol (BGP) propagation of routes through Amazon VPC Route Server.
+  **Transit options to your on-premises network** 
  +  Direct Connect with AWS Transit Gateway, over a private dedicated connection. For more information, see [AWS Direct Connect and AWS Transit Gateway](https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/aws-direct-connect-aws-transit-gateway.html).
  +  AWS Site-to-Site VPN with AWS Transit Gateway, over an IPsec VPN connection across the internet. For more information, see [AWS Transit Gateway and AWS Site-to-Site VPN](https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/aws-transit-gateway-vpn.html).

Amazon EVS doesn’t support connectivity through an Direct Connect private virtual interface (VIF), or through a Site-to-Site VPN connection that terminates directly in the Amazon EVS VPC. For the full list, see [Supported network configurations](#plan-network-supported).

The following diagram shows HCX private connectivity through a transit gateway.

![On-premises HCX connects through Direct Connect or Site-to-Site VPN and a transit gateway to the private HCX VLAN subnet.](https://docs.aws.amazon.com/evs/latest/userguide/images/hcx-private-landscape.drawio.png)


#### Internet connectivity architecture
<a name="plan-hcx-connectivity-internet"></a>

HCX internet connectivity uses these components:
+  **Amazon EVS network components** 
  + An isolated public HCX VLAN subnet for internet connectivity between Amazon EVS and your on-premises HCX appliances.
  + Network ACLs for traffic control.
  + Dynamic BGP propagation of routes through VPC Route Server.
+  **IPAM and public IP address management** 
  + IPAM manages public IPv4 address allocation from an Amazon-owned public IPAM pool.
  + A /28 public CIDR block from the IPAM pool is added to the VPC as a secondary CIDR. It is separate from the main VPC CIDR block.

HCX doesn’t support destination network address translation (DNAT). HCX needs an uplink network that is routable and has a default gateway IP address. Amazon EVS VLAN subnets have a default gateway like other VPC subnets, but they are always private subnets, even when you use CIDR blocks outside the RFC 1918 range. To give HCX internet connectivity without DNAT, Amazon EVS uses a public HCX VLAN subnet whose CIDR block matches an internet-routable CIDR block from IPAM. Traffic for each associated Elastic IP address goes directly to the HCX appliance that has the same IP address.

The following diagram shows HCX internet connectivity.

![On-premises HCX connects over the internet to Elastic IP addresses on the public HCX VLAN subnet.](https://docs.aws.amazon.com/evs/latest/userguide/images/hcx-public-landscape.drawio.png)


#### CIDR and Elastic IP address rules for internet connectivity
<a name="plan-hcx-connectivity-rules"></a>

These rules apply only to HCX internet connectivity.

##### IPAM and the public CIDR block
<a name="plan-hcx-connectivity-rules-ipam"></a>
+ Create an IPAM and a public IPv4 IPAM pool in the public scope, with **Amazon-owned** as the public IP source. The pool locale must be the Region of your VPC.
+ Your IPAM quota for the netmask length of Amazon-provided contiguous public IPv4 CIDR blocks must allow a /28. For more information, see [Quotas for your IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/quotas-ipam.html).
+ Provision a CIDR block from the pool with a minimum netmask length of /28.
+ You can use the IPAM Free Tier for these resources. You still pay for other services that you use with IPAM, such as NAT gateways and public IPv4 addresses beyond the free tier. For more information, see the [Amazon VPC pricing page](https://aws.amazon.com/vpc/pricing).

##### VPC and the public HCX VLAN subnet
<a name="plan-hcx-connectivity-rules-vpc"></a>
+ Add the public IPv4 CIDR block to your VPC as an additional CIDR. You must enter the CIDR block manually. Amazon EVS doesn’t support an IPAM-allocated CIDR block for the VPC, and one can cause Elastic IP address association to fail.
+ When you create the environment, use the same public CIDR block for the HCX uplink VLAN subnet. Because you add the public CIDR block to the VPC as an additional CIDR, it meets the rule that every VLAN CIDR block is inside a CIDR block associated with the VPC.
+ The public HCX VLAN CIDR block must have a netmask length of exactly /28. Amazon EVS returns a validation error for any other size.
+ The first usable address of the CIDR block is the default gateway that you configure on your HCX appliances.
+ Configure route server propagation for both the service access subnet and the public subnet that you create with the VPC.

##### Elastic IP addresses
<a name="plan-hcx-connectivity-rules-eip"></a>
+ Allocate at least two Elastic IP addresses from the IPAM pool, one for the HCX Manager and one for the HCX Interconnect (HCX-IX) appliance. Allocate one more for each HCX network appliance that you deploy, for example HCX Network Extension (HCX-NE). HCX internet connectivity fails if fewer than two are associated with the HCX VLAN subnet.
+ You can associate up to 13 Elastic IP addresses with the /28 public HCX VLAN subnet. They must all come from the same IPAM pool.
+ You can’t associate the first two addresses or the last address of the public CIDR block. They are reserved as the network, default gateway, and broadcast addresses, and Amazon EVS returns a validation error if you try.
+ When you allocate the Elastic IP addresses, enter the addresses within the IPAM pool manually, so that the reserved addresses aren’t allocated.
+ Amazon EVS supports Elastic IP address associations only with the HCX VLAN subnet. You can associate and disassociate them after deployment. Each association has its own association ID.

##### Security
<a name="plan-hcx-connectivity-rules-security"></a>

**Important**  
Associating an Elastic IP address with a VLAN subnet gives direct internet access to all resources on that subnet. We strongly recommend that you create a custom network ACL dedicated to the HCX VLAN subnet, with rules that allow only the inbound and outbound connections that HCX needs. For the HCX port requirements, see the [VMware HCX User Guide](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-11.html) in the Broadcom documentation.

Security groups don’t apply to traffic on the HCX public VLAN subnet. Use network ACLs for traffic control. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

#### Requirements for both options
<a name="plan-hcx-connectivity-prereqs"></a>

Before you use HCX with Amazon EVS, make sure that you meet the following requirements.
+ Your network meets the HCX underlay requirements. For more information, see [Network Underlay Minimum Requirements](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-11/vmware-hcx-user-guide-4-11/preparing-for-hcx-installations/network-underlay-minimum-requirements.html) in the Broadcom documentation.
+ VMware NSX is installed and configured in the environment. For more information, see the [VMware NSX Installation Guide](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide.html) in the Broadcom documentation.
+ HCX is activated and installed in the environment. For more information, see [About Getting Started with VMware HCX](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-11/getting-started-with-vmware-hcx-4-11/about-getting-started-with-vmware-hcx.html) in the Broadcom documentation.

Record your HCX values in the [Deployment prerequisite checklist](#evs-deployment-prereq-checklist). For the steps to set up HCX internet connectivity, see [Configure HCX public internet connectivity](#evs-env-hcx-internet-access). To check your environment before migration, see [Prepare for HCX migration](#hcx-prepare).

## Plan costs and licenses
<a name="plan-cost"></a>

An Amazon Elastic VMware Service environment has charges from more than one AWS service. You also bring your own VMware Cloud Foundation (VCF) subscription from Broadcom. Plan all of them before you deploy, so that your estimate matches your bill. For current prices in each AWS Region, see [Amazon EVS pricing](https://aws.amazon.com/evs/pricing).

### Billing dimensions
<a name="plan-cost-dimensions"></a>

The following table lists the charges for an Amazon EVS environment. The first three apply to every environment.


| Charge | What it covers | 
| --- | --- | 
| Amazon EC2 instances | The bare metal instances that run as your ESX hosts, billed at standard EC2 rates for the instance type. This is usually the largest part of the cost. | 
| Amazon VPC Route Server endpoints | The route server endpoints that your environment uses for dynamic routing between NSX and the VPC. | 
| Amazon EVS control plane | The Amazon EVS service charge. The rate varies by AWS Region and doesn’t depend on the instance type of your hosts. | 
| Windows Server license included (optional) | The Windows Server entitlements that you create for virtual machines, billed hourly for each vCPU. The rate is the same in every AWS Region. For more information, see [Windows Server entitlements](#entitlements). | 
| External storage (optional) | Datastores on Amazon FSx or on another storage product, billed by that service. | 
| Capacity Reservations (optional) | Reserved EC2 capacity, billed from the time that the reservation becomes active, whether or not hosts use it. For more information, see [Reserve capacity for hosts](#plan-capacity). | 

VCF subscriptions aren’t billed by AWS. You buy them from Broadcom or a qualified reseller. For more information, see [VCF licensing](#vcf-license-mgmt).

### Estimate costs with the AWS Pricing Calculator
<a name="plan-cost-calculator"></a>

In the [AWS Pricing Calculator](https://calculator.aws/), the **Amazon EVS** line item covers only the Amazon EVS control plane. To estimate the full cost, also add the EC2 instances for your hosts and the route server endpoints. For the number of hosts, see [Size your environment](#plan-sizing).

### Lower the cost of hosts
<a name="plan-cost-savings"></a>

You can lower the cost of the EC2 instances that run as your hosts with Savings Plans and Reserved Instances. For the options and an example, see [Amazon EVS pricing](https://aws.amazon.com/evs/pricing).

### Choose a tenancy
<a name="plan-cost-tenancy"></a>

When you create the VPC for your environment, keep the VPC **Tenancy** setting as `Default`. For the steps, see [Create a VPC with subnets and route tables](#network-create-vpc).

### Common cost questions
<a name="plan-cost-questions"></a>

Stop hosts to save cost  
You can’t stop hosts to save cost, for example on weekends. Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. For more information, see [Host maintenance and retirement events](#evs-host-maintenance).

Buy VCF through AWS   
You can’t buy VCF through AWS. You must bring your own VCF subscription. You can get it from Broadcom or from a qualified reseller. For more information, see [VCF licensing](#vcf-license-mgmt).

Next, plan your VCF subscription. See [VCF licensing](#vcf-license-mgmt).

**Topics**

### VCF licensing
<a name="vcf-license-mgmt"></a>

Amazon Elastic VMware Service (Amazon EVS) uses VMware Cloud Foundation (VCF) subscriptions with license portability entitlements that you bring to AWS (BYOS). You must have a valid and active VCF subscription to use Amazon EVS, whichever way you deploy VCF.

**Note**  
Amazon EVS doesn’t support perpetual vSphere licenses.

Your VCF subscription is available to Amazon EVS across all AWS Regions for license compliance.

#### License keys by deployment mode
<a name="vcf-license-mgmt-by-mode"></a>

VCF 9.x environments are deployed using Evaluation Mode. A valid VCF subscription must be provided before the evaluation period expires.

What you do and don’t do with license keys depends on the deployment mode, as the following table shows.


| Deployment mode | You do | You don’t | 
| --- | --- | --- | 
| VCF 9.x (Self-deployed) | Provide a valid VCF subscription before the evaluation period expires. After you install VCF, add the license keys that your VCF deployment needs in your VCF management appliance, and manage them there. | Provide license keys when you create the environment. | 
| VCF 5.2.x (EVS-deployed) | Provide a VCF solution key and a vSAN license key in the environment creation request. You use the vSphere license key as the VCF solution key. Manage the keys in SDDC Manager. | Use a license key that is already in use in another Amazon EVS environment. | 

Amazon EVS doesn’t validate license keys. To validate a license key, use the [Broadcom support portal](https://support.broadcom.com/web/ecx).

#### Key requirements at environment creation
<a name="vcf-license-mgmt-creation"></a>

The requirements in this section apply when you provide license keys in the environment creation request, which you do for VCF 5.2.x.

Each VCF license key can be used for only one Amazon EVS environment. Environment creation fails if you use a VCF license key that is already in use in another environment.

Your VCF solution key must have enough cores for the four initial hosts that Amazon EVS deploys when it creates the environment. The required core count depends on the instance type that you select, because each instance type has a different number of cores.


| Instance type | Cores per host | Minimum cores for four hosts (VCF solution key) | 
| --- | --- | --- | 
|  `i4i.metal`  | 64 | 256 | 
|  `i7i.metal-24xl`  | 48 | 192 | 
|  `i7i.metal-48xl`  | 96 | 384 | 

The vSAN license key must meet the capacity requirement for the instance type that you select.


| Instance type | Minimum vSAN capacity for four hosts (vSAN license key) | 
| --- | --- | 
|  `i4i.metal`  | 110 TiB | 
|  `i7i.metal-24xl`  | 82 TiB | 
|  `i7i.metal-48xl`  | 164 TiB | 

Environment creation fails if you use undersized license keys. Your environment also fails to deploy if you provide an expired license key.

For more information about VCF license keys in VCF 5.2.x, see [Managing License Keys in VMware Cloud Foundation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/license-management-admin.html) in the Broadcom documentation.

#### Manage license keys in SDDC Manager
<a name="license-mgmt"></a>

You are responsible for managing your VCF subscriptions. In environments where Amazon EVS deploys VCF 5.2.x, you manage your VCF solution and vSAN license keys in SDDC Manager.

**Important**  
Amazon EVS requires that you maintain valid VCF solution and vSAN license keys in SDDC Manager for the service to function properly. You assign keys to your hosts and vSAN cluster in the vSphere Client. You must also make sure that those keys appear on the licensing screen of the SDDC Manager user interface.

If you remove your license keys from SDDC Manager, or replace them with a key that is in use elsewhere, the environment status check fails. A failed status check prevents you from adding hosts to your Amazon EVS environment. For more information, see [Monitor environment status](#evs-env-status-check) and [Troubleshoot environment status checks](#troubleshoot-status-checks).

#### Add VCF license keys
<a name="add-vcf-license"></a>

To license hosts that you add after the initial deployment, or to license more environments, do the following.

1. In the Broadcom support portal, purchase more VCF license keys, or split or merge existing license keys.

1. Add the purchased license keys to the vCenter Server and SDDC Manager inventory.

1. When you add hosts, assign the license keys to the correct hosts in vSphere. Make sure that the keys have enough cores and vSAN storage capacity.

1. Assign new license keys to vCenter Server before the evaluation period of the key expires, so that they stay active.

Amazon EVS doesn’t support unlicensed hosts.

For more information, see [Configuring License Settings for Assets in the vSphere Client](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/license-management/managing-licenses/configuring-license-settings-for-assets-in-the-vsphere-client.html#GUID-670D0552-5880-441C-AEEA-AE78C52CA075-en) and [Create a New License](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/license-management/managing-licenses/create-a-new-license.html) in the Broadcom documentation. If you have problems with license keys that you added, see [Troubleshoot environment status checks](#troubleshoot-status-checks).

#### Remove VCF license keys
<a name="remove-vcf-license"></a>

After you delete hosts, you can remove VCF license keys from the SDDC Manager inventory to reduce your core and vSAN capacity.
+ To stay in compliance with the licensing models of the products that you use with vSphere, remove all unassigned license keys from the inventory.
+ If you split, merged or upgraded license keys in the Broadcom support portal, remove the old license keys.

For more information, see [Remove a license](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/license-management/managing-licenses/remove-a-license.html) in the Broadcom documentation.

Next, plan security, support, and recovery. See [Plan security, support, and recovery](#plan-security-support).

## Plan security, support, and recovery
<a name="plan-security-support"></a>

Amazon Elastic VMware Service is a self-managed service, so you operate the VMware software and many of the safeguards around it. Make these decisions before you deploy, in the following order.

1.  [Shared responsibility and support](#plan-shared-responsibility). Learn what AWS manages and what you manage in each deployment mode, and where to get help.

1.  [Account permissions and guardrails](#plan-account-guardrails). Prepare the AWS Identity and Access Management (IAM) permissions and check the account policies that can block a deployment.

1.  [Plan for resilience and recovery](#plan-resilience). Design for availability in one Availability Zone, and plan backup and disaster recovery.

For the security controls of a running environment, see [Security in Amazon Elastic VMware Service](#security).

**Topics**
+ [Shared responsibility and support](#plan-shared-responsibility)
+ [Account permissions and guardrails](#plan-account-guardrails)
+ [Plan for resilience and recovery](#plan-resilience)

### Shared responsibility and support
<a name="plan-shared-responsibility"></a>

Amazon Elastic VMware Service is a self-managed service. AWS manages the infrastructure that Amazon EVS provisions, and you manage the VMware Cloud Foundation (VCF) software that runs on it.

#### Who manages what
<a name="plan-shared-responsibility-table"></a>

The following table shows who manages each area in each deployment mode. In the mode columns, AWS or You shows the owner.


| Area | VCF 9.x (Self-deployed) | VCF 5.2.x (EVS-deployed) | Notes | 
| --- | --- | --- | --- | 
| Amazon EVS control plane and the AWS resources that Amazon EVS provisions |  AWS  |  AWS  | Includes the VLAN subnets that Amazon EVS creates, the bare metal hosts and their firmware upgrades. You maintain the VPC networking resources that you create, such as route tables, network ACLs, route servers, and gateways. | 
| Initial VCF installation | You |  AWS  | In Self-deployed mode, you install VCF on the hosts that Amazon EVS creates. In EVS-deployed mode, Amazon EVS installs VCF when it creates the environment. | 
| VCF upgrades and patches (vCenter, NSX, SDDC Manager or VCF Operations, ESX) | You | You | For more information, see [Update and upgrade](#evs-lifecycle-mgmt). | 
| VCF authentication, role-based access control, and password rotation for the management appliances | You | You | Harden the VCF management network according to Broadcom guidance and the security requirements of your organization. | 
| Host health monitoring and host replacement | You | You | For more information, see [Host maintenance and retirement events](#evs-host-maintenance) and [Plan for resilience and recovery](#plan-resilience). | 
| VCF subscriptions | You | You | For more information, see [VCF licensing](#vcf-license-mgmt). | 
| Secrets for connectors in AWS Secrets Manager | You rotate them | You rotate them | Amazon EVS encrypts the credentials that you store in Secrets Manager with AWS KMS and restricts service access to them with resource tags. For more information, see [Manage secrets](#evs-env-secret-rotation). | 
| Connector health and aggregate environment health |  AWS  |  AWS  | Amazon EVS monitors the connectors that you create and reports the health of the environment. For more information, see [Monitor environment status](#evs-env-status-check). | 
| Security monitoring of the VCF deployment | You | You | For more information, see [Security in Amazon Elastic VMware Service](#security). | 
| VM-level high availability, replication, and backup | You | You | For more information, see [Plan for resilience and recovery](#plan-resilience). | 

Security follows the AWS shared responsibility model, which divides security of the cloud from security in the cloud. For more information, see [Security in Amazon Elastic VMware Service](#security).

You can also work with AWS Partners to manage your environment.

#### Get help from AWS Support and Broadcom
<a name="plan-shared-responsibility-support"></a>

Amazon EVS requires an AWS Business Support, AWS Enterprise On-Ramp, or AWS Enterprise Support plan. AWS Business Support is the minimum tier that meets the requirement. If you run business-critical workloads, we recommend AWS Enterprise On-Ramp or AWS Enterprise Support. The support plan gives you continuous access to technical support and architectural guidance. For more information, see [Compare AWS Support Plans](https://aws.amazon.com/premiumsupport/plans).

**Important**  
Environment creation fails if your account isn’t enrolled in an AWS Business Support, AWS Enterprise On-Ramp, or AWS Enterprise Support plan.

 AWS provides support for Amazon EVS and its associated infrastructure services, including VCF. For VCF configuration guidance, or for issues with other VMware products such as VCF Operations, HCX or NSX, you can also contact Broadcom with your Broadcom support entitlement. For more information, see the [Broadcom Support Portal](https://support.broadcom.com/).

For troubleshooting steps and how to open a case, see [Get help from AWS Support and Broadcom](#get-help).

### Account permissions and guardrails
<a name="plan-account-guardrails"></a>

Amazon Elastic VMware Service creates and manages resources in your AWS account, so the permissions and policies in that account decide whether a deployment succeeds. Prepare the AWS Identity and Access Management (IAM) permissions and check the guardrails on the account before you create an environment.

#### IAM permissions for administrators
<a name="plan-account-guardrails-iam"></a>

The people and automation that create and manage Amazon EVS environments need an IAM role with permissions for Amazon EVS operations. We recommend that you create a role for this purpose and grant only the actions that your administrators need. For the steps, see [Create an IAM role for Amazon EVS administrators](#setup-iam-role). For example policies, see [Amazon EVS identity-based policy examples](#security-iam-id-based-policy-examples).

For guidance on writing the policies, such as least privilege and conditions, see [Policy best practices](#security_iam_service-with-iam-policy-best-practices).

To use the Amazon EVS console, a principal also needs a minimum set of permissions to list and view Amazon EVS resources. For the policy, see [Using the Amazon EVS console](#security-iam-id-based-policy-examples-console).

Amazon EVS doesn’t support resource-based policies or access control lists (ACLs). To give principals in other accounts access to your Amazon EVS resources, use an IAM role. For more information, see [I want to give other accounts access to my Amazon EVS resources](#security-iam-troubleshoot-cross-account-access).

To view all Amazon EVS actions, resources and condition keys, see [Actions, resources, and condition keys for Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html) in the *Service Authorization Reference*.

**Note**  
Amazon EVS doesn’t support the use of an instance profile to pass an IAM role to an EC2 instance.

#### Service-linked role
<a name="plan-account-guardrails-slr"></a>

Amazon EVS uses the service-linked role `AWSServiceRoleForAmazonEVS` to call other AWS services on your behalf. When you create an environment, Amazon EVS creates the role for you if the IAM principal that creates the environment has the `iam:CreateServiceLinkedRole` permission. Make sure that the role that creates the first environment in the account has this permission. Before you can delete the role, you must delete the environments and hosts that use it.

For more information, see [Using service-linked roles for Amazon EVS](#using-service-linked-roles).

#### Service control policies
<a name="plan-account-guardrails-scp"></a>

If your account is in an organization in AWS Organizations, service control policies (SCPs) limit the permissions that IAM principals in the account can use. Review the SCPs that apply to the account before you deploy. Make sure that they allow the actions in your IAM policy for Amazon EVS and for the other services that your deployment uses.

For the actions that the example policies grant, see [Permissions for an Amazon EVS organizational unit](#security-iam-id-based-policy-examples-ou).

#### Allowed AMIs and other EC2 account settings
<a name="plan-account-guardrails-amis"></a>

Amazon EVS launches the hosts as Amazon EC2 bare metal instances in your account. If you use the Allowed AMIs setting in your account, your image criteria must include the Amazon EVS AMIs, or you can’t create an environment or add a host. For the criteria to use, see [EC2 Allowed AMIs (only if you restrict AMI usage)](#evs-prereq-allowed-amis).

Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. For more information, see [Host maintenance and retirement events](#evs-host-maintenance).

#### Other account requirements
<a name="plan-account-guardrails-other"></a>

Check the following account requirements before you deploy.

 **Support plan**   
Environment creation fails if your account isn’t enrolled in an AWS Business Support, AWS Enterprise On-Ramp, or AWS Enterprise Support plan. For more information, see [Shared responsibility and support](#plan-shared-responsibility).

 **Quotas**   
Environment creation fails if the Amazon EVS host quota is too low for your VMware Cloud Foundation (VCF) topology. For the quotas to check, see [Size your environment](#plan-sizing).

 **EC2 key pair**   
Amazon EVS uses an EC2 key pair that you provide to connect to your hosts. For more information, see [Create a key pair for your Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html) in the *Amazon EC2 User Guide*.

For the security controls of a running environment, including data protection and infrastructure security, see [Security in Amazon Elastic VMware Service](#security). For the compliance programs that apply to Amazon EVS, see [Compliance validation for Amazon EVS](#compliance-validation).

### Plan for resilience and recovery
<a name="plan-resilience"></a>

An Amazon Elastic VMware Service environment runs in one AWS Availability Zone, so your design must handle host failures inside that zone and plan recovery outside it. You configure most of the VMware availability features yourself, and you’re responsible for the high availability of your virtual machines.

#### Design for a single Availability Zone
<a name="plan-resilience-single-az"></a>

An Amazon EVS environment runs in one Availability Zone. All VPC subnets of an Amazon EVS environment must be in the same Availability Zone, and so must its hosts and any external storage. Choose the Availability Zone before you create the environment. For more information, see [VPC and subnets](#plan-network-supported-vpc).

The following items keep the environment available inside the Availability Zone.


| Item | What it does | 
| --- | --- | 
| Number of hosts | VMware Cloud Foundation (VCF) requires a minimum number of hosts to keep vSAN quorum and to stay available during maintenance and host failures. For more information, see [Size your environment](#plan-sizing). | 
| NSX Edge cluster | An Active/Standby NSX Edge cluster with its two nodes on different hosts allows a quick failover if one NSX Edge node fails. In EVS-deployed mode, Amazon EVS deploys this cluster. In Self-deployed mode, you deploy it. For more information, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge). | 
| EC2 placement groups | Amazon EVS supports partition and cluster placement groups for hosts. A partition placement group places groups of hosts on separate hardware, which reduces the chance of correlated hardware failures. A cluster placement group places hosts in the same rack for low latency. For more information, see [Placement strategies](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-strategies.html) in the *Amazon EC2 User Guide*. | 
| Spare host and Capacity Reservations | A spare host and reserved EC2 capacity let you replace a failed host without waiting for capacity. For more information, see [Reserve capacity for hosts](#plan-capacity). | 

You choose the placement group when you add a host, so plan it before you add the first host.

#### VMware features that you configure
<a name="plan-resilience-vcf"></a>

You’re responsible for configuring the VMware components that run on Amazon EVS for the availability of your virtual machines. The following VCF features are available to you.

 **vSphere HA**   
Restarts virtual machines on other hosts when a host fails. For more information, see [High Availability Design for vCenter Server for VMware Cloud Foundation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vcf-design-5-2/vsphere-design-for-vmware-cloud-foundation/vcf-vcenter-server-design.html#GUID-B4380728-190C-428E-83AE-3F7667506948-en) in the Broadcom documentation.

 **vSAN failures to tolerate (FTT)**   
Sets how many host failures a virtual machine can withstand before it becomes inaccessible. For more information, see [Tolerate Additional Failures with Fault Domain in vSAN Cluster](https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration/expanding-and-managing-a-vsan-cluster/managing-fault-domains-in-vsan-clusters/tolerate-additional-failures-with-fault-domains-in-vsan-cluster.html) in the Broadcom documentation.

 **vSAN data protection**   
Recovers virtual machines from operational failures or ransomware attacks with native snapshots on the vSAN cluster. For more information, see [Using vSAN Data Protection](https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration/expanding-and-managing-a-vsan-cluster/using-vsan-data-protection.html) in the Broadcom documentation.

 **vSphere Replication**   
Replicates virtual machines asynchronously for disaster recovery and migration. For more information, see [How vSphere Replication Works](https://techdocs.broadcom.com/us/en/vmware-cis/live-recovery/vsphere-replication/9-0-3/using-vsphere-replication/about-vmware-vsphere-replication/how-vsphere-replication-works.html) in the Broadcom documentation.

The vSAN features apply to hosts with local NVMe. Diskless hosts are available only with VCF 9.x. On diskless hosts, the availability of your datastores depends on the external storage that you choose. For more information, see [External storage options for diskless hosts](#external-storage-diskless).

Before you design for a VCF feature, check whether Amazon EVS supports it. For example, see the [vSphere Fault Tolerance](#feat-vsphere-fault-tolerance) row in [VCF feature support on Amazon EVS](#vcf-feature-support).

#### Plan disaster recovery
<a name="plan-resilience-dr"></a>

Because an environment runs in one Availability Zone, plan how you recover if the Availability Zone or the environment is unavailable. You can use vSphere Replication for disaster recovery of your virtual machines.

Multi-AZ external storage doesn’t keep an Amazon EVS environment running through an Availability Zone failure, because the environment itself runs in one Availability Zone. For example, with a Multi-AZ Amazon FSx for NetApp ONTAP file system, you build a new environment in the other Availability Zone. Then you mount the volumes there and register the virtual machines again.

#### Plan backup
<a name="plan-resilience-backup"></a>

You can use third-party backup tools that you already know with Amazon EVS, such as Veeam. Check that the backup product supports the VCF version that you deploy.

#### Plan for host failures
<a name="plan-resilience-hosts"></a>

You’re responsible for monitoring host health and replacing failed hosts. AWS schedules maintenance and retirement events for EC2 instances, and it notifies you in advance. Replace an affected host before the event, because a reboot of a bare metal instance doesn’t preserve instance store data.

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API.

For more information, see [Host maintenance and retirement events](#evs-host-maintenance) and [Replace a failed host (Self-deployed)](#host-replace).

## Deployment prerequisite checklist
<a name="evs-deployment-prereq-checklist"></a>

Use the following tables to collect the values that you need to deploy an Amazon Elastic VMware Service environment that runs VMware Cloud Foundation (VCF). The checklist has one table for each topic. Your environment deployment fails if a required value is missing or does not meet its requirement.

### How to use this checklist
<a name="evs-prereq-how-to-use"></a>

Use the checklist as follows.
+ Fill in one copy of the checklist for each environment. Write your values in the **Your value** column.
+ Check the **Mode** column. `Both` rows apply to Self-deployed and EVS-deployed environments. Rows marked `Self-deployed` or `EVS-deployed` apply only to that mode. For the differences between the modes, see [Differences between deployment modes](#deployment-mode-differences).
+ Complete the tables in order. In an EVS-deployed environment, you need every value before you create the environment. In a Self-deployed environment, you need some values later, and the table says when.
+ Decide the network values carefully. You can’t change the VPC CIDR block, add VLAN subnets, or change the VLAN subnet CIDR blocks after you create the environment. For sizing, see [Plan your VPC and VLAN subnets](#plan-network-sizing).
+ Use the example values only as a pattern. The examples use one consistent address plan, with the VPC CIDR block `10.1.0.0/20`.

### Account, Region, and capacity
<a name="evs-prereq-account"></a>

Collect the following account, Region, and capacity values.


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
|  AWS account ID | Both | Access to an AWS account. You create the VPC, the subnets, and the environment in this account. | 111122223333 |  | 
|  AWS Region | Both | A Region where Amazon EVS is available. For the list, see [AWS Regions for Amazon EVS](#regions-reference). | US West (Oregon) |  | 
|  AWS Support plan | Both | An AWS Business Support, AWS Enterprise On-Ramp, or AWS Enterprise Support plan. Environment creation fails if your account isn’t enrolled in one of these plans. |  AWS Business Support |  | 
| Service quotas | Both | A quota for the number of hosts in an environment that is high enough for your VCF topology. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs). | The number of hosts that you deploy |  | 
| Instance type | Both | An instance type that Amazon EVS supports for your VCF version. All hosts in a cluster use the same instance type. For more information, see [Choose instance types](#plan-instance-types). |  `i4i.metal`  |  | 
| Amazon EC2 Capacity Reservation | Both | A Capacity Reservation. We recommend one, so that capacity is available when you add hosts. For more information, see [Reserve capacity for hosts](#plan-capacity). | cr-0123456789abcdef0 |  | 

### External storage for diskless hosts
<a name="evs-prereq-diskless"></a>

This section applies to Self-deployed environments that run VCF 9.x.

Diskless instance types, such as `r7i.metal-24xl` and `r7i.metal-48xl`, have no local storage, so the VCF management domain and your workloads run on external storage. Collect the following value if you use diskless hosts.


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| External storage | Self-deployed | A datastore on external storage in the same VPC and Availability Zone as the environment, sized for the VCF management domain, that the VCF Installer accepts as principal storage. For more information, see [External storage options for diskless hosts](#external-storage-diskless). | Amazon FSx for OpenZFS, 7,168 GiB, NFS v3 |  | 

### VCF subscriptions
<a name="evs-prereq-licenses"></a>

You bring your own VCF subscription in both modes. When you provide the license keys depends on the mode. For more information, see [VCF licensing](#vcf-license-mgmt).


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| Broadcom Site ID | EVS-deployed | The Site ID that Broadcom gives you for the Broadcom support portal. You enter it when you create the environment. | 01234567 |  | 
| VCF solution key | EVS-deployed | A valid, active VCF license key for the VCF stack that no other Amazon EVS environment uses. You enter it when you create the environment. | ABCDE-FGHIJ-KLMNO-PQRSTU-VWXYZ |  | 
| vSAN license key | EVS-deployed | A valid, active vSAN license key that no other Amazon EVS environment uses. You enter it when you create the environment. | ABCDE-FGHIJ-KLMNO-PQRSTU-VWXYZ |  | 
| VCF license keys | Self-deployed | License keys that you add in your VCF management appliance after you install VCF. You don’t enter them when you create the environment. | ABCDE-FGHIJ-KLMNO-PQRSTU-VWXYZ |  | 

### VPC and service access subnet
<a name="evs-prereq-vpc"></a>

Collect the following VPC and service access subnet values.


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| VPC ID | Both | The VPC that Amazon EVS deploys the environment into. | vpc-0abcdef1234567890 |  | 
| VPC CIDR block | Both | An IPv4 CIDR block of /22 or larger, large enough for the service access subnet and all VLAN subnets. Use a CIDR block that is unique across your environments. Each environment needs its own VPC. You can’t change it after you create the environment. | 10.1.0.0/20 |  | 
| Service access subnet ID | Both | A standard VPC subnet that Amazon EVS uses for service access. For more information, see [Concepts and components of Amazon EVS](#concepts). | subnet-abcdef1234567890e |  | 
| Service access subnet CIDR block | Both | A CIDR block inside a CIDR block that is associated with the VPC. We recommend a /24 netmask. | 10.1.0.0/24 |  | 
| Availability Zone | Both | One Availability Zone for all VPC subnets, identified by its Availability Zone ID. For more information, see [Create a VPC with subnets and route tables](#network-create-vpc). | usw2-az1 |  | 

### VLAN subnets
<a name="evs-prereq-vlans"></a>

Amazon EVS creates these VLAN subnets when you create the environment, in both modes. Every CIDR block must have a netmask from /28 to /24 and must be inside a CIDR block that is associated with the VPC. It must not overlap with any other subnet in the VPC, including the other VLAN subnets and the service access subnet. You can’t add VLAN subnets or change their CIDR blocks after you create the environment. For sizing guidance, see [Plan your VPC and VLAN subnets](#plan-network-sizing).


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| Host management VLAN CIDR block | Both | /28 to /24. | 10.1.1.0/24 |  | 
| vMotion VLAN CIDR block | Both | The same size as the host management VLAN. | 10.1.2.0/24 |  | 
| vSAN VLAN CIDR block | Both | The same size as the host management VLAN. | 10.1.3.0/24 |  | 
| VTEP VLAN CIDR block | Both | The same size as the host management VLAN. | 10.1.4.0/24 |  | 
| Edge VTEP VLAN CIDR block | Both | /28 to /24. | 10.1.5.0/24 |  | 
| Management VM VLAN CIDR block | Both | /28 to /24. | 10.1.6.0/24 |  | 
| HCX uplink VLAN CIDR block | Both | /28 to /24 for a private VMware Hybrid Cloud Extension (HCX) VLAN. Exactly /28 for a public HCX VLAN, using the public CIDR block in [HCX internet connectivity (optional)](#evs-prereq-hcx). | 10.1.7.0/24 |  | 
| NSX uplink VLAN CIDR block | Both | /28 to /24. The route server peer IP addresses come from this CIDR block. | 10.1.8.0/24 |  | 
| Expansion VLAN 1 CIDR block | Both | /28 to /24. | 10.1.9.0/24 |  | 
| Expansion VLAN 2 CIDR block | Both | /28 to /24. | 10.1.10.0/24 |  | 

### DNS, NTP, and the DHCP option set
<a name="evs-prereq-dns-ntp"></a>

Amazon EVS reads the DNS servers, the domain name and the NTP servers from the DHCP option set of your VPC. For the configuration steps, see [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp).


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| DHCP option set ID | Both | A DHCP option set with at least two DNS servers, your DNS domain name, and an NTP server. You can use Amazon Route 53 or your own DNS servers. | dopt-0a1b2c3d |  | 
| Primary DNS server IP address | Both | A reachable DNS server that answers queries. The VPC main route table must have a route to it. For Route 53, see [Create private hosted zones and a Resolver inbound endpoint](#network-dhcp-dns-zones). | 10.1.0.10 |  | 
| Secondary DNS server IP address | Both | A second reachable DNS server. | 10.1.0.20 |  | 
| NTP server IP addresses | Both | At least one NTP server, and up to four, in the DHCP option set. Amazon EVS hosts use the Amazon Time Sync Service address `169.254.169.123`. For the VCF Installer, see [Time (NTP) settings](#installer-settings-ntp). | IPv4 address of your NTP server |  | 
| DNS domain name | Both | A valid, unexpired domain name. The FQDN of each host and VCF management appliance, including this domain name, must not exceed 62 characters. An FQDN can contain only letters, digits, the minus sign (-), and periods between labels. |  `evs.example.com`  |  | 

### DNS records for hosts and appliances
<a name="evs-prereq-dns-records"></a>

Each host and each VCF management appliance needs an A record in a forward lookup zone and a pointer (PTR) record in a reverse lookup zone. The records must resolve through the DNS servers in your DHCP option set, with no duplicate host names or IP addresses.
+ In an EVS-deployed environment, create all records before you create the environment.
+ In a Self-deployed environment, create the ESX host records before you add each host, and the appliance records before you install VCF. For more information, see [Step 2: Create DNS records](#self-deployed-host-dns-records).

We recommend host names of 15 characters or fewer, as a NetBIOS practice.


| Record | Mode | Example IP address | Example host name | Your value | 
| --- | --- | --- | --- | --- | 
| ESX host 1 | Both | 10.1.1.11 | esxi01 |  | 
| ESX host 2 | Both | 10.1.1.12 | esxi02 |  | 
| ESX host 3 | Both | 10.1.1.13 | esxi03 |  | 
| ESX host 4 | Both | 10.1.1.14 | esxi04 |  | 
| vCenter Server appliance | Both | 10.1.6.10 | vc01 |  | 
| NSX Manager cluster | Both | 10.1.6.11 | nsx |  | 
| NSX Manager 1 appliance | Both | 10.1.6.16 | nsx01 |  | 
| NSX Manager 2 appliance (three-node cluster only) | Both | 10.1.6.17 | nsx02 |  | 
| NSX Manager 3 appliance (three-node cluster only) | Both | 10.1.6.18 | nsx03 |  | 
| NSX Edge 1 appliance | Both | 10.1.6.14 | edge01 |  | 
| NSX Edge 2 appliance | Both | 10.1.6.15 | edge02 |  | 
| SDDC Manager appliance | EVS-deployed | 10.1.6.12 | sddcm01 |  | 
| Cloud Builder appliance | EVS-deployed | 10.1.6.13 | cb01 |  | 
| Other VCF management appliances for your VCF version | Self-deployed | An address in the Management VM VLAN subnet | The host name that you configure during VCF installation |  | 

ESX host records use addresses in the host management VLAN subnet. VCF management appliance records use addresses in the Management VM VLAN subnet.

### Route tables, network ACL, and transit gateway
<a name="evs-prereq-routing"></a>

Collect the following route table, network ACL, and transit gateway values.


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| Main route table ID | Both | A main route table with routes to your DNS servers and to any on-premises systems that the deployment needs. Amazon EVS associates the VLAN subnets with the main route table when it creates them. | rtb-0123456789abcdef0 |  | 
| Network ACL ID | Both | A network ACL that allows the traffic in [Required traffic](#evs-env-nacl-rules). | acl-0f62c640e793a38a3 |  | 
| Transit gateway ID | Both | A transit gateway, which you need to connect the environment to on-premises networks. For the supported connections, see [Supported network configurations](#plan-network-supported). | tgw-0262a0e521EXAMPLE |  | 
| On-premises connectivity method | Both |  Direct Connect, AWS Site-to-Site VPN, or both, with the transit gateway. For more information, see [Private IP AWS Site-to-Site VPN with Direct Connect](https://docs.aws.amazon.com/vpn/latest/s2svpn/private-ip-dx.html). | Site-to-Site VPN with Direct Connect |  | 

### Amazon VPC Route Server
<a name="evs-prereq-route-server"></a>

For the design rules, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design).
+ In an EVS-deployed environment, you enter two route server peers when you create the environment, and Amazon EVS configures Border Gateway Protocol (BGP) on the NSX Edge nodes.
+ In a Self-deployed environment, you don’t enter peers when you create the environment. You configure BGP between the Tier-0 gateway and the route server after you install VCF. For more information, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge).


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| Route server ID | Both | A route server associated with your VPC. | rs-0a1b2c3d4e5f67890 |  | 
| Amazon-side ASN | Both | An autonomous system number (ASN) in the range 1 to 4294967295. We recommend a private ASN, 64512 to 65534 (16-bit) or 4200000000 to 4294967294 (32-bit). | 65022 |  | 
| NSX Edge Tier-0 ASN | Both | An ASN that matches the peer ASN on every route server peer. The NSX Edge default is 65000. | 65000 |  | 
| Route server endpoint IDs | Both | Two endpoints, both in the service access subnet. | rse-0123456789abcdef0, rse-fedcba9876543210f |  | 
| Route server peer 1 ID and IP address | Both | A unique, unused address from the NSX uplink VLAN CIDR block (`PeerAddress`). In an EVS-deployed environment, Amazon EVS assigns it to NSX Edge 1. | rsp-0123456789abcdef0, 10.1.8.10 |  | 
| Route server peer 2 ID and IP address | Both | A unique, unused address from the NSX uplink VLAN CIDR block. In an EVS-deployed environment, Amazon EVS assigns it to NSX Edge 2. | rsp-fedcba9876543210f, 10.1.8.200 |  | 
| Additional route server peers | Self-deployed | Two more route server peers. We recommend four peers in a full mesh, so that each Edge node peers with both endpoints. | Two more peer IDs |  | 
| Route server propagation | Both | Route server propagation that is enabled on the route table associated with the service access subnet. Each propagated route table must have at least one explicit subnet association. | rtb-0123456789abcdef0 |  | 

### EC2 key pair and Allowed AMIs
<a name="evs-prereq-ec2"></a>

Collect the following EC2 key pair value.


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| EC2 key pair name | Both | A unique key pair name. Amazon EVS uses the key pair to connect to your hosts. |  `my-ec2-key-pair`  |  | 

#### EC2 Allowed AMIs (only if you restrict AMI usage)
<a name="evs-prereq-allowed-amis"></a>

If you use Allowed AMIs in your account, your image criteria must include the Amazon EVS AMIs, or you can’t create an environment or add a host. For more information, see [Control the discovery and use of AMIs in Amazon EC2 with Allowed AMIs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-allowed-amis.html) in the *Amazon EC2 User Guide*.


| Item | Mode | Requirement | Image criteria | Your value | 
| --- | --- | --- | --- | --- | 
| Allowed AMIs criteria | Both | Image criteria that include the Amazon EVS AMIs. |  `ImageProvider: amazon` `ImageName: AmazonEVS-*`  |  | 

### HCX internet connectivity (optional)
<a name="evs-prereq-hcx"></a>

You need these values only for HCX internet connectivity.

For the rules, see [Plan HCX connectivity](#plan-hcx-connectivity).

For the steps that produce these values, see [Part 1 of Configure HCX public internet connectivity](#hcx-internet-config). You complete Part 1 before you create the environment.


| Item | Mode | Requirement | Example value | Your value | 
| --- | --- | --- | --- | --- | 
| IPAM ID | Both | An Amazon VPC IP Address Manager (IPAM) that provides public IPv4 addresses. | ipam-0123456789abcdef0 |  | 
| IPAM pool ID | Both | A public IPv4 IPAM pool with Amazon-owned addresses, in the Region of your VPC. | ipam-pool-0123456789abcdef0 |  | 
| Public HCX VLAN CIDR block | Both | A /28 public IPv4 CIDR block from the IPAM pool. Add it to the VPC as an additional CIDR by entering it manually, and use it for the HCX uplink VLAN subnet. | 203.0.113.0/28 |  | 
| Elastic IP addresses | Both | At least two Elastic IP addresses from the same IPAM pool, for HCX Manager and HCX Interconnect (HCX-IX), plus one for each HCX network appliance. Up to 13. For the reserved addresses, see [Plan HCX connectivity](#plan-hcx-connectivity). | eipalloc-0123456789abcdef0, eipalloc-0123456789abcdef1, eipalloc-0123456789abcdef2 |  | 

When you have all the values, build the network. For more information, see [Build the network foundation](#network-foundation).

# Deploy
<a name="deploy-overview"></a>

You can deploy an Amazon Elastic VMware Service environment in three ways.

The procedures in [Deploy in Self-deployed mode](#getting-started-self-deployed) deploy VCF 9.x. Pages in [Deploy in EVS-deployed mode](#evs-deployed-overview) cover VCF 5.2.x. A page with no line and outside those sections covers both.

## Choose how to deploy
<a name="deploy-choose-path"></a>

Each path creates an Amazon EVS environment with hosts in your own Amazon VPC. The paths differ in the VMware Cloud Foundation (VCF) version and in who installs VCF.


| Path | VCF version | Who installs VCF | Use it when | Start here | 
| --- | --- | --- | --- | --- | 
| EVS Deployment Orchestrator (EDO), a tool in the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs)  | 9.x | EDO installs VCF. You launch one AWS CloudFormation stack, and it builds the network, adds the hosts, and installs VCF in Self-deployed mode. | You want a complete VCF environment without running each step yourself, and EDO supports your VCF version and instance type. |  [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo)  | 
| Manual, Self-deployed mode | 9.x | You install VCF. You use the VCF Installer on hosts that Amazon EVS provisions. | You want to control each step, or you plan to use settings that EDO doesn’t support, such as Amazon FSx for OpenZFS as the principal storage. |  [Deploy in Self-deployed mode](#getting-started-self-deployed)  | 
| EVS-deployed mode | 5.2.x | Amazon EVS installs VCF when you create the environment. | You deploy VCF 5.2.x. |  [Deploy in EVS-deployed mode](#evs-deployed-overview)  | 

For how the two modes differ after deployment, see [Differences between deployment modes](#deployment-mode-differences). For the VCF versions and instance types that each mode supports, see [Supported VCF versions and instance types](#versions-provided).

## Before you begin
<a name="deploy-prerequisites"></a>

Every path needs the following.
+ An AWS account that is ready for Amazon EVS, with a support plan, the AWS CLI, and an IAM role for administrators. For instructions, see [Prepare your AWS account](#setting-up).
+ A plan for your IP address space, DNS records, host count, and VCF subscriptions. For a list of the decisions to make first, see [Deployment prerequisite checklist](#evs-deployment-prereq-checklist).

We recommend that you complete the checklist before you create any resources, so that you have every value that the deployment steps ask for.

If you plan to use VMware HCX over the internet, also set up the IPAM pool and the public CIDR block before you create the environment. For the task list, see [Build the network foundation](#network-foundation).

## Order of the sections
<a name="deploy-section-order"></a>

For the manual paths, follow the sections in this order.

1.  [Prepare your AWS account](#setting-up).

1.  [Build the network foundation](#network-foundation). Both modes use the same VPC, DHCP option set, Amazon VPC Route Server, network ACL, and route table setup.

1. Deploy the environment in one mode. Follow [Deploy in Self-deployed mode](#getting-started-self-deployed) to install VCF yourself (the procedures deploy VCF 9.x), or [Deploy in EVS-deployed mode](#evs-deployed-overview) to have Amazon EVS install VCF 5.2.x.

1. (Optional) [Connect external storage](#external-storage-connect) after the environment is running.

If you use EDO, you still prepare your AWS account first. EDO then creates the network resources and runs the Self-deployed steps for you. You can also deploy EDO into an existing VPC. For more information, see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo).

To create Amazon EVS resources from templates instead of the console or AWS CLI, see [Automate your deployment](#deploy-automate).

## Deployment tools
<a name="deploy-tools"></a>

The following table lists the released open-source tools that help you deploy Amazon EVS environments.


| Tool | VCF version | What it does | Link | 
| --- | --- | --- | --- | 
| EVS Deployment Orchestrator (EDO) | 9.0.2 and 9.1.0 | Deploys a complete VCF environment on Amazon EVS from one AWS CloudFormation stack. |  [Deploy folder of the Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy)  | 

**Topics**

## Automate your deployment
<a name="deploy-automate"></a>

You can create Amazon Elastic VMware Service resources step by step in the Amazon EVS console or AWS CLI, or you can use a tool that runs the steps for you. Use the following table to choose the method that fits your deployment.


| Method | VCF version | Use it when | 
| --- | --- | --- | 
| EVS Deployment Orchestrator (EDO), a tool in the [Solutions for Amazon EVS](https://github.com/aws/solutions-for-amazon-evs) repository on GitHub | 9.x | You want one AWS CloudFormation stack to build the network, add the hosts, and install VMware Cloud Foundation (VCF) from end to end. On diskless hosts, EDO uses Amazon FSx for NetApp ONTAP as the principal storage. For the supported VCF versions, instance types, and storage, see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo). | 
|  AWS CloudFormation templates | All | You want to describe Amazon EVS resources, such as an environment, in a template and create them the same way in several accounts and Regions. For more information, see [Create Amazon EVS resources with AWS CloudFormation](#creating-resources-with-cloudformation). | 
| Amazon EVS console and AWS CLI  | All | You want to control and check each step, or you deploy a configuration that the other methods don’t cover. For instructions, see [Deploy in Self-deployed mode](#getting-started-self-deployed) or [Deploy in EVS-deployed mode](#evs-deployed-overview). | 

Whichever method you use, you first prepare your AWS account. For instructions, see [Prepare your AWS account](#setting-up).

**Topics**
+ [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo)
+ [Create Amazon EVS resources with AWS CloudFormation](#creating-resources-with-cloudformation)

### Deploy with the EVS Deployment Orchestrator
<a name="deploy-with-edo"></a>

This page applies to Self-deployed environments that run VCF 9.x. For how EVS-deployed environments (VCF 5.2.x) differ, see [Differences between deployment modes](#deployment-mode-differences).

The EVS Deployment Orchestrator (EDO) is an open-source tool in the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs), where Amazon Elastic VMware Service provides tooling and solutions that you can use. EDO deploys a complete VMware Cloud Foundation (VCF) environment on Amazon EVS in Self-deployed mode. You launch one AWS CloudFormation stack, and EDO runs the deployment without further input. EDO is in the `Deploy` folder of the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy) on GitHub.

For the parameters, the blueprint fields, the commands, and the troubleshooting steps, see the [EDO README](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy/EVS-Deployment-Orchestrator).

**Note**  
Before you use EDO, complete [Prepare your AWS account](#setting-up). EDO creates the VPC and the network resources, so you don’t need to complete [Build the network foundation](#network-foundation) first.

#### What EDO builds
<a name="edo-what-it-builds"></a>

The stack starts a small runner instance in your account. The runner does the deployment work and creates the following resources.
+ A VPC with the networking that Amazon EVS needs, including DNS, a NAT gateway, Amazon VPC Route Server, and a security group. You can also deploy into an existing VPC that meets the requirements in the README.
+ An Amazon EVS environment with ESX hosts on bare metal Amazon EC2 instances in your account. You choose the instance type and the host count in a blueprint file.
+ For diskless hosts, an Amazon FSx for NetApp ONTAP file system and the NFS datastore that the management cluster uses as principal storage. If you deploy into your own VPC, you provide the FSx for NetApp ONTAP volume instead.
+ VCF 9.x, installed and configured, with vCenter Server, NSX Manager, SDDC Manager, and VCF Operations.
+ An NSX Edge cluster that connects the NSX overlay network to your VPC.
+ Appliance credentials, stored in AWS Secrets Manager.

The deployment takes about 4 to 7 hours. To remove everything that EDO created, you run one teardown command from the repository. For more information, see [Clean up](#edo-clean-up).

#### Supported VCF versions and instance types
<a name="edo-supported-configurations"></a>

You set the VCF version and the instance type in the blueprint. The SDDC Manager Open Virtualization Appliance (OVA) file that you upload must match the VCF version. EDO supports the following combinations.


| Instance type | VCF versions | Principal storage | 
| --- | --- | --- | 
|  `i4i.metal`  | 9.0.2, 9.1.0, 9.1.1 | vSAN on local NVMe storage | 
|  `i7i.metal-24xl`  | 9.0.2, 9.1.0, 9.1.1 | vSAN on local NVMe storage | 
|  `i7i.metal-48xl`  | 9.0.2, 9.1.0, 9.1.1 | vSAN on local NVMe storage | 
|  `r7i.metal-24xl`  | 9.0.2, 9.1.0, 9.1.1 | Amazon FSx for NetApp ONTAP over NFS | 
|  `r7i.metal-48xl`  | 9.0.2, 9.1.0, 9.1.1 | Amazon FSx for NetApp ONTAP over NFS | 

The `i4i` and `i7i` instance types have local NVMe storage, and EDO deploys vSAN on it. The `r7i` instance types are diskless, so EDO uses Amazon FSx for NetApp ONTAP as the principal storage of the management cluster. You choose the storage in the `storage` section of the blueprint. When EDO creates the VPC, it also creates the FSx for NetApp ONTAP file system. When you deploy into your own VPC, you create the file system first and set the ID of its NFS volume in the blueprint. EDO doesn’t support other external storage, such as Amazon FSx for OpenZFS. To use it, follow the manual steps in [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless). For every VCF version and instance type that Amazon EVS supports, see [Supported VCF versions and instance types](#versions-provided). For the storage options for diskless hosts, see [External storage options for diskless hosts](#external-storage-diskless).

**Important**  
Billing for the bare metal hosts starts when EDO provisions them, about 50 minutes after the deployment starts. The runner instance is also billed to your account until you tear down the deployment. For prices, see [Amazon EVS pricing](https://aws.amazon.com/evs/pricing).

#### Before you begin
<a name="edo-prerequisites"></a>

Before you begin, make sure that the following are true.
+ You prepared your AWS account, including an AWS support plan that meets Amazon EVS requirements. For instructions, see [Prepare your AWS account](#setting-up).
+ You have a Broadcom download token for the Broadcom online depot from the [Broadcom Support Portal](https://support.broadcom.com), under your VCF entitlement.
+ You downloaded the VCF SDDC Manager OVA for VCF 9.0.2, 9.1.0, or 9.1.1 from the Broadcom Support Portal.
+ You downloaded the Linux zip file of VMware Open Virtualization Format (OVF) Tool from the Broadcom Support Portal.
+ Your computer has the OVA and OVF Tool files and enough disk space for them. If you use the AWS CLI instead of the console, your computer also has AWS CLI version 2 and Python 3.9 or later.
+ You chose an AWS Region and Availability Zone that offer your instance type. For more information about Regions, see [AWS Regions for Amazon EVS](#regions-reference).
+ For diskless hosts in your own VPC, you created an Amazon FSx for NetApp ONTAP file system in that VPC with an NFS volume whose export policy allows read and write access as root from the management subnet of the hosts. The README lists the export policy rules.
+ Your service quotas cover the deployment. The repository includes a read-only quota check script. For more information about Amazon EVS quotas, see [Service quotas for Amazon EVS](#service-quotas-evs).

#### Deploy with EDO
<a name="edo-procedure"></a>

Follow the [EDO README](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy/EVS-Deployment-Orchestrator) for the exact commands in Bash or PowerShell, and for the console steps.

1. Clone the Solutions for Amazon EVS repository and open the `Deploy/EVS-Deployment-Orchestrator` folder.

1. Store your Broadcom download token as a secret in AWS Secrets Manager.

1. Copy the example blueprint and set every field that the blueprint marks as required, including the VCF version, the instance type, and the host names. For diskless hosts, also set the `storage` section.

1. (Optional) Run the quota check script against your blueprint.

1. Upload the blueprint, the SDDC Manager OVA, and the OVF Tool zip file to an Amazon S3 bucket in your account.

1. Create a CloudFormation stack from the EDO template, and pass the S3 locations, the secret name, and the Availability Zone as parameters.

1. Open the logs link on the **Outputs** tab of the stack, and watch the progress in Amazon CloudWatch Logs.

The first stack completes in about 5 minutes. EDO then creates a second stack for the deployment infrastructure. Don’t delete the second stack by hand, because the teardown command removes it.

The deployment is complete when the log shows `ALL STAGES COMPLETE`. If you set an Amazon SNS topic at launch, EDO also publishes a success or failure notification to it.

You have deployed a VCF environment on Amazon EVS. The vCenter, NSX, and SDDC Manager interfaces are reachable only from inside the VPC. To connect to them and to get the appliance credentials, follow the "After deployment" section of the README.

#### Clean up
<a name="edo-clean-up"></a>

To stop the charges, remove everything that EDO created by running the teardown command that the [EDO README](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy/EVS-Deployment-Orchestrator) describes.

**Important**  
Run the teardown before you delete either CloudFormation stack. EDO creates the hosts through the Amazon EVS API, so if you delete the stacks first, the hosts keep running and you keep paying for them. The teardown script is `destroy.py` in the EDO folder.

#### Next steps
<a name="edo-next-steps"></a>
+ Check the hosts, the VCF appliances, and the connectivity. For instructions, see [Step 7: Verify the environment](#self-deployed-verify).
+ Add more storage. For more information, see [Connect external storage](#external-storage-connect).
+ Operate the environment. For more information, see [Operate](#operate-overview).
+ Remove the deployment when you no longer need it. For instructions, see [Clean up](#edo-clean-up).

### Create Amazon EVS resources with AWS CloudFormation
<a name="creating-resources-with-cloudformation"></a>

Amazon EVS is integrated with AWS CloudFormation, a service that helps you model and set up your AWS resources so that you can spend less time creating and managing your resources and infrastructure. You create a template that describes all the AWS resources that you want, an Amazon EVS environment for example, and AWS CloudFormation takes care of provisioning and configuring those resources for you.

When you use AWS CloudFormation, you can reuse your template to set up your Amazon EVS resources consistently and repeatedly. Just describe your resources once, and then provision the same resources over and over in multiple AWS accounts and Regions.

#### Amazon EVS and AWS CloudFormation templates
<a name="working-with-templates"></a>

To provision and configure resources for Amazon EVS and related services, you must understand [AWS CloudFormation templates](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-guide.html). Templates are formatted text files in JSON or YAML. These templates describe the resources that you want to provision in your AWS CloudFormation stacks.

Amazon EVS supports creating environments in AWS CloudFormation. For more information, including examples of JSON and YAML templates for your environments, see [Amazon EVS resource type reference](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/AWS_EVS.html) in the * AWS CloudFormation Template Reference*.

#### Learn more about AWS CloudFormation
<a name="learn-more-cloudformation"></a>

To learn more about AWS CloudFormation, see the following resources:
+  [AWS CloudFormation](https://aws.amazon.com/cloudformation/) 
+  [AWS CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) 
+  [AWS CloudFormation Command Line Interface User Guide](https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/what-is-cloudformation-cli.html) 

## Prepare your AWS account
<a name="setting-up"></a>

Before you build the network for Amazon Elastic VMware Service, complete these one-time tasks in the AWS account where you plan to deploy. The tasks are the same for both deployment modes. After you finish, continue with [Build the network foundation](#network-foundation).

Before you begin, review your network plan. For the VPC CIDR block size and the VLAN subnets, see [Plan your VPC and VLAN subnets](#plan-network-sizing). For a full list of requirements, see [Deployment prerequisite checklist](#evs-deployment-prereq-checklist).

Complete the following tasks in order. The sections after this list describe each task.

1.  [Sign up for an AWS account](#sign-up-for-aws) 

1.  [Sign up for a supported AWS Support plan](#setting-up-aws-business-support) 

1.  [Check quotas](#check-quotas) 

1.  [Reserve Amazon EC2 capacity](#ec2-future-capacity-reservation) 

1.  [Set up the AWS CLI](#set-up-cli) 

1.  [Create an Amazon EC2 key pair](#create-ec2-key-pair) 

1.  [Create an IAM role for Amazon EVS administrators](#setting-up-iam) 

1.  [Prepare for VMware Cloud Foundation](#setting-up-vcf) 

1.  [Acquire a VCF subscription and license keys](#setting-up-vcf-licensing) 

When you have completed these tasks, your account is ready for Amazon EVS. Next, build the network. For instructions, see [Build the network foundation](#network-foundation).

### Sign up for an AWS account
<a name="sign-up-for-aws"></a>

To get started with AWS, you need an AWS account. For information about creating an AWS account, see [Getting started with an AWS account](https://docs.aws.amazon.com/accounts/latest/reference/getting-started.html) in the * AWS Account Management Reference Guide*.

You must create the VPC, the subnets, and the Amazon EVS environment in the same AWS account. Amazon EVS does not support cross-account sharing of VPC subnets or Amazon EVS environments.

### Sign up for a supported AWS Support plan
<a name="setting-up-aws-business-support"></a>

Amazon EVS requires that you enroll in an AWS Business, AWS Enterprise On-Ramp, or AWS Enterprise Support plan. These plans give you continuous access to technical support and architectural guidance. AWS Business Support is the minimum AWS Support tier that meets Amazon EVS requirements. If you run business-critical workloads, we recommend AWS Enterprise On-Ramp or AWS Enterprise Support, because these plans add faster response times and architectural guidance. For more information, see [Compare AWS Support Plans](https://aws.amazon.com/premiumsupport/plans).

**Important**  
Amazon EVS environment creation fails if you do not sign up for an AWS Business, AWS Enterprise On-Ramp, or AWS Enterprise Support plan.

### Check quotas
<a name="check-quotas"></a>

Your account must have the required minimum account-level quotas before you create an environment. For the quotas and how to request an increase, see [Service quotas for Amazon EVS](#service-quotas-evs).

**Important**  
Amazon EVS environment creation fails if the host count per environment quota is too low for your VCF deployment topology. For host count requirements, see the Broadcom documentation for [VCF 9.x](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html) or [VCF 5.2.x](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2.html).

### Reserve Amazon EC2 capacity
<a name="ec2-future-capacity-reservation"></a>

Amazon EVS launches Amazon EC2 bare metal instances that become the ESX hosts in your environment. We recommend that you request an Amazon EC2 Capacity Reservation, so that capacity is available when you add hosts. You can request a Capacity Reservation for immediate use or for a future date. For more information, see [Reserve capacity for hosts](#plan-capacity) and [Reserve compute capacity with EC2 On-Demand Capacity Reservations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html) in the *Amazon EC2 User Guide*.

### Set up the AWS CLI
<a name="set-up-cli"></a>

The AWS Command Line Interface (AWS CLI) is a command line tool for working with AWS services, including Amazon EVS. You also use it to authenticate IAM users or roles for access to Amazon EVS and other AWS resources from your local machine. To use the AWS CLI, you get an access key ID and a secret access key, and then you configure these credentials in the AWS CLI. For instructions, see [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html) in the * AWS Command Line Interface User Guide for Version 2*.

### Create an Amazon EC2 key pair
<a name="create-ec2-key-pair"></a>

Amazon EVS uses an Amazon EC2 key pair to connect to your hosts. You provide it when you create the environment in EVS-deployed mode, and when you add a host in Self-deployed mode. To create a key pair, see [Create a key pair for your Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html) in the *Amazon EC2 User Guide*.

### Create an IAM role for Amazon EVS administrators
<a name="setting-up-iam"></a>

Create an IAM role with a permissions policy that allows Amazon EVS operations, and let your administrators assume it. For instructions, see [Create an IAM role for Amazon EVS administrators](#setup-iam-role).

### Prepare for VMware Cloud Foundation
<a name="setting-up-vcf"></a>

Your environment must meet the VMware Cloud Foundation (VCF) infrastructure requirements for the VCF version that you deploy. For the VCF versions that Amazon EVS provides in each mode, see [Supported VCF versions and instance types](#versions-provided). For the VCF planning requirements, see the Planning and Preparation Workbook in the Broadcom documentation for [VCF 9.x](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/planning-and-preparation.html) or [VCF 5.2.x](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/planning-and-preparation-workbook-5-2.html).

If you plan to use diskless hosts, which have no local storage, you must provide external storage. The storage must be in the same VPC and Availability Zone as your environment. Diskless hosts are available only with VCF 9.x in Self-deployed mode. For the VCF versions and instance types that support diskless hosts, see [Supported VCF versions and instance types](#versions-provided). For storage options, requirements, and sizing, see [External storage options for diskless hosts](#external-storage-diskless).

If you plan to migrate workloads with VMware HCX, see [Plan HCX connectivity](#plan-hcx-connectivity) before you build the network. HCX internet connectivity needs a public IPAM pool and Elastic IP addresses that you prepare in advance.

### Acquire a VCF subscription and license keys
<a name="setting-up-vcf-licensing"></a>

You bring your own VCF subscription to run VCF on Amazon EVS, in both deployment modes. When and where you provide the license keys depends on the deployment mode and the instance type. For the license requirements and the steps for each mode, see [VCF licensing](#vcf-license-mgmt).

**Topics**

### Create an IAM role for Amazon EVS administrators
<a name="setup-iam-role"></a>

Create an IAM role with a permissions policy that allows Amazon Elastic VMware Service operations, so that your administrators use temporary credentials instead of long-term keys. You can use the IAM console or the AWS CLI.

Administrators assume the role by calling the AWS Security Token Service (AWS STS) `AssumeRole` API operation, which returns temporary security credentials. For more information, see [Create a role to delegate permissions to an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) in the * AWS Identity and Access Management User Guide*.

**Note**  
Amazon EVS does not support the use of an instance profile to pass an IAM role to an EC2 instance.

#### Before you begin
<a name="setup-iam-role-prereqs"></a>

Make sure that the following are true.
+ You’re signed in as an IAM principal that has permission to create IAM policies and roles and to attach policies to roles (`iam:CreatePolicy`, `iam:CreateRole`, and `iam:AttachRolePolicy`).
+ You’ve chosen the permissions for the policy. For an example policy for creating and managing an environment, see [Amazon EVS identity-based policy examples](#security-iam-id-based-policy-examples). For all Amazon EVS actions, resources, and condition keys, see [Actions, resources, and condition keys for Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html) in the *Service Authorization Reference*.
+ You know the AWS account ID of the account whose users assume the role.
+ Your IAM principal also has the `iam:CreateServiceLinkedRole` permission, so that Amazon EVS can create its service-linked role the first time that you create an environment. For more information, see [Using service-linked roles for Amazon EVS](#using-service-linked-roles).

#### Create the policy and the role
<a name="setup-iam-role-steps"></a>

You can create the policy and the role in the IAM console or with the AWS CLI. Choose the tab for the method that you use.

**Example**  

1. Open the [IAM console](https://console.aws.amazon.com/iam), choose **Policies** in the navigation pane, and then choose **Create policy**.

1. In the policy editor, enter a permissions policy that allows Amazon EVS operations.

1. Choose **Next**. Under **Policy name**, enter a name that identifies the policy, and review the permissions that the policy defines.

1. (Optional) Add tags to help identify, organize, or search for the policy, and then choose **Create policy**.

1. In the navigation pane, choose **Roles**, and then choose **Create role**.

1. For **Trusted entity type**, choose ** AWS account **.

1. Under **An AWS account **, specify the account whose users will perform Amazon EVS actions, and then choose **Next**.

1. On the **Add permissions** page, select the permissions policy that you created, and then choose **Next**.

1. Under **Role name**, enter a name that identifies the role.

1. Review the trust policy and confirm that the correct AWS account is listed as the principal.

1. (Optional) Add tags, and then choose **Create role**.

1. Verify the result. Choose the role, and confirm that the **Permissions** tab lists your policy and the **Trust relationships** tab lists the trusted account.

1. Copy the following contents to a trust policy JSON file. For the principal ARN, replace the example AWS account ID and `service-user` name with your own AWS account ID and IAM user name.

   ```json
   {
     "Version":"2012-10-17",		 	 	 
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::123456789012:user/service-user" 
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   ```

1. Create the role. Replace `evs-environment-role-trust-policy.json` with your trust policy file name.

   ```bash
   aws iam create-role \
     --role-name myAmazonEVSEnvironmentRole \
     --assume-role-policy-document file://evs-environment-role-trust-policy.json
   ```

1. Save your permissions policy to a JSON file, for example `evs-admin-policy.json`, and create the policy.

   ```bash
   POLICY_ARN=$(aws iam create-policy \
     --policy-name myAmazonEVSAdminPolicy \
     --policy-document file://evs-admin-policy.json \
     --query Policy.Arn \
     --output text)
   ```

   The command saves the policy ARN that `create-policy` returns in `POLICY_ARN`, so the next step uses the correct ARN in every partition.

1. Attach the policy to the role.

   ```bash
   aws iam attach-role-policy \
     --policy-arn $POLICY_ARN \
     --role-name myAmazonEVSEnvironmentRole
   ```

1. Verify that the policy is attached to the role.

   ```bash
   aws iam list-attached-role-policies \
     --role-name myAmazonEVSEnvironmentRole
   ```

   The response lists `myAmazonEVSAdminPolicy` under `AttachedPolicies`.

You have created an IAM role that your administrators can assume to work with Amazon EVS.

To troubleshoot access errors, see [Troubleshoot Amazon EVS identity and access](#security-iam-troubleshoot).

Next, build the network for your environment. See [Build the network foundation](#network-foundation).

## Build the network foundation
<a name="network-foundation"></a>

Amazon Elastic VMware Service deploys your environment into a VPC that you create and configure. Both deployment modes use the same network foundation, so you complete these tasks once before you follow the Self-deployed or the EVS-deployed path.

Before you start, plan your CIDR blocks and VLAN subnets. For more information, see [Plan your VPC and VLAN subnets](#plan-network-sizing).

Complete the tasks in the following table in order. Each task uses resources from the task before it. The **When** column shows which tasks you complete before you create the environment, after you create it, and after you delete it.


| Task | What it produces | When | 
| --- | --- | --- | 
|  [Create a VPC with subnets and route tables](#network-create-vpc)  | A VPC, a service access subnet, an optional public subnet, route tables, and the gateway routes on the main route table. | Before you create the environment. | 
|  [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp)  | Optional private hosted zones and a Resolver inbound endpoint, a DHCP option set with DNS servers, a domain name, and NTP servers, the DNS record requirements, and an optional instance to reach the VPC. | Before you create the environment. | 
|  [Set up VPC Route Server and on-premises connectivity](#network-route-server)  | A route server in Amazon VPC Route Server with two endpoints, peers, and propagation, and optional on-premises connectivity through a transit gateway. | Before you create the environment. Update the transit gateway route tables after you create it. | 
|  [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong)  | A network ACL with the rules that the VLAN subnets need. | Before you create the environment. Associate the network ACL with the VLAN subnets after you create the environment. Update the rules when your traffic rules change. | 
|  [Part 1 of Configure HCX public internet connectivity](#hcx-internet-config) (optional) | An IPAM, a public IPv4 pool, Elastic IP addresses, and a public CIDR block in your VPC for the public HCX VLAN subnet. You need them only if you use VMware HCX over the internet. | Before you create the environment, because you can’t change VLAN CIDR blocks later. Associate the Elastic IP addresses after you create the environment, as described in Part 2 of the same topic. | 
|  [Associate VLAN subnets with a route table](#evs-env-config-custom-rt)  | Explicit route table associations for every VLAN subnet. | After you create the environment. | 
|  [Clean up network resources](#cleanup)  | No new resources. Deletes the network resources that you created. | After you delete the environment. | 

When the network foundation is ready, continue with your deployment mode:
+ For VMware Cloud Foundation (VCF) 9.x, see [Deploy in Self-deployed mode](#getting-started-self-deployed).
+ For VCF 5.2.x, see [Deploy in EVS-deployed mode](#evs-deployed-overview).

**Topics**
+ [Create a VPC with subnets and route tables](#network-create-vpc)
+ [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp)
+ [Set up VPC Route Server and on-premises connectivity](#network-route-server)
+ [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong)
+ [Associate VLAN subnets with a route table](#evs-env-config-custom-rt)
+ [Clean up network resources](#cleanup)

### Create a VPC with subnets and route tables
<a name="network-create-vpc"></a>

Create the VPC that Amazon Elastic VMware Service deploys your environment into, with a private subnet for service access and route tables with the gateway routes that deployment needs. You can use the Amazon VPC console or the AWS CLI.

**Important**  
You can’t change the VPC CIDR block after you create the environment. The VPC CIDR block must have a minimum size of /22 and must leave room for the VLAN subnets that Amazon EVS creates and for future growth. For sizing guidance, see [Plan your VPC and VLAN subnets](#plan-network-sizing) and [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference).

#### Before you begin
<a name="network-create-vpc-prereqs"></a>

Make sure that the following are true.
+ You plan to create the VPC, the subnets, and the Amazon EVS environment in the same AWS account. Amazon EVS doesn’t support cross-account sharing of VPC subnets or environments.
+ You’ve chosen an AWS Region where Amazon EVS is available. For more information, see [AWS Regions for Amazon EVS](#regions-reference).
+ You’ve chosen one Availability Zone. An Amazon EVS environment runs in one Availability Zone. Identify the zone by its Availability Zone ID, for example `usw2-az1`, because zone names map to different zone IDs in different accounts. Create every subnet of the VPC in this zone. For more information, see [VPC and subnets](#plan-network-supported-vpc).
+ You’ve chosen the IPv4 CIDR block for the VPC and for each subnet. Amazon EVS doesn’t support IPv6.
+ You’ve decided whether you need a NAT gateway. In Self-deployed mode you must create one, because the VMware Cloud Foundation (VCF) Installer downloads the VCF software from the Broadcom online depot over the internet. In EVS-deployed mode, you need one only if your environment needs outbound internet access. A NAT gateway has a cost. For more information, see [Pricing for NAT gateways](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-pricing.html).

The private subnet that you create is the service access subnet that you provide to Amazon EVS when you create the environment. For more information, see [Concepts and components of Amazon EVS](#concepts).

#### Create the VPC and subnets
<a name="network-create-vpc-steps"></a>

You can create the VPC and subnets in the Amazon VPC console or with the AWS CLI. Choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc), and on the VPC dashboard choose **Create VPC**.

1. For **Resources to create**, choose **VPC and more**. Keep **Name tag auto-generation** selected to create Name tags, or clear it to enter your own Name tags.

1. For **IPv4 CIDR block**, enter the IPv4 CIDR block that you chose for the VPC.

1. Keep **Tenancy** as `Default`. EC2 instances that launch into this VPC then use the tenancy attribute that is specified at launch. Amazon EVS launches bare metal EC2 instances on your behalf.

1. For **Number of Availability Zones (AZs)**, choose **1**. Expand **Customize AZs** and choose the Availability Zone for your subnets.

1. For **Number of public subnets**, choose **1**. The NAT gateway needs this public subnet. In EVS-deployed mode without outbound internet access, choose **0**.

1. For **Number of private subnets**, choose **1**. This subnet is the service access subnet.

1. Expand **Customize subnets CIDR blocks** and enter the IP address ranges for your subnets. Leave enough space in the VPC CIDR block for the VLAN subnets that Amazon EVS creates.

1. For **NAT gateways**, choose **In 1 AZ**. Self-deployed mode requires the NAT gateway. In EVS-deployed mode, choose it only if you need outbound internet access over IPv4.

1. For **VPC endpoints**, choose **None**. Amazon EVS doesn’t support gateway VPC endpoints for Amazon S3. To connect to Amazon S3, set up an interface VPC endpoint. For more information, see [AWS PrivateLink for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html) in the * Amazon S3 User Guide*.

1. For **DNS options**, keep the defaults selected, because Amazon EVS requires DNS resolution for all VCF components. (Optional) To add a tag, expand **Additional tags**, choose **Add new tag**, and enter a tag key and value.

1. Choose **Create VPC**.

    Amazon VPC creates a main route table and implicitly associates the subnets with it.

1. Create the VPC and store its ID for the following commands. Replace the example addresses and names with your own values.

   ```bash
   VPC_ID=$(aws ec2 create-vpc \
     --cidr-block 10.10.0.0/16 \
     --instance-tenancy default \
     --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=evs-vpc}]' \
     --query 'Vpc.VpcId' \
     --output text)
   ```

1. Enable DNS hostnames and DNS support.

   ```bash
   aws ec2 modify-vpc-attribute \
     --vpc-id $VPC_ID \
     --enable-dns-hostnames
   aws ec2 modify-vpc-attribute \
     --vpc-id $VPC_ID \
     --enable-dns-support
   ```

1. Create the private subnet (the service access subnet) and store its ID.

   ```bash
   PRIVATE_SUBNET_ID=$(aws ec2 create-subnet \
     --vpc-id $VPC_ID \
     --cidr-block 10.10.200.0/24 \
     --availability-zone-id usw2-az1 \
     --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=evs-private-subnet}]' \
     --query 'Subnet.SubnetId' \
     --output text)
   ```

1. Create a public subnet for the NAT gateway and store its ID. Self-deployed mode requires the NAT gateway. In EVS-deployed mode, you can skip this step and the next two if you don’t need outbound internet access.

   ```bash
   PUBLIC_SUBNET_ID=$(aws ec2 create-subnet \
     --vpc-id $VPC_ID \
     --cidr-block 10.10.201.0/24 \
     --availability-zone-id usw2-az1 \
     --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=evs-public-subnet}]' \
     --query 'Subnet.SubnetId' \
     --output text)
   ```

1. Create an internet gateway and attach it to the VPC.

   ```bash
   IGW_ID=$(aws ec2 create-internet-gateway \
     --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=evs-igw}]' \
     --query 'InternetGateway.InternetGatewayId' \
     --output text)
   
   aws ec2 attach-internet-gateway \
     --vpc-id $VPC_ID \
     --internet-gateway-id $IGW_ID
   ```

1. Allocate an Elastic IP address and create a NAT gateway in the public subnet.

   ```bash
   EIP_ID=$(aws ec2 allocate-address \
     --domain vpc \
     --tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=Name,Value=evs-nat-eip}]' \
     --query 'AllocationId' \
     --output text)
   
   NAT_GW_ID=$(aws ec2 create-nat-gateway \
     --subnet-id $PUBLIC_SUBNET_ID \
     --allocation-id $EIP_ID \
     --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=evs-nat}]' \
     --query 'NatGateway.NatGatewayId' \
     --output text)
   ```

   The NAT gateway must be in the `available` state before you create a route that uses it. To check, run `aws ec2 describe-nat-gateways --nat-gateway-ids $NAT_GW_ID --query 'NatGateways[0].State'`.

1. Create a private route table and a public route table, and store their IDs.

   ```bash
   PRIVATE_RT_ID=$(aws ec2 create-route-table \
     --vpc-id $VPC_ID \
     --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=evs-private-rt}]' \
     --query 'RouteTable.RouteTableId' \
     --output text)
   
   PUBLIC_RT_ID=$(aws ec2 create-route-table \
     --vpc-id $VPC_ID \
     --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=evs-public-rt}]' \
     --query 'RouteTable.RouteTableId' \
     --output text)
   ```

1. Add the default routes. The public route table routes to the internet gateway, and the private route table routes to the NAT gateway.

   ```bash
   aws ec2 create-route \
     --route-table-id $PUBLIC_RT_ID \
     --destination-cidr-block 0.0.0.0/0 \
     --gateway-id $IGW_ID
   
   aws ec2 create-route \
     --route-table-id $PRIVATE_RT_ID \
     --destination-cidr-block 0.0.0.0/0 \
     --nat-gateway-id $NAT_GW_ID
   ```

1. Associate the route tables with your subnets.

   ```bash
   aws ec2 associate-route-table \
     --route-table-id $PRIVATE_RT_ID \
     --subnet-id $PRIVATE_SUBNET_ID
   
   aws ec2 associate-route-table \
     --route-table-id $PUBLIC_RT_ID \
     --subnet-id $PUBLIC_SUBNET_ID
   ```

    Amazon VPC also creates a main route table for the VPC. Subnets without an explicit association use it.

You have created the VPC, the service access subnet, and the route tables.

#### Configure the main route table
<a name="vpc-main-rt"></a>

During environment deployment, the Amazon EVS VLAN subnets are implicitly associated with the VPC main route table. For deployment to succeed, the main route table must route traffic to dependent services such as your DNS servers and on-premises systems. The main route table must also include a route for the VPC CIDR block.

**Important**  
Don’t use a custom route table for the VLAN subnets while you create the environment, because this can cause connectivity issues. Amazon EVS supports custom route tables only after the environment is created. After deployment, you must associate each VLAN subnet explicitly. For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).

##### Gateway route requirements
<a name="vpc-main-rt-reqs"></a>

Add routes on the main route table for the gateways that your connectivity needs.


| Gateway | Required | Route rules | 
| --- | --- | --- | 
| NAT gateway | Yes in Self-deployed mode, for the Broadcom online depot. In EVS-deployed mode, only for outbound internet access. | Place the NAT gateway in a public subnet with internet gateway access. Add routes from the private subnets and the Amazon EVS VLAN subnets to the NAT gateway. For more information, see [Work with NAT gateways](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-working-with.html). | 
| Transit gateway | For on-premises connectivity through Direct Connect or AWS Site-to-Site VPN | Add routes for your on-premises network ranges. Configure route propagation if you use Border Gateway Protocol (BGP). For more information, see [Set up VPC Route Server and on-premises connectivity](#network-route-server) and [Transit gateways in Amazon VPC Transit Gateways](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-transit-gateways.html). | 

To add a route in the Amazon VPC console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc) and choose **Route tables** in the navigation pane.

1. Select the route table for your VPC where **Main** is **Yes**.

1. On the **Routes** tab, choose **Edit routes**, and then choose **Add route**.

1. Enter the destination CIDR block and choose the target gateway.

1. Choose **Save changes**.

1. Find the ID of the main route table. If `$VPC_ID` isn’t set in your shell, replace it in the following commands with the ID of your VPC.

   ```bash
   MAIN_RT_ID=$(aws ec2 describe-route-tables \
     --filters Name=vpc-id,Values=$VPC_ID Name=association.main,Values=true \
     --query 'RouteTables[0].RouteTableId' \
     --output text)
   ```

1. Add a route. This example sends the default route to the NAT gateway. If `$NAT_GW_ID` isn’t set in your shell, replace it in the following commands with the ID of your NAT gateway.

   ```bash
   aws ec2 create-route \
     --route-table-id $MAIN_RT_ID \
     --destination-cidr-block 0.0.0.0/0 \
     --nat-gateway-id $NAT_GW_ID
   ```

You have added the routes to the main route table.

We recommend that you document your route table configurations, use consistent names, and test connectivity after each change, so that you can find routing problems quickly. For more information, see [Configure route tables](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html) in the * Amazon VPC User Guide*.

Next, configure DNS and NTP for the VPC. See [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp).

### Configure DNS and NTP with a DHCP option set
<a name="network-dhcp-dns-ntp"></a>

Amazon Elastic VMware Service reads your VPC DHCP option set. The option set provides the DNS servers for host IP address resolution, the domain name for DNS resolution, and the NTP servers for time synchronization. Create a DHCP option set with these values and associate it with your VPC, using the Amazon VPC console or the AWS CLI.

Your environment deployment fails if you don’t meet the following requirements.
+ The DHCP option set includes a primary DNS server IP address and a secondary DNS server IP address.
+ A DNS forward lookup zone has A records for each VMware Cloud Foundation (VCF) management appliance and Amazon EVS host in your deployment.
+ A DNS reverse lookup zone has PTR records for each VCF management appliance and Amazon EVS host in your deployment.
+ The VPC main route table has a route to your DNS servers.
+ Your domain name registration is valid and unexpired, and no duplicate host names or IP addresses exist.
+ Your security groups and network ACLs allow the traffic in [Required traffic](#evs-env-nacl-rules).

#### DNS server requirements
<a name="getting-started-config-dns"></a>

Both DNS server IP addresses must be reachable and must respond to queries when the Amazon EVS connector launches. All required forward A records and reverse PTR records must resolve through those servers at that time. For the host names and records that each mode needs, see [Step 2: Create DNS records](#self-deployed-host-dns-records) for Self-deployed mode and [Deployment prerequisite checklist](#evs-deployment-prereq-checklist) for EVS-deployed mode.

##### DNS for on-premises connectivity
<a name="getting-started-config-dns-on-prem"></a>

For on-premises connectivity, we recommend Route 53 private hosted zones with inbound resolver endpoints. This setup gives you hybrid DNS resolution, so resources in your VPC resolve names hosted on your on-premises network, and the other way around. You can also use your own DNS server with Route 53 outbound resolver endpoints. Using both Route 53 and a custom DNS server in the DHCP option set can cause unexpected behavior. For instructions, see [Creating a private hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zone-private-creating.html) and [Forwarding inbound DNS queries to your VPC](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-forwarding-inbound-queries.html) in the *Amazon Route 53 Developer Guide*.

**Important**  
If you use custom DNS domain names in a Route 53 private hosted zone, you must set the `enableDnsHostnames` and `enableDnsSupport` VPC attributes to `true`. The same applies if you use private DNS with interface VPC endpoints (AWS PrivateLink). For more information, see [DNS attributes for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#vpc-dns-support).

#### NTP server requirements
<a name="getting-started-config-ntp"></a>

A consistent and accurate time reference is required for many VCF tasks, including system logging and auditing, security operations, distributed system management, and troubleshooting. You can enter the IPv4 addresses of up to four NTP servers in the DHCP option set.

Amazon EVS hosts use the link-local Amazon Time Sync Service address `169.254.169.123` as their NTP source. The VCF Installer specification for diskless hosts in this guide uses `time.aws.com`. Diskless hosts are available only with VCF 9.x. For the NTP validation warnings that this causes, see [Validate and deploy](#getting-started-diskless-deploy).

For more information about NTP options in DHCP, see [RFC 2132](https://datatracker.ietf.org/doc/html/rfc2132#section-8.3). For time synchronization on EC2 instances, see [Precision clock and time synchronization in your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html) in the *Amazon EC2 User Guide*.

#### Create private hosted zones and a Resolver inbound endpoint
<a name="network-dhcp-dns-zones"></a>

If you don’t have DNS servers that the VPC can reach, use Route 53 private hosted zones. Create the zones and the endpoint before the DHCP option set, because the endpoint addresses become the DNS servers in the option set. You create the DNS records later. For Self-deployed mode, see [Step 2: Create DNS records](#self-deployed-host-dns-records). For EVS-deployed mode, see [Deployment prerequisite checklist](#evs-deployment-prereq-checklist).

1. Create a Route 53 private hosted zone for your domain name, for example `evs.example.com`, and associate it with your VPC. This zone is the forward lookup zone.

1. Create a second private hosted zone for reverse lookup, and associate it with your VPC. The zone must cover the address range of your Amazon EVS VLAN subnets, for example `10.10.in-addr.arpa` for 10.10.0.0/16.

1. Create a security group in your VPC that allows inbound TCP and UDP port 53 from the VPC CIDR block and from any network that queries the endpoint.

1. Create a Route 53 Resolver inbound endpoint in the service access subnet with two IP addresses in that subnet, and the security group from the previous step. The two addresses can be in the same subnet, so a Single-AZ VPC doesn’t need a second Availability Zone for the endpoint.

1. Wait until the endpoint status is operational, and note its two IP addresses.

Use the two endpoint IP addresses as the primary and secondary DNS servers in the DHCP option set. For instructions, see [Creating a private hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zone-private-creating.html) and [Forwarding inbound DNS queries to your VPC](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-forwarding-inbound-queries.html) in the *Amazon Route 53 Developer Guide*.

#### Create and associate the DHCP option set
<a name="network-dhcp-dns-ntp-steps"></a>

Create the DHCP option set and associate it with your VPC in the Amazon VPC console or with the AWS CLI. Choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc) and choose **DHCP option sets** in the navigation pane.

1. Choose **Create DHCP option set**.

1. For **Domain name**, enter your DNS domain name.

1. For **Domain name servers**, enter the primary and secondary DNS server IP addresses.

1. For **NTP servers**, enter the IPv4 addresses of up to four NTP servers.

1. Choose **Create DHCP option set**.

1. In the navigation pane, choose **Your VPCs** and select your VPC.

1. Choose **Actions**, **Edit VPC settings**.

1. For **DHCP option set**, choose the option set that you created, and then choose **Save**.

1. Create the DHCP option set and store its ID. Replace the example addresses and names with your own values.

   ```bash
   DHCP_OPTIONS_ID=$(aws ec2 create-dhcp-options \
     --dhcp-configurations \
       "Key=domain-name,Values=example.com" \
       "Key=domain-name-servers,Values=10.10.200.100,10.10.200.101" \
       "Key=ntp-servers,Values=10.10.200.20" \
     --query 'DhcpOptions.DhcpOptionsId' \
     --output text)
   ```

1. Associate the DHCP option set with your VPC. If `$VPC_ID` isn’t set in your shell, replace it in the following commands with the ID of your VPC.

   ```bash
   aws ec2 associate-dhcp-options \
     --dhcp-options-id $DHCP_OPTIONS_ID \
     --vpc-id $VPC_ID
   ```

1. Verify the association.

   ```bash
   aws ec2 describe-vpcs \
     --vpc-ids $VPC_ID \
     --query 'Vpcs[0].DhcpOptionsId'
   ```

   The output is the ID of the DHCP option set that you created.

You have configured DNS and NTP for your VPC. For more information, see [Create a DHCP option set](https://docs.aws.amazon.com/vpc/latest/userguide/DHCPOptionSet.html#CreatingaDHCPOptionSet) in the * Amazon VPC User Guide*.

#### Launch an instance to reach the VPC
<a name="network-access-instance"></a>

Later steps need a computer with a web browser that can reach the private IP addresses of your VPC and resolve the names in your private DNS zones. For example, you verify DNS records from it and open the VMware Host Client, the VCF Installer, and the vSphere Client from it. If your on-premises network doesn’t reach the VPC yet, launch an Amazon EC2 instance in the VPC for this purpose.

Launch the instance after you associate the DHCP option set. An instance that launches before the association doesn’t get the DNS servers and the domain name of the option set.

1. Launch an Amazon EC2 instance in the service access subnet, with no public IP address. Use a Windows instance if you want a desktop web browser on it.

1. Attach an instance profile that lets AWS Systems Manager manage the instance. The instance reaches Systems Manager through the NAT gateway, or through interface VPC endpoints for Systems Manager.

1. Connect to the instance with Session Manager, or to a Windows desktop with Fleet Manager Remote Desktop.

1. From the instance, run `nslookup` with your domain name. The output names the DNS server that answered, which must be one of the DNS servers in the option set.

For instructions, see [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) and [Connecting to a Windows Server managed instance using Remote Desktop](https://docs.aws.amazon.com/systems-manager/latest/userguide/fleet-manager-remote-desktop-connections.html) in the * AWS Systems Manager User Guide*.

#### Check DNS reachability
<a name="evs-env-dns-reachability"></a>

Amazon EVS needs a persistent connection to SDDC Manager and to the DNS servers in your DHCP option set to reach DNS records. If the connection to SDDC Manager becomes unavailable, Amazon EVS can no longer validate the environment status, and you might lose environment access. For instructions to troubleshoot this issue, see [Troubleshoot networking](#troubleshoot-networking).

Next, set up Amazon VPC Route Server. For instructions, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

### Set up VPC Route Server and on-premises connectivity
<a name="network-route-server"></a>

Amazon Elastic VMware Service uses Amazon VPC Route Server for dynamic routing between NSX and your VPC underlay network by using Border Gateway Protocol (BGP). Create a route server with endpoints and peers in your VPC, and then optionally connect your on-premises network through a transit gateway. You can use the Amazon VPC console or the AWS CLI. The VPC Route Server tutorial in the * Amazon VPC User Guide* has the steps for both.

Your environment deployment fails if you don’t meet the following requirements for VPC Route Server.
+ You must create at least two route server endpoints in the service access subnet.
+ The peer autonomous system number (ASN) on the route server peers must match the ASN of the NSX Tier-0 gateway.
+ Each route server peer must use a unique IP address from the NSX uplink VLAN subnet.
+ Every route table that you propagate to must have at least one explicit subnet association.

#### Before you begin
<a name="evs-env-rs-prereq"></a>

Make sure that the following are true.
+ You’ve created the VPC and the service access subnet. For instructions, see [Create a VPC with subnets and route tables](#network-create-vpc).
+ You have IAM permissions to manage VPC Route Server resources.
+ You’ve chosen the Amazon-side ASN for the route server. The value must be in the range 1 to 4294967295. You can use any private ASN, for example `65022`.
+ You’ve chosen the peer ASN for the NSX Tier-0 gateway. The default ASN for an NSX Edge appliance is `65000`.
+ You’ve chosen the peer IP addresses from the NSX uplink VLAN CIDR block that you pass as `initialVlans.nsxUplink` when you create the environment. Amazon EVS creates the NSX uplink VLAN subnet during environment creation, so it doesn’t exist yet. The addresses must fall within that CIDR block and must remain unused.

How the peer values are used differs by deployment mode:
+ In Self-deployed mode, you configure the NSX Edge Tier-0 gateway yourself. Use the Amazon-side ASN as the BGP neighbor (remote) ASN, and use the peer ASN as the Tier-0 local ASN. For more information, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge).
+ In EVS-deployed mode, Amazon EVS assigns the peer IP addresses to the NSX Edge nodes during environment deployment.

#### Create the route server, endpoints, and peers
<a name="evs-env-rs-steps"></a>

Follow the [VPC Route Server get started tutorial](https://docs.aws.amazon.com/vpc/latest/userguide/route-server-tutorial.html) in the * Amazon VPC User Guide* and use these Amazon EVS values:

1. Create a route server with the Amazon-side ASN that you chose.

   We recommend that you enable persistent routes with a persist duration of 1 to 5 minutes. Persistent routes keep the routes in the routing database of the route server even if all BGP sessions end. For more information, see [Create a route server](https://docs.aws.amazon.com/vpc/latest/userguide/route-server-tutorial-create.html).

1. Associate the route server with your VPC.

1. Create two route server endpoints, both in the service access subnet.

1. Create the route server peers. Use a unique IP address from the NSX uplink VLAN CIDR block for each peer, and set the peer ASN to the NSX Tier-0 ASN.

   We recommend four peers in a full mesh, so that each of the two NSX Edge nodes peers with both endpoints. A full mesh keeps routes propagating if one endpoint goes into maintenance. You must create at least two peers.

1. Enable route server propagation on the route table that is associated with the service access subnet. That route table must have at least one explicit subnet association. BGP route advertisement fails if a propagated route table has no explicit subnet association.

   If you configure HCX internet connectivity, also enable propagation on the route table of the public subnet. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity).

   If you use a NAT gateway or a transit gateway, make sure that the route server propagates NSX routes to the VPC route tables that need them.

1. Verify that both endpoints and all peers appear in the route server. BGP sessions stay down until the Amazon EVS environment is deployed and operational.

You have created the VPC Route Server resources that Amazon EVS needs.

**Note**  
For peer liveness detection, Amazon EVS supports only the default BGP keepalive mechanism. Amazon EVS does not support multi-hop Bidirectional Forwarding Detection (BFD).

#### Connect your on-premises network (optional)
<a name="getting-started-connect-on-prem"></a>

You can connect your on-premises data center to your VPC with Direct Connect and an associated transit gateway, or with an AWS Site-to-Site VPN attachment to a transit gateway. To reach on-premises systems during environment deployment, add the routes to the VPC main route table. For more information, see [Create a VPC with subnets and route tables](#network-create-vpc).
+ For Direct Connect, see [Direct Connect gateways and transit gateway associations](https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-connect-transit-gateways.html).
+ For AWS Site-to-Site VPN, see [AWS Site-to-Site VPN attachments in Amazon VPC Transit Gateways](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpn-attachments.html) in the * Amazon VPC Transit Gateways Guide*.

**Note**  
Amazon EVS does not support connectivity through an Direct Connect private virtual interface (VIF), or through an AWS Site-to-Site VPN connection that terminates directly in the underlay VPC. For the connection types that Amazon EVS supports, see [Supported network configurations](#plan-network-supported).

##### After you create the environment
<a name="getting-started-config-tgw-assoc"></a>

Complete these steps after you create the environment.

1. Update the transit gateway route tables with the VPC CIDR blocks that the Amazon EVS environment creates. For more information, see [Transit gateway route tables in Amazon VPC Transit Gateways](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-route-tables.html).

1. If you use Direct Connect, update your allowed prefixes so that the Direct Connect gateway sends and receives the updated routes from the VPC. For more information, see [Allowed prefixes interactions for Direct Connect gateways](https://docs.aws.amazon.com/directconnect/latest/UserGuide/allowed-to-prefixes.html).

#### Troubleshoot VPC Route Server
<a name="evs-env-rs-troubleshoot"></a>

If routes don’t propagate or BGP sessions don’t come up after deployment, see [Routes don’t propagate or BGP sessions don’t come up](#troubleshoot-rs-routes). If the environment deployment fails because of the route server configuration, see [The deployment fails because of the VPC Route Server configuration](#troubleshoot-rs-deploy-fails).

Next, create a network ACL for the VLAN subnets. For instructions, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

### Create a network ACL for VLAN subnet traffic
<a name="evs-env-nacl-cong"></a>

A network access control list (ACL) allows or denies specific inbound or outbound traffic at the subnet level. You use network ACLs to control traffic to and from your Amazon Elastic VMware Service VLAN subnets, because security groups don’t apply to them. You can use the default network ACL of your VPC, or create a custom network ACL with the Amazon VPC console or the AWS CLI.

**Important**  
You must use a network ACL to control traffic to and from Amazon EVS VLAN subnets. If your security groups and network ACLs don’t allow the traffic in the following table, environment deployment fails, and existing environments can have a degraded compliance status.

#### Required traffic
<a name="evs-env-nacl-rules"></a>

The following table lists the traffic that your network ACLs must allow.


| Traffic | Protocol and port | Between | 
| --- | --- | --- | 
| DNS queries from Amazon EVS | TCP and UDP 53 | Amazon EVS and the DNS servers in your DHCP option set | 
| HTTPS access from Amazon EVS | TCP 443 (HTTPS) | Amazon EVS and the host management VLAN subnet, and Amazon EVS and the management VLAN subnet | 
| SSH access from Amazon EVS | TCP 22 (SSH) | Amazon EVS and the host management VLAN subnet, and Amazon EVS and the management VLAN subnet | 
| Border Gateway Protocol (BGP) sessions with Amazon VPC Route Server | TCP 179 | The NSX Edge uplink IP addresses and the route server endpoint IP addresses | 
| NFS to Amazon FSx for OpenZFS, for diskless hosts with VMware Cloud Foundation (VCF) 9.x | TCP and UDP 111, 2049, and 20001 to 20003 | The host management VLAN subnet and the subnet of the storage, for example the service access subnet. For more information, see [Security group rules](#fsx-openzfs-security-group). | 
| NFS to Amazon FSx for NetApp ONTAP | TCP and UDP 111, 635, 2049, 4045, and 4046, and UDP 4049 | The host management VLAN subnet and the subnet of the storage. For more information, see [File system access control with Amazon VPC](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/limit-access-security-groups.html) in the *Amazon FSx for NetApp ONTAP User Guide*. | 
| iSCSI to Amazon FSx for NetApp ONTAP | TCP 3260 | The host management VLAN subnet and the subnet of the storage. | 

The security group that is attached to the route server endpoints must also allow inbound TCP 179.

In Self-deployed mode, your network ACLs must also allow the following traffic, which the VCF installation needs.
+ HTTPS (TCP 443) from the VM management VLAN subnet to the internet through your NAT gateway, so that the VCF Installer can download software from the Broadcom online depot. For more information, see [Create a VPC with subnets and route tables](#network-create-vpc).
+ DNS (TCP and UDP 53) from the host management VLAN subnet and the VM management VLAN subnet to the DNS servers in your DHCP option set. For example, the DNS servers can be the IP addresses of a Route 53 Resolver inbound endpoint. For more information, see [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp).
+ HTTPS (TCP 443) from your access instance to the host management VLAN subnet and the VM management VLAN subnet. This traffic lets you open the VMware Host Client and the VCF Installer. For more information, see [Launch an instance to reach the VPC](#network-access-instance).

Network ACLs are stateless, so you must also allow the return traffic for each rule. For how to allow return traffic on ephemeral ports, see [Ephemeral ports](https://docs.aws.amazon.com/vpc/latest/userguide/nacl-ephemeral-ports.html) in the * Amazon VPC User Guide*.

If you plan to configure VMware Hybrid Cloud Extension (VMware HCX) internet connectivity, your network ACL rules must also allow the inbound and outbound connections that the HCX components need. For the HCX port requirements, see the [VMware HCX User Guide](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-11.html) in the Broadcom documentation.

**Important**  
When you associate an Elastic IP address with a VLAN, all resources on that VLAN subnet get direct internet access. Configure network ACLs that restrict access as your security requirements need. We recommend a custom network ACL that is dedicated to the HCX public VLAN, so that its internet-facing rules don’t affect other VLAN subnets.

#### Create the network ACL and its rules
<a name="evs-env-nacl-steps"></a>

You can create the network ACL in the Amazon VPC console or with the AWS CLI. Choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc) and choose **Network ACLs** in the navigation pane.

1. Choose **Create network ACL**.

1. Enter a name, choose your VPC, and then choose **Create network ACL**.

1. Select the network ACL, choose the **Inbound rules** tab, and then choose **Edit inbound rules**.

1. Add a rule for each row in the required traffic table and for any traffic that your workloads need, and then choose **Save changes**.

1. Choose the **Outbound rules** tab, choose **Edit outbound rules**, add the matching outbound rules and return traffic rules, and then choose **Save changes**.

1. Create the network ACL and store its ID. If `$VPC_ID` isn’t set in your shell, replace it in the following commands with the ID of your VPC.

   ```bash
   NACL_ID=$(aws ec2 create-network-acl \
     --vpc-id $VPC_ID \
     --tag-specifications 'ResourceType=network-acl,Tags=[{Key=Name,Value=evs-vlan-nacl}]' \
     --query 'NetworkAcl.NetworkAclId' \
     --output text)
   ```

1. Add an entry for each rule. This example allows inbound HTTPS from the VPC CIDR block. Replace the example addresses with your own values.

   ```bash
   aws ec2 create-network-acl-entry \
     --network-acl-id $NACL_ID \
     --ingress \
     --rule-number 100 \
     --protocol tcp \
     --port-range From=443,To=443 \
     --cidr-block 10.10.0.0/16 \
     --rule-action allow
   ```

1. Repeat the previous step for each inbound rule. For outbound rules, use `--egress` instead of `--ingress`.

1. Verify the rules.

   ```bash
   aws ec2 describe-network-acls \
     --network-acl-ids $NACL_ID \
     --query 'NetworkAcls[0].Entries'
   ```

   The output includes the entries that you added, with their rule numbers, protocols, port ranges, and actions.

You have created a network ACL with the rules that the Amazon EVS VLAN subnets need. For more information, see [Create a network ACL for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-network-acl.html) in the * Amazon VPC User Guide*.

#### Associate the network ACL with the VLAN subnets
<a name="evs-env-nacl-associate"></a>

Amazon EVS creates the VLAN subnets when you create the environment. Until you change the association, the VLAN subnets use the default network ACL of the VPC. In EVS-deployed mode with HCX internet connectivity, you choose the network ACL for the HCX VLAN when you create the environment. For more information, see [Create an environment](#getting-started-create-env).

After you create the environment, associate your network ACL with the VLAN subnets in the Amazon VPC console or with the AWS CLI. Choose the tab for the method that you use.

**Example**  

1. In the Amazon VPC console, choose **Network ACLs** and select your network ACL.

1. Choose the **Subnet associations** tab, and then choose **Edit subnet associations**.

1. Select the Amazon EVS VLAN subnets, and then choose **Save changes**.

1. Find the current network ACL association ID for a VLAN subnet. Replace the example subnet ID with the ID of your VLAN subnet.

   ```bash
   aws ec2 describe-network-acls \
     --filters Name=association.subnet-id,Values=subnet-01234a1b2cde1234f \
     --query 'NetworkAcls[0].Associations[?SubnetId==`subnet-01234a1b2cde1234f`].NetworkAclAssociationId' \
     --output text
   ```

1. Replace the association with your network ACL. Replace the example association ID with the ID that the previous command returns. If `$NACL_ID` isn’t set in your shell, replace it in the following commands with the ID of your network ACL.

   ```bash
   aws ec2 replace-network-acl-association \
     --association-id aclassoc-0123456789abcdef0 \
     --network-acl-id $NACL_ID
   ```

1. Repeat these steps for each VLAN subnet.

Your VLAN subnets now use the network ACL that you created.

To check which network ACL a VLAN subnet uses, see [Troubleshoot networking](#troubleshoot-networking).

Next, create the environment in your deployment mode. For VCF 9.x, see [Deploy in Self-deployed mode](#getting-started-self-deployed). For VCF 5.2.x, see [Deploy in EVS-deployed mode](#evs-deployed-overview). After you create the environment, associate the VLAN subnets with a route table. For instructions, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).

### Associate VLAN subnets with a route table
<a name="evs-env-config-custom-rt"></a>

After your environment deploys, you must explicitly associate each Amazon Elastic VMware Service VLAN subnet with a route table in your VPC. NSX connectivity fails if your VLAN subnets are not explicitly associated with a VPC route table. You can use the Amazon VPC console or the AWS CLI.

#### Why explicit association is required
<a name="evs-env-config-custom-rt-why"></a>

During environment deployment, the Amazon EVS VLAN subnets are implicitly associated with the VPC main route table. That is why the main route table must route to dependent services such as DNS and on-premises systems while you create the environment. For more information, see [Create a VPC with subnets and route tables](#network-create-vpc).

Amazon EVS supports a custom route table only after the environment is created. The route table that you associate lets AWS resources communicate with virtual machines on NSX network segments. Route server propagation in Amazon VPC Route Server also needs it, because Border Gateway Protocol (BGP) route advertisement fails if a propagated route table has no explicit subnet association.

We recommend that you associate the VLAN subnets with a custom route table, because it gives you granular control over routing for specific subnets and gateways. For steps to create one, see [Create a route table for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc-route-table.html) in the * Amazon VPC User Guide*.

#### Associate the VLAN subnets
<a name="getting-started-associate-vlans"></a>

You can associate the VLAN subnets in the Amazon VPC console or with the AWS CLI. Choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc).

1. In the navigation pane, choose **Route tables**.

1. Choose the route table that you want to associate with the Amazon EVS VLAN subnets. If you created a public VLAN for VMware Hybrid Cloud Extension (VMware HCX), associate the HCX public VLAN subnet with a public route table that routes to an internet gateway.

1. Choose the **Subnet associations** tab.

1. Under **Explicit subnet associations**, choose **Edit subnet associations**.

1. Select all of the Amazon EVS VLAN subnets.

1. Choose **Save associations**.

1. Verify that the VLAN subnets appear under **Explicit subnet associations**.

1. Identify the Amazon EVS VLAN subnet IDs in your VPC. If `$VPC_ID` isn’t set in your shell, replace it in the following commands with the ID of your VPC.

   ```bash
   aws ec2 describe-subnets \
     --filters Name=vpc-id,Values=$VPC_ID \
     --query 'Subnets[].[SubnetId,CidrBlock]' \
     --output table
   ```

1. Associate each VLAN subnet with the route table. If you created a public HCX VLAN, associate the HCX public VLAN subnet with a public route table that routes to an internet gateway. Replace the example route table ID and subnet ID with your own values.

   ```bash
   aws ec2 associate-route-table \
     --route-table-id rtb-0123456789abcdef0 \
     --subnet-id subnet-01234a1b2cde1234f
   ```

1. Verify the associations.

   ```bash
   aws ec2 describe-route-tables \
     --route-table-ids rtb-0123456789abcdef0 \
     --query 'RouteTables[0].Associations[].SubnetId'
   ```

   The output lists the IDs of the VLAN subnets that you associated.

You have explicitly associated the VLAN subnets with a route table.

Next, return to your deployment path.
+ In Self-deployed mode, record the VLAN IDs as described at the end of [Step 1: Create the environment](#self-deployed-create-env), and then continue with [Step 2: Create DNS records](#self-deployed-host-dns-records).
+ In EVS-deployed mode, continue with the remaining tasks in [Next steps](#verify-env-evs-deployed-next).

If you connect on premises through a transit gateway, update the transit gateway route tables next. For more information, see [Set up VPC Route Server and on-premises connectivity](#network-route-server). For connectivity issues, see [Troubleshoot networking](#troubleshoot-networking).

### Clean up network resources
<a name="cleanup"></a>

After you delete your Amazon Elastic VMware Service environment, delete the network resources that you created for it, so that they don’t continue to incur costs. You delete them in reverse order of creation, using the Amazon VPC console or the AWS CLI as described in the linked Amazon VPC topics.

**Warning**  
You can’t recover these resources after you delete them. Make sure that no other workload uses the VPC, the route server, the network ACL, the gateways, or the subnets before you delete them.

#### Before you begin
<a name="cleanup-prereqs"></a>

Make sure that you’ve deleted the Amazon EVS environment. For instructions, see the page for your deployment mode.
+ For a Self-deployed environment, see [Clean up a Self-deployed environment](#self-deployed-cleanup).
+ For an EVS-deployed environment, see [Delete an EVS-deployed environment](#getting-started-cleanup-env-hosts).

Environment deletion also deletes the Amazon EVS VLAN subnets and the AWS Secrets Manager secrets that Amazon EVS created. Amazon EVS VLAN subnets can be deleted only when the environment is deleted. AWS resources that you created yourself are not deleted.

#### Delete the network resources
<a name="cleanup-steps"></a>

To delete the remaining resources, do the following.

1.  (HCX internet connectivity only) Delete the IPAM resources in this order.

   1. Release the Elastic IP address allocations from the public IPAM pool. For instructions, see [Release an allocation](https://docs.aws.amazon.com/vpc/latest/ipam/release-alloc-ipam.html).

   1. Disassociate the public CIDR block from your VPC. Use the [disassociate-vpc-cidr-block](https://docs.aws.amazon.com/cli/latest/reference/ec2/disassociate-vpc-cidr-block.html) command with the association ID of the public CIDR block.

   1. Deprovision the public IPv4 CIDR block from the IPAM pool. For instructions, see [Deprovision CIDRs from a pool](https://docs.aws.amazon.com/vpc/latest/ipam/depro-pool-cidr-ipam.html).

   1. Delete the public IPAM pool. For instructions, see [Delete a pool](https://docs.aws.amazon.com/vpc/latest/ipam/delete-pool-ipam.html).

   1. Delete the IPAM. For instructions, see [Delete an IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/delete-ipam.html).

1. Terminate the instances and delete the file systems that you created in the VPC subnets, such as the instance from [Launch an instance to reach the VPC](#network-access-instance) and your external storage. You can’t delete a subnet while network interfaces exist in it.

1. If you attached the VPC to a transit gateway, delete the VPC attachment. For instructions, see [Amazon VPC attachments in Amazon VPC Transit Gateways](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-vpc-attachments.html).

1.  Delete the Amazon VPC Route Server components. For instructions, see [Route server cleanup](https://docs.aws.amazon.com/vpc/latest/userguide/route-server-tutorial-cleanup.html) in the * Amazon VPC User Guide*.

1.  Delete the custom network ACL. For instructions, see [Delete a network ACL for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/delete-network-acl.html).

1. Delete the NAT gateway, wait until its state is `deleted`, and then release its Elastic IP address. A NAT gateway and an allocated Elastic IP address continue to incur costs until you delete or release them. For instructions, see [Work with NAT gateways](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-working-with.html) and [Elastic IP addresses](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-eips.html).

1.  Disassociate and delete the subnet route tables that you created. For instructions, see [Subnet route tables](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html).

1. If you use Route 53 for DNS, delete the inbound resolver endpoints in the service access subnet, and then delete the private hosted zones. You can’t delete the service access subnet while the endpoints exist.

1. Detach the internet gateway from the VPC, and then delete it. For instructions, see [Internet gateways](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html).

1.  Delete the VPC subnets, including the service access subnet. For instructions, see [Delete a subnet](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-deleting.html).

1.  Delete the VPC, and then delete the DHCP option set that you created for it. For instructions, see [Delete your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/delete-vpc.html) and [DHCP option sets](https://docs.aws.amazon.com/vpc/latest/userguide/DHCPOptionSet.html).

1. If you have Amazon EC2 Capacity Reservations that you no longer need, cancel them. For instructions, see [Cancel a Capacity Reservation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservations-release.html) in the *Amazon EC2 User Guide*.

You have deleted the network resources for your Amazon EVS environment. To confirm, open the Amazon VPC console and check that the VPC no longer appears under **Your VPCs**.

## Deploy in Self-deployed mode
<a name="getting-started-self-deployed"></a>

In Self-deployed mode, you create an Amazon Elastic VMware Service environment, add hosts, and then install and configure VMware Cloud Foundation (VCF) yourself. Amazon EVS provisions the AWS networking and the VLAN subnets. To deploy VCF 9.x, you use the VCF Installer and connect VCF back to Amazon EVS with connectors.

Charges for the Amazon EC2 bare-metal instances begin when you add hosts. For more information, see [Billing](#self-deployed-costs).

You can also run these steps with the EVS Deployment Orchestrator (EDO), a tool in the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs). For more information, see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo). For the VCF versions and instance types that Self-deployed mode supports, see [Supported VCF versions and instance types](#versions-provided). If you want Amazon EVS to install VCF 5.2.x, see [Deploy in EVS-deployed mode](#evs-deployed-overview). To install VCF 5.x yourself in Self-deployed mode, follow the Broadcom documentation for your VCF version. The steps in this guide cover VCF 9.x.

### Steps to deploy
<a name="self-deployed-process"></a>

Before you begin, prepare your AWS account and build the network foundation. For more information, see [Prepare your AWS account](#setting-up), [Build the network foundation](#network-foundation), and [Deployment prerequisite checklist](#evs-deployment-prereq-checklist).

Then complete these steps in order.

1.  [Step 1: Create the environment](#self-deployed-create-env). Amazon EVS provisions your VLAN subnets.

1.  [Step 2: Create DNS records](#self-deployed-host-dns-records). Create A and PTR records for your ESX hosts and VCF management appliances.

1.  [Step 3: Add hosts](#self-deployed-add-hosts). Add EC2 bare-metal hosts to your environment.

1. Install VCF on your hosts. The procedure depends on the storage of your instance type.
   + For hosts with local storage that use vSAN, see [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf), and then [Step 4b (local storage): Run VCF bringup](#sd-install-bringup).
   + For diskless hosts that use external storage, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless), and then [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec). Diskless hosts have no local storage.

1.  [Step 5: Configure NSX networking](#self-deployed-nsx-edge). Create the NSX Edge cluster and the Tier-0 gateway, and configure routing to your VPC.

1.  [Step 6: Create connectors](#self-deployed-create-connectors). Create connectors so that Amazon EVS can monitor your deployment.

1.  [Step 7: Verify the environment](#self-deployed-verify). Confirm that your hosts, management appliances, and connectors are healthy.

For help to choose between local storage and diskless hosts, see [Choose instance types](#plan-instance-types). When you no longer need the environment, see [Clean up a Self-deployed environment](#self-deployed-cleanup).

### Billing
<a name="self-deployed-costs"></a>

After you add hosts to your environment, you accrue AWS charges for the EC2 bare-metal instances, as for any other EC2 instance. The charges apply whether or not you have installed VCF on the hosts.

If you have created an environment in Self-deployed mode but not yet added hosts or installed VCF, AWS might contact you at the email address of your AWS account. AWS asks you to either complete setup or remove the environment.

To stop accruing charges for hosts that you no longer use, delete those hosts. For more information, see [Clean up a Self-deployed environment](#self-deployed-cleanup). For prices, see the [Amazon EVS pricing page](https://aws.amazon.com/evs/pricing). For the billing dimensions, see [Plan costs and licenses](#plan-cost).

**Topics**

### Step 1: Create the environment
<a name="self-deployed-create-env"></a>

In Self-deployed mode, environment creation provisions the Amazon Elastic VMware Service VLAN subnets that you specify. It doesn’t deploy VMware Cloud Foundation (VCF) or create hosts. You can create the environment with the Amazon EVS console or the AWS CLI. The steps are the same for hosts with local storage and for diskless hosts.

In Self-deployed mode, you don’t select peers for Amazon VPC Route Server when you create the environment. You configure Border Gateway Protocol (BGP) peering between the NSX Edge Tier-0 gateway and your route server yourself, after you install VCF. For more information, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge) and [Set up VPC Route Server and on-premises connectivity](#network-route-server).

**Important**  
You can’t add VLAN subnets or change their CIDR blocks after you create the environment. Size each VLAN subnet before you start. For sizing guidance, see [Plan your VPC and VLAN subnets](#plan-network-sizing).

Before you begin, complete the AWS account and network prerequisites. For more information, see [Prepare your AWS account](#setting-up) and [Build the network foundation](#network-foundation).

If you plan to use VMware HCX over the internet, also complete Part 1 of [Configure HCX public internet connectivity](#evs-env-hcx-internet-access) first. The public HCX VLAN subnet uses a public CIDR block that you add to the VPC before you create the environment.

To create the environment in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs). Check that the Region selector in the upper right of the console shows the AWS Region where you want to create the environment.

1. In the navigation pane, choose **Environments**.

1. Choose **Create environment**.

1. On the **Configure environment** step, do the following.

   1. Review the ** AWS account requirements** panel to confirm that your account meets the Amazon EVS prerequisites.

   1. (Optional) For **Name**, enter an environment name.

   1. For **VCF version**, choose **Self-deployed**.

   1. Choose **Next**.

1. On the **Configure networks and connectivity** step, do the following.

   1. For **VPC**, choose the VPC that you created.

   1. For **Service access subnet**, choose the private subnet that you created.

   1. (Optional) For **Service access security group - optional**, choose up to two security groups that control communication between the Amazon EVS control plane and your VPC. Amazon EVS uses the default security group if you don’t choose a security group. The security groups that you choose must provide connectivity to the Amazon EVS VLAN subnets.

   1. Under **Management connectivity**, enter CIDR blocks for the Amazon EVS VLAN subnets. Amazon EVS creates these VLAN subnets as part of environment creation.

   1. Under **Expansion VLANs**, enter CIDR blocks for additional Amazon EVS VLAN subnets that you can use to extend your VCF deployment.

   1. Under **Workload/VCF connectivity**, enter the CIDR block for the NSX uplink VLAN.

   1. Choose **Next**.

1. (Optional) On the **Add tags** step, add tags and choose **Next**. Tags that you add to the Amazon EVS environment don’t propagate to underlying AWS resources such as EC2 instances.

1. On the **Review and create** step, review your configuration and choose **Create environment**.

   An information alert on the **Review** page confirms: "Your environment infrastructure will be provisioned now. After creation, add hosts and deploy VCF from the Environment Detail page."

1. Open a terminal session.

1. Run the `aws evs create-environment` command, and specify `--vcf-version SELF_DEPLOYED`.

   In Self-deployed mode, don’t specify the following parameters, because they cause a validation error: `--license-info`, `--hosts`, `--vcf-hostnames`, `--site-id`, and `--connectivity-info`.

   The following example creates an Amazon EVS environment in Self-deployed mode. Replace the example IDs, names, and CIDR blocks with your own values, and size the CIDR blocks for your VPC. The example CIDR blocks assume the 10.10.0.0/16 VPC from the earlier network steps. Each VLAN CIDR block must be inside a CIDR block of your VPC. Replace `<region-code>` with the Region where you want to create the environment.

   ```bash
   aws evs create-environment \
       --environment-name my-self-deployed-env \
       --vpc-id vpc-0abcdef1234567890 \
       --service-access-subnet-id subnet-01234a1b2cde1234f \
       --vcf-version SELF_DEPLOYED \
       --terms-accepted \
       --initial-vlans '{
         "vmkManagement":   { "cidr": "10.10.0.0/24" },
         "vmManagement":    { "cidr": "10.10.1.0/24" },
         "vMotion":         { "cidr": "10.10.2.0/24" },
         "vSan":            { "cidr": "10.10.3.0/24" },
         "vTep":            { "cidr": "10.10.4.0/24" },
         "edgeVTep":        { "cidr": "10.10.5.0/24" },
         "nsxUplink":       { "cidr": "10.10.6.0/24" },
         "hcx":             { "cidr": "10.10.7.0/24" },
         "expansionVlan1":  { "cidr": "10.10.8.0/24" },
         "expansionVlan2":  { "cidr": "10.10.9.0/24" }
       }' \
       --region <region-code>
   ```

   The `vSan` CIDR block is a required input, also for diskless hosts that don’t use vSAN.

1. Don’t run the command again while the environment is being created, because a repeated command can create a second environment. Check the environment state with `aws evs get-environment` until `environmentState` is `CREATED`.

Environment creation takes about 25 to 30 minutes. When the environment reaches the `CREATED` state, you have created the environment and its VLAN subnets.

#### Prepare the VLAN subnets for VCF
<a name="self-deployed-create-env-after"></a>

After the environment is in the `CREATED` state, do the following.

1. Associate the Amazon EVS VLAN subnets with the route table that you want them to use. The VCF Installer needs outbound internet access from the VM management VLAN subnet, so use a route table with a default route to your NAT gateway. For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).

1. Record the VLAN IDs that Amazon EVS assigned. The VLAN IDs can differ for each environment, so read them from your environment. You can find them in the Amazon EVS console on the **Networks and connectivity** tab of your environment, or with the following command. Replace `<environment-id>` with the ID of your environment, and replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs list-environment-vlans \
       --environment-id <environment-id> \
       --query 'environmentVlans[].[functionName,vlanId,cidr]' \
       --output table \
       --region <region-code>
   ```

For how to use the VLAN IDs, see [Step 3: Add hosts](#self-deployed-add-hosts).

Next step: [Step 2: Create DNS records](#self-deployed-host-dns-records).

### Step 2: Create DNS records
<a name="self-deployed-host-dns-records"></a>

Before you add hosts and install VMware Cloud Foundation (VCF), create DNS records for each ESX host and for each VCF management appliance that you plan to deploy. Create a forward (A) record and a reverse (PTR) record for each one. Amazon Elastic VMware Service performs a DNS lookup of the fully qualified domain name (FQDN) of each host during host creation. Host creation fails if the records don’t already exist.

**Important**  
Create the A and PTR records for every host before you add that host. If the records don’t resolve, the host transitions to `CREATE_FAILED`.

The host FQDN is `<host-name>.<domain>`. `<host-name>` is the name that you pass when you add the host. `<domain>` is the domain name in the DHCP option set of your VPC. For more information, see [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp).

#### Plan host names
<a name="self-deployed-dns-names"></a>

When you plan host names, include the length of your domain name.
+ The FQDN of each host and VCF management appliance must not exceed 62 characters. The FQDN is the host name combined with the domain name from your VPC DHCP option set. A longer host FQDN causes an error when you add the host.
+ We recommend that you keep each host name to 15 characters or fewer, to follow NetBIOS naming limits. A longer host name doesn’t cause a deployment failure.

#### Record requirements
<a name="self-deployed-dns-requirements"></a>

Host records must meet these requirements.
+ The A record IP address is within the host management (`vmkManagement`) VLAN CIDR block that you specified when you created the environment.
+ The A record IP address matches the management IP address that the host gets. In the example deployment in this guide, Amazon EVS gave the hosts the next free addresses of the host management VLAN subnet in order, starting at `.11`. We recommend that you create the host A records in the order in which you add the hosts, starting at that address.
+ Each A record has a matching PTR record in your reverse lookup zone.
+ Both forward and reverse lookups succeed from the Amazon EVS service access subnet.

VCF management appliance records (for vCenter Server, NSX Manager, and the other appliances for your VCF version) must meet these requirements.
+ The FQDNs match the host names that you configure during VCF installation.
+ The IP addresses fall within the range of the appropriate VLAN subnet.
+ The records resolve from the Amazon EVS management VLAN and from any network from which you reach the VCF management appliances.

The names and number of appliances depend on your VCF version and topology. For the full list, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html). For the records that the diskless example needs, with example addresses, see [DNS records for the example](#getting-started-diskless-dns).

#### Create the records
<a name="self-deployed-dns-create"></a>

If you use Route 53 private hosted zones, create the records in the forward and reverse zones that you created with the network foundation. For more information, see [Create private hosted zones and a Resolver inbound endpoint](#network-dhcp-dns-zones).

1. Create an A record in a forward zone for each host and each VCF management appliance.

1. Create a PTR record in a reverse zone for each A record. A PTR record name is the IP address in reverse order, under the reverse zone. For example, with a reverse zone named `10.10.in-addr.arpa`, the PTR record for 10.10.0.11 has the name `11.0.10.10.in-addr.arpa` and points to `esx01.evs.example.com.`.

#### Verify the records
<a name="self-deployed-dns-verify"></a>

Verify the records from an instance in your VPC, such as the instance in [Launch an instance to reach the VPC](#network-access-instance). Query each DNS server in your DHCP option set for the forward and reverse record of every host and appliance. The following example verifies one host against two DNS servers.

```bash
nslookup esx01.evs.example.com 10.10.200.100
nslookup esx01.evs.example.com 10.10.200.101
nslookup 10.10.0.11 10.10.200.100
nslookup 10.10.0.11 10.10.200.101
```

Each command must return the expected name or address. Don’t add a host until all lookups succeed for it. If you use Route 53, also wait until Route 53 reports the record changes as in sync.

You have created and verified the DNS records. For troubleshooting, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

Next step: [Step 3: Add hosts](#self-deployed-add-hosts).

### Step 3: Add hosts
<a name="self-deployed-add-hosts"></a>

Add enough hosts to serve the topology of your target VMware Cloud Foundation (VCF) version. Charges for the Amazon EC2 bare-metal instances begin when you add a host. For more information, see [Billing](#self-deployed-costs). You can add hosts with the Amazon Elastic VMware Service console or the AWS CLI.

For minimum host counts, including vSAN requirements, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html).

Before you begin, make sure that the following are true.
+ The A and PTR records of each host resolve. For more information, see [Step 2: Create DNS records](#self-deployed-host-dns-records).
+ Your **Host count per EVS environment** quota is at least the number of hosts that the environment will have after you add hosts. Host creation fails if this quota is lower. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs).
+ You have an Amazon EC2 key pair. Amazon EVS requires the key pair name when you add a host.

#### Choose the instance type and ESX version
<a name="self-deployed-host-choices"></a>

All hosts in a VCF cluster must use the same instance type. For the instance types and ESX versions available to your account, see [Supported VCF versions and instance types](#versions-provided) or run `aws evs get-versions`.

If you don’t specify an ESX version, Amazon EVS uses the default ESX version for Self-deployed mode, which `aws evs get-versions` reports as `defaultEsxVersion`. We recommend that you specify the ESX version explicitly, so that it matches the VCF Installer build that you use. Confirm that the ESX version is compatible with your VCF release in the [Broadcom Interoperability Matrix](https://interopmatrix.broadcom.com/Interoperability?col=1).

Diskless hosts require external storage that you prepare before you install VCF. For more information, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

**Note**  
To add diskless hosts, specify a diskless instance type and an ESX 9.x version. Add only the first diskless host at this step. You add the other diskless hosts in [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless), while the VCF software downloads.

#### Add a host
<a name="self-deployed-host-create"></a>

To add a host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**, and then choose your environment.

1. Choose the **Hosts** tab, and then choose **Create host**.

1. Enter the host details. Use the short host name from your DNS records (such as `esx01`), the instance type, the ESX version, and your Amazon EC2 key pair.

1. Choose **Create host**.

1. Repeat these steps for each host that your VCF topology requires.

1. Run the `aws evs create-environment-host` command. The following example adds a host that runs ESX 9.0.2 to an Amazon EVS environment. Replace the values in angle brackets with your own values. Replace the example host name, key pair name, and instance type with your own values. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs create-environment-host \
       --environment-id <environment-id> \
       --esx-version ESXi-9.0.2.0.25148076 \
       --host '{
           "hostName": "esx01",
           "keyName": "my-ec2-key-pair",
           "instanceType": "i4i.metal"
       }' \
       --region <region-code>
   ```

1. Repeat this command for each host that your VCF topology requires. Don’t run the command again for a host that is in the `CREATING` state.

1. Check the host state with the following command.

   ```bash
   aws evs list-environment-hosts \
       --environment-id <environment-id> \
       --query 'environmentHosts[].[hostName,hostState,ipAddress]' \
       --output table \
       --region <region-code>
   ```

A host is ready when its state is `CREATED`. Confirm that the IP address of each host equals the address in its A record. If a host stays in `CREATING` or moves to `CREATE_FAILED`, check its DNS records first. For more information, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

**Important**  
Hosts that you add to this environment receive the tag `DoNotDelete-EVS-[<environmentId>]-[<hostname>]`. Don’t delete, stop, or shut down these hosts outside of Amazon EVS. Doing so causes Amazon EVS to lose visibility into the host and can put your environment into an impaired state.

Amazon EVS stores the ESX root password of each host in AWS Secrets Manager. Each host has its own secret. You need the root password to sign in to the VMware Host Client in the next step.

#### Host networking settings
<a name="self-deployed-host-networking"></a>

Amazon EVS assigns a VLAN ID to each network function in your environment. To find the VLAN IDs, see the VLAN IDs step in [Prepare the VLAN subnets for VCF](#self-deployed-create-env-after). When you install VCF, apply these host networking settings.

 **Host management**   
The host management port group `Management` must remain untagged with VLAN ID `0`. `aws evs list-environment-vlans` also lists a VLAN ID for `vmkManagement`. Don’t use that VLAN ID on the host.

 **VM management**   
On the host where you run the VCF Installer, set the `VM Network` port group to the VM management (`vmManagement`) VLAN ID. For instructions, see [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf) for hosts with local storage, and [Prepare the first host](#getting-started-diskless-prep-host) for diskless hosts.

 **Other networks**   
Use the Amazon EVS-assigned VLAN IDs for the vMotion, vSAN, host overlay (`vTep`), Edge overlay (`edgeVTep`), and NSX uplink networks in your deployment specification. For the MTU of each network, see [Amazon EVS network settings for VCF](#sd-install-network).

 **Uplink teaming**   
Use a failover teaming policy (an active uplink with a standby uplink) for the distributed switch port groups, rather than a load-balancing policy. In NSX, set the teaming policy to `FAILOVER_ORDER`.

The VCF Installer migrates host networking to a distributed switch during bringup. You don’t need to enable SSH on the hosts.

Next step: For hosts with local storage, see [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf). For diskless hosts, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

### Step 4a (local storage): Prepare the VCF Installer
<a name="self-deployed-install-vcf"></a>

After your hosts are in the `CREATED` state and your DNS records resolve, you install VMware Cloud Foundation (VCF) with the VCF Installer. This step prepares an ESX host and a temporary datastore for the VCF Installer appliance, deploys the appliance, and downloads the VCF software. In [Step 4b (local storage): Run VCF bringup](#sd-install-bringup), you run bringup.

Use these steps for hosts with local storage that use vSAN. For diskless hosts, the datastore preparation, bringup storage, and reclaim steps differ. For instructions, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

The installation itself follows the standard Broadcom VCF process. For the installation guidance for your target VCF version, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html). In the VCF Installer, you also provide some settings that are specific to Amazon Elastic VMware Service.

**Tip**  
To run this installation end to end, including the Amazon EVS-specific settings, you can use the EVS Deployment Orchestrator (EDO), a tool in the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs). For more information, see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo).

Before you begin, make sure that you have the following.
+ A Broadcom account with a valid VCF entitlement.
+ The VCF Installer OVA file for your target VCF version, downloaded from the [Broadcom Support Portal](https://support.broadcom.com/).
+ A computer with a web browser that can reach the private IP addresses of your VPC, to open the VMware Host Client and the VCF Installer.

#### Amazon EVS network settings for VCF
<a name="sd-install-network"></a>

To find the VLAN ID of each network function, see the VLAN IDs step in [Prepare the VLAN subnets for VCF](#self-deployed-create-env-after). For how to apply the VLAN IDs on the hosts, see [Host networking settings](#self-deployed-host-networking).

The following table lists the MTU and use of each network function.


| Network function | MTU | Used for | 
| --- | --- | --- | 
| Host management (`vmkManagement`) | 1500 | ESX host management | 
| VM management (`vmManagement`) | 1500 | VCF management appliances (vCenter Server, NSX Manager, SDDC Manager, and VCF Operations) | 
| vMotion | 8500 | vMotion traffic | 
| vSAN | 8500 | vSAN storage traffic | 
| Host overlay (`vTep`) | 8500 | Host overlay (Geneve) tunnel endpoints | 
| Edge overlay (`edgeVTep`) | 8500 | NSX Edge overlay tunnel endpoints | 
| NSX uplink (`nsxUplink`) | 1500 | Tier-0 gateway north-south uplink | 

**Important**  
Configure jumbo frames (MTU 8500) on the vMotion, vSAN, and overlay (TEP) networks. The management and uplink networks use MTU 1500. The MTU must be consistent across the network path, or vSAN and overlay traffic fails.

#### Prepare the VCF Installer host
<a name="sd-install-hosts"></a>

On the ESX host where you run the VCF Installer appliance, set the `VM Network` port group to the Amazon EVS-assigned VM management VLAN ID. This setting lets the VCF Installer appliance communicate on the VM management VLAN.

1. Find the VLAN ID of the VM management (`vmManagement`) network. For instructions, see the VLAN IDs step in [Prepare the VLAN subnets for VCF](#self-deployed-create-env-after).

1. Sign in to the VMware Host Client of the host at `https://<host-fqdn>` as `root`, with the root password from AWS Secrets Manager.

1. On the host, set the `VM Network` port group VLAN ID to the VM management VLAN ID. Leave the host management port group `Management` untagged with VLAN ID `0`.

#### Prepare a temporary datastore for the VCF Installer
<a name="sd-install-datastore"></a>

The VCF Installer appliance needs a temporary datastore, because Amazon EVS hosts have no local VMFS datastores and the vSAN datastore doesn’t exist until bringup completes. Use the host that you prepared in the previous section.

1. In the Amazon EC2 console, create an encrypted General Purpose SSD (`gp3`) Amazon EBS volume in the same Availability Zone as the host. Size it to hold the VCF Installer appliance and the VCF installation bundles, at least 256 GB.

1. Attach the EBS volume to the ESX host. For instructions, see [Attach an Amazon EBS volume to an instance](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-attaching-volume.html) in the *Amazon EBS User Guide*.

1. Using the VMware Host Client or the vSphere APIs, create a local VMFS datastore on the attached EBS volume.

The EBS volume accrues storage charges until you delete it in [Reclaim the temporary datastore](#sd-install-reclaim).

#### Deploy the VCF Installer
<a name="sd-install-deploy"></a>

To deploy the VCF Installer, do the following.

1. Deploy the VCF Installer OVA onto the local VMFS datastore.

1. Attach the appliance to the `VM Network` port group.

1. Set its management IP address to the SDDC Manager address from your DNS plan, and set the appliance passwords and NTP servers. For the values to use, see [VCF Installer settings for Amazon EVS](#vcf-installer-settings).

The VCF Installer appliance becomes SDDC Manager during bringup, so it uses the SDDC Manager address. On VCF 9.0 and 9.1, VCF Operations is a separate appliance.

#### Connect the software depot
<a name="sd-install-depot"></a>

The VCF Installer downloads the VCF software from a software depot. Connect the VCF Installer to the Broadcom online depot, and then download the install binaries for your VCF version.

 **Broadcom online depot** 

1. Make sure that your Broadcom account has a valid VCF entitlement for the VCF version that you plan to deploy.

1. In the VCF Installer, open the depot settings, and configure the Broadcom online depot with your Broadcom credential. The credential depends on your VCF version.
   + For VCF 9.0, generate a Broadcom download token from the [Broadcom Support Portal](https://support.broadcom.com/), and enter the token in the VCF Installer.
   + For VCF 9.1, the VCF Installer authenticates to the Broadcom online depot with an activation code. Generate a software depot ID in the VCF Installer, and register it in the VCF Business Services console. Enter the activation code that the console returns in the VCF Installer, and then review and accept the certificate thumbprint. For more information, see [Broadcom knowledge base article 443647](https://knowledge.broadcom.com/external/article/443647).

1. Select the VCF version that you want, and download the install binaries. Downloading the binaries pulls that version of the VCF software into the local depot of the VCF Installer.

1. Wait until every download shows success before you validate the deployment specification.

**Note**  
Connecting to the Broadcom online depot and downloading software requires outbound internet access from the VCF Installer. The NAT gateway in your network foundation provides this access. For more information, see [Create a VPC with subnets and route tables](#network-create-vpc).

If the depot connection succeeds but downloads fail with a 403 error, check that your Broadcom entitlement covers the VCF version that you selected. For more information, see [Broadcom knowledge base article 440278](https://knowledge.broadcom.com/external/article/440278).

The Amazon EVS custom add-on depot is a different depot. It provides the Amazon EVS add-on for ESX cluster images, and you don’t use it in this step. For more information, see [Use the Amazon EVS custom add-on depot](#addon-depot).

When every download shows success, you have prepared the VCF Installer and downloaded the VCF software.

Next step: [Step 4b (local storage): Run VCF bringup](#sd-install-bringup).

### Step 4b (local storage): Run VCF bringup
<a name="sd-install-bringup"></a>

With the VMware Cloud Foundation (VCF) software downloaded, you create your VCF deployment specification, validate it, and run the deployment in the VCF Installer. Bringup deploys the VCF management appliances and creates the vSAN datastore. After bringup, you reclaim the temporary datastore.

Use these steps for hosts with local storage that use vSAN. For diskless hosts, see [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec).

**Important**  
In Self-deployed mode, Amazon Elastic VMware Service doesn’t manage VCF installation. For VCF-specific requests, you can use your active VCF subscription entitlements to contact Broadcom directly through the Broadcom Support Portal. For more information about support boundaries, see [Shared responsibility and support](#plan-shared-responsibility).

Before you begin, complete [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf).

#### Amazon EVS settings for the management cluster
<a name="sd-install-bringup-settings"></a>

When you configure the management cluster in the deployment specification, apply these Amazon EVS-specific settings.

 **vSAN**   
Use vSAN ESA (Express Storage Architecture) with failures to tolerate (FTT) set to at least 1. Amazon EVS doesn’t support vSAN OSA (Original Storage Architecture).

 **Uplink teaming**   
Use a failover teaming policy (an active uplink with a standby uplink) for the distributed switch port groups, rather than a load-balancing policy. Set the NSX teaming policy to `FAILOVER_ORDER`.

 **EVC mode**   
Set the cluster Enhanced vMotion Compatibility (EVC) mode to match your instance type. For `i7i.metal-24xl` and `i7i.metal-48xl`, use `INTEL_SAPPHIRERAPIDS`. For `i4i.metal`, don’t set an EVC mode in the specification. The EVS Deployment Orchestrator (EDO) leaves it unset for this instance type, because setting `INTEL_ICELAKE` can cause bringup to fail on VCF 9.1.

 **Networks**   
Use the Amazon EVS-assigned VLAN IDs and the MTU values in [Amazon EVS network settings for VCF](#sd-install-network).

 **Passwords**   
Set the passwords for the VCF management appliances. Each appliance enforces its own password complexity requirements, which VCF defines. If an appliance rejects a password, the validation error states the requirement that the password must meet. For the password rules, see [VCF Installer settings for Amazon EVS](#vcf-installer-settings).

#### Run bringup
<a name="sd-install-bringup-run"></a>

**Important**  
Amazon EVS doesn’t support running ESX outside of a full VCF deployment, or workload virtual machines on the Amazon EVS VLAN subnets (the underlay). Deploy workload virtual machines onto NSX overlay networks. For more information, see [Amazon EVS architecture](#architecture).

To run bringup, do the following.

1. In the VCF Installer, create the deployment specification for your management domain. Apply the Amazon EVS-specific settings in the previous section.

1. Run validation against the specification, and resolve any errors that it reports. Some standard VCF Installer validation checks don’t apply to the Amazon EVS network environment, such as the gateway ping check, the ESX host thumbprint check, and, on hosts with local storage, the vSAN ESA certified disk check. For how to handle them, see [VCF Installer settings for Amazon EVS](#vcf-installer-settings).

1. Run the deployment.

Bringup is the longest part of the installation and accounts for most of the setup time. When you use vSAN, forming the datastore and deploying the management appliances can take several hours.

**Tip**  
When you run bringup through the VCF Installer wizard, use the wizard to identify and correct specification errors. The wizard shows validation problems more clearly than the API. To generate an example bringup specification, use the [spec-generator tool](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy/EVS-Deployment-Orchestrator/spec-generator) in the `Deploy` folder of the [Solutions for Amazon EVS](https://github.com/aws/solutions-for-amazon-evs) repository on GitHub.

#### Reclaim the temporary datastore
<a name="sd-install-reclaim"></a>

When bringup finishes, the management appliances run on the vSAN datastore, and the temporary VMFS datastore is empty. Reclaim the temporary datastore to stop accruing storage charges.

1. In the vSphere Client, unmount the temporary VMFS datastore from the host.

1. In the Amazon EC2 console or with the AWS CLI, detach the EBS volume from the host instance. For instructions, see [Detach an Amazon EBS volume from an instance](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-detaching-volume.html) in the *Amazon EBS User Guide*.

1. Wait until the volume state is `available`. To check the volume state, see [View Amazon EBS volume information](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-describing-volumes.html).

1. Delete the volume with the EC2 console or the AWS CLI. For instructions, see [Delete an Amazon EBS volume](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-deleting-volume.html) in the *Amazon EBS User Guide*.

You have installed VCF on your hosts. If bringup fails, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

Next step: [Step 5: Configure NSX networking](#self-deployed-nsx-edge).

### Step 4a (diskless): Prepare diskless hosts and the VCF Installer
<a name="getting-started-diskless"></a>

Diskless hosts have no local storage for VMware Cloud Foundation (VCF). They boot from an Amazon EBS volume, and vSAN and local VMFS datastores aren’t available, so you provide external storage for the management domain. For the instance types that are diskless, see [Supported VCF versions and instance types](#versions-provided). In this step, you create the external storage, prepare the first host, deploy the VCF Installer appliance, download the VCF software, and add the remaining hosts. In [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec), you create the deployment specification and deploy VCF.

The example on this page deploys a VCF 9.x management domain on two r7i.metal-24xl hosts. It uses Amazon FSx for OpenZFS as the only storage, with a Network File System (NFS) v3 principal datastore and no vSAN. The VCF Installer accepts a management domain with two hosts and an NFS principal datastore.

**Note**  
You can use Amazon FSx for NetApp ONTAP or Everpure Cloud Dedicated instead. Follow the same sequence, and replace the storage creation and datastore mount steps with the procedures for your storage. For more information, see [External storage options for diskless hosts](#external-storage-diskless).

#### Before you begin
<a name="getting-started-diskless-prereqs"></a>

Make sure that you meet the following prerequisites.
+ You created an environment with `SELF_DEPLOYED` as the VCF version. For more information, see [Step 1: Create the environment](#self-deployed-create-env).
+ You associated the Amazon Elastic VMware Service VLAN subnets with a route table that has a default route to your NAT gateway. The VCF Installer downloads software from the Broadcom online depot, so the VM management VLAN subnet needs outbound internet access. For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).
+ You created and verified the forward (A) and reverse (PTR) DNS records for every host and VCF appliance. For more information, see [Step 2: Create DNS records](#self-deployed-host-dns-records).
+ You added the first diskless host, and it is in the `CREATING` or `CREATED` state. For more information, see [Step 3: Add hosts](#self-deployed-add-hosts). You can add the remaining hosts later in this step, while the VCF software downloads.
+ You have a computer or an instance with a web browser that can reach the private IP addresses of your VPC. You use it to open the VMware Host Client and the VCF Installer. For more information, see [Launch an instance to reach the VPC](#network-access-instance).
+ You have a Broadcom account with a VCF entitlement, and you downloaded the VCF Installer Open Virtualization Appliance (OVA) file for your VCF version from the [Broadcom Support Portal](https://support.broadcom.com/).
+ You chose the size and type of your external storage. For more information, see [External storage options for diskless hosts](#external-storage-diskless) and [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore).

The examples on this page and in Step 4b (diskless) use the following values. Replace the example addresses and names with your own values.


| Item | Example value | 
| --- | --- | 
| Instance type |  `r7i.metal-24xl`  | 
| VPC CIDR block | 10.10.0.0/16 | 
| Domain name in the DHCP option set |  `evs.example.com`  | 
| Service access subnet | 10.10.200.0/24 | 
| DNS servers in the DHCP option set | 10.10.200.100 and 10.10.200.101 | 
| Host management VLAN subnet | 10.10.0.0/24 | 
| VM management VLAN subnet | 10.10.1.0/24 | 
| Host names and addresses |  `esx01` (10.10.0.11) and `esx02` (10.10.0.12) | 
| VCF Installer, which becomes SDDC Manager |  `sddcm` (10.10.1.12) | 
| FSx for OpenZFS endpoint IP address | 10.10.200.50 | 

##### DNS records for the example
<a name="getting-started-diskless-dns"></a>

Create an A record and a PTR record for each name in the following table before you add the first host. The example uses the `evs.example.com` zone. The specification in Step 4b (diskless) gives no IP addresses for the appliances, so the VCF Installer takes them from these records. For the record requirements and the verification steps, see [Step 2: Create DNS records](#self-deployed-host-dns-records).


| Name | IP address | Use | 
| --- | --- | --- | 
|  `esx01`  | 10.10.0.11 | First host | 
|  `esx02`  | 10.10.0.12 | Second host | 
|  `vc`  | 10.10.1.10 | vCenter Server | 
|  `nsx`  | 10.10.1.11 | NSX Manager virtual IP address | 
|  `sddcm`  | 10.10.1.12 | VCF Installer, which becomes SDDC Manager | 
|  `nsx01`  | 10.10.1.16 | NSX Manager node | 
|  `vcfops`  | 10.10.1.19 | VCF Operations | 
|  `vcfopscol`  | 10.10.1.23 | VCF Operations collector | 
|  `vsp-platform`  | 10.10.1.25 | VCF services platform | 
|  `vsp-instance`  | 10.10.1.26 | VCF services platform | 
|  `vsp-fleet`  | 10.10.1.27 | VCF services platform | 
|  `vcf-vidb`  | 10.10.1.28 | VCF identity database | 
|  `vcf-license`  | 10.10.1.29 | VCF license server | 

Keep these addresses outside the address ranges in the specification. In the example, the VCF services platform pool uses 10.10.1.100 to 10.10.1.113, and the NFS network range uses 10.10.0.20 to 10.10.0.29.

#### Create the external storage
<a name="getting-started-diskless-storage"></a>

Create your external storage while the first host is created. A host reaches the `CREATED` state in about 9 minutes, and a file system becomes available in 10 to 25 minutes.

For Amazon FSx for OpenZFS, create a file system and two NFS volumes. Use `vcf-principal` as the principal datastore and `installer-staging` as the temporary datastore for the VCF Installer appliance. Note the endpoint IP address of the file system. For instructions, see [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore).

#### Prepare the first host
<a name="getting-started-diskless-prep-host"></a>

The VCF Installer expects its appliance and the principal datastore on the first host. Diskless hosts have no local storage, so you use the two NFS volumes for both. This procedure replaces the Amazon EBS-backed temporary datastore that hosts with local storage use.

Amazon EVS stores the ESX root password of each host in AWS Secrets Manager. Each host has its own secret. To find the secret, search Secrets Manager for the environment ID. The value of the secret is a plain string and not JSON. The `credentials` list of the environment stays empty in Self-deployed mode.

In the VMware Host Client, do the following:

1. Open `https://esx01.evs.example.com` in your browser. The certificate is self-signed, so your browser shows a warning.

1. Sign in as `root` with the root password from Secrets Manager.

1. Set the `VM Network` port group to the VM management VLAN ID, and keep the host management port group untagged with VLAN ID `0`. Without this VLAN ID, the VCF Installer appliance can’t reach the VM management VLAN subnet. For how to find the VLAN ID, see [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference).

1. Choose **Storage**, **Datastores**, and **New datastore**, and then choose **Mount NFS datastore**.

1. Enter the following values, and finish the wizard.
   + For **NFS server**, enter the endpoint IP address of the file system.
   + For **NFS share**, enter `/fsx/installer-staging`.
   + For **NFS version**, choose NFS 3.
   + For **Name**, enter a name such as `fsx-staging`.

1. Repeat the previous two steps to mount the principal datastore with the share `/fsx/vcf-principal` and a name such as `fsx-principal`.

You have mounted both datastores on the first host. Both datastores mount through the host management VMkernel adapter (`vmk0`). Don’t create a separate NFS VMkernel adapter.

Each datastore shows the capacity of the file system, or the volume quota if you set one. For example, a 7,168 GiB file system with no volume quota appears as a 7,168 GiB datastore. The VCF Installer checks this size during validation. For more information, see [Size the file system](#fsx-openzfs-sizing).

The principal datastore must be mounted on the first host before you validate the specification. Otherwise, the VCF Installer reports that it can’t find the datastore on the host. For more information, see [article 414088](https://knowledge.broadcom.com/external/article/414088) in the Broadcom knowledge base. If a mount fails, check the security group of the file system and the NFS export list. For more information, see [Security group rules](#fsx-openzfs-security-group) and [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

#### Deploy the VCF Installer
<a name="getting-started-diskless-installer"></a>

Deploy the VCF Installer OVA file onto the first host, on the staging datastore.

In the VMware Host Client, do the following:

1. Choose **Virtual Machines**, and then choose **Create / Register VM**.

1. Choose **Deploy a virtual machine from an OVF or OVA file**, enter a name such as `sddcm`, and select the OVA file.

1. Choose the `fsx-staging` datastore.

1. On the deployment options page, map the network to `VM Network`, choose thin disk provisioning, and turn on **Power on automatically**.

1. On the additional settings page, enter the appliance values. The following table shows example values.


<table>
<thead>
  <tr><th>Setting</th><th>Example value</th></tr>
</thead>
<tbody>
  <tr><td>Host Name</td><td> <code>sddcm.evs.example.com</code> </td></tr>
  <tr><td>Network 1 IPv4 Address</td><td>10.10.1.12</td></tr>
  <tr><td>Network 1 Subnet Mask</td><td>255.255.255.0</td></tr>
  <tr><td>Network Default IPv4 Gateway</td><td>10.10.1.1</td></tr>
  <tr><td>DNS Domain Name</td><td> <code>evs.example.com</code> </td></tr>
  <tr><td>Domain Search Path</td><td> <code>evs.example.com</code> </td></tr>
  <tr><td>Domain Name Servers</td><td>10.10.200.100,10.10.200.101</td></tr>
  <tr><td>NTP Servers</td><td> <code>time.aws.com</code> </td></tr>
  <tr><td>Root user password and local user password</td><td>Your own passwords</td></tr>
</tbody>
</table>


1. Finish the wizard.

The VCF Installer appliance becomes SDDC Manager after deployment. Its IP address and host name must match the SDDC Manager record in your DNS plan. The passwords must be at least 15 characters long and contain an uppercase letter, a lowercase letter, a digit, and one of the special characters `! % @ $ ^ # ? *`. For more information, see [VCF Installer settings for Amazon EVS](#vcf-installer-settings).

The import of the OVA file onto the NFS datastore takes about 2 minutes. The VCF Installer web service responds within a few minutes after the appliance starts. To verify, open `https://sddcm.evs.example.com` in your browser, and sign in as `admin@local` with the local user password that you entered in the wizard.

Right after the first start, the inventory page of the VCF Installer can show an error for more than 30 minutes. The error doesn’t stop you from connecting the Broadcom online depot or from validating the specification. Continue with the next section, and don’t wait for the inventory page.

#### Connect the software depot
<a name="getting-started-diskless-depot"></a>

The VCF Installer downloads the VCF software from a software depot. Connect the VCF Installer to the Broadcom online depot, and then download the install binaries for your VCF version.

 **Broadcom online depot** 

1. Make sure that your Broadcom account has a valid VCF entitlement for the VCF version that you plan to deploy.

1. In the VCF Installer, open the depot settings, and configure the Broadcom online depot with your Broadcom credential. The credential depends on your VCF version.
   + For VCF 9.0, generate a Broadcom download token from the [Broadcom Support Portal](https://support.broadcom.com/), and enter the token in the VCF Installer.
   + For VCF 9.1, the VCF Installer authenticates to the Broadcom online depot with an activation code. Generate a software depot ID in the VCF Installer, and register it in the VCF Business Services console. Enter the activation code that the console returns in the VCF Installer, and then review and accept the certificate thumbprint. For more information, see [Broadcom knowledge base article 443647](https://knowledge.broadcom.com/external/article/443647).

1. Select the VCF version that you want, and download the install binaries. Downloading the binaries pulls that version of the VCF software into the local depot of the VCF Installer.

1. Wait until every download shows success before you validate the deployment specification.

**Note**  
Connecting to the Broadcom online depot and downloading software requires outbound internet access from the VCF Installer. The NAT gateway in your network foundation provides this access. For more information, see [Create a VPC with subnets and route tables](#network-create-vpc).

If the depot connection succeeds but downloads fail with a 403 error, check that your Broadcom entitlement covers the VCF version that you selected. For more information, see [Broadcom knowledge base article 440278](https://knowledge.broadcom.com/external/article/440278).

The Amazon EVS custom add-on depot is a different depot. It provides the Amazon EVS add-on for ESX cluster images, and you don’t use it in this step. For more information, see [Use the Amazon EVS custom add-on depot](#addon-depot).

The catalog lists install and patch images of each component, and several builds of your VCF version. Download the install image of the newest build of each component within the VCF version that you selected. The VCF Installer downloads three bundles at a time.

#### Add the remaining hosts
<a name="getting-started-diskless-host2"></a>

Add the other hosts while the downloads run. All hosts in a cluster must use the same instance type and the same ESX version. Before you add each host, verify its DNS records. For more information, see [Step 2: Create DNS records](#self-deployed-host-dns-records).

Diskless hosts support only ESX 9.x versions. Choose an ESX version that Amazon EVS provides for the instance type and that matches your VCF Installer build. To list the ESX versions for the instance type, run `aws evs get-versions`. For more information, see [Supported VCF versions and instance types](#versions-provided).

You can add a host by using the Amazon EVS console or the AWS CLI.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**, and then choose your environment.

1. Choose the **Hosts** tab, and then choose **Create host**.

1. Enter the short host name from your DNS records, such as `esx02`. Choose your Amazon EC2 key pair. Choose the same instance type and ESX version as for the first host.

1. Choose **Create host**.

1. Run the `aws evs create-environment-host` command.

   In the following command, replace `<esx-version>` with the ESX version of the first host, as listed in the `get-versions` response. Replace the example environment ID, host name, and key pair name with your own values. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs create-environment-host \
       --environment-id env-abcde12345 \
       --esx-version <esx-version> \
       --host '{
           "hostName": "esx02",
           "keyName": "my-ec2-key-pair",
           "instanceType": "r7i.metal-24xl"
       }' \
       --region <region-code>
   ```

1. Check the host state. The create command doesn’t return a host ID, so use the host name to find the host.

   Replace the example environment ID with your own value, and replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs list-environment-hosts \
       --environment-id env-abcde12345 \
       --query 'environmentHosts[].[hostName,hostState,ipAddress]' \
       --output table \
       --region <region-code>
   ```

Don’t run the create command again for a host that is in the `CREATING` state. A host usually reaches `CREATED` in about 9 minutes. When a host is `CREATED`, confirm that its IP address equals the address in its A record. If a host stays in `CREATING`, check its DNS records first. For more information, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

You don’t prepare the other hosts. The VCF Installer mounts the datastores on them during deployment. Get the root password of each host from Secrets Manager, because you need it in the specification.

When every host is `CREATED` and every download shows success, you have prepared the diskless hosts and the VCF Installer.

Next step: [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec).

**Topics**

#### Configure FSx for OpenZFS as principal storage for diskless hosts
<a name="config-fsx-openzfs-datastore"></a>

This page applies to Self-deployed environments that run VCF 9.x on diskless hosts. For how EVS-deployed environments (VCF 5.2.x) differ, see [Differences between deployment modes](#deployment-mode-differences).

Diskless hosts need external storage for the VMware Cloud Foundation (VCF) management domain. You size and create an Amazon FSx for OpenZFS file system and two Network File System (NFS) volumes, and you set the security group rules that they need. You can grow the capacity later. You mount the volumes on the first host and reference the principal volume in the VCF Installer specification, as described in [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless) and [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec). For an overview of the storage options, see [External storage options for diskless hosts](#external-storage-diskless).

##### Before you begin
<a name="fsx-openzfs-prereqs"></a>

Make sure that you meet the following prerequisites.
+ You created an Amazon Elastic VMware Service environment in Self-deployed mode in your VPC. For more information, see [Step 1: Create the environment](#self-deployed-create-env).
+ You know the CIDR block of the host management (`vmkManagement`) VLAN subnet. You use it in the NFS export list and in the security group.
+ You know the ID of the service access subnet that Amazon EVS uses.
+ You or your storage administrator have permissions to create and manage FSx for OpenZFS file systems in your VPC. For more information, see [Identity and access management for Amazon FSx for OpenZFS](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/security-iam.html) in the *Amazon FSx for OpenZFS User Guide*.

##### Size the file system
<a name="fsx-openzfs-sizing"></a>

The VCF Installer checks the capacity of the principal datastore during validation. The datastore size that the ESX host reports depends on the file system and on the volume quota. A volume with no quota appears on the ESX host with the capacity of the file system. Provision enough capacity for the Installer requirement plus the amount that the Installer advises to stay under 80 percent utilization.

For the two-host management domain specification in [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec), the VCF Installer reported the following values.


| Installer check | Value | 
| --- | --- | 
| Required capacity for the management domain | 5,398 GB | 
| Recommended capacity to stay under 80 percent utilization | 6,748 GB | 

The VCF Installer reports capacity in GB. Amazon FSx capacity and the recommendation that follows are in GiB.

For a new deployment of this size, we recommend that you provision at least 7,168 GiB. A file system with 7,168 GiB of SSD storage capacity and no volume quota appears on the ESX host as a 7,168 GiB datastore. At this size, both Installer capacity checks pass and the Installer shows no capacity advisory. If you set a volume quota, keep it above 6,748 GB.

After the two-host management domain is deployed, the principal datastore holds about 284 GiB. The datastores are thin provisioned, so the file system leaves a large amount of free space for workloads. Because both volumes share the file system, the second volume can show a slightly smaller size than the principal volume. Use the size of the principal datastore for capacity planning.

**Important**  
You can increase the SSD storage capacity of a file system later, but you can’t decrease it. Plan the capacity before you deploy VCF. For more information, see [Increase the capacity](#fsx-openzfs-grow).

For prices, see the [Amazon FSx for OpenZFS pricing page](https://aws.amazon.com/fsx/openzfs/pricing).

##### Create the file system
<a name="fsx-openzfs-create-fs"></a>

Create a file system in the service access subnet of your environment. You can use the Amazon FSx console or the AWS CLI.

Before you create the file system, create a security group in your VPC with the rules in [Security group rules](#fsx-openzfs-security-group).

Create the file system with the option to copy tags to volumes turned off. If this option is on and the file system has a `Name` tag, creating a volume with a `Name` tag that has a different value fails with a `BadRequest` error.

To create the file system in the Amazon FSx console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon FSx console](https://console.aws.amazon.com/fsx).

1. Choose **Create file system**, and then choose **Amazon FSx for OpenZFS**.

1. For **Deployment type**, choose **Single-AZ (HA)**, and then choose **Single-AZ 2**.

1. For **Storage class**, choose **SSD (provisioned)**, and for **SSD storage capacity**, enter the capacity from [Size the file system](#fsx-openzfs-sizing).

1. For **Throughput capacity**, choose 640 MBps.

1. For **VPC**, choose the VPC of your environment, and for **Subnet**, choose the Amazon EVS service access subnet.

1. For **Security group**, choose a security group that allows inbound NFS traffic from the host management VLAN subnet. For the rules, see [Security group rules](#fsx-openzfs-security-group).

1. Turn off the option to copy tags to volumes.

1. Under **Root volume configuration**, set the data compression type to **LZ4**.

1. For the NFS exports of the root volume, specify the host management VLAN subnet CIDR block as the client, and set the options to `rw,crossmnt,no_root_squash`.

1. Review the settings, and choose **Create file system**.

1. Run the `aws fsx create-file-system` command. The following example creates a Single-AZ (HA) generation 2 file system. For `--subnet-ids`, specify the Amazon EVS service access subnet. For `Clients`, specify the CIDR block of the host management VLAN subnet. Replace the values in angle brackets with your own values. Replace the example CIDR block with your own value. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws fsx create-file-system \
       --file-system-type OPENZFS \
       --storage-capacity 7168 \
       --storage-type SSD \
       --subnet-ids <subnet-id> \
       --security-group-ids <security-group-id> \
       --open-zfs-configuration '{
           "DeploymentType": "SINGLE_AZ_HA_2",
           "ThroughputCapacity": 640,
           "AutomaticBackupRetentionDays": 0,
           "CopyTagsToVolumes": false,
           "RootVolumeConfiguration": {
               "RecordSizeKiB": 128,
               "DataCompressionType": "LZ4",
               "NfsExports": [{
                   "ClientConfigurations": [{
                       "Clients": "10.10.0.0/24",
                       "Options": ["rw", "crossmnt", "no_root_squash"]
                   }]
               }]
           }
       }' \
       --region <region-code>
   ```

Keep the `no_root_squash` option, because ESX mounts NFS datastores as the root user. The example turns off automatic backups. Choose the backup and maintenance settings that match your requirements. For more information, see [Creating FSx for OpenZFS file systems](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/creating-file-systems.html) in the *Amazon FSx for OpenZFS User Guide*.

A Single-AZ (HA) generation 2 file system with 7,168 GiB of SSD storage and 640 MBps of throughput becomes available in 10 to 25 minutes. Wait until the file system is available before you create the volumes. The endpoint IP address appears when the file system is available.

If you created the file system with the option to copy tags to volumes turned on, turn it off with the following command. The change causes no interruption. Replace `<file-system-id>` with the ID of your file system, and replace `<region-code>` with the Region where you created the environment.

```bash
aws fsx update-file-system \
    --file-system-id <file-system-id> \
    --open-zfs-configuration CopyTagsToVolumes=false \
    --region <region-code>
```

**Note**  
An Amazon EVS environment runs in one Availability Zone. For more information, see [VPC and subnets](#plan-network-supported-vpc). A Multi-AZ file system keeps the storage available if an Availability Zone fails, but it doesn’t keep your Amazon EVS environment running. Choose the deployment type for your availability and cost requirements. For more information, see [Availability and durability](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/availability-durability.html) in the *Amazon FSx for OpenZFS User Guide*.

###### Security group rules
<a name="fsx-openzfs-security-group"></a>

The FSx for OpenZFS file system doesn’t start outbound connections. The NFS datastores mount through the host management VMkernel adapter, so allow the following inbound traffic from the CIDR block of the host management VLAN subnet.


| Purpose | Protocol | Ports | 
| --- | --- | --- | 
| NFS and related services | TCP and UDP | 111, 2049, and 20001 to 20003 | 
| Connectivity checks | Internet Control Message Protocol (ICMP) | All | 

For more information, see [File System Access Control with Amazon VPC](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/limit-access-security-groups.html) in the *Amazon FSx for OpenZFS User Guide*.

##### Create the volumes
<a name="fsx-openzfs-create-volumes"></a>

Create two child volumes of the root volume.

 `vcf-principal`   
The principal datastore for the VCF management domain. The VCF Installer deploys the management appliances onto this volume.

 `installer-staging`   
The temporary datastore for the VCF Installer appliance. You mount it on the first host before you deploy the VCF Installer. During deployment, the VCF Installer moves its appliance to the principal datastore.

Create each volume with the following settings.

 **Parent volume**   
Use the root volume of the file system.

 **NFS exports**   
Use the host management VLAN subnet CIDR block with the options `rw,crossmnt,no_root_squash`.

 **Storage capacity quota**   
Use none (`-1`). The volume is then limited by the quota of the root volume, which equals the file system capacity. The describe output of a volume without a quota shows no quota value. A smaller quota limits the datastore size that ESX reports, so keep any quota above 6,748 GB.

 **Record size**   
The examples on this page use 16 KiB for `vcf-principal` and 128 KiB for `installer-staging`.

 **Tags**   
If you add a `Name` tag, make sure that the option to copy tags to volumes is turned off on the file system.

To create the volumes in the Amazon FSx console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon FSx console](https://console.aws.amazon.com/fsx).

1. In the navigation pane, choose **File systems**, and then choose your file system.

1. Choose the **Volumes** tab, and then choose **Create volume**.

1. For **Volume name**, enter `vcf-principal`.

1. For **Parent volume**, choose the root volume.

1. Apply the settings in the previous list, and choose **Confirm**.

1. Repeat these steps for `installer-staging`.

1. Get the ID of the root volume. Replace `<file-system-id>` with the ID of your file system, and replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws fsx describe-file-systems \
       --file-system-ids <file-system-id> \
       --query 'FileSystems[0].OpenZFSConfiguration.RootVolumeId' \
       --output text \
       --region <region-code>
   ```

1. Create the principal volume. Replace `<root-volume-id>` with the ID of the root volume from the previous step. Replace the example CIDR block with the CIDR block of your host management VLAN subnet.

   ```bash
   aws fsx create-volume \
       --volume-type OPENZFS \
       --name vcf-principal \
       --open-zfs-configuration '{
           "ParentVolumeId": "<root-volume-id>",
           "StorageCapacityQuotaGiB": -1,
           "RecordSizeKiB": 16,
           "DataCompressionType": "LZ4",
           "NfsExports": [{
               "ClientConfigurations": [{
                   "Clients": "10.10.0.0/24",
                   "Options": ["rw", "crossmnt", "no_root_squash"]
               }]
           }]
       }' \
       --region <region-code>
   ```

1. Repeat the command with `--name installer-staging` and `"RecordSizeKiB": 128`.

You have created the two volumes.

##### Get the endpoint IP address
<a name="fsx-openzfs-endpoint"></a>

You use the endpoint IP address of the file system as the NFS server when you mount the volumes on the host and in the VCF Installer specification. To get the endpoint IP address in the Amazon FSx console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon FSx console](https://console.aws.amazon.com/fsx).

1. In the navigation pane, choose **File systems**, and then choose your file system.

1. Choose the **Network & security** tab.

1. Copy the **Endpoint IP address**.

1. Run the following command. Replace `<file-system-id>` with the ID of your file system, and replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws fsx describe-file-systems \
       --file-system-ids <file-system-id> \
       --query 'FileSystems[0].OpenZFSConfiguration.EndpointIpAddress' \
       --output text \
       --region <region-code>
   ```

The NFS export path of a volume is `/fsx/<volume-name>`. For example, the path of the principal volume is `/fsx/vcf-principal`.

##### Increase the capacity
<a name="fsx-openzfs-grow"></a>

If you need more capacity before you deploy VCF, complete the following steps in order. After you deploy VCF, the principal datastore holds your management appliances, and you can’t unmount a datastore while virtual machines use it.

1. Increase the SSD storage capacity of the file system. Each increase must be at least 10 percent of the current capacity. After a change, you must wait six hours before you can change the capacity again. Replace `<file-system-id>` with the ID of your file system, and replace the example capacity with your own value. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws fsx update-file-system \
       --file-system-id <file-system-id> \
       --storage-capacity 9216 \
       --region <region-code>
   ```

1. If you set a storage capacity quota on the volume, raise the quota. Each volume has its own quota, and a volume doesn’t use the new capacity until you raise it. Replace `<volume-id>` with the ID of the volume.

   ```bash
   aws fsx update-volume \
       --volume-id <volume-id> \
       --open-zfs-configuration StorageCapacityQuotaGiB=9216 \
       --region <region-code>
   ```

1. Unmount the NFS datastore from the host and mount it again, so that ESX reads the new size. A storage rescan doesn’t update the reported size.

To delete the file system when you no longer need it, see [Clean up a Self-deployed environment](#self-deployed-cleanup).

Next step: Mount the volumes on the first host, as described in [Prepare the first host](#getting-started-diskless-prep-host).

### Step 4b (diskless): Create the specification and deploy
<a name="deploy-diskless-spec"></a>

In this step, you describe the VMware Cloud Foundation (VCF) management domain in a JSON deployment specification, validate it in the VCF Installer, and deploy VCF. Then you verify the cluster and the storage in the vSphere Client. The example is a two-host VCF 9.x management domain on r7i.metal-24xl hosts, with one NFS principal datastore on Amazon FSx for OpenZFS and no vSAN.

Before you begin, make sure that the following are true.
+ You completed [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).
+ Every host is in the `CREATED` state.
+ The principal datastore is mounted on the first host, and every download shows success.

#### Create the deployment specification
<a name="getting-started-diskless-spec"></a>

To create the specification, do the following.

1. List the VLAN IDs that Amazon Elastic VMware Service assigned to your environment. The VLAN IDs can differ for each environment, so read them from your environment.

   Replace the example environment ID with your own value. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs list-environment-vlans \
       --environment-id env-abcde12345 \
       --query 'environmentVlans[].[functionName,vlanId,cidr]' \
       --output table \
       --region <region-code>
   ```

   You need the VM management (`vmManagement`), vMotion, and host overlay (`vTep`) VLAN IDs. The output also lists a VLAN ID for `vmkManagement`. Don’t use it, because the host management network is untagged and uses VLAN ID `0`.

1. Save the following specification to a file. It’s for a two-host management domain that uses one NFS datastore and no vSAN.

   Replace the example addresses and names with your own values.

   ```json
   {
     "version": "<vcf-spec-version>",
     "vcfInstanceName": "evs-vcf",
     "sddcId": "evs-m01",
     "ceipEnabled": false,
     "skipEsxThumbprintValidation": true,
     "skipGatewayPingValidation": true,
     "workflowType": "VCF",
     "dnsSpec": {
       "subdomain": "evs.example.com",
       "nameservers": ["10.10.200.100", "10.10.200.101"]
     },
     "ntpServers": ["time.aws.com"],
     "hostSpecs": [
       {
         "hostname": "esx01.evs.example.com",
         "credentials": { "username": "root", "password": "<esx01-root-password>" }
       },
       {
         "hostname": "esx02.evs.example.com",
         "credentials": { "username": "root", "password": "<esx02-root-password>" }
       }
     ],
     "networkSpecs": [
       {
         "networkType": "MANAGEMENT",
         "ipAddressVersion": "IPv4",
         "subnet": "10.10.0.0/24",
         "gateway": "10.10.0.1",
         "vlanId": "0",
         "mtu": 1500,
         "portGroupKey": "DVPG_FOR_MANAGEMENT",
         "teamingPolicy": "failover_explicit",
         "activeUplinks": ["uplink1"],
         "standbyUplinks": ["uplink2"]
       },
       {
         "networkType": "VM_MANAGEMENT",
         "ipAddressVersion": "IPv4",
         "subnet": "10.10.1.0/24",
         "gateway": "10.10.1.1",
         "vlanId": "<vm-management-vlan-id>",
         "mtu": 1500,
         "portGroupKey": "DVPG_FOR_VM_MANAGEMENT",
         "teamingPolicy": "failover_explicit",
         "activeUplinks": ["uplink1"],
         "standbyUplinks": ["uplink2"]
       },
       {
         "networkType": "VMOTION",
         "ipAddressVersion": "IPv4",
         "subnet": "10.10.2.0/24",
         "gateway": "10.10.2.1",
         "vlanId": "<vmotion-vlan-id>",
         "mtu": 8500,
         "includeIpAddressRanges": [
           { "startIpAddress": "10.10.2.10", "endIpAddress": "10.10.2.20" }
         ],
         "portGroupKey": "DVPG_FOR_VMOTION",
         "teamingPolicy": "failover_explicit",
         "activeUplinks": ["uplink1"],
         "standbyUplinks": ["uplink2"]
       },
       {
         "networkType": "NFS",
         "ipAddressVersion": "IPv4",
         "subnet": "10.10.0.0/24",
         "gateway": "10.10.0.1",
         "vlanId": "0",
         "mtu": 1500,
         "includeIpAddressRanges": [
           { "startIpAddress": "10.10.0.20", "endIpAddress": "10.10.0.29" }
         ],
         "portGroupKey": "DVPG_FOR_NFS",
         "teamingPolicy": "failover_explicit",
         "activeUplinks": ["uplink1"],
         "standbyUplinks": ["uplink2"]
       }
     ],
     "vspClusterSpec": {
       "ipv4Pool": {
         "ipRange": { "startIpAddress": "10.10.1.100", "endIpAddress": "10.10.1.113" }
       },
       "platformFqdn": "vsp-platform.evs.example.com",
       "instanceFqdn": "vsp-instance.evs.example.com",
       "fleetFqdn": "vsp-fleet.evs.example.com",
       "systemUserPassword": "<vsp-system-password>",
       "size": "small",
       "name": "evs-m01-vmsp-01",
       "internalClusterCidrIpv4": "198.18.0.0/15"
     },
     "nsxtSpec": {
       "transportVlanId": "<vtep-vlan-id>",
       "ipAddressPoolSpec": {
         "name": "evs-m01-cl01-tep01",
         "description": "ESX host overlay TEP IP pool",
         "subnets": [
           {
             "cidr": "10.10.4.0/24",
             "gateway": "10.10.4.1",
             "ipAddressPoolRanges": [
               { "start": "10.10.4.10", "end": "10.10.4.30" }
             ]
           }
         ]
       },
       "nsxtManagerSize": "medium",
       "vipFqdn": "nsx.evs.example.com",
       "rootNsxtManagerPassword": "<nsx-root-password>",
       "nsxtAdminPassword": "<nsx-admin-password>",
       "nsxtAuditPassword": "<nsx-audit-password>",
       "useExistingDeployment": false,
       "nsxtManagers": [
         { "hostname": "nsx01.evs.example.com" }
       ]
     },
     "vcfOperationsSpec": {
       "applianceSize": "small",
       "useExistingDeployment": false,
       "adminUserPassword": "<vcf-operations-admin-password>",
       "nodes": [
         {
           "hostname": "vcfops.evs.example.com",
           "rootUserPassword": "<vcf-operations-root-password>",
           "type": "<primary-node-type>"
         }
       ]
     },
     "vcfOperationsCollectorSpec": {
       "applianceSize": "small",
       "hostname": "vcfopscol.evs.example.com",
       "rootUserPassword": "<vcf-operations-collector-root-password>",
       "useExistingDeployment": false
     },
     "licenseServerSpec": { "hostname": "vcf-license.evs.example.com" },
     "vidbSpec": { "hostname": "vcf-vidb.evs.example.com" },
     "saltSpec": {},
     "saltRaasSpec": {},
     "telemetryAcceptorSpec": {},
     "fleetLcmSpec": { "hostname": "vsp-fleet.evs.example.com" },
     "sddcLcmSpec": { "hostname": "vsp-instance.evs.example.com" },
     "fleetDepotSpec": {},
     "vcenterSpec": {
       "vcenterHostname": "vc.evs.example.com",
       "adminUserSsoPassword": "<vcenter-sso-password>",
       "rootVcenterPassword": "<vcenter-root-password>",
       "vmSize": "small",
       "ssoDomain": "vsphere.local",
       "useExistingDeployment": false
     },
     "clusterSpec": {
       "datacenterName": "EVS-Datacenter",
       "clusterName": "EVS-Mgmt-Cluster",
       "clusterEvcMode": "INTEL_SAPPHIRERAPIDS"
     },
     "datastoreSpec": {
       "nfsDatastoreSpec": {
         "datastoreName": "fsx-principal",
         "nasVolume": {
           "serverName": ["<fsx-endpoint-ip>"],
           "path": "/fsx/vcf-principal",
           "readOnly": false,
           "enableBindToVmknic": false
         }
       }
     },
     "dvsSpecs": [
       {
         "dvsName": "evs-m01-cl01-vds01",
         "networks": ["MANAGEMENT", "VM_MANAGEMENT", "VMOTION", "NFS"],
         "mtu": 8500,
         "nsxtSwitchConfig": {
           "transportZones": [
             { "name": "evs-overlay-TZ", "transportType": "OVERLAY" },
             { "name": "evs-vlan-TZ", "transportType": "VLAN" }
           ],
           "hostSwitchOperationalMode": "STANDARD"
         },
         "vmnicsToUplinks": [
           { "id": "vmnic1", "uplink": "uplink1" },
           { "id": "vmnic0", "uplink": "uplink2" }
         ],
         "nsxTeamings": [
           {
             "policy": "FAILOVER_ORDER",
             "activeUplinks": ["uplink1"],
             "standByUplinks": ["uplink2"]
           }
         ],
         "lagSpecs": null
       }
     ],
     "sddcManagerSpec": {
       "hostname": "sddcm.evs.example.com",
       "localUserPassword": "<sddc-manager-local-password>",
       "useExistingDeployment": true
     }
   }
   ```

1. Replace the placeholders as described in the following table.


<table>
<thead>
  <tr><th>Placeholder</th><th>Value</th></tr>
</thead>
<tbody>
  <tr><td> <code>&lt;vcf-spec-version&gt;</code> </td><td>A specification version that your VCF Installer supports. For more information, see <b>Version string</b> in the points that follow.</td></tr>
  <tr><td> <code>&lt;esx01-root-password&gt;</code>, <code>&lt;esx02-root-password&gt;</code> </td><td>The ESX root passwords from the Amazon EVS secrets in AWS Secrets Manager.</td></tr>
  <tr><td> <code>&lt;fsx-endpoint-ip&gt;</code> </td><td>The endpoint IP address of the file system. For more information, see <a href="#fsx-openzfs-endpoint">Get the endpoint IP address</a>.</td></tr>
  <tr><td> <code>&lt;vm-management-vlan-id&gt;</code>, <code>&lt;vmotion-vlan-id&gt;</code>, <code>&lt;vtep-vlan-id&gt;</code> </td><td>The VLAN IDs from the <code>list-environment-vlans</code> output. Keep the quotation marks around the values.</td></tr>
  <tr><td> <code>&lt;primary-node-type&gt;</code> </td><td>The value that the VCF Installer API defines for the primary VCF Operations node. For the value of <code>nodes.type</code>, see <a href="https://developer.broadcom.com/xapis/vcf-installer-api/latest/data-structures/VcfOperationsSpec">VcfOperationsSpec</a> in the VCF Installer API reference.</td></tr>
  <tr><td> <code>&lt;sddc-manager-local-password&gt;</code> </td><td>The local user password that you entered for the VCF Installer OVA.</td></tr>
  <tr><td>All other passwords</td><td>Passwords that you choose. For the rules, see <a href="#vcf-installer-settings">VCF Installer settings for Amazon EVS</a>.</td></tr>
</tbody>
</table>


The following points apply to this specification.

Uplinks  
An r7i.metal-24xl host has the following uplinks.  


<table>
<thead>
  <tr><th>Instance type</th><th>Elastic Network Adapter uplinks</th><th>Speed</th></tr>
</thead>
<tbody>
  <tr><td> <code>r7i.metal-24xl</code> </td><td> <code>vmnic0</code>, <code>vmnic1</code> </td><td>37.5 Gbps each</td></tr>
</tbody>
</table>

 `vmnic0` is the uplink of the standard switch `vSwitch0`, and `vmnic1` is unused. Map `vmnic1` to `uplink1` and `vmnic0` to `uplink2` in `vmnicsToUplinks`, as in the example. The VCF Installer moves `vmnic0` from `vSwitch0` to the distributed switch. Use only these two uplinks.  
If you use r7i.metal-48xl, check the adapter names and speeds in the VMware Host Client before you fill in `vmnicsToUplinks`.

No vSAN  
The `datastoreSpec` contains only `nfsDatastoreSpec`, and the `networks` list of the distributed switch has no vSAN network. The specification has no `vsanSpec` and no reference to local devices.

NFS network  
You must declare a network of type `NFS` in `networkSpecs` and in the `networks` list of the distributed switch. The following rules apply to the NFS network.  
+ If the NFS network is missing, the VCF Installer rejects the specification with the errors `MISSING_NET_SPEC_ENTRY` and `MISSING_DVS_SPEC_NET_ENTRY`.
+ Declare the NFS network on the host management subnet, as in the example.
+ Give the NFS network an address range in `includeIpAddressRanges`, with at least one address for each host. The Installer assigns each host one address from this range. Without the range, the Installer reports `NOT_ENOUGH_IPS` and `MISSING_INCLUSION_RANGE`.
+ The datastore stays mounted through `vmk0`, and `enableBindToVmknic` is `false`.

Version string  
Replace `<vcf-spec-version>` with a specification version that your VCF Installer supports. The `version` field of the specification is described in [SddcSpec](https://developer.broadcom.com/xapis/vcf-installer-api/latest/data-structures/SddcSpec) in the VCF Installer API reference. Don’t enter a build number. If the version isn’t supported, the Installer rejects the specification and lists the specification versions that it supports.

Cluster EVC mode  
The example sets the Enhanced vMotion Compatibility (EVC) mode of the cluster, `clusterEvcMode`, to `INTEL_SAPPHIRERAPIDS`. The VCF Installer doesn’t apply this value, so you set the mode after deployment, as described in [Set the cluster EVC mode](#getting-started-diskless-evc).

NTP servers  
The example uses `time.aws.com`. The diskless hosts use the link-local Amazon Time Sync Service address `169.254.169.123` as their NTP source. Validation reports the NTP warnings that [Validate and deploy](#getting-started-diskless-deploy) lists, and you can acknowledge them.

Skipped validation checks  
The example sets `skipEsxThumbprintValidation` and `skipGatewayPingValidation` to `true`. For more information, see [VCF Installer settings for Amazon EVS](#vcf-installer-settings).

Appliance sizes  
The example uses one NSX Manager node of size `medium` and the `small` size for the other appliances. The VCF Installer accepts this configuration on two hosts.

#### Validate and deploy
<a name="getting-started-diskless-deploy"></a>

Validation checks the specification and deploys nothing.

In the VCF Installer, do the following:

1. Choose **Deploy Using JSON Spec**, and upload your specification file.

1. Review the summary and the JSON preview, and choose **Next**. The Installer validates the specification and shows the results.

1. Resolve any errors, and acknowledge the warnings that you accept.

1. Choose **Deploy**.

Validation of a two-host specification with an NFS principal datastore takes less than 10 minutes and runs 15 checks. Expect 13 checks to pass and two checks to show warnings, with no errors. The following warnings are expected with this configuration.


| Warning | Meaning | 
| --- | --- | 
|  `IP_REACHABLE_ESXI_HOST` for each host | The check pings `time.aws.com`, which doesn’t answer. NTP doesn’t require Internet Control Message Protocol (ICMP) traffic. You can acknowledge it. | 
|  `HOST_NTP_VALIDATION` for each host | The host reads the NTP source `169.254.169.123`, which is the Amazon EVS link-local time source, and the specification uses `time.aws.com`. You can acknowledge it. | 
|  `HOST_NTP_VALIDATION` for the VCF Installer appliance | The `/etc/ntp.conf` file of the appliance holds an Amazon EC2 DNS name that `time.aws.com` resolved to, and the check expects `time.aws.com`. You can acknowledge it. | 

The Installer shows no capacity warning when the datastore is 7,168 GiB. If you provision less capacity, the Installer can advise a larger datastore. For more information, see [Size the file system](#fsx-openzfs-sizing). If the validation shows errors, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

Deployment of a two-host management domain takes about 4.5 hours. Don’t start a second deployment while one is running.

During deployment, the VCF Installer moves its own appliance from the staging datastore to the principal datastore. This task shows no progress in the VCF Installer for about 35 minutes. You can see it as a `RelocateVM` task in vCenter Server. Don’t treat this pause as a stall.

#### Set the cluster EVC mode
<a name="getting-started-diskless-evc"></a>

The VCF Installer doesn’t apply the EVC mode from the specification, so you set it after deployment. To set the mode, do the following.

1. When the VCF Installer reports success, sign in to the vSphere Client as `administrator@vsphere.local` with the password from your specification.

1. Choose the cluster, choose **Configure**, choose **VMware EVC**, and set the mode to **Intel Sapphire Rapids**. The hosts report Intel Sapphire Rapids as their highest EVC mode.

#### Verify the result
<a name="getting-started-diskless-verify"></a>

In the vSphere Client, do the following:

1. Confirm that the cluster `EVS-Mgmt-Cluster` shows all hosts as connected, and that vSphere High Availability (HA) and Distributed Resource Scheduler (DRS) are on.

1. Confirm that vSAN isn’t enabled, and that the hosts have no local datastore other than the ESX system volumes.

1. Confirm that the datastore `fsx-principal` holds the management appliances, including the VCF Installer appliance.

1. Confirm that the datastores `fsx-principal` and `fsx-staging` are mounted on all hosts. If a datastore is missing on a host, mount it as in [Prepare the first host](#getting-started-diskless-prep-host).

1. Create a test virtual machine on the principal datastore, and migrate it between hosts with vMotion in both directions. Because the datastore is shared, vMotion moves only the memory and the state of the virtual machine.

In SDDC Manager, confirm that NFS is the primary datastore type of the cluster.

After deployment, the hosts can show a yellow warning status in vCenter Server. The warning is a vSphere HA configuration issue that says the host couldn’t reach the isolation address. The cluster stays green, and HA and DRS stay on. VCF sets the vSphere HA options `das.ignoreInsufficientHbDatastore` and `das.ignoreRedundantNetWarning` to `true`, so one NFS datastore and one network don’t raise warnings.

You have deployed the VCF management domain on diskless hosts with external storage.

Next step: [Step 5: Configure NSX networking](#self-deployed-nsx-edge). Then continue with [Step 6: Create connectors](#self-deployed-create-connectors) and [Step 7: Verify the environment](#self-deployed-verify).

When you no longer need the environment, see [Clean up a Self-deployed environment](#self-deployed-cleanup).

**Important**  
Deleting the environment doesn’t delete your external storage. The file system and its volumes continue to incur charges until you delete them.

### Step 5: Configure NSX networking
<a name="self-deployed-nsx-edge"></a>

After the VMware Cloud Foundation (VCF) Installer completes bringup, NSX Manager is operational. The NSX Edge cluster and the Tier-0 gateway aren’t yet configured for connectivity with Amazon VPC Route Server. In this step, you deploy the Edge cluster, create the Tier-0 gateway, and configure Border Gateway Protocol (BGP) peering with the route server in NSX Manager.

You can create your overlay networks with Tier-0 and Tier-1 gateways on the NSX Edges directly, or by configuring NSX VPCs, a centralized NSX transit gateway, and Edge clusters. For design guidance, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design).

**Note**  
NSX defines its own **VPC** and **transit gateway** abstractions, which are different from Amazon VPC and AWS Transit Gateway. On this page, VPC and transit gateway refer to the AWS resources unless the name starts with NSX.

Before you begin, confirm that the following are in place.
+ VCF installation completed successfully, and NSX Manager and your VCF management appliances are accessible.
+ Your route server is created with endpoints, peers, and propagations. For more information, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).
+ You have the two route server endpoint IP addresses. Both endpoints are in the service access subnet, which provides redundancy.
+ You have the route server peer IP addresses that you chose from the NSX uplink VLAN subnet. You assign them to the Tier-0 external interfaces.
+ You have chosen two private BGP autonomous system numbers (ASNs). They must match the values that you configured on the route server peers.
  + NSX Edge Tier-0 local ASN (for example, `65000`)
  + VPC Route Server (remote) ASN (for example, `65022`)

    Private ASNs are in the range 64512 to 65534 (16-bit) or 4200000000 to 4294967294 (32-bit).

#### Prepare the Edge networking objects
<a name="sd-nsx-prepare-objects"></a>

The NSX Edge nodes need the following vSphere and NSX objects before you deploy the Edge cluster. Create them in the vSphere Client and in NSX Manager.
+ A distributed port group with VLAN trunking (VLAN range 0 to 4094) for the Edge uplink and Edge TEP traffic.
+ An IP pool for the Edge tunnel endpoint (TEP) addresses. Take the addresses from the Edge overlay (`edgeVTep`) VLAN subnet of your environment.
+ A VLAN transport zone for the Edge uplinks.

If you deploy with the EVS Deployment Orchestrator (EDO), a tool in the [Solutions for Amazon EVS repository](https://github.com/aws/solutions-for-amazon-evs), EDO creates these objects. If you install VCF with the VCF Installer, you create the objects yourself. For more information, see [Deploy with the EVS Deployment Orchestrator](#deploy-with-edo).

#### Create an uplink profile
<a name="sd-nsx-uplink-profile"></a>

Before you deploy the NSX Edge cluster, create an uplink profile in NSX Manager. The uplink profile defines the teaming policy, transport VLAN, and maximum transmission unit (MTU) that Edge transport nodes use for overlay (Geneve) traffic.

For version-specific field labels and screenshots, see the Broadcom documentation in the following table.


| VCF version | Broadcom documentation | 
| --- | --- | 
| VCF 9.0 |  [Create an Uplink Profile](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/transport-zones-and-transport-nodes/configuring-profiles/create-an-uplink-profile.html)  | 
| VCF 9.1 |  [Add an Uplink Profile](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/advanced-network-management/transport-zones-and-transport-nodes/configuring-profiles/create-an-uplink-profile.html)  | 

Configure the profile with the following values for Amazon Elastic VMware Service.


| Parameter | Value | 
| --- | --- | 
| Name | For example, `edge-uplink-profile`  | 
| Teaming policy | Failover Order | 
| Active Uplinks |  `uplink1`  | 
| Standby Uplinks | Leave empty. NSX Edge VMs don’t support standby uplinks. | 
| Transport VLAN | The Amazon EVS-assigned Edge TEP (`edgeVTep`) VLAN ID. To find the VLAN ID, see the VLAN IDs step in [Prepare the VLAN subnets for VCF](#self-deployed-create-env-after). | 
| MTU |  `8500`  | 

Use the uplink name `uplink1` when you configure each Edge transport node in the next section.

#### Deploy the NSX Edge cluster
<a name="sd-nsx-edge-cluster"></a>

Deploy two Edge transport nodes and group them into an Edge cluster with the NSX Manager UI. The workflow order differs by VCF version. The following table lists the workflow for each VCF version and the Broadcom documentation for the complete procedure.


| VCF version | Workflow | Broadcom documentation | 
| --- | --- | --- | 
| VCF 9.0 | Create the Edge transport nodes one by one, and then create the Edge cluster. |  [Create an Edge Transport Node](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/installing-nsx-edge/create-an-edge-transport-node.html) and [Create an Edge Cluster](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/installing-nsx-edge/create-an-edge-cluster.html)  | 
| VCF 9.1 | Start an Edge cluster workflow, and add the nodes within it. |  [Create an NSX Edge Cluster and Add Edge Nodes](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/advanced-network-management/installing-nsx-edge/create-an-edge-transport-node.html)  | 

When you configure the Edge nodes for your Amazon EVS environment, use the following values.


| Parameter | Value | 
| --- | --- | 
| Name | A unique name for each Edge node (for example, `edge-node-01` and `edge-node-02`). | 
| Form factor | Large (recommended) | 
| Compute and datastore | A compute resource and datastore appropriate for your deployment. | 
| Management IP | An IP address from the Amazon EVS VM management VLAN subnet. | 
| Default gateway | The default gateway for the Amazon EVS VM management VLAN subnet. | 
| Management interface | The VM management port group. | 
| Transport networking | The Amazon EVS-assigned transport VLAN and the uplink profile that you created, for datapath connectivity. | 

#### Create the Tier-0 gateway
<a name="sd-nsx-tier0"></a>

To create the Tier-0 gateway, do the following.

1. In NSX Manager, choose **Networking**, and then choose **Tier-0 Gateways**.

1. Choose **Add Tier-0 Gateway** and configure the following.

   1. For **Name**, enter a name, for example `evs-tier0-gw`.

   1. For **HA mode**, choose **Active-Standby**, and set the failover mode to **Non-preemptive**.

   1. For **Edge cluster**, choose the Edge cluster that you created.

1. Save the Tier-0 gateway.

1. Expand **Interfaces**, and add one external interface for each Edge node. Each interface is on the NSX uplink VLAN, through the VLAN transport zone that you created.

   1. For **Edge node**, choose Edge node 1 for the first interface and Edge node 2 for the second.

   1. For **IP address**, enter a VPC Route Server peer IP address that you chose in [Set up VPC Route Server and on-premises connectivity](#network-route-server), with the prefix length of the NSX uplink VLAN subnet. Each interface uses a different peer IP address.

1. Save the interfaces.

The VPC Route Server accepts BGP sessions only from the peer IP addresses that you created, so each external interface must use one of them. The BGP neighbors in the next section use these interfaces as source interfaces.

**Note**  
Make sure that both the Tier-0 and Tier-1 gateways have **Non-preemptive** failover. Non-preemptive is the NSX default. It avoids the BGP session drop that occurs when a recovered Edge node fails back to the preferred node.

#### Configure BGP
<a name="sd-nsx-bgp"></a>

We recommend a full mesh, in which each of the two Edge nodes peers with both route server endpoints, for four BGP sessions in total. A full mesh keeps routes propagating if a route server endpoint goes into maintenance. At minimum, configure two sessions, with each Edge node peering with one endpoint.

The Tier-0 external interfaces and the route server endpoints are in different subnets, so these are multihop BGP sessions.

To configure BGP, do the following.

1. Select the Tier-0 gateway and expand **BGP**.

1. Turn on **BGP** and set the **Local AS** number to the NSX Edge Tier-0 ASN that you chose (for example, `65000`).

1. Under **BGP Neighbors**, configure peering between the Edge nodes and the route server endpoints. For a full mesh, add the four neighbors in the following table. Set the **Remote AS** of every neighbor to the VPC Route Server ASN (for example, `65022`) and the **Address family** to IPv4 Unicast.


<table>
<thead>
  <tr><th>Neighbor</th><th>Neighbor address</th><th>Source interface</th></tr>
</thead>
<tbody>
  <tr><td>Edge 1 to endpoint 1</td><td> <code>&lt;route-server-endpoint-1-ip&gt;</code> </td><td>External interface on Edge node 1</td></tr>
  <tr><td>Edge 1 to endpoint 2</td><td> <code>&lt;route-server-endpoint-2-ip&gt;</code> </td><td>External interface on Edge node 1</td></tr>
  <tr><td>Edge 2 to endpoint 1</td><td> <code>&lt;route-server-endpoint-1-ip&gt;</code> </td><td>External interface on Edge node 2</td></tr>
  <tr><td>Edge 2 to endpoint 2</td><td> <code>&lt;route-server-endpoint-2-ip&gt;</code> </td><td>External interface on Edge node 2</td></tr>
</tbody>
</table>


   Set the BGP multihop limit to at least 2. Make sure that the Tier-0 gateway can reach each route server endpoint IP address, for example with a static route to the endpoint through the uplink gateway.

1. Choose **Save**.

1. Wait for the BGP sessions to establish.

1. To advertise only overlay network routes to the VPC Route Server, configure route redistribution and an outbound route filter on the Tier-0 gateway.

   1. On the Tier-0 gateway, expand **Route Re-Distribution**. Turn on redistribution into BGP for your overlay network route types, for example **Tier-1 Connected** (workload segment subnets), **Tier-1 NAT**, and **Tier-1 Static Routes**.

      Don’t select **External Interface Subnet**, and don’t select Tier-0 **Static Routes**. Redistributing **External Interface Subnet** advertises the NSX uplink subnet to AWS. Redistributing **Static Routes** advertises the route server endpoint host routes again.

   1. (Optional) Apply an outbound route filter for RFC 1918 networks to limit the CIDR blocks that are advertised to your VPC. Create an IP prefix list that permits `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16` (including more-specific routes within them) and denies all other prefixes. Apply it as the out filter on each BGP neighbor.

#### Verify BGP peering
<a name="sd-nsx-verify"></a>

To verify BGP peering, do the following.

1. In NSX Manager, choose **Networking**, choose **Tier-0 Gateways**, and then choose **BGP**. Under **BGP Neighbors**, confirm that the neighbors show status **Established**.

1. In the [Amazon VPC console](https://console.aws.amazon.com/vpc), go to **Route Server**, **Routes**, and confirm that NSX overlay routes appear.

1. In NSX Manager, verify that VPC routes are learned under **Routing**, **Forwarding Table**.

If a BGP session doesn’t establish, or routes don’t appear in the VPC route table, see [NSX Edge and BGP symptoms](#troubleshoot-sd-nsx).

You have configured NSX networking when the BGP sessions are established and routes are propagating.

Next step: [Step 6: Create connectors](#self-deployed-create-connectors).

### Step 6: Create connectors
<a name="self-deployed-create-connectors"></a>

A connector is an Amazon Elastic VMware Service sub-resource that represents a persistent connection from Amazon EVS to a specific VMware Cloud Foundation (VCF) management appliance. Amazon EVS uses connectors to monitor your deployment. Create them after VCF is installed and its management appliances are reachable over your VCF management network. You can create connectors with the Amazon EVS console or the AWS CLI. For more information, see [Manage connectors and credentials](#connectors-overview).

#### Why the Operations Manager connector matters
<a name="self-deployed-connector-why"></a>

VCF 9.x requires an Operations Manager (`OPERATIONS_MANAGER`) connector. Amazon EVS uses this connector to connect to and stay in sync with your VMware deployment, and to monitor vSAN capacity. If the required connector isn’t created or becomes unreachable, Amazon EVS reports impaired environment health through AWS Health notifications.

You can also create a vCenter (`VCENTER`) connector. Amazon EVS uses it to monitor VM lifecycle events for entitled VMs. For the connector types and their required secret keys, see [Create a connector](#evs-env-create-connector).

#### Store the appliance credentials
<a name="self-deployed-connector-secret"></a>

**Important**  
Before you create a connector, store the credentials for the target VCF management appliance in AWS Secrets Manager. Tag the secret with `EvsAccess=true`. If you encrypted the secret with your own AWS KMS key, also tag the key with `EvsAccess=true`. Without this tag, Amazon EVS can’t access the secret and connector creation fails.

To store the credentials, do the following.

1. In the VCF management appliance, create a dedicated read-only user for Amazon EVS. For Operations Manager and vCenter, use credentials with the minimum permissions that the appliance type requires. Don’t use credentials with elevated or administrative permissions.

1. In AWS Secrets Manager, create a secret with the keys `username` and `password` that contain the credentials of that user. Each connector maps to a single appliance FQDN, so create a separate secret for each appliance.

1. Add the tag `EvsAccess=true` to the secret, and to the AWS KMS key if you use your own key.

#### Create the Operations Manager connector
<a name="self-deployed-connector-create"></a>

To create the connector in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select your environment.

1. Select the **Connectors** tab.

1. Choose **Create connector**.

1. For **Appliance FQDN**, enter the fully qualified domain name of the VCF Operations appliance.

1. For the Secrets Manager dropdown, select the **Secret** that contains the appliance credentials.

1. Choose **Create connector**.

1. Run the `aws evs create-environment-connector` command. The secret identifier can be the secret name or ARN. Replace `<environment-id>` with the ID of your environment. Replace the example FQDN and secret name with your own values.

   ```bash
   aws evs create-environment-connector \
       --environment-id <environment-id> \
       --type OPERATIONS_MANAGER \
       --appliance-fqdn vcfops.evs.example.com \
       --secret-identifier vcfops-credentials
   ```

1. List the connectors of the environment.

   ```bash
   aws evs list-environment-connectors \
       --environment-id <environment-id>
   ```

Connector creation is asynchronous and doesn’t validate appliance reachability or credentials. After the connector state reaches `ACTIVE`, the reachability check status changes from `UNKNOWN` to `PASSED` or `FAILED` within 10 minutes. You have created the connector when its state is `ACTIVE` and its reachability check is `PASSED`.

To create a vCenter connector, repeat these steps with the vCenter Server FQDN, its secret, and the `VCENTER` type.

If connector creation fails or the reachability check stays `FAILED`, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

Next step: [Step 7: Verify the environment](#self-deployed-verify).

### Step 7: Verify the environment
<a name="self-deployed-verify"></a>

After you have added hosts, installed VMware Cloud Foundation (VCF), configured NSX networking, and created at least one connector, verify that the environment works. You can check the Amazon Elastic VMware Service resources with the Amazon EVS console or the AWS CLI. These checks apply to hosts with local storage and to diskless hosts.

#### Check the hosts and connectors
<a name="self-deployed-verify-evs"></a>

To check the hosts and connectors, do the following.

1. Confirm that your hosts are in the `CREATED` state. In the Amazon EVS console, choose **Environments**, choose your environment, and then choose the **Hosts** tab. You can also run `aws evs list-environment-hosts --environment-id <environment-id>`.

1. Confirm that the connectors that you created are in the `ACTIVE` state and that their reachability checks are `PASSED`. In the Amazon EVS console, choose the **Connectors** tab. You can also run `aws evs list-environment-connectors --environment-id <environment-id>`.

1. If you created an Operations Manager connector, confirm that its `OPERATIONS_MANAGER_REACHABILITY` check shows `PASSED`. The **Environment status** of a Self-deployed environment can show `UNKNOWN`. For what this means and how to read connector health, see [Monitor environment status](#evs-env-status-check).

#### Check the VCF appliances and connectivity
<a name="self-deployed-verify-vcf"></a>

To check the VCF appliances and connectivity, do the following.

1. Confirm that your VCF management appliances, such as vCenter Server, NSX Manager, SDDC Manager, and VCF Operations, are reachable from the management VLAN.

1. In NSX Manager, confirm that the Border Gateway Protocol (BGP) neighbors of the Tier-0 gateway show status **Established**.

1. In the Amazon VPC console, confirm that NSX overlay routes appear in the Amazon VPC Route Server routes. For instructions, see [Verify BGP peering](#sd-nsx-verify).

For diskless hosts, also complete the storage and cluster checks in [Step 4b (diskless): Create the specification and deploy](#deploy-diskless-spec).

You have verified the environment. If a check fails, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

Next step: To migrate your workloads to the environment, see [Migrate workloads with HCX](#migrate-evs-hcx).

### Clean up a Self-deployed environment
<a name="self-deployed-cleanup"></a>

When you no longer need your Amazon Elastic VMware Service environment, delete its connectors, hosts, and the environment itself, to stop the charges for the hosts. You can use the Amazon EVS console or the AWS CLI. Then delete the network resources that you created outside of Amazon EVS.

**Warning**  
Deleting the hosts and the environment deletes the VMware Cloud Foundation (VCF) installation that runs in the environment. You can’t undo this action. Back up or migrate any workloads that you want to keep before you start.

#### Delete the connectors
<a name="self-deployed-cleanup-connectors"></a>

Delete all connectors of the environment. For instructions, see [Delete a connector](#evs-env-delete-connector).

#### Delete the hosts and the environment
<a name="self-deployed-cleanup-env"></a>

Delete all hosts before you delete the environment, because you can’t delete an environment that still has hosts. To delete the hosts and the environment in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the hosts to delete.

1. Select the **Hosts** tab.

1. Select a host and choose **Delete**. Repeat this step for each host in the environment.

1. At the top of the **Environments** page, choose **Delete** and then **Delete environment**.

1. Find the ID of the environment.

   ```bash
   aws evs list-environments
   ```

1. Delete each host in the environment. Replace `<environment-id>` with the ID of your environment, and replace the example host name `esx01` with the name of the host.

   ```bash
   aws evs delete-environment-host \
       --environment-id <environment-id> \
       --host-name esx01
   ```

1. Repeat the previous step for the remaining hosts in your environment.

1. Delete the environment.

   ```bash
   aws evs delete-environment --environment-id <environment-id>
   ```

Environment deletion also deletes the Amazon EVS VLAN subnets and the AWS Secrets Manager secrets that Amazon EVS created. It doesn’t delete the VPC, the route server, or other AWS resources that you created outside of Amazon EVS. These resources can continue to incur costs.

To remove a single host from a running cluster instead, see [Delete a host (VCF 9.x)](#host-delete).

#### Delete the remaining resources
<a name="self-deployed-cleanup-other"></a>

To delete the remaining resources, do the following.

1. If you have Amazon EC2 Capacity Reservations that you no longer need, cancel them. For more information, see [Cancel a Capacity Reservation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservations-release.html) in the *Amazon EC2 User Guide*.

1. If you used diskless hosts, delete your external storage, such as the file system and its volumes, to stop accruing storage charges. Deleting the environment doesn’t delete external storage.

1. Delete the network resources, such as the route server, the network ACL, the route tables, the subnets, and the VPC. For instructions, see [Clean up network resources](#cleanup).

You have cleaned up the environment.

## Deploy in EVS-deployed mode
<a name="evs-deployed-overview"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Deploy in Self-deployed mode](#getting-started-self-deployed).

In EVS-deployed mode, Amazon Elastic VMware Service deploys VMware Cloud Foundation (VCF) 5.2.x on your hosts when you create the environment. You supply the network, license keys, and host names, and Amazon EVS builds the VCF environment in your VPC. For a side-by-side comparison with Self-deployed mode, see [Differences between deployment modes](#deployment-mode-differences).

### What Amazon EVS deploys for you
<a name="evs-deployed-what-evs-deploys"></a>

When you create an environment in EVS-deployed mode, Amazon EVS does the following:
+ Creates the Amazon EVS VLAN subnets in your VPC from the CIDR blocks that you specify.
+ Adds the four hosts that the environment requires for initial deployment.
+ Deploys the VCF management appliances (vCenter Server, NSX Manager, NSX Edge nodes, and SDDC Manager) with the DNS host names that you specify.
+ Stores the VCF credentials as secrets in AWS Secrets Manager in your account, encrypted with the AWS KMS key that you choose.

Diskless hosts aren’t available with VCF 5.2.x. For the VCF versions, ESX versions, and instance types that each VCF version supports, see [Supported VCF versions by instance type](#current-vcf-versions).

### Before you begin
<a name="evs-deployed-before-you-begin"></a>

Before you create the environment, complete the prerequisites in [Before you begin](#create-env-evs-deployed-before-you-begin).

### Tasks in this section
<a name="evs-deployed-tasks"></a>

Do these tasks in order:

1.  [Create an environment](#getting-started-create-env). Use the Amazon EVS console or the AWS CLI to create the environment, including license keys.

1.  [Verify environment creation](#verify-env-creation). Check that the environment and its hosts are ready.

1.  [Retrieve VCF credentials and access appliances](#access-vcf). Get the VCF credentials from Secrets Manager and sign in to the management appliances.

When you no longer need the environment, see [Delete an EVS-deployed environment](#getting-started-cleanup-env-hosts).

### Next steps
<a name="evs-deployed-next-steps"></a>

After you verify the environment, associate the Amazon EVS VLAN subnets with a route table. For instructions, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt). Then you can migrate workloads with VMware HCX. For more information, see [Migrate workloads with HCX](#migrate-evs-hcx).

**Topics**

### Create an environment
<a name="getting-started-create-env"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Deploy in Self-deployed mode](#getting-started-self-deployed).

Create an Amazon Elastic VMware Service environment in EVS-deployed mode with the Amazon EVS console or the AWS CLI. In this mode, environment creation adds the hosts and deploys VMware Cloud Foundation (VCF) on them. In Self-deployed mode, you add hosts and install VCF after you create the environment.

**Important**  
You can configure an environment only when you create it. You can’t change the environment settings after you create it. You can’t add VLAN subnets or change their CIDR blocks after you create the environment. For an overview of all environment settings, see the [Amazon EVS API Reference Guide](https://docs.aws.amazon.com/evs/latest/APIReference/Welcome.html).

#### Before you begin
<a name="create-env-evs-deployed-before-you-begin"></a>

Complete the following tasks before you create the environment.
+ Check that you meet the Amazon EVS service requirements. Environment deployment fails if they aren’t met. For more information, see [Prepare your AWS account](#setting-up).
+ Build the network foundation in the same Region and Availability Zone where you create the environment. This includes the VPC, the service access subnet, DNS and NTP, a route server in Amazon VPC Route Server with two endpoints and two peers, and a network ACL. For more information, see [Build the network foundation](#network-foundation).
+ Size the Amazon EVS VLAN subnets. You can’t add VLAN subnets or change their CIDR blocks after you create the environment. For more information, see [Plan your VPC and VLAN subnets](#plan-network-sizing) and [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference).
+ If you plan to use HCX over the internet, create a custom network ACL for the HCX VLAN. We recommend a dedicated network ACL so that you can control HCX internet traffic separately. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).
+ If you plan to use HCX over the internet, create the IPAM, the Elastic IP addresses, and the public CIDR block in your VPC. For the steps, see Part 1 of [Configure HCX public internet connectivity](#evs-env-hcx-internet-access).
+ Have your Broadcom Site ID, a VCF solution key (VMware vSphere 8 Enterprise Plus for VCF), and a vSAN license key. For the license requirements, see [License key requirements](#license-keys-evs-deployed).
+ Create a DNS A record in a forward lookup zone and a PTR record in a reverse lookup zone for each host and each VCF management appliance. For the list of records, see [Deployment prerequisite checklist](#evs-deployment-prereq-checklist).
+ Create an Amazon EC2 key pair for SSH access to the hosts.

##### License key requirements
<a name="license-keys-evs-deployed"></a>

Your license keys must meet the following requirements.
+ The VCF solution key must have sufficient cores, and the vSAN license key must have sufficient vSAN capacity. The thresholds depend on the instance type. For more information, see [VCF licensing](#vcf-license-mgmt).
+ Neither key can be in use by an existing Amazon EVS environment.
+ Amazon EVS doesn’t validate license keys. To validate them, go to [Broadcom support](https://support.broadcom.com/web/ecx).
+ You must keep a valid VCF solution key and vSAN license key in SDDC Manager for the service to function. If you manage the keys in the vSphere Client after deployment, make sure that they also appear on the licensing screen of SDDC Manager.
+ Your VCF subscription and your environment ID are available to Amazon EVS across all AWS Regions for license compliance.

#### Create the environment
<a name="create-env-evs-deployed-procedure"></a>

Environment creation can take several hours.

**Important**  
The hosts that Amazon EVS deploys in your account incur charges. For prices, see [Amazon EVS pricing](https://aws.amazon.com/evs/pricing). To stop the charges, delete the environment as described in [Delete an EVS-deployed environment](#getting-started-cleanup-env-hosts).

You can create the environment by using the Amazon EVS console or the AWS CLI.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs). Check that the AWS Region in the upper right of the console is the Region where you want to create the environment. To change it, choose the Region name and choose another Region.

1. In the navigation pane, choose **Environments**.

1. Choose **Create environment**.

1. On the **Validate Amazon EVS requirements** page, check that the service requirements are met, and then do the following.

   1. (Optional) For **Name**, enter an environment name.

   1. For **Environment version**, choose your VCF version. For the versions that Amazon EVS provides, see [Supported VCF versions and instance types](#versions-provided). The list also includes a **Self-deployed** option. If you choose it, the wizard follows the Self-deployed flow instead of these steps. For more information, see [Deploy in Self-deployed mode](#getting-started-self-deployed).

   1. For **Site ID**, enter your Broadcom Site ID.

   1. For **VCF Solution key**, enter a VCF solution key (VMware vSphere 8 Enterprise Plus for VCF).

   1. For **vSAN license key**, enter a vSAN license key.

   1. For **VCF license terms**, select the check box. You confirm that you have purchased, and will continue to maintain, VCF software licenses that cover all physical processor cores in the Amazon EVS environment. Information about your VCF software in Amazon EVS is shared with Broadcom to verify license compliance.

   1. Choose **Next**.

1. On the **Specify host details** page, add the four hosts that Amazon EVS requires for initial deployment. For each host, do the following.

   1. Choose **Add host details**.

   1. For **DNS hostname**, enter the host name for the host.

   1. For **instance type**, choose the Amazon EC2 instance type.

   1. For **ESX host version**, Amazon EVS uses the default ESX version for the VCF version that you chose. For more information, see [Supported VCF versions and instance types](#versions-provided).

   1. For **SSH key pair**, choose an SSH key pair for SSH access to the host.

   1. Choose **Add host**.

1. On the **Configure networks and connectivity** page, do the following.

   1. For **HCX connectivity requirements**, choose whether to use HCX with private connectivity or over the internet. For help with the choice, see [Plan HCX connectivity](#plan-hcx-connectivity).

   1. For **VPC**, choose the VPC that you created.

   1. (HCX internet connectivity only) For **HCX network ACL**, choose the network ACL to associate with the HCX VLAN.

   1. For **Service access subnet**, choose the private subnet that you created with the VPC.

   1. (Optional) For **Security group**, choose up to two security groups that control communication between the Amazon EVS control plane and the VPC. If you choose none, Amazon EVS uses the default security group. The security groups must allow connectivity to your DNS servers and the Amazon EVS VLAN subnets.

   1. Under **Management connectivity**, enter the CIDR blocks for the Amazon EVS VLAN subnets. For a public HCX VLAN, the netmask length must be exactly /28. For a private HCX VLAN and all other VLAN subnets, the netmask length must be /28 to /24. You can’t add VLAN subnets or change their CIDR blocks after you create the environment.

   1. Under **Expansion VLANs**, enter the CIDR blocks for additional Amazon EVS VLAN subnets. You can use them to expand VCF capabilities, such as NSX Federation.

   1. Under **Workload/VCF connectivity**, enter the CIDR block for the NSX uplink VLAN. Then choose the two route server peer IDs that peer with the route server endpoints over the NSX uplink. For more information, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

   1. Choose **Next**.

1. On the **Specify Management DNS hostnames** page, do the following.

   1. Under **Management appliance DNS hostnames**, enter the DNS host names for the virtual machines that host the VCF management appliances. If you use Route 53 for DNS, also choose the hosted zone that contains your DNS records.

   1. Under **Credentials**, choose the AWS managed KMS key for Secrets Manager or a customer managed KMS key that you provide. The key encrypts the VCF credentials for SDDC Manager, NSX Manager, and vCenter. Customer managed KMS keys have usage costs. For more information, see [AWS KMS pricing](https://aws.amazon.com/kms/pricing).

   1. Choose **Next**.

1. (Optional) On the **Add tags** page, add tags for the environment and choose **Next**.

1. On the **Review and create** page, review your configuration and choose **Create environment**.

1. Open a terminal session.

1. Run the `aws evs create-environment` command. Specify the following parameters.
   + For `--vpc-id`, specify the VPC that you created, with a minimum IPv4 CIDR range of /22.
   + For `--service-access-subnet-id`, specify the ID of the private subnet that you created with the VPC.
   + For `--vcf-version`, specify the VCF version. For the versions that Amazon EVS provides, see [Supported VCF versions and instance types](#versions-provided).
   + With `--terms-accepted`, you confirm that you have purchased, and will continue to maintain, VCF software licenses that cover all physical processor cores in the Amazon EVS environment. Information about your VCF software in Amazon EVS is shared with Broadcom to verify license compliance.
   + For `--license-info`, specify your VCF solution key (VMware vSphere 8 Enterprise Plus for VCF) and your vSAN license key.
   + For `--initial-vlans`, specify the CIDR blocks for the Amazon EVS VLAN subnets that Amazon EVS creates for you. The VCF management appliances are deployed on these VLAN subnets. For a public HCX VLAN, the netmask length must be exactly /28. For a private HCX VLAN and all other VLAN subnets, the netmask length must be /28 to /24. You can’t add VLAN subnets or change their CIDR blocks after you create the environment.
   + Use `hcxNetworkAclId` in `--initial-vlans` only for HCX internet connectivity. Specify the custom network ACL for the public HCX VLAN.
   + For `--hosts`, specify the four hosts that Amazon EVS requires for environment deployment. For each host, include the DNS host name, the EC2 SSH key name, and the EC2 instance type. The dedicated host ID is optional.
   + For `--connectivity-info`, specify the two route server peer IDs that you created. For more information, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).
   + For `--vcf-hostnames`, specify the DNS host names for the virtual machines that host the VCF management appliances.
   + For `--site-id`, specify your Broadcom Site ID. Broadcom gives you this ID at the close of your software contract or contract renewal.
   + (Optional) For `--region`, specify the Region for the environment. If you don’t specify it, your default Region is used.

     The following is a sample request. Replace the example IDs, names, and CIDR blocks with your own values. Each VLAN CIDR block must be inside a CIDR block of your VPC. The sample uses a private HCX VLAN subnet. For HCX over the internet, use a public /28 CIDR block from your IPAM pool for `hcx`, set `isHcxPublic` to `true`, and set `hcxNetworkAclId` to the ID of your network ACL (it starts with `acl-`). For the steps, see [Configure HCX public internet connectivity](#evs-env-hcx-internet-access). Replace `<region-code>` with the Region where you want to create the environment.

     ```bash
     aws evs create-environment \
     --environment-name testEnv \
     --vpc-id vpc-1234567890abcdef0 \
     --service-access-subnet-id subnet-01234a1b2cde1234f \
     --vcf-version VCF-5.2.2 \
     --terms-accepted \
     --license-info "{
           \"solutionKey\": \"00000-00000-00000-abcde-11111\",
           \"vsanKey\": \"00000-00000-00000-abcde-22222\"
         }" \
        --initial-vlans "{
           \"vmkManagement\": {
             \"cidr\": \"10.10.0.0/24\"
           },
           \"vmManagement\": {
             \"cidr\": \"10.10.1.0/24\"
           },
           \"vMotion\": {
             \"cidr\": \"10.10.2.0/24\"
           },
           \"vSan\": {
             \"cidr\": \"10.10.3.0/24\"
           },
           \"vTep\": {
             \"cidr\": \"10.10.4.0/24\"
           },
           \"edgeVTep\": {
             \"cidr\": \"10.10.5.0/24\"
           },
           \"nsxUplink\": {
             \"cidr\": \"10.10.6.0/24\"
           },
           \"hcx\": {
             \"cidr\": \"10.10.7.0/24\"
           },
           \"expansionVlan1\": {
             \"cidr\": \"10.10.8.0/24\"
           },
           \"expansionVlan2\": {
               \"cidr\": \"10.10.9.0/24\"
           }
         }" \
     --hosts "[
         {
           \"hostName\": \"esx01\",
           \"keyName\": \"sshKey-04-05-45\",
           \"instanceType\": \"i4i.metal\",
           \"dedicatedHostId\": \"h-07879acf49EXAMPLE\"
         },
         {
           \"hostName\": \"esx02\",
           \"keyName\": \"sshKey-04-05-45\",
           \"instanceType\": \"i4i.metal\",
           \"dedicatedHostId\": \"h-07878bde50EXAMPLE\"
         },
         {
           \"hostName\": \"esx03\",
           \"keyName\": \"sshKey-04-05-45\",
           \"instanceType\": \"i4i.metal\",
           \"dedicatedHostId\": \"h-07877eio51EXAMPLE\"
         },
         {
           \"hostName\": \"esx04\",
           \"keyName\": \"sshKey-04-05-45\",
           \"instanceType\": \"i4i.metal\",
           \"dedicatedHostId\": \"h-07863ghi52EXAMPLE\"
         }
       ]" \
     --connectivity-info "{
         \"privateRouteServerPeerings\": [\"rsp-1234567890abcdef0\",\"rsp-abcdef01234567890\"]
       }" \
       --vcf-hostnames "{
         \"vCenter\": \"vcf-vc01\",
         \"nsx\": \"vcf-nsx\",
         \"nsxManager1\": \"vcf-nsxm01\",
         \"nsxManager2\": \"vcf-nsxm02\",
         \"nsxManager3\": \"vcf-nsxm03\",
         \"nsxEdge1\": \"vcf-edge01\",
         \"nsxEdge2\": \"vcf-edge02\",
         \"sddcManager\": \"vcf-sddcm01\",
         \"cloudBuilder\": \"vcf-cb01\"
       }" \
     --site-id my-site-id \
     --region <region-code>
     ```

     The following is a sample response.

     ```json
     {
         "environment": {
             "environmentId": "env-abcde12345",
             "environmentState": "CREATING",
             "stateDetails": "The environment is being initialized, this operation may take some time to complete.",
             "createdAt": "2025-04-13T12:03:39.718000+00:00",
             "modifiedAt": "2025-04-13T12:03:39.718000+00:00",
             "environmentArn": "arn:aws:evs:us-east-2:111122223333:environment/env-abcde12345",
             "environmentName": "testEnv",
             "vpcId": "vpc-1234567890abcdef0",
             "serviceAccessSubnetId": "subnet-01234a1b2cde1234f",
             "vcfVersion": "VCF-5.2.2",
             "termsAccepted": true,
             "licenseInfo": [
                 {
                     "solutionKey": "00000-00000-00000-abcde-11111",
                     "vsanKey": "00000-00000-00000-abcde-22222"
                 }
             ],
             "siteId": "my-site-id",
             "connectivityInfo": {
                 "privateRouteServerPeerings": [
                     "rsp-1234567890abcdef0",
                     "rsp-abcdef01234567890"
                 ]
             },
             "vcfHostnames": {
                 "vCenter": "vcf-vc01",
                 "nsx": "vcf-nsx",
                 "nsxManager1": "vcf-nsxm01",
                 "nsxManager2": "vcf-nsxm02",
                 "nsxManager3": "vcf-nsxm03",
                 "nsxEdge1": "vcf-edge01",
                 "nsxEdge2": "vcf-edge02",
                 "sddcManager": "vcf-sddcm01",
                 "cloudBuilder": "vcf-cb01"
             }
         }
     }
     ```

     The ARN in the sample uses the `aws` partition. In other partitions, the ARN starts with that partition name.

You have started the creation of an Amazon EVS environment.

#### What happens during creation
<a name="create-env-evs-deployed-after"></a>

The following happens when you create the environment.
+ Amazon EVS creates the Amazon EVS VLAN subnets and implicitly associates them with the main route table of the VPC. After deployment completes, you must explicitly associate the VLAN subnets with a route table for NSX connectivity. For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).
+ Hosts that are created with the environment receive the tag `DoNotDelete-EVS-[<environmentId>]-[<hostname>]`.
+ Tags that you add to the environment don’t propagate to the underlying AWS resources, such as EC2 instances. You can tag those resources with the console of the respective service or the AWS CLI.

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API.

**Note**  
Amazon EVS deploys a recent bundled version of VCF that might not include individual product updates (async patches).

After deployment completes, we recommend that you review and update the individual products so that they include the latest fixes. You can use the Broadcom Async Patch Tool (AP Tool) or the SDDC Manager in-product lifecycle management (LCM) automation. For more information, see [Update and upgrade](#evs-lifecycle-mgmt).

**Important**  
You must upgrade NSX outside of SDDC Manager.

#### Next step
<a name="create-env-evs-deployed-next"></a>

 [Verify environment creation](#verify-env-creation) 

### Verify environment creation
<a name="verify-env-creation"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Deploy in Self-deployed mode](#getting-started-self-deployed).

After you create an environment, check that Amazon Elastic VMware Service finished the deployment and that the environment and its hosts are ready.

Environment creation can take several hours. If the environment is still being created, wait and check again.

#### Check the environment and its hosts
<a name="verify-env-evs-deployed-procedure"></a>

You can check the environment by using the Amazon EVS console or the AWS CLI.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment.

1. Choose the **Details** tab.

1. Check that the **Environment status** is **Passed** and the **Environment state** is **Created**. If the **Environment state** still shows **Creating**, refresh the page.

1. Choose the **Hosts** tab and check that the four hosts that you specified are listed.

1. Open a terminal session.

1. Run the following command with your environment ID and the Region that contains your resources.

   Replace the example environment ID with your own value, and replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs get-environment --environment-id env-abcde12345 --region <region-code>
   ```

   The environment is ready to use when `environmentState` is `CREATED`. If it still shows `CREATING`, run the command again. The following is a sample response.

   ```json
   {
       "environment": {
           "environmentId": "env-abcde12345",
           "environmentState": "CREATED",
           "createdAt": "2025-04-13T13:39:49.546000+00:00",
           "modifiedAt": "2025-04-13T13:40:39.355000+00:00",
           "environmentArn": "arn:aws:evs:us-east-2:111122223333:environment/env-abcde12345",
           "environmentName": "testEnv",
           "vpcId": "vpc-1234567890abcdef0",
           "serviceAccessSubnetId": "subnet-01234a1b2cde1234f",
           "vcfVersion": "VCF-5.2.2",
           "termsAccepted": true,
           "licenseInfo": [
               {
                   "solutionKey": "00000-00000-00000-abcde-11111",
                   "vsanKey": "00000-00000-00000-abcde-22222"
               }
           ],
           "siteId": "my-site-id",
           "checks": [],
           "connectivityInfo": {
               "privateRouteServerPeerings": [
                   "rsp-1234567890abcdef0",
                   "rsp-abcdef01234567890"
               ]
           },
           "vcfHostnames": {
               "vCenter": "vcf-vc01",
               "nsx": "vcf-nsx",
               "nsxManager1": "vcf-nsxm01",
               "nsxManager2": "vcf-nsxm02",
               "nsxManager3": "vcf-nsxm03",
               "nsxEdge1": "vcf-edge01",
               "nsxEdge2": "vcf-edge02",
               "sddcManager": "vcf-sddcm01",
               "cloudBuilder": "vcf-cb01"
           },
           "credentials": []
       }
   }
   ```

   The ARN in the sample uses the `aws` partition. In other partitions, the ARN starts with that partition name.

1. List the hosts in the environment and check that the four hosts that you specified are listed.

   Replace the example environment ID with your own value, and replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs list-environment-hosts --environment-id env-abcde12345 --region <region-code>
   ```

The environment is ready when its state is **Created** (`CREATED`). In the console, its status also shows **Passed**.

#### If the environment status is Failed
<a name="verify-env-evs-deployed-failed"></a>

When the environment status is **Failed**, the console shows the results of the environment status checks. The checks are **Key re-use**, **Host count**, **Key coverage**, and **Reachability**. For what each check means, see [Monitor environment status](#evs-env-status-check). For causes and solutions, see [Troubleshoot environment status checks](#troubleshoot-status-checks).

#### Next steps
<a name="verify-env-evs-deployed-next"></a>

After you verify the environment, complete the following tasks.
+ Explicitly associate the Amazon EVS VLAN subnets with a route table in your VPC. NSX connectivity requires this association. For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).
+ Retrieve the VMware Cloud Foundation (VCF) credentials and sign in to the management appliances. For more information, see [Retrieve VCF credentials and access appliances](#access-vcf).
+ If you chose HCX internet connectivity, associate Elastic IP addresses with the public HCX VLAN subnet. For more information, see [Configure HCX public internet connectivity](#evs-env-hcx-internet-access).
+ If you connect to on-premises networks through a transit gateway, update the transit gateway route tables. For more information, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

### Retrieve VCF credentials and access appliances
<a name="access-vcf"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Deploy in Self-deployed mode](#getting-started-self-deployed).

Amazon Elastic VMware Service uses AWS Secrets Manager to create, encrypt, and store managed secrets in your account. The secrets hold the VMware Cloud Foundation (VCF) credentials that you need to access the VCF management appliances (vCenter Server, NSX, and SDDC Manager) and the ESX root password. Retrieve the credentials with the console or the AWS CLI, and then sign in to the appliances.

**Important**  
Amazon EVS doesn’t provide managed rotation of your secrets. We recommend that you rotate your secrets on a regular schedule so that they aren’t long-lived. For more information, see [Manage secrets](#evs-env-secret-rotation).

#### Retrieve the VCF credentials
<a name="access-vcf-retrieve"></a>

You can retrieve the credentials by using the Amazon EVS console or the AWS CLI.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Choose your environment ID to open the environment details page.

1. Choose the **Management appliances** tab. The tab shows the VCF management appliances in your environment with their DNS host names and related credentials.

1. Note the credentials for the appliance that you want to access. Then, in the Secrets Manager console, open that secret and retrieve the secret value. For the steps, see [Get secrets from AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html) in the * AWS Secrets Manager User Guide*.

1. Open a terminal session.

1. Find the secret for the appliance. The following command lists the secrets in your account in the Region of the environment.

   Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws secretsmanager list-secrets --region <region-code>
   ```

1. Retrieve the secret value.

   In the following command, replace `<secret-name>` with the name or ARN of the secret. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws secretsmanager get-secret-value --secret-id <secret-name> --region <region-code>
   ```

   For more information, see [Get secrets from AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html) in the * AWS Secrets Manager User Guide*.

#### Sign in to the VCF management appliances
<a name="access-vcf-sign-in"></a>

After you retrieve the credentials, use them to sign in to SDDC Manager, vCenter Server, and NSX Manager. Use the DNS host names that you specified when you created the environment. For the steps to sign in to SDDC Manager and vCenter Server, see the following pages in the Broadcom documentation:
+  [Log in to the SDDC Manager User Interface](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/getting-started-with-sddc-manager-admin/log-in-to-the-sddc-manager-dashboard-admin.html) 
+  [How to Use and Configure Your vSphere Client](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/using-the-vsphere-client/working-with-the-vsphere-client.html#GUID-CE128B59-E236-45FF-9976-D134DADC8178-en) 

You have retrieved the VCF credentials and signed in to the VCF management appliances.

You monitor the VCF components in SDDC Manager, not in the Amazon EVS console. For more information, see [Monitor environment status](#evs-env-status-check).

To reach the console of an ESX host for troubleshooting, use the Amazon EC2 serial console. For more information, see [Troubleshoot hosts](#troubleshoot-hosts).

#### Next steps
<a name="access-vcf-next"></a>

After you sign in to the appliances, continue with the following tasks.
+ If you haven’t done so, associate the Amazon EVS VLAN subnets with a route table. NSX connectivity requires this association. For instructions, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).
+ Migrate your workloads to Amazon EVS with VMware HCX. For more information, see [Migrate workloads with HCX](#migrate-evs-hcx).

### Delete an EVS-deployed environment
<a name="getting-started-cleanup-env-hosts"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Deploy in Self-deployed mode](#getting-started-self-deployed).

Delete the Amazon Elastic VMware Service hosts and then the environment with the Amazon EVS console or the AWS CLI. After that, delete the network resources that you created for the environment. For a Self-deployed environment, see [Clean up a Self-deployed environment](#self-deployed-cleanup).

**Warning**  
Deleting the hosts and the environment deletes the VMware Cloud Foundation (VCF) installation that runs in your Amazon EVS environment. You can’t undo this action.

You must delete all hosts in an environment before you can delete the environment. To remove one host and keep the environment, see [Delete a host (VCF 5.x)](#evs-env-delete-host).

#### Delete the hosts and the environment
<a name="cleanup-evs-deployed-procedure"></a>

You can delete the hosts and the environment by using the Amazon EVS console or the AWS CLI.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the hosts to delete.

1. Choose the **Hosts** tab.

1. Select a host and choose **Delete**. Repeat this step for each host in the environment.

1. Wait until the **Hosts** tab shows no hosts.

1. At the top of the **Environments** page, choose **Delete** and then **Delete environment**.

1. If you have Amazon EC2 Capacity Reservations that you no longer need, cancel them. For more information, see [Cancel a Capacity Reservation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservations-release.html) in the *Amazon EC2 User Guide*.

1. Open a terminal session.

1. Identify the environment that contains the hosts to delete.

   ```bash
   aws evs list-environments
   ```

   The following is a sample response.

   ```json
   {
       "environmentSummaries": [
           {
               "environmentId": "env-abcde12345",
               "environmentName": "testEnv",
               "vcfVersion": "VCF-5.2.2",
               "environmentState": "CREATED",
               "createdAt": "2025-04-13T14:42:41.430000+00:00",
               "modifiedAt": "2025-04-13T14:43:33.412000+00:00",
               "environmentArn": "arn:aws:evs:us-east-2:111122223333:environment/env-abcde12345"
           },
           {
               "environmentId": "env-edcbaEXAMPLE",
               "environmentName": "testEnv2",
               "vcfVersion": "VCF-5.2.2",
               "environmentState": "CREATED",
               "createdAt": "2025-04-13T13:39:49.546000+00:00",
               "modifiedAt": "2025-04-13T13:52:13.342000+00:00",
               "environmentArn": "arn:aws:evs:us-east-2:111122223333:environment/env-edcbaEXAMPLE"
           }
       ]
   }
   ```

   The ARNs in the sample use the `aws` partition. In other partitions, the ARN starts with that partition name.

1. Delete a host from the environment. The following is a sample `aws evs delete-environment-host` request.

   ```bash
   aws evs delete-environment-host \
   --environment-id env-abcde12345 \
   --host-name esx01
   ```

1. Repeat the previous step for each remaining host in the environment.

1. Wait until the following command returns no hosts.

   ```bash
   aws evs list-environment-hosts --environment-id env-abcde12345
   ```

1. Delete the environment.

   ```bash
   aws evs delete-environment --environment-id env-abcde12345
   ```

1. If you have Amazon EC2 Capacity Reservations that you no longer need, cancel them. For more information, see [Cancel a Capacity Reservation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservations-release.html) in the *Amazon EC2 User Guide*.

You have deleted the hosts and the environment.

#### What environment deletion removes
<a name="cleanup-evs-deployed-what-remains"></a>

Environment deletion also deletes the Amazon EVS VLAN subnets and the AWS Secrets Manager secrets that Amazon EVS created.

**Important**  
Environment deletion doesn’t delete the AWS resources that you created, such as the VPC, the service access subnet, the route server, and the network ACL. These resources can continue to incur charges until you delete them.

#### Next step
<a name="cleanup-evs-deployed-next"></a>

Delete the network resources that you created for the environment, including IPAM resources if you configured HCX internet connectivity. For more information, see [Clean up network resources](#cleanup).

## Connect external storage
<a name="external-storage-connect"></a>

You can add external storage to the ESX hosts in a running Amazon Elastic VMware Service environment. You add each datastore in vSphere after the environment is running, next to the storage that the hosts already use. You can use the procedures on this page with VCF 5.x and VCF 9.x environments, in either deployment mode.

Using Amazon FSx for NetApp ONTAP as the principal storage of diskless hosts is available only with VCF 9.x in Self-deployed mode.

Choose the procedure by storage service and protocol.


| Storage service | Protocol | Datastore type in vSphere | Procedure | 
| --- | --- | --- | --- | 
| Amazon FSx for NetApp ONTAP | NFS v3 or NFS v4.1 | NFS datastore |  [Configure FSx for ONTAP as an NFS datastore](#config-fsx-nfs-datastore)  | 
| Amazon FSx for NetApp ONTAP | iSCSI | VMFS datastore |  [Configure FSx for ONTAP as an iSCSI datastore](#config-fsx-iscsi-datastore)  | 
| Amazon FSx for NetApp ONTAP | NVMe/TCP | VMFS datastore |  [Configure FSx for ONTAP as an NVMe/TCP datastore](#config-fsx-nvme-datastore)  | 

To decide which protocol to use and how FSx for ONTAP connects to your environment, read [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap) first.

Diskless hosts are available only with VMware Cloud Foundation (VCF) 9.x. For diskless hosts, external storage is the principal storage. You create it during deployment, as described in [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless) and [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore).

**Topics**
+ [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap)
+ [Configure FSx for ONTAP as an NFS datastore](#config-fsx-nfs-datastore)
+ [Configure FSx for ONTAP as an iSCSI datastore](#config-fsx-iscsi-datastore)
+ [Configure FSx for ONTAP as an NVMe/TCP datastore](#config-fsx-nvme-datastore)

### Use Amazon FSx for NetApp ONTAP with Amazon EVS
<a name="fsx-ontap"></a>

Amazon FSx for NetApp ONTAP is a storage service that you use to launch and run fully managed ONTAP file systems in the cloud. Amazon Elastic VMware Service supports FSx for ONTAP as an NFS, iSCSI, or NVMe/TCP datastore for your ESX hosts and as guest-connected storage for virtual machines.

For more information about FSx for ONTAP, see the [FSx for ONTAP User Guide](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/what-is-fsx-ontap.html).

#### How FSx for ONTAP connects to your environment
<a name="fsx-ontap-architecture"></a>

You create the FSx for ONTAP file system in the same VPC as your Amazon EVS environment. The file system is deployed in the Amazon EVS service access subnet. For more information, see [Service access subnet](#concepts-service-access-subnet).

The ESX hosts reach the file system through the endpoints of a storage virtual machine (SVM). NFS datastores use the NFS DNS name of the SVM. iSCSI and NVMe/TCP datastores use the iSCSI logical interface (LIF) IP address of the SVM.

A security group on the file system controls this traffic. It must permit the traffic for your protocol to the Amazon EVS host VMkernel management VLAN subnet. For the ports that FSx for ONTAP uses, see [File system access control with Amazon VPC](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/limit-access-security-groups.html) in the *FSx for ONTAP User Guide*.

You can also send storage traffic over an Amazon EVS expansion VLAN to keep it apart from other traffic.

#### Single-AZ and Multi-AZ file systems
<a name="fsx-ontap-multi-az"></a>

An Amazon EVS environment runs in one Availability Zone. For more information, see [VPC and subnets](#plan-network-supported-vpc). You can use either a Single-AZ or a Multi-AZ FSx for ONTAP file system.

If you choose a Multi-AZ file system, you must associate it with every VPC route table that your Amazon EVS host subnets use. FSx for ONTAP adds routes to each associated route table, and a file system supports a limited number of routes. For the route rules and limits, see [Availability and durability](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/high-availability-AZ.html) and [Quotas](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/limits.html) in the *FSx for ONTAP User Guide*.

During a Multi-AZ failover, NFS datastores reconnect without action from you because the endpoint IP address stays the same. For iSCSI and NVMe/TCP datastores, the failover is transparent only if you configure multipathing on the ESX hosts.

**Important**  
A Multi-AZ file system doesn’t keep your Amazon EVS environment running if its Availability Zone fails, because the environment runs in one Availability Zone. To recover, you create an environment in another Availability Zone, mount the volumes, and register the virtual machines again. For recovery planning, see [Resilience for Amazon EVS](#disaster-recovery-resiliency).

#### Choose a storage protocol
<a name="fsx-ontap-protocol-choice"></a>

All three protocols use the same kind of FSx for ONTAP file system. They differ in the datastore type and in the setup on ONTAP and on the ESX hosts.


| Protocol | Datastore type | Setup on ONTAP | Setup on each ESX host | Requirement | 
| --- | --- | --- | --- | --- | 
| NFS v3 or NFS v4.1 | NFS | A volume with a junction path | None beyond mounting the datastore | None | 
| iSCSI | VMFS | A LUN, an initiator group, and a LUN mapping | A software iSCSI adapter and dynamic discovery | None | 
| NVMe/TCP | VMFS | A namespace and a subsystem with the host NQNs | A software NVMe over TCP adapter and a controller | A second-generation file system with 6 or fewer HA pairs | 

For instructions, see [Configure FSx for ONTAP as an NFS datastore](#config-fsx-nfs-datastore), [Configure FSx for ONTAP as an iSCSI datastore](#config-fsx-iscsi-datastore), or [Configure FSx for ONTAP as an NVMe/TCP datastore](#config-fsx-nvme-datastore).

#### Supported FSx for NetApp ONTAP capabilities
<a name="fsx-ontap-supported-capabilities"></a>

The following FSx for NetApp ONTAP capabilities are available for use with Amazon EVS.


| Capability | Description | 
| --- | --- | 
| External NFS v3 datastores | Mount FSx for ONTAP volumes as NFS v3 datastores for VMware vSphere on Amazon EVS. | 
| External NFS v4.1 datastores | Mount FSx for ONTAP volumes as NFS v4.1 datastores for VMware vSphere on Amazon EVS, with added security and performance features over NFS v3. | 
| External NVMe datastores | Use FSx for ONTAP as NVMe-based external datastores for VMware vSphere on Amazon EVS, for block storage for demanding workloads. | 
| iSCSI datastores | Configure FSx for ONTAP as iSCSI-based VMFS datastores for VMware vSphere on Amazon EVS. | 
| Guest-mounted iSCSI disks | Present FSx for ONTAP iSCSI LUNs directly to guest virtual machines running on Amazon EVS as in-guest connected storage. | 
| NetApp SnapCenter Plug-in for VMware vSphere | Provide application-consistent backup and restore operations for VMs and datastores that run on Amazon EVS. | 
| Deployment using Amazon EVS expansion VLAN | Deploy FSx for ONTAP external datastores using an Amazon EVS expansion VLAN for dedicated storage network traffic. | 

### Configure FSx for ONTAP as an NFS datastore
<a name="config-fsx-nfs-datastore"></a>

You can mount an Amazon FSx for NetApp ONTAP volume as an NFS datastore on the ESX hosts in your Amazon Elastic VMware Service environment. In this procedure, you use the Amazon FSx console and the vSphere Client to create the datastore.

#### Before you begin
<a name="fsx-evs-prereqs-nfs"></a>

Make sure that you meet the following prerequisites.
+ Your Amazon EVS environment is deployed in your VPC. For more information, see [Deploy](#deploy-overview).
+ You have access to the vSphere Client for your Amazon EVS environment.
+ You or your storage administrator have the permissions to create and manage FSx for ONTAP file systems in your VPC. For more information, see [Identity and access management for Amazon FSx for NetApp ONTAP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/security-iam.html).

#### In the Amazon FSx console, create an FSx for NetApp ONTAP file system
<a name="create-fsx-file-system-nfs"></a>

You can use a Single-AZ or Multi-AZ file system. If you choose Multi-AZ, read the route table requirement in [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap) first. For the full file system creation workflow, see [Creating FSx for ONTAP file systems](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/creating-file-systems.html) in the *Amazon FSx for NetApp ONTAP User Guide*.

When you reach the networking and volume configuration pages of the FSx creation wizard, apply the following Amazon EVS settings.

1. Select the Amazon EVS VPC that has connectivity to your Amazon EVS VLAN subnets.

1. Select a security group that permits all required FSx for ONTAP NFS traffic to the Amazon EVS host VMkernel management VLAN subnet.

1. Select the Amazon EVS service access subnet for your file system. For more information, see [Service access subnet](#concepts-service-access-subnet).

1. For **Junction path**, enter a name that identifies this volume in vSphere, such as `/vol1`.

1. Within **Default volume configuration**, set **Storage efficiency** to **Enabled**.

#### In the Amazon FSx console, get the NFS DNS name
<a name="create-fsx-volume-nfs"></a>

To get the NFS DNS name of the storage virtual machine (SVM), do the following.

1. Open the [Amazon FSx console](https://console.aws.amazon.com/fsx).

1. In the navigation pane, choose **File systems**.

1. Choose the file system that you created.

1. Choose the **Storage virtual machines** tab.

1. Choose the storage virtual machine.

1. Choose the **Endpoints** tab.

1. Copy the NFS DNS name. You use it in the vSphere Client.

#### In the vSphere Client, create the NFS datastore
<a name="attach-fsx-volume-vsphere-nfs"></a>

To create the datastore from the FSx for ONTAP volume, do the following.

1. Follow the instructions in [Create an NFS Datastore in vSphere Environment](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage-8-0/working-with-datastores-in-vsphere-storage-environment/creating-vsphere-datastores.html#GUID-B52657D0-248D-4A99-99CC-D35B350461D5-en) in the Broadcom documentation.

1. For the **Server** setting, enter the NFS DNS name of the SVM that you copied.

1. For the folder, enter the junction path of the volume.

1. Select the ESX hosts that must have access to the datastore.

#### In the vSphere Client, verify the datastore
<a name="verify-fsx-volume-vsphere-nfs"></a>

To verify the datastore, do the following.

1. In the vSphere Client, open the **Datastores** view.

1. Choose the new datastore.

1. Confirm that the datastore is accessible and that its capacity matches the size of the FSx for ONTAP volume.

1. On the **Hosts** tab of the datastore, confirm that every ESX host that you selected is listed.

You have configured FSx for ONTAP as an NFS datastore for your Amazon EVS environment.

If the hosts can’t mount the datastore, check the security group rules and the routes between the service access subnet and the VMkernel management VLAN subnet. For help, see [Troubleshoot networking](#troubleshoot-networking).

### Configure FSx for ONTAP as an iSCSI datastore
<a name="config-fsx-iscsi-datastore"></a>

You can present Amazon FSx for NetApp ONTAP block storage to the ESX hosts in your Amazon Elastic VMware Service environment over iSCSI, and use it as a VMFS datastore. You use the Amazon FSx console, the NetApp ONTAP CLI, and the vSphere Client. To use NVMe/TCP instead, see [Configure FSx for ONTAP as an NVMe/TCP datastore](#config-fsx-nvme-datastore).

#### Before you begin
<a name="fsx-evs-prereqs-iscsi"></a>

Make sure that you meet the following prerequisites.
+ Your Amazon EVS environment is deployed in your VPC. For more information, see [Deploy](#deploy-overview).
+ You have access to the vSphere Client for your Amazon EVS environment.
+ You or your storage administrator have the permissions to create and manage FSx for ONTAP file systems in your VPC. For more information, see [Identity and access management for Amazon FSx for NetApp ONTAP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/security-iam.html).
+ You can run commands in the NetApp ONTAP CLI of the file system.

#### In the Amazon FSx console, create an FSx for NetApp ONTAP file system
<a name="create-fsx-file-system-iscsi"></a>

You can use a Single-AZ or Multi-AZ file system. If you choose Multi-AZ, read the route table requirement in [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap) first. For the full file system creation workflow, see [Creating FSx for ONTAP file systems](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/creating-file-systems.html) in the *Amazon FSx for NetApp ONTAP User Guide*.

When you reach the networking and volume configuration pages of the FSx creation wizard, apply the following Amazon EVS settings.

1. Select the Amazon EVS VPC that has connectivity to your Amazon EVS VLAN subnets.

1. Select a security group that permits all required FSx for ONTAP iSCSI traffic to the Amazon EVS host VMkernel management VLAN subnet.

1. Select the Amazon EVS service access subnet for your file system. For more information, see [Service access subnet](#concepts-service-access-subnet).

1. Within **Default volume configuration**, set **Storage efficiency** to **Enabled**.

After the file system is created, copy the iSCSI IP address of the storage virtual machine (SVM). You can find it in the [Amazon FSx console](https://console.aws.amazon.com/fsx) on the **Endpoints** tab of the SVM. You use it for dynamic discovery in vSphere.

#### In the vSphere Client, configure the iSCSI adapter
<a name="config-iscsi-adapter"></a>

For each ESX host, you must configure the software iSCSI adapter so that the host can access iSCSI storage. We recommend that you use the host management VMkernel adapter (`vmk0`) for iSCSI traffic. For instructions, see [Add or Remove the Software iSCSI Adapter](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage-8-0/configuring-iscsi-and-iser-adapters-and-storage-with-esxi/configure-the-software-iscsi-adapter-with-esxi/add-or-remove-the-software-iscsi-adapter.html) in the Broadcom documentation.

After you configure the adapter, copy the iSCSI Qualified Name (IQN) of the adapter on each host. You use the IQNs when you create the initiator group.

#### In the ONTAP CLI, create an iSCSI LUN
<a name="config-iscsi-lun"></a>

FSx for ONTAP can create logical unit numbers (LUNs) for iSCSI access, which provide shared block storage to your ESX hosts. You use the NetApp ONTAP CLI to create a LUN.

**Note**  
We recommend that you set the LUN size to 90% of the volume size.

The following is a sample command. Replace the values in angle brackets with your own values.

```bash
lun create -vserver <svm-name> \
-path /vol/<volume-name>/<lun-name> \
-size <datastore-capacity> \
-ostype vmware
```

For more information, see [Creating an iSCSI LUN](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/create-iscsi-lun.html) in the *FSx for ONTAP User Guide*.

#### In the ONTAP CLI, create an initiator group and map the LUN
<a name="config-initiator-group"></a>

Next, create an initiator group (`igroup`) that contains the ESX hosts, and map the LUN to it. Replace the values in angle brackets with your own values.

1. Create the initiator group. For `-initiator`, use the IQNs that you copied from the iSCSI adapters. The following is a sample command.

   ```bash
   igroup create <svm-name> \
   -igroup <igroup-name> \
   -protocol iscsi \
   -ostype vmware \
   -initiator <iqn-1>,<iqn-2>,<iqn-3>,<iqn-4>
   ```

1. Confirm that the `igroup` exists.

   ```bash
   lun igroup show
   ```

1. Map the LUN to the initiator group. The following is a sample command.

   ```bash
   lun mapping create -vserver <svm-name> \
   -path /vol/<volume-name>/<lun-name> \
   -igroup <igroup-name> \
   -lun-id <lun-id>
   ```

1. Confirm that the LUN is created, online, and mapped.

   ```bash
   lun show -path /vol/<volume-name>/<lun-name> -fields state,mapped,serial-hex
   ```

For more information, see [Provisioning iSCSI for Linux](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/mount-iscsi-luns-linux.html) or [Provisioning iSCSI for Windows](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/mount-iscsi-windows.html) in the *FSx for ONTAP User Guide*.

#### In the vSphere Client, configure dynamic discovery of the iSCSI LUN
<a name="config-dynamic-discovery-vsphere"></a>

To let the ESX hosts see the iSCSI LUN, configure dynamic discovery on each host. For the iSCSI server, enter the iSCSI IP address of the SVM that you copied after you created the file system. For instructions, see [Configure Dynamic or Static Discovery for iSCSI and iSER on ESX Host](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage-8-0/configuring-iscsi-and-iser-adapters-and-storage-with-esxi/configure-dynamic-or-static-discovery-for-iscsi-and-iser-on-esxi-host.html#GUID-4ED3304A-ED4F-4692-825F-83637E04D592-en) in the Broadcom documentation.

#### In the vSphere Client, create a VMFS datastore
<a name="create-vmfs"></a>

Virtual Machine File System (VMFS) datastores are repositories for virtual machines. To create the datastore, follow the instructions in [Create a vSphere VMFS Datastore](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage-8-0/working-with-datastores-in-vsphere-storage-environment/creating-vsphere-datastores.html#GUID-5AC611E0-7CEB-4604-A03C-F600B1BA2D23-en) in the Broadcom documentation. When you are prompted to select a device, choose the iSCSI LUN that you configured.

You have configured FSx for ONTAP as an iSCSI datastore for your Amazon EVS environment. To confirm, open the datastore in the vSphere Client and check that every ESX host in the initiator group is listed on its **Hosts** tab.

For a Multi-AZ file system, configure multipathing on the ESX hosts so that the datastore stays available during a failover. For more information, see [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap).

### Configure FSx for ONTAP as an NVMe/TCP datastore
<a name="config-fsx-nvme-datastore"></a>

You can present Amazon FSx for NetApp ONTAP block storage to the ESX hosts in your Amazon Elastic VMware Service environment over NVMe/TCP, and use it as a VMFS datastore. You use the Amazon FSx console, the NetApp ONTAP CLI, and the vSphere Client. To use iSCSI instead, see [Configure FSx for ONTAP as an iSCSI datastore](#config-fsx-iscsi-datastore).

**Important**  
NVMe/TCP requires a second-generation FSx for ONTAP file system with six or fewer HA pairs. Choose a second-generation file system when you create it.

#### Before you begin
<a name="fsx-evs-prereqs-nvme"></a>

Make sure that you meet the following prerequisites.
+ Your Amazon EVS environment is deployed in your VPC. For more information, see [Deploy](#deploy-overview).
+ You have access to the vSphere Client for your Amazon EVS environment.
+ You or your storage administrator have the permissions to create and manage FSx for ONTAP file systems in your VPC. For more information, see [Identity and access management for Amazon FSx for NetApp ONTAP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/security-iam.html).
+ You can run commands in the NetApp ONTAP CLI of the file system and the ESX shell of each host.

#### In the Amazon FSx console, create an FSx for NetApp ONTAP file system
<a name="create-fsx-file-system-nvme"></a>

You can use a Single-AZ or Multi-AZ file system. If you choose Multi-AZ, read the route table requirement in [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap) first. For the full file system creation workflow, see [Creating FSx for ONTAP file systems](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/creating-file-systems.html) in the *Amazon FSx for NetApp ONTAP User Guide*.

When you reach the networking and volume configuration pages of the FSx creation wizard, apply the following Amazon EVS settings.

1. Select the Amazon EVS VPC that has connectivity to your Amazon EVS VLAN subnets.

1. Select a security group that permits all required FSx for ONTAP NVMe/TCP traffic to the Amazon EVS host VMkernel management VLAN subnet.

1. Select the Amazon EVS service access subnet for your file system. For more information, see [Service access subnet](#concepts-service-access-subnet).

1. Within **Default volume configuration**, set **Storage efficiency** to **Enabled**.

#### In the vSphere Client, configure the NVMe/TCP adapter
<a name="config-nvme-tcp-adapter"></a>

For each ESX host, you must configure the software NVMe over TCP adapter so that the host can access NVMe/TCP storage. We recommend that you use the host management VMkernel adapter (`vmk0`) for NVMe/TCP traffic. For instructions, see [Configuring NVMe over TCP on ESXi](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage/about-vmware-nvme-storage/configuring-nvme-over-tcp-on-esxi.html) in the Broadcom documentation.

After you configure the adapter, copy the NVMe Qualified Name (NQN) of each ESX host. To get the host NQN, run the following command on the host.

```bash
esxcli nvme info get
```

You use the NQNs when you add hosts to the subsystem.

#### In the ONTAP CLI, create an NVMe namespace and subsystem
<a name="config-nvme-namespace"></a>

FSx for ONTAP can create NVMe namespaces for NVMe/TCP access, which provide shared block storage to your ESX hosts. You use the NetApp ONTAP CLI to create a namespace and a subsystem, map the namespace to the subsystem, and add the host NQNs. Replace the values in angle brackets with your own values.

1. Create the namespace. The following is a sample command.

   ```bash
   vserver nvme namespace create -vserver <svm-name> \
   -path /vol/<volume-name>/<namespace-name> \
   -size <datastore-capacity> \
   -ostype vmware
   ```

1. Create the subsystem.

   ```bash
   vserver nvme subsystem create -vserver <svm-name> \
   -subsystem <subsystem-name> \
   -ostype vmware
   ```

1. Map the namespace to the subsystem.

   ```bash
   vserver nvme subsystem map add -vserver <svm-name> \
   -subsystem <subsystem-name> \
   -path /vol/<volume-name>/<namespace-name>
   ```

1. Add the NQN of an ESX host to the subsystem. Repeat this command for each host that needs access to the namespace.

   ```bash
   vserver nvme subsystem host add -vserver <svm-name> \
   -subsystem <subsystem-name> \
   -host-nqn <host-nqn>
   ```

1. Confirm that the namespace exists.

   ```bash
   vserver nvme namespace show -vserver <svm-name>
   ```

For more information, see [Provisioning NVMe/TCP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/provision-nvme-linux.html) in the *FSx for ONTAP User Guide*.

#### In the vSphere Client, add a controller for the NVMe namespace
<a name="config-nvme-discovery-vsphere"></a>

To let the ESX hosts see the NVMe namespace, add a controller on the NVMe over TCP adapter of each host.

1. Get the iSCSI IP address of the storage virtual machine (SVM). FSx for ONTAP uses the same SVM endpoints for iSCSI and NVMe/TCP. You can find the address in the [Amazon FSx console](https://console.aws.amazon.com/fsx) on the **Endpoints** tab of the SVM, or by running the following command in the ONTAP CLI. Replace the values in angle brackets with your own values.

   ```bash
   network interface show -vserver <svm-name> -data-protocol nvme-tcp
   ```

1. In the vSphere Client, add a controller on the NVMe over TCP adapter of each host, and use that IP address as the target address. For instructions, see [Configuring NVMe over TCP on ESXi](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage/about-vmware-nvme-storage/configuring-nvme-over-tcp-on-esxi.html) in the Broadcom documentation.

#### In the vSphere Client, create a VMFS datastore
<a name="create-vmfs-nvme"></a>

Virtual Machine File System (VMFS) datastores are repositories for virtual machines. To create the datastore, follow the instructions in [Create a vSphere VMFS Datastore](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-storage-8-0/working-with-datastores-in-vsphere-storage-environment/creating-vsphere-datastores.html#GUID-5AC611E0-7CEB-4604-A03C-F600B1BA2D23-en) in the Broadcom documentation. When you are prompted to select a device, choose the NVMe namespace that you configured.

You have configured FSx for ONTAP as an NVMe/TCP datastore for your Amazon EVS environment. To confirm, open the datastore in the vSphere Client and check that every ESX host that you added to the subsystem is listed on its **Hosts** tab.

For a Multi-AZ file system, configure multipathing on the ESX hosts so that the datastore stays available during a failover. For more information, see [Use Amazon FSx for NetApp ONTAP with Amazon EVS](#fsx-ontap).

# Operate
<a name="operate-overview"></a>

After you deploy an Amazon Elastic VMware Service environment, you operate it. Amazon EVS is a self-managed service, so you’re responsible for the VMware Cloud Foundation (VCF) software on your hosts, for monitoring host health, and for replacing hosts that fail. For who manages what in each deployment mode, see [Shared responsibility and support](#plan-shared-responsibility).

The procedures in [Deploy in Self-deployed mode](#getting-started-self-deployed) deploy VCF 9.x. Pages in [Deploy in EVS-deployed mode](#evs-deployed-overview) cover VCF 5.2.x. A page with no line and outside those sections covers both.

## Day 2 tasks by area
<a name="operate-overview-tasks"></a>

The following table lists the tasks in this chapter in the order that you usually need them.


| Area | Tasks | 
| --- | --- | 
|  [Monitor your environment](#monitor-overview)  | Check environment status, get notified about host events, monitor hosts and VCF, and log Amazon EVS API calls with AWS CloudTrail. | 
|  [Manage hosts](#host-management)  | Prepare for EC2 maintenance events, and add, replace, or delete hosts. | 
|  [Update and upgrade](#evs-lifecycle-mgmt)  | Update ESX and VCF, use the Amazon EVS custom add-on depot, and upgrade VCF. | 
|  [Manage connectors and credentials](#connectors-overview)  | Create, update, and delete connectors, rotate vCenter connector credentials, and manage the secrets that Amazon EVS creates. | 
|  [Windows Server entitlements](#entitlements)  | Create, activate, and delete Windows Server entitlements for your VMs. | 
|  [Migrate workloads with HCX](#migrate-evs-hcx)  | Prepare for VMware HCX migration, configure HCX internet connectivity, and verify migrations. | 
|  [Use VCF features on Amazon EVS](#operate-vcf-features)  | Set up VCF features where Amazon EVS adds steps of its own, such as NSX Federation. | 

**Topics**

## Monitor your environment
<a name="monitor-overview"></a>

You monitor an Amazon Elastic VMware Service environment with AWS tools and with VMware Cloud Foundation (VCF) tools. AWS tools report on the environment, the EC2 instances that run your hosts, and Amazon EVS API calls. VCF tools report on the VCF components and the workloads that run on them.

The following table shows each monitoring source, what it tells you, and where to read more.


| Source | What it tells you | More information | 
| --- | --- | --- | 
| Amazon EVS environment status | Whether Amazon EVS can reach your VCF management appliances, and the results of the environment status checks. |  [Monitor environment status](#evs-env-status-check)  | 
|  AWS Health | Scheduled maintenance and retirement events for the EC2 instances that run your hosts, and other events that affect your environment. |  [Get notified about host events](#monitor-notifications)  | 
| Amazon CloudWatch and VCF Operations | EC2 status checks and alarms for your hosts, Amazon EVS usage metrics, and the health of VCF components and workloads. |  [Monitor hosts and VCF](#monitor-observability)  | 
|  AWS CloudTrail | Who called which Amazon EVS API operation, and when. |  [Log API calls with CloudTrail](#logging-using-cloudtrail)  | 

**Topics**
+ [Monitor environment status](#evs-env-status-check)
+ [Get notified about host events](#monitor-notifications)
+ [Monitor hosts and VCF](#monitor-observability)
+ [Log API calls with CloudTrail](#logging-using-cloudtrail)

### Monitor environment status
<a name="evs-env-status-check"></a>

You can monitor the status of your Amazon Elastic VMware Service environment and its underlying AWS resources with the Amazon EVS console or the AWS CLI. The environment status tells you whether the environment has issues that need your attention. Which status checks Amazon EVS runs depends on the deployment mode and on the connectors that you create.

**Note**  
You monitor VMware Cloud Foundation (VCF) components with VCF tools, not with the Amazon EVS console or AWS CLI.

In EVS-deployed environments (VCF 5.2.x), you use SDDC Manager. For more information, see [Getting started with SDDC Manager](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/getting-started-with-sddc-manager-admin.html) in the Broadcom documentation. For how AWS monitoring compares with VCF Operations, see [Monitor hosts and VCF](#monitor-observability).

#### Environment status checks
<a name="evs-env-status-checks-ref"></a>

The environment status is one of `PASSED`, `FAILED`, or `UNKNOWN`. Each check also reports `PASSED`, `FAILED`, or `UNKNOWN`, and a failed check shows when the impairment began.

The following table lists the checks. The columns show the check type and what it verifies. Connector checks apply only after you create a connector of that type.


| Check type | What it verifies | 
| --- | --- | 
|  `KEY_REUSE`  | The VCF license key is not used by another Amazon EVS environment. | 
|  `KEY_COVERAGE`  | The VCF license key allocates enough vCPU cores for all deployed hosts. | 
|  `REACHABILITY`  | The Amazon EVS control plane has a persistent connection to SDDC Manager. | 
|  `HOST_COUNT`  | The environment meets the minimum host count. | 
|  `SDDC_MANAGER_REACHABILITY`  | Amazon EVS reaches SDDC Manager through the SDDC Manager connector. | 
|  `SDDC_MANAGER_HOST_COUNT`  | The host count that SDDC Manager reports meets the Amazon EVS minimum. | 
|  `SDDC_MANAGER_KEY_COVERAGE`  | The VCF license key in SDDC Manager covers all deployed hosts. | 
|  `SDDC_MANAGER_KEY_REUSE`  | The VCF license key in SDDC Manager is not used by another Amazon EVS environment. | 
|  `OPERATIONS_MANAGER_REACHABILITY`  | Amazon EVS reaches VCF Operations through the Operations Manager connector. | 
|  `VCENTER_REACHABILITY`  | Amazon EVS reaches vCenter Server through the vCenter connector. | 
|  `VCENTER_VM_SYNC`  | The vCenter connector can synchronize the VM inventory from vCenter Server. | 
|  `VCENTER_VM_EVENT`  | The vCenter connector can receive VM lifecycle events from vCenter Server. | 
|  `CONNECTOR_HEALTH`  | The combined health of all connectors in the environment. | 

The SDDC Manager connector applies to VCF 5.2.x, and the Operations Manager connector applies to VCF 9.0 and 9.1. For the connector types, see [Manage connectors and credentials](#connectors-overview). For the definitions of the check types, see [Check](https://docs.aws.amazon.com/evs/latest/APIReference/API_Check.html) in the *Amazon Elastic VMware Service API Reference*.

In a Self-deployed environment that has no connector, the environment status can show `UNKNOWN` with no check results. After you create a connector and its state is `ACTIVE`, its reachability check changes from `UNKNOWN` to `PASSED` or `FAILED` within 10 minutes. For more information, see [Step 6: Create connectors](#self-deployed-create-connectors).

#### View environment status and resources
<a name="evs-env-view-status"></a>

To check your environment status and view the underlying resources in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Choose your environment ID to open the environment details page.

1. Under **Details**, view the **Environment status**.

   If your environment is healthy, the status shows as **Passed**. If there are issues, the status shows as **Failed**. When the status is **Failed**, open the popover to see the result of each check.

1. To view the resources in your environment, choose one of the following tabs.
   +  **Hosts** shows the hosts in your environment.
   +  **Networks & connectivity** shows the VPC, Amazon EVS VLAN subnets, and VPC Route Server resources of your environment.
   +  **Management appliances** shows the VCF management appliances in your environment with their DNS host names and related credentials.
   +  **Connectors** shows the connectors of your environment, their state, and their reachability check result.
   +  **Tags** shows the tags of your environment.

1. To list all environments with their state and status, run the following command.

   ```bash
   aws evs list-environments \
       --query 'environmentSummaries[*].[environmentId,environmentState,environmentStatus]' \
       --output table
   ```

1. To view the status and check results of one environment, run the following command.

   ```bash
   aws evs get-environment \
       --environment-id env-abcde12345 \
       --query 'environment.[environmentStatus,checks]'
   ```

1. To list the hosts of the environment, run the following command.

   ```bash
   aws evs list-environment-hosts \
       --environment-id env-abcde12345
   ```

1. To list the VLANs of the environment, run the following command.

   ```bash
   aws evs list-environment-vlans \
       --environment-id env-abcde12345
   ```

1. To list the connectors of the environment with their state and reachability check result, run the following command.

   ```bash
   aws evs list-environment-connectors \
       --environment-id env-abcde12345
   ```

For more information about the API operations, see [ListEnvironments](https://docs.aws.amazon.com/evs/latest/APIReference/API_ListEnvironments.html), [GetEnvironment](https://docs.aws.amazon.com/evs/latest/APIReference/API_GetEnvironment.html), [ListEnvironmentHosts](https://docs.aws.amazon.com/evs/latest/APIReference/API_ListEnvironmentHosts.html), and [ListEnvironmentVlans](https://docs.aws.amazon.com/evs/latest/APIReference/API_ListEnvironmentVlans.html) in the *Amazon Elastic VMware Service API Reference*.

You have viewed the status, the check results, and the resources of your environment. If a check fails, see [Troubleshoot environment status checks](#troubleshoot-status-checks). To get notified when Amazon EVS reports impaired environment health, see [Get notified about host events](#monitor-notifications).

### Get notified about host events
<a name="monitor-notifications"></a>

 AWS Health tells you in advance when an EC2 instance that runs one of your Amazon Elastic VMware Service hosts is scheduled for maintenance or retirement. You can route these events with Amazon EventBridge to an Amazon SNS topic, so that your operations team gets them by email or another channel. You can set up the rule with the EventBridge console or the AWS CLI.

#### Events that affect your environment
<a name="monitor-notifications-events"></a>

 AWS sends you the following notifications for an Amazon EVS environment.

 **EC2 scheduled events**   
If an instance that runs one of your hosts is affected by a scheduled event, AWS notifies you in advance. AWS sends an email to the address that’s associated with your AWS account, and sends an AWS Health event. The events include `system-maintenance` and `instance-retirement` events. For what to do when you get one, see [Host maintenance and retirement events](#evs-host-maintenance).

 **Impaired environment health**   
If a required connector is not created or becomes unreachable, Amazon EVS reports impaired environment health through AWS Health. For more information, see [Monitor environment status](#evs-env-status-check).

 **Windows Server entitlements**   
If the connection to vCenter Server is not restored within the 8-hour grace period, Amazon EVS removes the entitlements and sends you an AWS Health notification. For more information, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

 AWS Health events identify the affected EC2 instance. To find the host that runs on an instance, compare the instance ID with the `ec2InstanceId` value that `aws evs list-environment-hosts` returns.

#### Before you begin
<a name="monitor-notifications-before"></a>

Before you create the rule, complete the following tasks.
+ Create an Amazon SNS topic and subscribe your email address or another endpoint to it. For more information, see [Creating an Amazon SNS topic](https://docs.aws.amazon.com/sns/latest/dg/sns-create-topic.html) in the *Amazon Simple Notification Service Developer Guide*.
+ Make sure that the access policy of the topic allows EventBridge to publish to it. For more information, see [Amazon SNS permissions](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-use-resource-based.html#eb-sns-permissions) in the *Amazon EventBridge User Guide*.
+ Create the rule in each AWS Region where you have an Amazon EVS environment. EventBridge rules receive events for your own AWS account only.

#### Event pattern
<a name="monitor-notifications-pattern"></a>

The following event pattern matches AWS Health scheduled change events for Amazon EC2. This includes maintenance and retirement events for the instances that run your hosts.

```json
{
  "source": ["aws.health"],
  "detail-type": ["AWS Health Event"],
  "detail": {
    "service": ["EC2"],
    "eventTypeCategory": ["scheduledChange"]
  }
}
```

To also receive EC2 issue events, add `issue` to the `eventTypeCategory` list. For the fields that you can match, see [Reference: AWS Health events Amazon EventBridge schema](https://docs.aws.amazon.com/health/latest/ug/aws-health-events-eventbridge-schema.html) in the * AWS Health User Guide*.

The pattern matches events for every EC2 instance in your account, not only your Amazon EVS hosts.

#### Create the rule
<a name="monitor-notifications-create"></a>

To create the rule in the Amazon EventBridge console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EventBridge console](https://console.aws.amazon.com/events).

1. Create a rule on the default event bus with the event pattern from the previous section. For the steps, see [Creating rules in Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule-visual.html) in the *Amazon EventBridge User Guide*.

1. For the target, choose your Amazon SNS topic.

1. Create the rule.

1. Save the event pattern from the previous section in a file named `evs-host-events.json`.

1. Create the rule. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws events put-rule \
       --name evs-host-events \
       --event-pattern file://evs-host-events.json \
       --region <region-code>
   ```

1. Add your Amazon SNS topic as the target of the rule. Replace `<region-code>` with the Region where you created the environment. Replace the example account ID with your own value.

   ```bash
   aws events put-targets \
       --rule evs-host-events \
       --targets 'Id=1,Arn=arn:aws:sns:<region-code>:111122223333:evs-host-events' \
       --region <region-code>
   ```

   In other partitions, such as AWS GovCloud (US), replace `arn:aws` with the partition of your account, for example `arn:aws-us-gov`.

1. Confirm that the rule exists. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws events describe-rule \
       --name evs-host-events \
       --region <region-code>
   ```

You have created a rule that sends EC2 scheduled change events to your Amazon SNS topic. To check the pattern without waiting for an event, use the sample events in the EventBridge console. For more information, see [Configuring an EventBridge rule to send notifications about events in AWS Health](https://docs.aws.amazon.com/health/latest/ug/creating-event-bridge-events-rule-for-aws-health.html) in the * AWS Health User Guide*.

To receive events from all accounts in an organization, see [Aggregating AWS Health events](https://docs.aws.amazon.com/health/latest/ug/aggregate-events.html) in the * AWS Health User Guide*.

Next step: Plan how you respond to each event type. For more information, see [Host maintenance and retirement events](#evs-host-maintenance).

### Monitor hosts and VCF
<a name="monitor-observability"></a>

You monitor an Amazon Elastic VMware Service environment at two layers. Amazon CloudWatch and the EC2 status checks report on the EC2 instances that run your hosts. VMware Cloud Foundation (VCF) tools, such as VCF Operations, report on VCF components and workloads. You’re responsible for monitoring host health in both deployment modes, so we recommend that you set up both layers.

#### What each tool monitors
<a name="monitor-observability-compare"></a>

The following table shows what you monitor, the tool that reports it, and where to read more.


| What you monitor | Tool | More information | 
| --- | --- | --- | 
| Hardware and software problems of the EC2 instance that runs a host | EC2 status checks |  [Monitor host instances with EC2 status checks and CloudWatch](#monitor-observability-ec2)  | 
| Alarms when a status check fails on a host | Amazon CloudWatch |  [Monitor host instances with EC2 status checks and CloudWatch](#monitor-observability-ec2)  | 
| Usage of Amazon EVS resources that have quotas (environments and hosts) | Amazon CloudWatch |  [Service quotas for Amazon EVS](#service-quotas-evs)  | 
| Scheduled maintenance and retirement of host instances |  AWS Health |  [Get notified about host events](#monitor-notifications)  | 
| Amazon EVS environment status and connector health | Amazon EVS console or AWS CLI  |  [Monitor environment status](#evs-env-status-check)  | 
| VCF components, such as vCenter Server, NSX, and vSAN, and the workloads that run on them | VCF Operations and the VCF management tools |  [Monitor VCF components and workloads with VCF tools](#monitor-observability-vcf)  | 
| User activity inside VCF, such as changes in vSphere or NSX Manager | VCF logs and VCF Operations |  [Log API calls with CloudTrail](#logging-using-cloudtrail)  | 

#### Monitor host instances with EC2 status checks and CloudWatch
<a name="monitor-observability-ec2"></a>

Amazon EC2 runs automated checks on every running EC2 instance to identify hardware and software issues. You can view the results of these status checks in the EC2 console or with the AWS CLI to identify specific and detectable problems. For more information, see [View status checks for Amazon EC2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_status.html) in the *Amazon EC2 User Guide* and [describe-instance-status](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-status.html) in the * AWS CLI Command Reference*.

To find the EC2 instance of a host, run `aws evs list-environment-hosts` and read the `ec2InstanceId` value of the host.

You can create a CloudWatch alarm that warns you when status checks fail on a specific instance. For more information, see [Create CloudWatch alarms for Amazon EC2 instances that fail status checks](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating_status_check_alarms.html) in the *Amazon EC2 User Guide*.

**Important**  
Use the alarm only to notify you, and don’t add an alarm action that stops, reboots, terminates, or recovers the instance.

Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. For why EC2 automatic recovery doesn’t apply to Amazon EVS hosts, see [Why EC2 automatic recovery doesn’t apply](#evs-host-auto-recovery).

Amazon CloudWatch also collects AWS usage metrics for Amazon EVS resources that have quotas. For more information, see [CloudWatch usage metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Usage-Metrics.html) in the *Amazon CloudWatch User Guide*.

#### Monitor VCF components and workloads with VCF tools
<a name="monitor-observability-vcf"></a>

Amazon EVS doesn’t monitor the VCF components inside your environment. You can keep using the VCF management and monitoring tools that you already use, such as VCF Operations, for monitoring and for centralized VCF logging.

The VCF tools that you use differ by deployment mode.
+ In Self-deployed environments that run VCF 9.x, VCF Operations is a separate appliance that you deploy with VCF. For more information, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html).
+ In EVS-deployed environments (VCF 5.2.x), you monitor VCF components in SDDC Manager. For more information, see [Getting started with SDDC Manager](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/getting-started-with-sddc-manager-admin.html) in the Broadcom documentation.

On diskless hosts (VCF 9.x only), your datastores are on external storage, such as Amazon FSx. You monitor the capacity and health of that storage with the tools of the storage service. For more information, see [External storage options for diskless hosts](#external-storage-diskless). For the VCF versions and instance types that each mode supports, see [Supported VCF versions and instance types](#versions-provided).

#### Where to get help
<a name="monitor-observability-support"></a>

For problems with the Amazon EVS service, the EC2 instances, or the AWS network, contact AWS Support. For VCF configuration guidance, you can also use the Broadcom documentation and your Broadcom support entitlement. For who to contact for which problem, see [Get help from AWS Support and Broadcom](#get-help).

### Log API calls with CloudTrail
<a name="logging-using-cloudtrail"></a>

Amazon Elastic VMware Service is integrated with AWS CloudTrail, a service that records the actions that an IAM user, an IAM role, or an AWS service takes in Amazon EVS. With the information that CloudTrail collects, you can find the request that was made to Amazon EVS, who made it, when, and from which IP address.

CloudTrail captures all AWS API calls for Amazon EVS as events. These include calls from the Amazon EVS console and calls from your code to the Amazon EVS API operations. To learn more about CloudTrail, see the [AWS CloudTrail User Guide](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html).

**Note**  
Amazon EVS doesn’t log user activity for components outside AWS, such as activity within your VMware Cloud Foundation (VCF) environment.

VMware consoles such as the vSphere Client and NSX Manager log these activities. For centralized VCF logging with a VCF monitoring solution such as VCF Operations, see [Monitor hosts and VCF](#monitor-observability).

#### Amazon EVS information in CloudTrail
<a name="service-name-info-in-cloudtrail"></a>

CloudTrail is turned on in your AWS account when you create the account. When activity occurs in Amazon EVS, CloudTrail records it in an event, together with the events of other AWS services, in **Event history**. You can view, search, and download recent events in your AWS account, even if you don’t create a trail. For more information, see [Working with CloudTrail event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html) in the * AWS CloudTrail User Guide*.

For an ongoing record of events in your AWS account, including events for Amazon EVS, create a trail. A *trail* delivers log files to an Amazon S3 bucket that you specify. By default, a trail that you create in the console applies to all AWS Regions in the partition. You can also configure other AWS services to analyze and act on the event data that CloudTrail collects. For more information, see the following topics in the * AWS CloudTrail User Guide*.
+  [Overview for creating a trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) 
+  [CloudTrail supported services and integrations](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-aws-service-specific-topics.html#cloudtrail-aws-service-specific-topics-integrations) 
+  [Configuring Amazon SNS notifications for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/getting_notifications_top_level.html) 
+  [Receiving CloudTrail log files from multiple Regions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/receive-cloudtrail-log-files-from-multiple-regions.html) 
+  [Receiving CloudTrail log files from multiple accounts](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html) 

CloudTrail logs all Amazon EVS actions. The actions are documented in the [Amazon Elastic VMware Service API Reference](https://docs.aws.amazon.com/evs/latest/APIReference/Welcome.html). For example, calls to the `CreateEnvironment`, `GetEnvironment`, and `DeleteEnvironment` actions generate entries in the CloudTrail log files.

Every event or log entry contains information about who generated the request. The identity information helps you determine the following:
+ Whether the request was made with root user or IAM user credentials.
+ Whether the request was made with temporary security credentials for a role or a federated user.
+ Whether the request was made by another AWS service.

For more information, see [CloudTrail userIdentity element](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html) in the * AWS CloudTrail User Guide*.

#### Find Amazon EVS events
<a name="cloudtrail-find-evs-events"></a>

To list recent Amazon EVS events in **Event history** with the AWS CLI, filter on the event source. Replace `<region-code>` with the Region where you created the environment.

```bash
aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventSource,AttributeValue=evs.amazonaws.com \
    --max-results 20 \
    --region <region-code>
```

To find who deleted a host, filter on the event name, for example `DeleteEnvironmentHost`. Replace `<region-code>` with the Region where you created the environment.

```bash
aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteEnvironmentHost \
    --region <region-code>
```

#### Understanding Amazon EVS log file entries
<a name="understanding-service-name-entries"></a>

CloudTrail log files contain one or more log entries. An event represents a single request from any source. It includes information about the requested action, the date and time of the action, the request parameters, and other details. CloudTrail log files aren’t an ordered stack trace of the public API calls, so the entries don’t appear in any specific order.

## Manage hosts
<a name="host-management"></a>

Because Amazon Elastic VMware Service is a self-managed service, you’re responsible for monitoring host health and for replacing a host when it fails. You add, replace, and delete hosts with the Amazon EVS console or the AWS CLI, and you bring them into or out of the cluster with VMware Cloud Foundation (VCF) tools. The VCF steps differ by VCF version, so each version has its own pages.

For the steps that apply to every version, see [Add a host](#host-add-overview) and [Delete a host](#host-delete-overview).

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API.

The following table shows the page for each host task by deployment mode.


<table>
<thead>
  <tr><th>Task</th><th>VCF 9.x (Self-deployed)</th><th>VCF 5.2.x (EVS-deployed)</th></tr>
</thead>
<tbody>
  <tr><td>Prepare for EC2 maintenance and retirement events</td><td colspan="2"> <a href="#evs-host-maintenance">Host maintenance and retirement events</a> </td></tr>
  <tr><td>Add a host</td><td> <a href="#host-add">Add a host (VCF 9.x)</a> </td><td> <a href="#evs-env-create-host">Add a host (VCF 5.x)</a> </td></tr>
  <tr><td>Replace a failed or retiring host</td><td> <a href="#host-replace">Replace a failed host (Self-deployed)</a> </td><td> <a href="#host-replace-evs-deployed">Replace a host (EVS-deployed)</a> </td></tr>
  <tr><td>Delete a host</td><td> <a href="#host-delete">Delete a host (VCF 9.x)</a> </td><td> <a href="#evs-env-delete-host">Delete a host (VCF 5.x)</a> </td></tr>
  <tr><td>Add or replace a diskless host</td><td> <a href="#host-add">Add a host (VCF 9.x)</a> and <a href="#host-replace">Replace a failed host (Self-deployed)</a> </td><td>Not available</td></tr>
</tbody>
</table>


For the VCF versions and instance types that each mode supports, see [Supported VCF versions and instance types](#versions-provided).

Before you add or replace hosts, check your **Host count per EVS environment** quota. Host creation fails if this quota is lower than the number of hosts that you want in the environment. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs). If a host task fails, see [Troubleshoot hosts](#troubleshoot-hosts).

**Topics**
+ [Host maintenance and retirement events](#evs-host-maintenance)
+ [Add a host](#host-add-overview)
+ [Add a host (VCF 9.x)](#host-add)
+ [Replace a failed host (Self-deployed)](#host-replace)
+ [Delete a host](#host-delete-overview)
+ [Delete a host (VCF 9.x)](#host-delete)
+ [Add a host (VCF 5.x)](#evs-env-create-host)
+ [Replace a host (EVS-deployed)](#host-replace-evs-deployed)
+ [Delete a host (VCF 5.x)](#evs-env-delete-host)

### Host maintenance and retirement events
<a name="evs-host-maintenance"></a>

Amazon Elastic VMware Service hosts run on EC2 bare metal instances, and AWS performs scheduled maintenance on these instances. You’re responsible for the VMware Cloud Foundation (VCF) software on the host, for monitoring host health, and for replacing a host before a scheduled event or after a host failure. What a scheduled event does to your data depends on whether the host has local storage or is diskless.

For more information about managing ESX hosts in VCF, see the [Broadcom documentation for VCF 9.x](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html), or [Host Management](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/host-management-admin.html) in the Broadcom documentation for VCF 5.2.x.

#### Checking health of the underlying EC2 instance
<a name="evs-host-ec2-instance-health"></a>

Amazon EC2 runs automated status checks on every running instance, and you can create a CloudWatch alarm that warns you when a check fails. For the steps, see [Monitor hosts and VCF](#monitor-observability).

#### About AWS scheduled maintenance for EC2 instances
<a name="evs-host-maintenance-about"></a>

 AWS performs scheduled maintenance on the underlying EC2 instances for reliability, availability, and performance. EC2 bare metal instances are subject to the same types of scheduled events as other EC2 instances. AWS can schedule events to reboot, stop, and retire your instances because of underlying hardware issues or scheduled maintenance. These events don’t occur often. For more information, see [Types of scheduled events](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html#types-of-scheduled-events) in the *Amazon EC2 User Guide*.

If a scheduled event affects one of your instances, AWS notifies you in advance. AWS sends an email to the address that’s associated with your AWS account, and sends an AWS Health event. To route these events to your team with Amazon EventBridge, see [Get notified about host events](#monitor-notifications). For more information about the events, see [Scheduled events for Amazon EC2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html) in the *Amazon EC2 User Guide*.

You can reschedule an event so that it occurs at a date and time that suits you, up to the event deadline. For more information, see [Reschedule a scheduled event for an EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/reschedule-event.html) in the *Amazon EC2 User Guide*.

**Important**  
Before a scheduled reboot event, place the host in maintenance mode in the vSphere Client.

#### Preparing for AWS scheduled `system-maintenance` and `instance-retirement` events
<a name="evs-host-preparing-aws-scheduled-maintenance"></a>

 AWS schedules two types of `system-maintenance` events, network maintenance and power maintenance.
+ During network maintenance, the scheduled instances lose network connectivity for a brief period. Normal network connectivity is restored after maintenance is complete.
+ During power maintenance, the scheduled instances are taken offline for a brief period, and then rebooted. When EC2 reboots a bare metal instance that has instance store volumes, the instance store data is not preserved.

 AWS schedules EC2 `instance-retirement` events when it detects degradation of the underlying hardware that hosts your instances.

The following table shows how to respond by host storage type.


| Host storage | VCF version | What to do | 
| --- | --- | --- | 
| Local storage (vSAN on instance store volumes) | VCF 5.2.x and VCF 9.x | Replace the host with a new host before the event occurs. If you wait for the event and EC2 must reboot the instance, you lose the vSAN data that’s stored on the instance store volumes. | 
| Diskless hosts | VCF 9.x | Diskless hosts have no instance store volumes, and your data stays on your external storage. | 

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API.

##### Replace an Amazon EVS host
<a name="evs-replace-host"></a>

The replacement procedure depends on the deployment mode.
+ For Self-deployed environments that run VCF 9.x, see [Replace a failed host (Self-deployed)](#host-replace).
+ For EVS-deployed environments (VCF 5.2.x), see [Replace a host (EVS-deployed)](#host-replace-evs-deployed).

#### Why EC2 automatic recovery doesn’t apply
<a name="evs-host-auto-recovery"></a>

Amazon EC2 offers two kinds of instance recovery.
+ Simplified automatic recovery doesn’t support metal instance sizes, and doesn’t support instances that attach instance store volumes at launch.
+ CloudWatch action-based recovery supports metal instances, but instance store data isn’t preserved. On hosts with local storage, vSAN keeps its data on the instance store volumes.

For more information, see [Automatic instance recovery](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html) in the *Amazon EC2 User Guide*. Don’t configure a recover action for an Amazon EVS host. Replace a failed or retiring host instead.

#### Using EC2 On-Demand Capacity Reservations
<a name="evs-host-maintenance-odcr"></a>

You can use EC2 On-Demand Capacity Reservations to make sure that your cluster has enough capacity during maintenance periods. You can reserve capacity in a specific Availability Zone for any duration. For the reservation settings that Amazon EVS hosts use, see [Reserve capacity for hosts](#plan-capacity). For more information, see [Reserve compute capacity with EC2 On-Demand Capacity Reservations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html) and [Create a Capacity Reservation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservations-create.html) in the *Amazon EC2 User Guide*.

A Capacity Reservation gives you access to EC2 capacity in an Availability Zone, and a spare host adds a layer of redundancy. With Dedicated Hosts, a spare host keeps your mission-critical workloads running if a host needs maintenance or has an issue.

**Note**  
If you use EC2 On-Demand Capacity Reservations or EC2 Dedicated Hosts, we recommend that you keep a spare host for mission-critical workloads.

#### Troubleshooting
<a name="evs-maintenance-troubleshooting"></a>

For host problems, such as cluster image remediation failures and commissioning validation failures, see [Troubleshoot hosts](#troubleshoot-hosts). For who to contact, AWS Support or Broadcom, see [Get help from AWS Support and Broadcom](#get-help).

### Add a host
<a name="host-add-overview"></a>

1. In VMware Cloud Foundation (VCF), you add the host to a cluster with VCF tools. The steps depend on the VCF version that the cluster runs. == Before you begin Before you create the host, make sure that the following are true.
   + A DNS forward lookup zone contains an A record for the host, and a reverse lookup zone contains a PTR record for the host. For more information, see [DNS records for hosts and appliances](#evs-prereq-dns-records).
   + Your **Host count per EVS environment** quota is at least the number of hosts that the environment will have after you add hosts. Host creation fails if the quota is lower. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs).
   + You have an Amazon EC2 key pair. Amazon EVS requires the key pair name when you create a host.
   + You know the instance type and the ESX version that the target cluster uses. For the versions that are available to your account, run `aws evs get-versions` or see [Supported VCF versions and instance types](#versions-provided). == Choose the procedure Each procedure covers both parts, including the console and AWS CLI steps for creating the host.


|  |  | 
| --- |--- |
| VCF version of the cluster | Procedure | 
| VCF 9.x |  [Add a host (VCF 9.x)](#host-add). Use this procedure also to add a diskless host. | 
| VCF 5.x |  [Add a host (VCF 5.x)](#evs-env-create-host)  | 

If a host stays in the `CREATING` state or moves to `CREATE_FAILED`, check its DNS records first. For more information, see [Troubleshoot hosts](#troubleshoot-hosts).

### Add a host (VCF 9.x)
<a name="host-add"></a>

After you deploy VMware Cloud Foundation (VCF) in a Self-deployed environment, you can add hosts to increase capacity and workload resiliency. You create the host with the Amazon Elastic VMware Service console or the AWS CLI, and then you add it to a cluster with VCF 9.x tools. You can add hosts with local storage or diskless hosts.

To add the first hosts of a new environment, see [Step 3: Add hosts](#self-deployed-add-hosts) instead. For EVS-deployed environments (VCF 5.2.x), see [Add a host (VCF 5.x)](#evs-env-create-host).

**Important**  
All hosts in a cluster must use the same instance type and the same ESX version. After you add a host, you can change its ESX version only with vSphere Lifecycle Manager.

#### Before you begin
<a name="host-add-prereqs"></a>

Before you begin, make sure that the following are true.
+ You created the forward (A) and reverse (PTR) DNS records for the new host, and they resolve. For more information, see [Step 2: Create DNS records](#self-deployed-host-dns-records).
+ Your **Host count per EVS environment** quota is at least the number of hosts that the environment will have after you add hosts. Host creation fails if the quota is lower. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs).
+ You have the name of an Amazon EC2 key pair. Amazon EVS requires the key pair name when you add a host.
+ You know the instance type and ESX version of the hosts in the target cluster. For the versions available to your account, run `aws evs get-versions` or see [Supported VCF versions and instance types](#versions-provided).
+ For a diskless host, the new host can reach the external storage of the cluster. For more information, see [Step 2: Connect a diskless host to external storage](#diskless-host-add-storage).

#### Step 1: Create the host
<a name="host-add-create"></a>

To create the host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**, and then choose your environment.

1. Choose the **Hosts** tab, and then choose **Create host**.

1. Enter the short host name from your DNS records (such as `esx05`), the instance type and ESX version of the target cluster, and your Amazon EC2 key pair.

1. Choose **Create host**.

1. To verify completion, check that the **Host state** has changed to **Created**.

1. Run the `aws evs create-environment-host` command. The following example adds a host that runs ESX 9.0.2. Replace the example environment ID, host name, and key pair name with your own values. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs create-environment-host \
       --environment-id env-abcde12345 \
       --esx-version ESXi-9.0.2.0.25148076 \
       --host '{
           "hostName": "esx05",
           "keyName": "my-ec2-key-pair",
           "instanceType": "i4i.metal"
       }' \
       --region <region-code>
   ```

1. Check the host state. The create command doesn’t return a host ID, so use the host name to find the host. Replace the example environment ID with your own value. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs list-environment-hosts \
       --environment-id env-abcde12345 \
       --query 'environmentHosts[].[hostName,hostState,ipAddress]' \
       --output table \
       --region <region-code>
   ```

Don’t run the create command again for a host that is in the `CREATING` state. A host usually reaches `CREATED` in about 9 minutes. When the host is `CREATED`, confirm that its IP address equals the address in its A record. If the host stays in `CREATING` or moves to `CREATE_FAILED`, check its DNS records first. For more information, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

Amazon EVS stores the ESX root password of each host in AWS Secrets Manager. Each host has its own secret. You need the root password to add the host to VCF.

#### Step 2: Connect a diskless host to external storage
<a name="diskless-host-add-storage"></a>

Skip this step for hosts with local storage.

Diskless hosts have no local storage for VCF, so the datastores of the cluster are on external storage. Before you add a diskless host to a cluster, make sure that the host can reach the external storage of the cluster, and mount the datastores on the host.
+ For Amazon FSx for OpenZFS, the NFS export list of each volume and the security group of the file system must allow the host management VLAN subnet. For more information, see [Configure FSx for OpenZFS as principal storage for diskless hosts](#config-fsx-openzfs-datastore).
+ For other external storage, see [External storage options for diskless hosts](#external-storage-diskless).

#### Step 3: Add the host to a cluster in VCF
<a name="host-add-vcf"></a>

 **In VCF** 

1. Add the host to the VCF inventory with the ESX root password from Secrets Manager.

1. Add the host to the target cluster.

For the procedures, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html).

You have added the host when it is connected in the cluster and shows no configuration errors. To check the environment after the change, see [Monitor environment status](#evs-env-status-check). If adding the host to the cluster fails, see [Troubleshoot hosts](#troubleshoot-hosts).

### Replace a failed host (Self-deployed)
<a name="host-replace"></a>

Replace a host in a Self-deployed environment when it fails, or before an EC2 `system-maintenance` or `instance-retirement` event occurs. You add a new host with the Amazon Elastic VMware Service console or the AWS CLI. You then move the cluster workloads to it with VMware Cloud Foundation (VCF) 9.x tools and delete the old host. For when to replace a host, see [Host maintenance and retirement events](#evs-host-maintenance). For EVS-deployed environments (VCF 5.2.x), see [Replace a host (EVS-deployed)](#host-replace-evs-deployed).

**Warning**  
Replace a host with local storage before the scheduled event occurs. If EC2 reboots the instance during the event, you lose the vSAN data that’s stored on its instance store volumes.

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API.

Diskless hosts have no instance store volumes, and your data stays on your external storage.

#### Before you begin
<a name="host-replace-prereqs"></a>

Before you begin, make sure that the following are true.
+ Your **Host count per EVS environment** quota allows one more host than the environment has. You might need to request a quota increase for maintenance that requires host replacement. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs).
+ You created the A and PTR DNS records for the new host. For more information, see [Step 2: Create DNS records](#self-deployed-host-dns-records).
+ You know the instance type and ESX version of the hosts in the cluster. The new host must use the same instance type and ESX version.
+ Any spare host or Capacity Reservation that you keep for replacements is available. For more information, see [Reserve capacity for hosts](#plan-capacity).

#### Step 1: Add the new host
<a name="host-replace-add"></a>

Create the new host, and for a diskless host, connect it to the external storage of the cluster. For the steps, see [Step 1: Create the host](#host-add-create) and [Step 2: Connect a diskless host to external storage](#diskless-host-add-storage). Get the ESX root password of the new host from AWS Secrets Manager.

#### Step 2: Move the workloads and remove the old host from VCF
<a name="host-replace-vcf"></a>

 **In VCF** 

1. Add the new host to the cluster that contains the host to be replaced.

1. If the old host still runs, place it in maintenance mode in the vSphere Client and migrate its VMs to the other hosts.

1. Remove the old host from the cluster and from the VCF inventory.

For the procedures, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html).

#### Step 3: Delete the old host
<a name="host-replace-delete"></a>

To delete the old host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**, and then choose your environment.

1. Choose the **Hosts** tab.

1. Select the old host and choose **Delete** > **Delete host**.

1. To verify completion, check that the old host no longer appears in the **Hosts** tab, or that its **Host state** is **Deleted**.

1. Run the `delete-environment-host` command. Replace the example environment ID and host name with your own values. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs delete-environment-host \
       --environment-id env-abcde12345 \
       --host-name esx01 \
       --region <region-code>
   ```

1. To verify completion, list the hosts and confirm that the old host is no longer listed or that its state is `DELETED`. Replace the example environment ID with your own value. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs list-environment-hosts \
       --environment-id env-abcde12345 \
       --query 'environmentHosts[].[hostName,hostState]' \
       --output table \
       --region <region-code>
   ```

You have replaced the host. To check the environment after the change, see [Monitor environment status](#evs-env-status-check). If a step fails, see [Troubleshoot hosts](#troubleshoot-hosts).

### Delete a host
<a name="host-delete-overview"></a>

1. In Amazon EVS, you delete the host with the Amazon EVS console or the AWS CLI. WARNING: Remove the host in VCF before you delete it in Amazon EVS. If you delete a host without removing it first, stale data stays in vCenter Server and SDDC Manager. Before you delete a host, make sure that the cluster keeps enough hosts for your VCF topology.

   <a name="host-delete-overview-choose"></a>== Choose the procedure Each procedure covers both parts, including the console and AWS CLI steps for deleting the host.


|  |  | 
| --- |--- |
| VCF version of the cluster | Procedure | 
| VCF 9.x |  [Delete a host (VCF 9.x)](#host-delete)  | 
| VCF 5.x |  [Delete a host (VCF 5.x)](#evs-env-delete-host)  | 

To delete every host and the environment, see [Clean up a Self-deployed environment](#self-deployed-cleanup) for Self-deployed environments, or [Delete an EVS-deployed environment](#getting-started-cleanup-env-hosts) for EVS-deployed environments.

### Delete a host (VCF 9.x)
<a name="host-delete"></a>

You can delete a host from a Self-deployed environment when you no longer need it. You remove the host from its cluster with VMware Cloud Foundation (VCF) 9.x tools first, and then delete it with the Amazon Elastic VMware Service console or the AWS CLI. Before you delete a host, make sure that the cluster keeps enough hosts for your VCF topology.

For minimum host counts, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html).

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API. Deleting a host from the Amazon EC2 console can leave your environment in an inconsistent state.

To delete every host and the environment, see [Clean up a Self-deployed environment](#self-deployed-cleanup) instead. For EVS-deployed environments (VCF 5.2.x), see [Delete a host (VCF 5.x)](#evs-env-delete-host).

#### Step 1: Remove the host from VCF
<a name="host-delete-vcf"></a>

 **In VCF** 

1. Place the host in maintenance mode in the vSphere Client, and migrate its VMs to the other hosts in the cluster.

1. Remove the host from its cluster and from the VCF inventory.

For the procedures, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html).

For a diskless host, the datastores of the cluster are on external storage, so removing the host doesn’t delete them.

#### Step 2: Delete the host
<a name="host-delete-evs"></a>

To delete the host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**, and then choose your environment.

1. Choose the **Hosts** tab.

1. Select the host and choose **Delete** > **Delete host**. Repeat this step for each host that you want to delete.

1. To verify completion, check that the host no longer appears in the **Hosts** tab, or that its **Host state** is **Deleted**.

1. Run the `delete-environment-host` command. Replace the example environment ID and host name with your own values. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs delete-environment-host \
       --environment-id env-abcde12345 \
       --host-name esx05 \
       --region <region-code>
   ```

1. To verify completion, list the hosts and confirm that the host is no longer listed or that its state is `DELETED`. Replace the example environment ID with your own value. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs list-environment-hosts \
       --environment-id env-abcde12345 \
       --query 'environmentHosts[].[hostName,hostState]' \
       --output table \
       --region <region-code>
   ```

You have deleted the host. To check the environment after the change, see [Monitor environment status](#evs-env-status-check).

### Add a host (VCF 5.x)
<a name="evs-env-create-host"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Manage hosts](#host-management).

After your Amazon Elastic VMware Service environment is deployed, you can add hosts to increase capacity and workload resiliency. You create the host with the Amazon EVS console or the AWS CLI, and then you commission it and add it to the cluster in SDDC Manager.

For host count requirements for your VMware Cloud Foundation (VCF) deployment topology, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2.html).

**Important**  
If you create a host with an ESX version that’s newer than the ESX version of your vSphere cluster image, you can’t downgrade the host. You must delete the host and create it again with the correct ESX version.

#### Before you begin
<a name="evs-env-create-host-prereqs"></a>

Before you begin, make sure that the following are true.
+ A DNS forward lookup zone contains an A record for the host, and a reverse lookup zone contains a PTR record for the host. For more information, see [DNS records for hosts and appliances](#evs-prereq-dns-records).
+ Your **Host count per EVS environment** quota is at least the number of hosts that the environment will have after you add hosts. Host creation fails if the quota is lower. To raise the quota, request a quota increase. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs).
+ You know the ESX version that your target vSphere cluster uses. Select this version when you create the host, or an older version if it’s unavailable. If you choose an older version, upgrade the host with vSphere Lifecycle Manager, which can require host reboots and increase the time it takes to commission the host. If you don’t specify an ESX version, Amazon EVS uses the default ESX version of your environment’s VCF version. For more information, see [Supported VCF versions and instance types](#versions-provided).
+ If you plan to use an EC2 partition placement group for your hosts, see the placement group guidance in [Resilience for Amazon EVS](#disaster-recovery-resiliency).

The following conditions can make the SDDC Manager steps fail. Check them before you add the host.
+ A new host has the latest available version of the Amazon EVS custom vendor add-on, which provides important host functionality. If the hosts in your cluster use an older add-on version, adding the new host to the vSphere cluster causes cluster image remediation to fail. For the fix, see [Troubleshoot hosts](#troubleshoot-hosts).
+ If you updated your ESX version after the environment was deployed, SDDC Manager can fail during VCF host validation in the commission hosts step. For the fix, see [Troubleshoot hosts](#troubleshoot-hosts).

#### Create the host
<a name="evs-env-create-host-evs"></a>

To create the host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment where you want to create the host.

1. Choose the **Hosts** tab.

1. Choose **Create host**.

1. Enter the host name, the instance type, the ESX version, and your Amazon EC2 key pair, and then choose **Create host**.

1. To verify completion, check that the **Host state** has changed to **Created**.

1. Run the `create-environment-host` command. The following example adds a host that runs ESX 8.0 U3g. Replace the example environment ID, host name, and key pair name with your own values.

   ```bash
   aws evs create-environment-host \
       --environment-id "env-abcde12345" \
       --esx-version "ESXi-8.0U3g-24859861" \
       --host '{
           "hostName": "esxi-host-05",
           "keyName": "my-ec2-key-pair",
           "instanceType": "i4i.metal"
       }'
   ```

1. To verify completion, check that the state of the host is `CREATED`. Replace the example environment ID with your own value.

   ```bash
   aws evs list-environment-hosts \
       --environment-id "env-abcde12345"
   ```

#### Commission the host in SDDC Manager
<a name="evs-env-create-host-sddc"></a>

You must commission the host and assign it to a cluster in the SDDC Manager user interface.

 **In SDDC Manager** 

1. Commission the new host. For more information, see [Commission Hosts](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/host-management-admin/commission-hosts-admin.html) in the Broadcom documentation.

1. Add the new host to the cluster. For more information, see [How to Add an ESX Host to Your vSphere Cluster by Using the Quickstart Workflow](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/organizing-your-inventory/creating-and-configuring-clusters/use-quickstart-to-add---host-to-a-cluster.html) in the Broadcom documentation.

You have added a host to your environment and to the cluster. To check the environment after the change, see [Monitor environment status](#evs-env-status-check). If commissioning or cluster image remediation fails, see [Troubleshoot hosts](#troubleshoot-hosts).

### Replace a host (EVS-deployed)
<a name="host-replace-evs-deployed"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Manage hosts](#host-management).

Replace a host when it fails, or before an EC2 `system-maintenance` or `instance-retirement` event occurs. You create the new host with the Amazon Elastic VMware Service console or the AWS CLI, move it into the cluster in SDDC Manager, and then delete the old host. For when to replace a host, see [Host maintenance and retirement events](#evs-host-maintenance).

**Warning**  
Replace a host with local storage before the scheduled event occurs. If EC2 reboots the instance during the event, you lose the vSAN data that’s stored on its instance store volumes.

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API.

**Important**  
Amazon EVS hosts use the Amazon EVS custom vendor add-on to provide important host functionality. A new host has the latest available version of the add-on. If your environment uses hosts with an older add-on version, adding the new host to your vSphere cluster causes cluster image remediation to fail. For the fix, see [Troubleshoot hosts](#troubleshoot-hosts).

**Note**  
If you updated your ESX version after deployment, SDDC Manager can fail during VMware Cloud Foundation (VCF) host validation in the commission hosts step. For the fix, see [Troubleshoot hosts](#troubleshoot-hosts).

#### Before you begin
<a name="host-replace-evs-deployed-prereqs"></a>

Before you begin, make sure that the following are true.
+ Your **Host count per EVS environment** quota allows one more host than the environment has. Host creation fails if the quota is lower than the number of hosts in the environment. You might need to request a quota increase for maintenance that requires host replacement. For more information, see [Service quotas for Amazon EVS](#service-quotas-evs).
+ The A and PTR DNS records for the new host exist. For more information, see [Deployment prerequisite checklist](#evs-deployment-prereq-checklist).
+ You know the ESX version that your vSphere cluster uses. For more information, see [Add a host (VCF 5.x)](#evs-env-create-host).

#### Step 1: Create the new host
<a name="host-replace-evs-deployed-create"></a>

To create the new host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the host to be replaced.

1. Choose the **Hosts** tab.

1. Choose **Create host**.

1. Enter the host name, the instance type, the ESX version, and your Amazon EC2 key pair, and then choose **Create host**.

1. To verify completion, check that the **Host state** has changed to **Created**.

1. Run the `create-environment-host` command. The following example adds a host that runs ESX 8.0 U3g. Replace the example environment ID, host name, and key pair name with your own values.

   ```bash
   aws evs create-environment-host \
       --environment-id "env-abcde12345" \
       --esx-version "ESXi-8.0U3g-24859861" \
       --host '{
           "hostName": "esxi-host-05",
           "keyName": "my-ec2-key-pair",
           "instanceType": "i4i.metal"
       }'
   ```

1. To verify completion, check that the state of the new host is `CREATED`. Replace the example environment ID with your own value.

   ```bash
   aws evs list-environment-hosts \
       --environment-id "env-abcde12345"
   ```

#### Step 2: Retrieve the ESX root password
<a name="host-replace-evs-deployed-creds"></a>

1. Retrieve the ESX root password of the new host from AWS Secrets Manager. For more information, see [Retrieve VCF credentials and access appliances](#access-vcf) and [Get secrets from AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html) in the * AWS Secrets Manager User Guide*.

1. Sign in to the VMware Host Client of the new host as `root` with this password, to confirm that the password works before you commission the host.

#### Step 3: Move the new host into the cluster
<a name="host-replace-evs-deployed-sddc"></a>

 **In SDDC Manager** 

1. Commission the new host, using the ESX root password that you retrieved in the previous step. For more information, see [Commission Hosts](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/host-management-admin/commission-hosts-admin.html) in the Broadcom documentation.

1. Add the new host to the cluster that contains the host to be replaced. For more information, see [How to Add an ESX Host to Your vSphere Cluster by Using the Quickstart Workflow](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/organizing-your-inventory/creating-and-configuring-clusters/use-quickstart-to-add---host-to-a-cluster.html) in the Broadcom documentation.

1. Decommission the old host to remove it from SDDC Manager. For more information, see [Decommission Hosts](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/host-management-admin/decommission-hosts-admin.html) in the Broadcom documentation.

#### Step 4: Delete the old host
<a name="host-replace-evs-deployed-delete"></a>

To delete the old host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Return to the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. On the **Hosts** tab, select the old host and choose **Delete** > **Delete host**.

1. Run the `delete-environment-host` command. Replace the example environment ID and host name with your own values.

   ```bash
   aws evs delete-environment-host \
       --environment-id "env-abcde12345" \
       --host-name "esxi-host-01"
   ```

1. To verify completion, list the hosts and confirm that the old host is no longer listed or is in the `DELETED` state.

   ```bash
   aws evs list-environment-hosts \
       --environment-id "env-abcde12345"
   ```

You have replaced the host. To check the environment after the change, see [Monitor environment status](#evs-env-status-check). If a step fails, see [Troubleshoot hosts](#troubleshoot-hosts).

### Delete a host (VCF 5.x)
<a name="evs-env-delete-host"></a>

This page applies to EVS-deployed environments (VCF 5.2.x). For Self-deployed environments that run VCF 9.x, see [Manage hosts](#host-management).

You can delete an Amazon Elastic VMware Service host from your environment when you no longer need it. You decommission the host in SDDC Manager first, and then delete it with the Amazon EVS console or the AWS CLI. Before you delete a host, make sure that your environment keeps enough hosts for your VMware Cloud Foundation (VCF) deployment topology.

For minimum host counts, see the [Broadcom documentation for VCF](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2.html).

**Warning**  
If you delete a host without decommissioning it, stale data stays in vCenter Server and SDDC Manager, and you might need extra work to clean it up. Decommission the host before you delete it in the Amazon EVS console or with the Amazon EVS API.

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API. Deleting a host from the Amazon EC2 console can leave your environment in an inconsistent state.

#### Step 1: Decommission the host
<a name="evs-env-delete-host-sddc"></a>

 **In SDDC Manager** 

1. Remove the host from its cluster.

1. Decommission the host. For more information, see [Decommission Hosts](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/host-management-admin/decommission-hosts-admin.html) in the Broadcom documentation.

#### Step 2: Delete the host
<a name="evs-env-delete-host-evs"></a>

To delete the host in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the host to delete.

1. Choose the **Hosts** tab.

1. Select the host and choose **Delete** > **Delete host**. Repeat this step for each host that you want to delete.

1. To verify completion, check that the host no longer appears on the **Hosts** tab, or that its **Host state** is **Deleted**.

1. Run the `delete-environment-host` command. Replace the example environment ID and host name with your own values.

   ```bash
   aws evs delete-environment-host \
       --environment-id env-abcde12345 \
       --host-name esxi-host-05
   ```

1. To verify completion, list the hosts of the environment and confirm that the host is no longer listed or that its state is `DELETED`.

   ```bash
   aws evs list-environment-hosts \
       --environment-id env-abcde12345 \
       --query 'environmentHosts[].[hostName,hostState]' \
       --output table
   ```

You have deleted the host. In SDDC Manager, confirm that the host no longer appears in the host inventory. To check the environment after the change, see [Monitor environment status](#evs-env-status-check).

## Update and upgrade
<a name="evs-lifecycle-mgmt"></a>

You’re responsible for patching, updating, and upgrading the VMware Cloud Foundation (VCF) software in your Amazon Elastic VMware Service environment. This includes ESX, vCenter Server, vSAN, NSX, SDDC Manager, and VCF Operations, and any third-party integrations that you add, such as data protection solutions. For the full split of responsibilities, see [Shared responsibility and support](#plan-shared-responsibility).

You have full control over your VCF software stack, so you can tune it for your applications.

### Who updates what
<a name="evs-lifecycle-who-updates"></a>

The following table lists each part of the environment and who maintains it after deployment. The two mode columns name the owner, either AWS or you.


| Part of the environment | VCF 9.x (Self-deployed) | VCF 5.2.x (EVS-deployed) | Notes | 
| --- | --- | --- | --- | 
| EC2 bare metal hosts and their firmware |  AWS  |  AWS  |  AWS performs scheduled maintenance on the underlying EC2 instances. For more information, see [Host maintenance and retirement events](#evs-host-maintenance). | 
|  AWS networking that Amazon EVS uses (VPC route tables, security groups, network ACLs, VPC Route Server, internet gateways, NAT gateways, transit gateways) | You | You |  AWS deploys the environment with the network settings that you provide. You maintain them afterward. | 
| Initial VCF installation | You |  AWS  | In Self-deployed environments, you install VCF yourself with the VCF Installer. In EVS-deployed environments, Amazon EVS installs VCF when it creates the environment. | 
| Patches, updates, and upgrades of VCF software (ESX, vCenter Server, vSAN, NSX, SDDC Manager, and other components) | You | You | See [VMware software updates](#vmware-updates). | 
| NSX configuration (segments, distributed firewall rules, load balancers, added Tier-1 gateways) and integrated solutions such as VMware HCX | You | You | In EVS-deployed environments, Amazon EVS creates the first Tier-0 and Tier-1 gateways. See the note after this table. | 
| Health monitoring and remediation of ESX hosts | You | You | For more information, see [Manage hosts](#host-management). | 
| External storage for diskless hosts, including its capacity, availability, backups, and patching | You | Not available | For more information, see [External storage options for diskless hosts](#external-storage-diskless). | 

**Note**  
In EVS-deployed environments, Amazon EVS creates and configures a Tier-0 gateway and a Tier-1 gateway when it deploys the environment. Amazon EVS supports a single Tier-0 gateway. Don’t modify these gateways or the NSX Edge node VMs, because changes can affect connectivity.

### Software depots
<a name="evs-lifecycle-depots"></a>

You get VCF software and the Amazon EVS custom vendor add-on from different software depots. Each depot serves a different purpose, and you can’t use one in place of another.


| Depot | Who runs it | What it provides | Where you use it | Access | 
| --- | --- | --- | --- | --- | 
| Broadcom online depot | Broadcom | VCF software for installation, patches, and upgrades. | The VCF Installer when you install VCF 9.x. For Day 2 updates, see the Broadcom documentation for your VCF version. | A Broadcom credential with a valid VCF entitlement, and outbound internet access. For the install steps, see [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf). | 
| Amazon EVS custom add-on depot |  AWS  | The add-on that ESX hosts need to run on Amazon EC2. | vSphere Lifecycle Manager (vLCM), as a download source for cluster images. | A URL with an access token that you get with the `GetDepotUrl` API action. For the steps, see [Use the Amazon EVS custom add-on depot](#addon-depot). | 

The Amazon EVS custom add-on depot doesn’t provide VCF software. The Broadcom online depot doesn’t provide the add-on.

### VMware software updates
<a name="vmware-updates"></a>

**Note**  
In EVS-deployed environments, if you update the ESX version after the environment deploys, SDDC Manager can fail during VCF host validation in the commission hosts step. For the steps to fix this issue, see [Troubleshoot hosts](#troubleshoot-hosts).

For the VCF and ESX versions that Amazon EVS provides, see [Supported VCF versions and instance types](#versions-provided). When you add a host, you can choose any available ESX version. After you add a host, you change its ESX version only with vSphere Lifecycle Manager.

Review the VCF software version after deployment, and update it as needed. Follow a regular maintenance schedule for updates and patches. You can get VCF updates through the [Broadcom support portal](https://support.broadcom.com/web/ecx).

Some patches, updates, and upgrades can affect the workloads in your environment. Before you change your VCF software, we recommend that you take the following steps:
+ Check software compatibility in the [Broadcom Interoperability Matrix](https://interopmatrix.broadcom.com/Interoperability?col=1) and hardware compatibility with EC2 instances in the [Broadcom Compatibility Guide](https://compatibilityguide.broadcom.com/).
+ Read the VCF lifecycle documentation for your version, so that you understand how the change affects your environment. For VCF 9.x, see the [Broadcom documentation for VCF 9.0](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html). For VCF 5.2.x, see [VCF Lifecycle Management Guide](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vmware-cloud-foundation-lifecycle-management.html) and [VCF 5.2.x release notes](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vcf-release-notes.html) in the Broadcom documentation.
+ Test the change in a staging environment before you apply it to production.
+ When you update an ESX cluster image in vLCM, include the add-on from the Amazon EVS custom add-on depot. For the steps, see [Use the Amazon EVS custom add-on depot](#addon-depot).

#### Updates in EVS-deployed environments
<a name="evs-lifecycle-5x-updates"></a>

Amazon EVS deploys a recent bundled version of VCF 5.2.x that might not include individual product updates, known as async patches. After deployment, we recommend that you review and update individual products with the Broadcom Async Patch Tool (AP Tool) or the SDDC Manager lifecycle management automation. You must upgrade NSX outside of SDDC Manager.

vSAN disk prechecks can fail during a cluster upgrade, because Amazon EVS uses vSAN Express Storage Architecture (ESA) and those prechecks don’t apply to vSAN ESA. For more information, see [article 369423](https://knowledge.broadcom.com/external/article/369423/upgrade-prechecks-fails-with-esxi-cluste.html) in the Broadcom knowledge base.

### ESX host lifecycle and maintenance
<a name="esxi-host-updates"></a>

You are responsible for ESX host lifecycle management and maintenance, including monitoring host health and fixing host issues. For more information, see [Manage hosts](#host-management).

 AWS performs scheduled maintenance on the underlying EC2 bare metal instances for reliability, availability, and performance. For more information, see [Host maintenance and retirement events](#evs-host-maintenance).

**Topics**

### Use the Amazon EVS custom add-on depot
<a name="addon-depot"></a>

The Amazon Elastic VMware Service custom add-on depot provides the Amazon EVS custom vendor add-on for ESX cluster images. You get the depot URL with the `GetDepotUrl` API action, and then add the URL as a download source in vSphere Lifecycle Manager (vLCM). You use the AWS CLI or the API for this task, because the Amazon EVS console doesn’t provide the URL.

The Amazon EVS custom add-on depot doesn’t provide VMware Cloud Foundation (VCF) software. To compare the software depots, see [Software depots](#evs-lifecycle-depots).

#### Amazon EVS AMI and custom vendor add-on
<a name="evs-env-ami-maintenance"></a>

Amazon EVS deploys ESX hosts with a custom Amazon EVS Amazon Machine Image (AMI). The AMI contains the add-on with the packages that ESX needs to run on Amazon EC2.

When you add a host to your environment, the host has the latest available version of the add-on. If the other hosts in your cluster use an older add-on version, adding the new host can fail with an error that the host isn’t compatible with your cluster image. For the steps to fix this issue, see [Troubleshoot hosts](#troubleshoot-hosts).

When you update an ESX cluster image in vLCM, include the add-on from the Amazon EVS custom add-on depot. Every host then keeps the add-on.

#### How the depot URL and token work
<a name="addon-depot-how-it-works"></a>

When you create an Amazon EVS environment, Amazon EVS provisions an access token for the Amazon EVS custom add-on depot. The `GetDepotUrl` API action returns a URL that includes this token. The token stays active until you rotate it.

The URL of the Amazon EVS custom add-on depot is a public address. To sync the add-on, vCenter Server needs outbound internet access to that address.

**Important**  
Protect the URL and the access token like a credential, and don’t share them outside the people who manage your environment.

The URL of the Amazon EVS custom add-on depot includes the access token. If you think that the token was exposed, rotate it. For the steps, see [Rotate the depot access token](#addon-depot-rotate).

#### Before you begin
<a name="addon-depot-prereqs"></a>
+ An Amazon EVS environment in the `CREATED` state.
+ IAM permissions to call the `evs:GetDepotUrl` action on the environment. See [IAM permissions](#addon-depot-iam).
+ A vSphere user with the `VMware vSphere Lifecycle Manager > Configure` privilege.

#### Get the depot URL
<a name="addon-depot-get-url"></a>

1. Run the `get-depot-url` command for your environment.

   ```bash
   aws evs get-depot-url --environment-id env-abcde12345
   ```

1. Note the URL of the Amazon EVS custom add-on depot in the response.

   The following is example output.

   ```json
   {
       "depotUrl": "https://example.cloudfront.net/EXAMPLE-TOKEN/depot/vmw-depot-index.xml",
       "token": "EXAMPLE-TOKEN"
   }
   ```

#### Add the depot URL to vSphere Lifecycle Manager
<a name="addon-depot-use-url"></a>

 **In the vSphere Client** 

1. Sign in to the vSphere Client.

1. Go to vSphere Lifecycle Manager.

1. Add the URL of the Amazon EVS custom add-on depot as a new download source.

1. Sync the download source so that the add-on appears in the vLCM depot.

For detailed steps, see [Add a New Download Source](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/managing-host-and-cluster-lifecycle/working-with-vsphere-lifecycle-manager-depots/add-a-new-download-source.html) in the Broadcom documentation.

You have added the Amazon EVS custom add-on depot when the add-on is available in vLCM for your cluster images.

#### Rotate the depot access token
<a name="addon-depot-rotate"></a>

1. Run the `get-depot-url` command with the `--rotate` flag.

   ```bash
   aws evs get-depot-url --environment-id env-abcde12345 --rotate
   ```

1. In vLCM, replace the old download source with the new depot URL from the response.

**Important**  
After you rotate the token, the old URLs of the Amazon EVS custom add-on depot stop working within 5 minutes. You must add the new URL in vLCM before you sync the add-on again.

#### IAM permissions
<a name="addon-depot-iam"></a>

To get the depot URL, your IAM identity needs permission to call the `evs:GetDepotUrl` action on the environment. The following example policy allows the action on one environment. Replace `<region-code>` with the Region where you created the environment.

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "evs:GetDepotUrl",
            "Resource": "arn:aws:evs:<region-code>:111122223333:environment/env-abcde12345"
        }
    ]
}
```

The example uses the `aws` partition. In other partitions, replace `aws` in the ARN with your partition, for example `aws-us-gov`. Anyone with this permission can get the depot URL, so grant it only to the people who manage the environment. For more policy examples, see [Amazon EVS identity-based policy examples](#security-iam-id-based-policy-examples).

Next step: to plan an update of your VCF software, see [Update and upgrade](#evs-lifecycle-mgmt).

## Manage connectors and credentials
<a name="connectors-overview"></a>

An Amazon Elastic VMware Service connector lets Amazon EVS communicate with one VMware Cloud Foundation (VCF) management appliance in your environment. Amazon EVS uses connectors to stay in sync with your VCF deployment and to track Windows Server entitlements. A connector that stops working affects your environment health.

Each connector uses the fully qualified domain name (FQDN) of the appliance and credentials that you store in an AWS Secrets Manager secret. For a short definition, see [Amazon EVS connectors](#concepts-connector).

### Connector types
<a name="connectors-overview-types"></a>

The following table lists the connector types and the deployment mode that uses each one.


| Connector type | VCF 9.x (Self-deployed) | VCF 5.2.x (EVS-deployed) | What Amazon EVS uses it for | 
| --- | --- | --- | --- | 
| Operations Manager (`OPERATIONS_MANAGER`) | Required | Not available | Connects to VCF Operations to stay in sync with your VMware deployment and to monitor vSAN capacity. | 
| SDDC Manager (`SDDC_MANAGER`) | Not available | Required | Connects to SDDC Manager to validate host counts and license key coverage, and to monitor vSAN capacity. | 
| vCenter (`VCENTER`) | Optional | Optional | Connects to vCenter Server to monitor VM lifecycle events for entitled VMs. You need it for Windows Server entitlements. For more information, see [Windows Server entitlements](#entitlements). | 

The VCF Operations and SDDC Manager connectors are the management connectors. Each environment needs the management connector for its mode.

### Reachability checks and environment health
<a name="connectors-overview-health"></a>

Amazon EVS runs a reachability check against each appliance through its connector. The check runs every 10 minutes after the connector becomes `ACTIVE`, and its status is `PASSED`, `FAILED`, or `UNKNOWN`.
+ If the management connector isn’t created or loses reachability, Amazon EVS reports impaired environment health through AWS Health notifications. For the SDDC Manager connector (VCF 5.2.x), Amazon EVS also can’t validate host counts and license key coverage. For more information, see [Monitor environment status](#evs-env-status-check).
+ If the vCenter connector loses reachability, the Windows Server entitlements that depend on it enter the `AT_RISK` status. If you don’t restore reachability within 8 hours, Amazon EVS removes the entitlements. For more information, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

A reachability check can fail when the password of the connector user changes or expires in VCF but the secret still holds the old password. To avoid this, update the secret whenever you change the credentials of the connector user. For the steps, see [Rotate vCenter connector credentials](#rotate-vcenter-credentials).

### Connector and credential tasks
<a name="connectors-overview-tasks"></a>
+ To create a connector of any type, see [Create a connector](#evs-env-create-connector).
+ To change the appliance FQDN or the secret of a connector, see [Update a connector](#evs-env-update-connector).
+ To delete a connector, see [Delete a connector](#evs-env-delete-connector).
+ To give a vCenter connector new credentials after you rotate them in vCenter Server, see [Rotate vCenter connector credentials](#rotate-vcenter-credentials).
+ To learn which secrets Amazon EVS creates and who rotates them, see [Manage secrets](#evs-env-secret-rotation).

**Topics**

### Create a connector
<a name="evs-env-create-connector"></a>

Create a connector so that Amazon Elastic VMware Service can communicate with a VMware Cloud Foundation (VCF) management appliance, such as VCF Operations, SDDC Manager, or vCenter Server. You can create a connector with the Amazon EVS console or the AWS CLI. The procedure is the same for every connector type. Only the type, the appliance, and the secret keys differ.

For what each connector type does, see [Manage connectors and credentials](#connectors-overview).

**Important**  
Use credentials with the minimum permissions that the appliance type requires.

For vCenter Server and VCF Operations, create a dedicated read-only user. For SDDC Manager, scope the API key to the read-only access that Amazon EVS requires. Avoid credentials with elevated or administrative permissions.

#### Connector types and secret keys
<a name="evs-env-create-connector-types"></a>

Choose the connector type for your appliance and deployment mode. The secret that you create must contain the keys in the **Required secret keys** column.


| Connector type | VCF version | Required secret keys | Appliance | 
| --- | --- | --- | --- | 
|  `OPERATIONS_MANAGER`  | VCF 9.0 and 9.1 (Self-deployed) |  `username` and `password`  | VCF Operations. This is the required management connector for VCF 9.x. Amazon EVS also uses it to monitor vSAN capacity. | 
|  `SDDC_MANAGER`  | VCF 5.2.x (EVS-deployed) |  `apiKey`  | SDDC Manager. This is the required management connector for VCF 5.2.x. Amazon EVS also uses it to monitor vSAN capacity. | 
|  `VCENTER`  | VCF 5.2.x, 9.0, and 9.1 (both modes) |  `username` and `password`  | vCenter Server. Needed for Windows Server entitlements. | 

#### Connector rules
<a name="evs-env-create-connector-rules"></a>

The following rules apply when you create a connector.
+ Each connector maps to a single appliance FQDN. The FQDN must match the domain name that you used when you created the Amazon EVS environment, and it must be unique across all connectors in the environment.
+ You can create only one vCenter connector for each environment.
+ Connector creation is asynchronous and doesn’t validate appliance reachability or credentials. After the connector state reaches `ACTIVE`, the reachability check status changes from `UNKNOWN` to `PASSED` or `FAILED` within 10 minutes.

#### Before you begin
<a name="evs-env-create-connector-before"></a>
+ Your VCF management appliance is installed and reachable over your VCF management network.
+ In the appliance, you have created a dedicated user, or for SDDC Manager an API key, with read-only access for Amazon EVS.
+ In AWS Secrets Manager, you have created a secret that contains the keys for your connector type, with the credentials of that user. Create a separate secret for each appliance.
+ You have added the tag `EvsAccess=true` to the secret. If you encrypted the secret with your own AWS KMS key, you have also added the `EvsAccess=true` tag to the AWS KMS key. Without this tag, Amazon EVS can’t access the secret and connector creation fails.
+ For a vCenter connector, you meet the requirements in [Windows Server entitlements](#entitlements).

#### Create the connector
<a name="evs-env-create-connector-procedure"></a>

To create the connector in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment where you want to create the connector.

1. Choose the **Connectors** tab.

1. Choose **Create connector**.

1. For **Appliance FQDN**, enter the fully qualified domain name of the appliance.

1. For **Secret**, choose the secret that contains the appliance credentials.

1. Choose **Create connector**.

1. To verify, check that the connector state is **Active** and the reachability check result is **Passed**.

1. Run the `create-environment-connector` command. For `--type`, use `OPERATIONS_MANAGER`, `SDDC_MANAGER`, or `VCENTER`. For `--secret-identifier`, use the secret name or ARN.

   The following example creates a vCenter connector. The example ARN uses the `aws` partition. In other partitions, replace `aws` with your partition. Replace `<region-code>` with the Region where you created the environment.

   ```bash
   aws evs create-environment-connector \
       --environment-id env-abcde12345 \
       --type VCENTER \
       --appliance-fqdn vcenter.example.com \
       --secret-identifier arn:aws:secretsmanager:<region-code>:111122223333:secret:vcenter-creds-AbCdEf
   ```

1. To verify, run the `list-environment-connectors` command, and check that the connector state is `ACTIVE` and the reachability check result is `PASSED`.

   ```bash
   aws evs list-environment-connectors \
       --environment-id env-abcde12345
   ```

You have created the connector when its state is `ACTIVE` and its reachability check is `PASSED`. If the required management connector isn’t created or becomes unreachable, Amazon EVS reports impaired environment health through AWS Health notifications.

If connector creation fails or the reachability check stays `FAILED`, see [Troubleshoot environment status checks](#troubleshoot-status-checks).

#### Next steps for a vCenter connector
<a name="create-vcenter-connector"></a>

After the vCenter connector is `ACTIVE` and its reachability check is `PASSED`, you can create Windows Server entitlements for your VMs. For the steps, see [Create Windows Server entitlements](#create-entitlements).

### Update a connector
<a name="evs-env-update-connector"></a>

Update a connector to change the appliance fully qualified domain name (FQDN) or to point the connector to a different AWS Secrets Manager secret. For example, you update the FQDN when the appliance endpoint changes. You can update a connector of any type with the Amazon Elastic VMware Service console or the AWS CLI.

If you only changed the credentials in the existing secret, you don’t need to update the connector, because the connector already points to that secret. For credential rotation, see [Rotate vCenter connector credentials](#rotate-vcenter-credentials).

#### Update rules
<a name="evs-env-update-connector-rules"></a>

The following rules apply when you update a connector.
+ You can update only one property of a connector at a time.
+ The connector must be in the `ACTIVE` or `UPDATE_FAILED` state. If the connector is in the `CREATING` or `UPDATING` state, wait for it to reach the `ACTIVE` state before you update it.
+ A new FQDN must be valid and must match the domain name that you used when you created the Amazon EVS environment. It must also be unique across all connectors in the environment.
+ A new secret must contain the keys for the connector type and carry the `EvsAccess=true` tag. For the keys, see [Connector types and secret keys](#evs-env-create-connector-types).

#### Update the connector
<a name="evs-env-update-connector-procedure"></a>

To update the connector in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the connector.

1. Choose the **Connectors** tab.

1. Select the connector that you want to update.

1. Choose **Actions**, and then choose **Update Secret** or **Update FQDN**.

1. Do one of the following:

   1. For **Update Secret**, select the secret with the appliance credentials, and choose **Update**.

   1. For **Update FQDN**, enter the new FQDN, and choose **Update**.

1. To verify, check that the connector state has returned to **Active** from **Updating**.

1. Run the `update-environment-connector` command with either `--secret-identifier` or `--appliance-fqdn`.

   The example ARN uses the `aws` partition. In other partitions, replace `aws` with your partition. Replace `<region-code>` with the Region where you created the environment.

   To update the secret:

   ```bash
   aws evs update-environment-connector \
       --environment-id env-abcde12345 \
       --connector-id cnctr-abcde12345 \
       --secret-identifier arn:aws:secretsmanager:<region-code>:111122223333:secret:vcenter-creds-AbCdEf
   ```

   To update the FQDN:

   ```bash
   aws evs update-environment-connector \
       --environment-id env-abcde12345 \
       --connector-id cnctr-abcde12345 \
       --appliance-fqdn vcenter.example.com
   ```

1. To verify, run the `list-environment-connectors` command, and check that the connector state is `ACTIVE`.

   ```bash
   aws evs list-environment-connectors \
       --environment-id env-abcde12345
   ```

You have updated the connector when its state is `ACTIVE` again. After the update, check that the reachability check result is `PASSED`. If the update fails or the reachability check stays `FAILED`, see [Troubleshoot environment status checks](#troubleshoot-status-checks).

### Delete a connector
<a name="evs-env-delete-connector"></a>

Delete a connector when you no longer need it. You can delete a connector with the Amazon Elastic VMware Service console or the AWS CLI.

**Important**  
The management connector of your environment is the SDDC Manager connector for VMware Cloud Foundation (VCF) 5.2.x or the VCF Operations connector for VCF 9.x. If you delete the management connector, Amazon EVS reports impaired environment health through AWS Health notifications. If you delete the SDDC Manager connector (VCF 5.2.x), Amazon EVS also can’t validate host counts and license key coverage.

#### Before you begin
<a name="_before_you_begin"></a>

Before you delete a connector, check the following requirements.
+ The connector must be in the `ACTIVE`, `CREATE_FAILED`, or `UPDATE_FAILED` state.
+ You must delete all entitlements that are associated with a vCenter connector before you can delete the connector. For the steps, see [Delete Windows Server entitlements](#delete-entitlements).

#### Delete the connector
<a name="evs-env-delete-connector-procedure"></a>

To delete the connector in the Amazon EVS console or with the AWS CLI, use the following steps.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the connector.

1. Choose the **Connectors** tab.

1. Select the connector that you want to delete.

1. Choose **Delete Connector**.

1. Confirm the deletion.

1. To verify, check that the connector no longer appears in the list.

1. Run the `delete-environment-connector` command.

   ```bash
   aws evs delete-environment-connector \
       --environment-id env-abcde12345 \
       --connector-id cnctr-abcde12345
   ```

1. To verify, run the `list-environment-connectors` command, and check that the connector is no longer listed.

   ```bash
   aws evs list-environment-connectors \
       --environment-id env-abcde12345
   ```

You have deleted the connector when it no longer appears in the list.

To create a connector again, see [Create a connector](#evs-env-create-connector).

### Rotate vCenter connector credentials
<a name="rotate-vcenter-credentials"></a>

A vCenter connector authenticates with vCenter Server by using the vCenter user credentials that you store in an AWS Secrets Manager secret. When you rotate the password of that vCenter user in VMware Cloud Foundation (VCF), you must give the connector the new credentials so that it can keep authenticating.

**Important**  
If the connector can’t authenticate, its reachability check fails and your Windows Server entitlements enter the `AT_RISK` status. If you don’t restore reachability within 8 hours, Amazon Elastic VMware Service removes the entitlements. To avoid this, give the connector the new credentials as soon as you change the password in vCenter Server.

You have two options:
+  **Update the existing secret.** Put the new credentials in the secret that the connector already uses. You don’t need to update the connector.
+  **Use a different secret.** Store the new credentials in another secret, and then update the connector to point to it.

#### Update the existing secret
<a name="rotate-vcenter-credentials-same-secret"></a>

 **In the vSphere Client** 

1. Change the password of the vCenter user that the connector uses.

 **In the AWS console** 

1. In Secrets Manager, update the value of the secret that the connector uses. Keep the keys `username` and `password`, and enter the new password. For the steps, see [Modify an AWS Secrets Manager secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/manage_update-secret.html) in the * AWS Secrets Manager User Guide*.

1. Wait up to 10 minutes for the next reachability check.

1. In the Amazon EVS console, on the **Connectors** tab of your environment, check that the reachability check result is **Passed**. You can also run `aws evs list-environment-connectors --environment-id env-abcde12345`.

You have rotated the credentials when the reachability check result is `PASSED`.

#### Use a different secret
<a name="rotate-vcenter-credentials-new-secret"></a>

1. Change the password of the vCenter user in the vSphere Client.

1. Create a secret with the keys `username` and `password` that contain the new credentials. Add the tag `EvsAccess=true` to the secret, and to the AWS KMS key if you encrypted the secret with your own key.

1. Update the connector to use the new secret. For the console and AWS CLI steps, see [Update a connector](#evs-env-update-connector).

1. Check that the connector state is `ACTIVE` and the reachability check result is `PASSED`.

If the reachability check stays `FAILED`, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

To rotate the secrets that Amazon EVS creates, see [Manage secrets](#evs-env-secret-rotation).

### Manage secrets
<a name="evs-env-secret-rotation"></a>

Amazon Elastic VMware Service stores credentials for your environment as secrets in AWS Secrets Manager in your account, and connectors read the secrets that you create. You are responsible for the lifecycle of these secrets, including rotation, because Amazon EVS doesn’t provide managed rotation.

#### Secrets in an Amazon EVS environment
<a name="evs-env-secret-types"></a>

The secrets in your account depend on the deployment mode and on the connectors that you create. The following table lists the secrets for each VMware Cloud Foundation (VCF) version and who creates them.


| Secret | VCF 9.x (Self-deployed) | VCF 5.2.x (EVS-deployed) | Who creates it | 
| --- | --- | --- | --- | 
| ESX root password of each host | Yes | Yes | Amazon EVS, when it creates the host. | 
| Credentials of the VCF management appliances, such as vCenter Server, NSX, and SDDC Manager | No | Yes | Amazon EVS, when it deploys the environment and installs VCF. In Self-deployed mode, you set these passwords yourself when you install VCF. | 
| Connector credentials for VCF Operations, SDDC Manager, or vCenter Server | Yes | Yes | You, before you create a connector. For more information, see [Create a connector](#evs-env-create-connector). | 

In EVS-deployed environments, Amazon EVS encrypts the secrets that it creates with the AWS KMS key that you choose when you create the environment. You can choose the AWS managed key for Secrets Manager or a customer managed key.

When you delete the environment, Amazon EVS also deletes the secrets that it created.

To get a secret value, see [Get secrets from AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html) in the * AWS Secrets Manager User Guide*. For EVS-deployed environments, see also [Retrieve VCF credentials and access appliances](#access-vcf).

#### How Amazon EVS accesses connector secrets
<a name="evs-env-secret-access"></a>

Amazon EVS can read a connector secret only when the secret has the tag `EvsAccess=true`. If you encrypt the secret with your own AWS KMS key, the key also needs the `EvsAccess=true` tag. Amazon EVS uses its service-linked role for this access, and the role’s permissions are scoped to resources with this tag. For more information, see [AWS managed policies for Amazon EVS](#security-iam-awsmanpol).

#### Rotate secrets
<a name="evs-env-secret-rotate"></a>

We recommend that you rotate secrets regularly on a set schedule, so that no secret is long-lived. For more information, see [Rotation schedules](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotate-secrets_schedule.html) in the * AWS Secrets Manager User Guide*.

When you change a password in VCF, also update the matching secret, so that the secret holds the password that works. For connector secrets, this matters most, because a connector with old credentials fails its reachability check.
+ To give a vCenter connector new credentials, see [Rotate vCenter connector credentials](#rotate-vcenter-credentials).
+ To point any connector to a different secret, see [Update a connector](#evs-env-update-connector).

In Self-deployed environments, you are also responsible for password rotation on all VCF management appliances. For the full split of responsibilities, see [Shared responsibility and support](#plan-shared-responsibility).

## Windows Server entitlements
<a name="entitlements"></a>

A Windows Server entitlement gives a virtual machine (VM) in your Amazon Elastic VMware Service environment Windows Server licensing coverage through AWS. You pay only for what you use, per vCPU-hour.

### Choose how to license Windows Server
<a name="entitlements-license-choice"></a>

You have two ways to license Windows Server for the VMs in your environment:

 **Bring your own licenses**   
If you have eligible Windows Server licenses with portability rights, you can bring them to your Windows VMs. You don’t need Windows Server entitlements for those VMs.

 **Use Windows Server entitlements**   
For VMs without portability rights, you consume Windows Server licenses through AWS and pay per vCPU-hour.

For prices, see [Amazon EVS pricing](https://aws.amazon.com/evs/pricing).

### How Windows Server entitlements work
<a name="entitlements-concepts-components"></a>

Windows Server entitlements use the following resources.

 **vCenter connector**   
A vCenter connector is a persistent connection between Amazon EVS and vCenter Server. Amazon EVS uses the connector to identify your VMs and monitor their Windows Server entitlement usage.

 **Windows Server entitlement**   
An entitlement gives a specific VM Windows Server licensing coverage offered by AWS. After you create an entitlement, Amazon EVS starts to monitor the Windows Server entitlement usage of that VM.

 **Windows Server activation**   
You activate Windows Server on each entitled VM against an Amazon EVS activation endpoint that you create in your VPC. You perform this activation yourself.

#### When the connector loses contact with vCenter
<a name="entitlements-grace-period"></a>

Amazon EVS needs the vCenter connector to reach vCenter Server. If the connector reachability check fails, the entitlements of all Windows Server VMs under that vCenter Server change to `AT_RISK`. You have 8 hours from the first failed check to restore the connection.
+ If you restore the connection within 8 hours, Amazon EVS keeps your entitlements without further action.
+ If you don’t restore the connection within 8 hours, Amazon EVS removes the entitlements and changes their status to `ENTITLEMENT_REMOVED`. License usage tracking stops as of the time the connection was lost.

For causes and solutions, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

### Before you begin
<a name="entitlements-prerequisites"></a>

Before you set up Windows Server entitlements, make sure that you meet the following prerequisites.
+ You have an active Amazon EVS environment.
+ You have a dedicated vCenter user for the connector to use. We recommend that you give this user a ReadOnly role. Don’t use credentials with elevated or administrative permissions.
+ You have created a secret in AWS Secrets Manager that holds the credentials of the dedicated vCenter user. The secret must contain two keys, `username` and `password`.
+ You have added the tag `EvsAccess=true` to the secret. If you encrypted the secret with your own AWS KMS key, you have also added the tag `EvsAccess=true` to the AWS KMS key.
+ The VMs that you want to cover run a supported Windows Server guest operating system. Amazon EVS supports Windows Server 2016 or later.

### Setup workflow
<a name="entitlements-concepts-workflow"></a>

To use Windows Server entitlements, complete these steps in order:

1. Create a vCenter connector so that Amazon EVS can communicate with your vCenter Server appliance. For instructions, see [Create a connector](#evs-env-create-connector).

1. Create Windows Server entitlements for the VMs that you want to cover. For instructions, see [Create Windows Server entitlements](#create-entitlements).

1. Activate Windows Server on the entitled VMs. For instructions, see [Activate Windows Server on entitled VMs](#activate-windows-server).

After setup, you can also do the following:
+ Rotate the vCenter credentials that the connector uses. For instructions, see [Rotate vCenter connector credentials](#rotate-vcenter-credentials).
+ Delete entitlements when you no longer need Windows Server licensing from AWS for a VM. For instructions, see [Delete Windows Server entitlements](#delete-entitlements).
+ Resolve entitlement issues. For more information, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

**Topics**

### Create Windows Server entitlements
<a name="create-entitlements"></a>

After you create a vCenter connector, you can create Windows Server entitlements for one or more virtual machines (VMs) in your Amazon Elastic VMware Service environment. After you create an entitlement, Amazon EVS starts to monitor the Windows Server entitlement usage of that VM. You can create entitlements for up to 100 VMs in a single request. You can use the Amazon EVS console or the AWS CLI.

#### Before you begin
<a name="create-entitlements-prereqs"></a>
+ You meet the requirements in [Before you begin](#entitlements-prerequisites) for [Windows Server entitlements](#entitlements).
+ Make sure that the vCenter connector is in the `ACTIVE` state and its reachability check is `PASSED`. To create a connector, see [Create a connector](#evs-env-create-connector).
+ You have the VM ID (Managed Object ID) of each VM. To find it, sign in to vCenter and select the VM. The ID appears in the URL in the browser address bar.

#### Create entitlements
<a name="create-entitlements-procedure"></a>

To create entitlements in the Amazon EVS console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the VMs.

1. Select the **Entitlements** tab.

1. Choose **Add**.

1. For product type, keep the default, **Windows server**.

1. Add the VMs that you want to create entitlements for. You can enter the VM IDs as text or upload a CSV file. The CSV file has a single column that contains only VM IDs.

1. Choose **Add entitlement**.

1. To verify, check that the entitlement **Status** changes to **Created**.

1. Open a terminal session.

1. Create the entitlements. The following command is an example. Replace the values in angle brackets with your own values. Replace the example VM IDs with your own values.

   ```bash
   aws evs create-entitlement \
       --environment-id <environment-id> \
       --connector-id <connector-id> \
       --entitlement-type WINDOWS_SERVER \
       --vm-ids vm-001 vm-002 vm-003
   ```

1. To verify, list the entitlements of the environment and check that the entitlement status is Created. Replace the values in angle brackets with your own values.

   ```bash
   aws evs list-vm-entitlements \
       --environment-id <environment-id> \
       --connector-id <connector-id> \
       --entitlement-type WINDOWS_SERVER
   ```

You have created Windows Server entitlements for your VMs.

When you create an entitlement, Amazon EVS validates the configuration of each VM. If a VM fails validation, its entitlement status changes to `CREATE_FAILED` with error details. After you fix the cause, you must create the entitlement for that VM again. For causes and solutions, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

#### Next step
<a name="create-entitlements-next"></a>

Activate Windows Server on the entitled VMs. For instructions, see [Activate Windows Server on entitled VMs](#activate-windows-server).

### Activate Windows Server on entitled VMs
<a name="activate-windows-server"></a>

Amazon Elastic VMware Service provides Windows Server activation for VMs that have Windows Server entitlements. You create an Amazon EVS Windows Server activation VPC endpoint in the VPC of your environment, and then configure each entitled VM to activate against it.

#### Before you begin
<a name="activate-windows-server-prereqs"></a>
+ You have an active Amazon EVS environment. You can create the VPC endpoint only when the environment is active.
+ You have created Windows Server entitlements for the VMs. For instructions, see [Create Windows Server entitlements](#create-entitlements).
+ You can connect to each Windows Server VM and open PowerShell.

#### Step 1. Create the activation VPC endpoint
<a name="activate-windows-server-endpoint"></a>

1. Identify the VPC where the Amazon EVS environment is deployed.

1. In the same VPC, create a VPC endpoint with the following settings:

   1. For **Type**, choose ** AWS services**.

   1. For **Service name**, search for and select `com.amazonaws.<region-code>.evs-windows-server-activation`. Replace `<region-code>` with the Region where you created the environment.

   1. For **VPC**, choose the VPC of your Amazon EVS environment.

   1. For **Subnets**, choose the subnets from which your Windows VMs make outbound connections.

   1. For **Security groups**, choose a security group that allows inbound TCP port `1688` from the security group or CIDR of your Windows VMs.

1. Note the private DNS name of the VPC endpoint.

#### Step 2. Check that the VM has a GVLK
<a name="activate-windows-server-gvlk"></a>

The Amazon EVS activation endpoint uses Microsoft Key Management Service (KMS) activation. This KMS is a Windows licensing service and is separate from AWS Key Management Service (AWS KMS). Each VM must have a Generic Volume License Key (GVLK) installed.

1. Connect to the Windows Server VM and open **PowerShell**.

1. Run the following command.

   ```powershell
   cscript C:\Windows\System32\slmgr.vbs /dlv | findstr /C:"Product Key Channel"
   ```

1. If the output does not show `Volume:GVLK`, find the GVLK for your version and edition of Windows Server in [KMS client activation keys](https://learn.microsoft.com/en-us/windows-server/get-started/kms-client-activation-keys?tabs=windows1110ltsc%2Cwindows81%2Cserver2025%2Cversion1803#windows-server-ltsc) on the Microsoft website. Then install it. In the following command, replace `<gvlk>` with the GVLK.

   ```powershell
   cscript C:\Windows\System32\slmgr.vbs /ipk <gvlk>
   ```

#### Step 3. Activate Windows Server
<a name="activate-windows-server-activate"></a>

1. On the VM, in **PowerShell**, set the activation server to the VPC endpoint. In the following command, replace `<vpc-endpoint-private-dns-name>` with the private DNS name of the VPC endpoint.

   ```powershell
   cscript C:\Windows\System32\slmgr.vbs /skms <vpc-endpoint-private-dns-name>:1688
   ```

   The output confirms that the activation server was set.

1. Activate Windows Server.

   ```powershell
   cscript C:\Windows\System32\slmgr.vbs /ato
   ```

   The output includes `Product activated successfully.` 

1. Verify the activation.

   ```powershell
   cscript C:\Windows\System32\slmgr.vbs /dli
   ```

   The output includes these values:
   +  `Volume activation expiration: 259200 minute(s) (180 day(s))` or a value close to it
   +  `Registered KMS machine name: <vpc-endpoint-private-dns-name>:1688` 

1. Repeat these steps on each entitled VM.

You have activated Windows Server on your entitled VMs.

#### Next steps
<a name="activate-windows-server-next"></a>

The following pages cover tasks that you might do after activation.
+ To fix entitlement status issues, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).
+ To stop using Windows Server licensing from AWS for a VM, see [Delete Windows Server entitlements](#delete-entitlements).

#### If the activation command returns an error
<a name="activate-windows-server-errors"></a>

If `cscript C:\Windows\System32\slmgr.vbs /ato` returns an error, first make sure that the VM has a GVLK ([Step 2](#activate-windows-server-gvlk)). Then check that the VM can reach the VPC endpoint on port 1688.

In the following command, replace `<vpc-endpoint-private-dns-name>` with the private DNS name of the VPC endpoint.

```powershell
Test-NetConnection -ComputerName <vpc-endpoint-private-dns-name> -Port 1688
```

The output shows `TcpTestSucceeded : True` when the connection works. The following is example output.

```text
ComputerName     : <vpc-endpoint-private-dns-name>
RemoteAddress    : <vpc-endpoint-ip-address>
RemotePort       : 1688
InterfaceAlias   : Ethernet 2
SourceAddress    : 10.0.110.93
TcpTestSucceeded : True
```

If `TcpTestSucceeded` is `False`, make sure that the security group of the VPC endpoint allows inbound TCP port 1688 from the security group or CIDR of the VM. For entitlement status issues, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

### Delete Windows Server entitlements
<a name="delete-entitlements"></a>

When you no longer need Windows Server licensing from AWS for a VM in your Amazon Elastic VMware Service environment, delete its Windows Server entitlement. You can use the Amazon EVS console or the AWS CLI. You can delete up to 100 entitlements at a time.

**Important**  
When you delete an entitlement, Amazon EVS stops tracking Windows Server entitlement usage for that VM. After deletion, the VM no longer has a Windows Server entitlement through AWS. To cover the VM again, create a new entitlement.

#### Delete entitlements
<a name="delete-entitlements-procedure"></a>

To delete entitlements in the Amazon EVS console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment that contains the VMs.

1. Select the **Entitlements** tab.

1. Select the VMs whose entitlements you want to delete.

1. Choose **Delete**.

1. Confirm the deletion.

1. To verify, check that the entitlements for those VMs no longer appear in the list.

1. Open a terminal session.

1. Delete the entitlements. The following command is an example. Replace the values in angle brackets with your own values. Replace the example VM IDs with your own values.

   ```bash
   aws evs delete-entitlement \
       --environment-id <environment-id> \
       --connector-id <connector-id> \
       --entitlement-type WINDOWS_SERVER \
       --vm-ids vm-003 vm-001
   ```

1. To verify, list the entitlements and check that the deleted VMs no longer appear. Replace the values in angle brackets with your own values.

   ```bash
   aws evs list-vm-entitlements \
       --environment-id <environment-id> \
       --connector-id <connector-id> \
       --entitlement-type WINDOWS_SERVER
   ```

You have deleted the Windows Server entitlements for the selected VMs.

If you no longer need the vCenter connector, you can delete it. See [Delete a connector](#evs-env-delete-connector).

## Migrate workloads with HCX
<a name="migrate-evs-hcx"></a>

After you deploy Amazon Elastic VMware Service, you can use VMware Hybrid Cloud Extension (HCX) to migrate workloads to your environment. You can use HCX over private connectivity or over the public internet. For how HCX works, see [Getting Started with VMware HCX](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-10/getting-started-with-vmware-hcx-4-10.html) in the Broadcom documentation.

### Migration workflow
<a name="migrate-evs-hcx-workflow"></a>

To migrate workloads with HCX, complete these steps in order.

1. Choose private or internet connectivity before you create the environment. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity).

1. For internet connectivity, create the IPAM pool, Elastic IP addresses, and VPC CIDR block before you create the environment. After you create the environment, associate the Elastic IP addresses with the public HCX VLAN subnet. For instructions, see [Configure HCX public internet connectivity](#evs-env-hcx-internet-access).

1. Install and activate HCX in the environment. For the requirements, see [Plan HCX connectivity](#plan-hcx-connectivity).

1. Check the HCX VLAN subnet, network ACL, route table association, and uplink port group. For instructions, see [Prepare for HCX migration](#hcx-prepare).

1. Turn on the optional HCX features that you need and verify HCX connectivity. For instructions, see [Optimize and verify HCX migration](#hcx-optimize-verify).

### HCX connectivity options
<a name="migrate-evs-hcx-connectivity"></a>

For a comparison of private and internet connectivity and the two architectures, see [Plan HCX connectivity](#plan-hcx-connectivity).

### HCX migration setup
<a name="hcx-migration-setup"></a>

For the checks before you migrate, see [Prepare for HCX migration](#hcx-prepare). For WAN optimization, Mobility Optimized Networking, and connectivity checks, see [Optimize and verify HCX migration](#hcx-optimize-verify).

**Topics**

### Prepare for HCX migration
<a name="hcx-prepare"></a>

Before you connect VMware HCX to your Amazon Elastic VMware Service environment, check that the HCX VLAN subnet is ready and create the HCX uplink port group. You can do the AWS checks with the console or the AWS CLI. The port group step is in the vSphere Client.

#### Before you begin
<a name="hcx-prepare-prereqs"></a>
+ You have chosen private or internet connectivity and meet the HCX requirements. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity).
+ For internet connectivity, you have associated Elastic IP addresses with the public HCX VLAN subnet. For instructions, see [Configure HCX public internet connectivity](#evs-env-hcx-internet-access).

#### Step 1. Check the status of the HCX VLAN subnet
<a name="check-hcx-config"></a>

Amazon EVS creates a VLAN subnet for HCX as part of every environment deployment. Check that it is ready.

To check the HCX VLAN subnet in the Amazon EVS console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment.

1. Select the **Networks and connectivity** tab.

1. Under **VLANs**, find the HCX VLAN and check that **State** is **Created**. For internet connectivity, also check that **Public** is **true**.

1. Run the following command with the ID of your environment and the Region that contains your resources.

   ```bash
   aws evs list-environment-vlans --region <region-code> --environment-id <environment-id>
   ```

1. In the output, find the VLAN with a `functionName` of `hcx`. Check that `vlanState` is `CREATED`. For internet connectivity, also check that `isPublic` is `true`.

   The following is an example of the HCX VLAN in the output.

   ```json
   {
     "environmentVlans": [
       {
         "vlanId": 80,
         "cidr": "203.0.113.0/28",
         "availabilityZone": "us-east-2c",
         "functionName": "hcx",
         "subnetId": "subnet-01234a1b2cde1234f",
         "vlanState": "CREATED",
         "stateDetails": "VLAN successfully created",
         "eipAssociations": [
           {
             "associationId": "eipassoc-09876543210abcdef",
             "allocationId": "eipalloc-0123456789abcdef0",
             "ipAddress": "203.0.113.3"
           },
           {
             "associationId": "eipassoc-12345678901abcdef",
             "allocationId": "eipalloc-1234567890abcdef1",
             "ipAddress": "203.0.113.4"
           }
         ],
         "isPublic": true,
         "networkAclId": "acl-0123456789abcdef0"
       }
     ]
   }
   ```

#### Step 2. Check that the HCX VLAN subnet has a network ACL
<a name="check-nacl-config"></a>

Check that the HCX VLAN subnet is associated with a network ACL. To create one, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

**Important**  
Associating an Elastic IP address with a VLAN subnet gives direct internet access to all resources on that subnet. Security groups don’t apply to traffic on Amazon EVS VLAN subnets, so use a network access control list (ACL) to restrict access. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

To check the network ACL in the Amazon VPC console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc).

1. In the navigation pane, choose **Network ACLs**.

1. Select the network ACL for your VLAN subnets.

1. Select the **Subnet associations** tab.

1. Check that the HCX VLAN subnet is in the list.

1. Run the following command with the HCX VLAN subnet ID in the `Values` filter.

   ```bash
   aws ec2 describe-network-acls --filters "Name=subnet-id,Values=subnet-abcdefg9876543210"
   ```

1. Check that the output returns the correct network ACL.

#### Step 3. Check the route table association
<a name="check-hcx-vlan-rtb-association"></a>

Amazon EVS requires that every VLAN subnet is explicitly associated with a route table in your VPC. A VLAN subnet without an explicit association uses the main route table implicitly. In that case, Amazon EVS doesn’t work correctly. For internet connectivity, the public HCX VLAN subnet must be associated with a public route table that has an internet gateway as a target.

To check the route table association in the Amazon VPC console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc).

1. In the navigation pane, choose **Route tables**.

1. Choose the route table for your VLAN subnets.

1. Select the **Subnet associations** tab.

1. Under **Explicit subnet associations**, check that all VLAN subnets are in the list.

1. If a VLAN subnet is missing, choose **Edit subnet associations** and add it.

1. Open a terminal session.

1. Check which route table is explicitly associated with a VLAN subnet. Replace the example subnet ID with your own value.

   ```bash
   aws ec2 describe-route-tables \
     --filters "Name=association.subnet-id,Values=subnet-01234a1b2cde1234f"
   ```

1. If no route table is returned, associate the VLAN subnet with a route table. Replace the example route table ID and subnet ID with your own values.

   ```bash
   aws ec2 associate-route-table \
     --route-table-id rtb-0123456789abcdef0 \
     --subnet-id subnet-01234a1b2cde1234f
   ```

For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).

#### Step 4. (Internet connectivity) Check the Elastic IP address associations
<a name="check-hcx-internet-eip-assoc"></a>

For internet connectivity, each HCX network appliance needs an Elastic IP address from the IPAM pool, associated with the public HCX VLAN subnet. You need at least two, one for the HCX Manager and one for the HCX Interconnect (HCX-IX) appliance. HCX internet connectivity fails if fewer than two Elastic IP addresses from the IPAM pool are associated with the public HCX VLAN subnet.

To check the associations in the Amazon EVS console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment.

1. On the **Networks and connectivity** tab, select the HCX public VLAN.

1. On the **EIP associations** tab, check that the Elastic IP addresses are listed.

1. Run the `list-environment-vlans` command from [Step 1](#check-hcx-config).

1. In the `eipAssociations` array of the HCX VLAN, check that each Elastic IP address is listed. Each entry has these fields:  
 `associationId`   
The ID of the association.  
 `allocationId`   
The allocation ID of the Elastic IP address.  
 `ipAddress`   
The IP address assigned on the VLAN.

To add missing associations, see [Configure HCX public internet connectivity](#evs-env-hcx-internet-access).

#### Step 5. Create a distributed port group with the HCX uplink VLAN ID
<a name="hcx-create-port-group"></a>

In the vSphere Client, do the following:

1. Add a distributed port group to the vSphere Distributed Switch. Follow the steps in [Add a Distributed Port Group](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-networking-8-0/basic-networking-with-vnetwork-distributed-switches/dvport-groups/add-a-distributed-port-group.html) in the Broadcom documentation.

1. For the VLAN setting, enter the HCX VLAN ID that you found in [Step 1](#check-hcx-config).

1. In the failover settings, set `uplink1` as an active uplink and `uplink2` as a standby uplink. This gives active/standby failover.

You have checked the HCX VLAN subnet and created the HCX uplink port group.

#### Next step
<a name="hcx-prepare-next"></a>

Turn on the optional HCX features and verify HCX connectivity. For instructions, see [Optimize and verify HCX migration](#hcx-optimize-verify).

### Configure HCX public internet connectivity
<a name="evs-env-hcx-internet-access"></a>

You can migrate workloads to Amazon Elastic VMware Service with VMware HCX over the internet by associating Elastic IP addresses with the public HCX VLAN subnet. You create the IPAM pool, the Elastic IP addresses and the VPC CIDR block before you create the environment, and associate the addresses after. You can use the consoles or the AWS CLI.

**Important**  
Associating an Elastic IP address with a VLAN subnet gives direct internet access to all resources on that subnet. Security groups don’t apply to traffic on Amazon EVS VLAN subnets, so use a network access control list (ACL) to restrict access. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

#### Before you begin
<a name="evs-env-hcx-internet-access-prereqs"></a>
+ Choose internet connectivity and review the CIDR and Elastic IP address rules in [Plan HCX connectivity](#plan-hcx-connectivity).
+ Check that your IPAM quota for the netmask length of Amazon-provided contiguous public IPv4 CIDR blocks allows a /28. For more information, see [Quotas for your IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/quotas-ipam.html).

#### Rules for the public HCX VLAN subnet
<a name="evs-env-hcx-internet-access-considerations-operation"></a>

For the CIDR block and Elastic IP address rules, see [CIDR and Elastic IP address rules for internet connectivity](#plan-hcx-connectivity-rules).

#### Part 1. Before you create the environment
<a name="hcx-internet-config"></a>

Do these steps before you create the environment. You can’t change the VLAN CIDR blocks after you create it.

After you create the environment, continue with [Part 2](#getting-started-eip-assoc).

##### Step 1. Create an IPAM
<a name="evs-env-hcx-internet-access-create-ipam"></a>

Follow the steps in [Create an IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/create-ipam.html) in the *Amazon VPC IP Address Manager User Guide*.

You have created an IPAM.

**Note**  
You can use the IPAM Free Tier for these resources. You still pay for other AWS services that you use with IPAM, such as NAT gateways and public IPv4 addresses beyond the free tier. For more information, see the [Amazon VPC pricing page](https://aws.amazon.com/vpc/pricing).

Amazon EVS doesn’t support private IPv6 Global Unicast Address (GUA) CIDR blocks.

##### Step 2. Create a public IPv4 IPAM pool
<a name="evs-env-hcx-internet-access-create-pool"></a>

The AWS CLI commands in Steps 2 to 4 store values in the shell variables `SCOPE_ID`, `POOL_ID`, and `CIDR_BLOCK`. Run them in the same terminal session.

To create the pool in the IPAM console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [IPAM console](https://console.aws.amazon.com/ipam).

1. In the navigation pane, choose **Pools**.

1. Choose the public scope. For more information about scopes, see [How IPAM works](https://docs.aws.amazon.com/vpc/latest/ipam/how-it-works-ipam.html).

1. Choose **Create pool**.

1. (Optional) Add a **Name** tag and a **Description** for the pool.

1. Under **Address family**, choose **IPv4**.

1. Under **Resource planning**, keep **Plan IP space within the scope** selected.

1. Under **Locale**, choose the AWS Region of your VPC.

1. Under **Service**, choose **EC2 (EIP/VPC)**. CIDR blocks from this pool are advertised for Elastic IP addresses.

1. Under **Public IP source**, choose **Amazon-owned**.

1. Under **CIDRs to provision**, choose **Add Amazon-owned public CIDR**, and choose a netmask length. The minimum is /28.

1. Choose **Create pool**.

1. Get the public scope ID of your IPAM.

   ```bash
   SCOPE_ID=$(aws ec2 describe-ipam-scopes \
     --filters Name=ipam-scope-type,Values=public \
     --query 'IpamScopes[0].IpamScopeId' \
     --output text)
   ```

1. Create a pool in the public scope. Use the Region of your VPC for `--locale`.

   ```bash
   aws ec2 create-ipam-pool \
     --ipam-scope-id $SCOPE_ID \
     --address-family ipv4 \
     --no-auto-import \
     --locale {{region-code}} \
     --description "Public IPv4 pool for HCX" \
     --tag-specifications 'ResourceType=ipam-pool,Tags=[{Key=Name,Value=evs-hcx-public-pool}]' \
     --public-ip-source amazon \
     --aws-service ec2
   ```

1. Store the pool ID.

   ```bash
   POOL_ID=$(aws ec2 describe-ipam-pools \
     --filters Name=tag:Name,Values=evs-hcx-public-pool \
     --query 'IpamPools[0].IpamPoolId' \
     --output text)
   ```

1. Provision a CIDR block from the pool. The minimum netmask length is /28.

   ```bash
   aws ec2 provision-ipam-pool-cidr \
     --ipam-pool-id $POOL_ID \
     --netmask-length 28
   ```

You have created a public IPv4 IPAM pool.

##### Step 3. Allocate Elastic IP addresses from the IPAM pool
<a name="evs-env-hcx-internet-access-allocate-eip"></a>

**Important**  
Enter each address manually. If you let IPAM choose the address, IPAM can allocate an address that Amazon EVS reserves, and the association with the VLAN subnet then fails. Don’t allocate the first two addresses or the last address of the CIDR block.

To allocate the addresses in the Amazon VPC console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc).

1. In the navigation pane, choose **Elastic IPs**.

1. Choose **Allocate Elastic IP address**.

1. Select **Allocate using an IPv4 IPAM pool**, and select the Amazon-owned public IPv4 pool that you created.

1. Under **Allocate IPAM method**, choose **Manually input address within the IPAM pool**.

1. Enter the address to allocate, and choose **Allocate**.

1. Repeat for each address that you need. You need at least two, plus one for each HCX network appliance.

1. Allocate each address from the pool. Use the pool ID from [Step 2](#evs-env-hcx-internet-access-create-pool) (`POOL_ID`). Replace the example addresses with addresses from your CIDR block.

   ```bash
   aws ec2 allocate-address \
     --domain vpc \
     --tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=Name,Value=evs-hcx-manager-eip}]' \
     --ipam-pool-id $POOL_ID \
     --address xx.xx.xxx.3
   
   aws ec2 allocate-address \
     --domain vpc \
     --tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=Name,Value=evs-hcx-ix-eip}]' \
     --ipam-pool-id $POOL_ID \
     --address xx.xx.xxx.4
   
   aws ec2 allocate-address \
     --domain vpc \
     --tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=Name,Value=evs-hcx-ne-eip}]' \
     --ipam-pool-id $POOL_ID \
     --address xx.xx.xxx.5
   ```

You have allocated the Elastic IP addresses.

##### Step 4. Add the public CIDR block to the VPC
<a name="evs-env-hcx-internet-access-add-cidr"></a>

Amazon EVS uses this CIDR block to connect VMware HCX to your network.

**Important**  
You must enter the CIDR block manually. Amazon EVS doesn’t support an IPAM-allocated CIDR block for the VPC, and one can cause the Elastic IP address association to fail.

To add the CIDR block in the Amazon VPC console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc).

1. In the navigation pane, choose **Your VPCs**.

1. Select your VPC, and choose **Actions**, **Edit CIDRs**.

1. Choose **Add new IPv4 CIDR**.

1. Select **IPv4 CIDR manual input**.

1. Enter the CIDR block from the public IPAM pool.

1. Get the CIDR block that you provisioned in the pool. Use the pool ID from [Step 2](#evs-env-hcx-internet-access-create-pool) (`POOL_ID`).

   ```bash
   CIDR_BLOCK=$(aws ec2 get-ipam-pool-cidrs \
     --ipam-pool-id $POOL_ID \
     --query 'IpamPoolCidrs[0].Cidr' \
     --output text)
   ```

1. Add the CIDR block to your VPC. Replace `<vpc-id>` with the ID of your VPC.

   ```bash
   aws ec2 associate-vpc-cidr-block \
     --vpc-id <vpc-id> \
     --cidr-block $CIDR_BLOCK
   ```

You have added the public CIDR block to your VPC.

##### Step 5. Create the environment
<a name="evs-env-hcx-internet-access-create-env"></a>

When you create the environment, use the public CIDR block for the HCX uplink VLAN subnet. For Self-deployed environments, see [Step 1: Create the environment](#self-deployed-create-env). For EVS-deployed environments, see [Create an environment](#getting-started-create-env).

#### Part 2. After you create the environment
<a name="getting-started-eip-assoc"></a>

After you create the environment, associate the Elastic IP addresses with the public HCX VLAN subnet.

##### Associate Elastic IP addresses with the HCX VLAN subnet
<a name="evs-env-hcx-internet-access-associate-eip"></a>

Associate one Elastic IP address from the IPAM pool for each HCX appliance. HCX internet connectivity fails if fewer than two Elastic IP addresses from the IPAM pool are associated with the public HCX VLAN subnet.

To associate the addresses in the Amazon EVS console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment.

1. On the **Networks and connectivity** tab, select the HCX public VLAN.

1. Choose **Associate EIP to VLAN**.

1. Select the Elastic IP addresses to associate.

1. Choose **Associate EIPs**.

1. On the **EIP associations** tab, check that the addresses are listed.

1. Run the `associate-eip-to-vlan` command with these values:  
 `environment-id`   
The ID of your environment.  
 `vlan-name`   
Must be `hcx`.  
 `allocation-id`   
The allocation ID of the Elastic IP address.  

   ```bash
   aws evs associate-eip-to-vlan \
     --environment-id <environment-id> \
     --vlan-name "hcx" \
     --allocation-id "eipalloc-0429268f30c4a34f7"
   ```
The command returns the VLAN with the new association in the `eipAssociations` array:  

   ```json
   {
       "vlan": {
           "vlanId": 80,
           "cidr": "203.0.113.0/28",
           "functionName": "hcx",
           "vlanState": "CREATED",
           "eipAssociations": [
               {
                   "associationId": "eipassoc-09e966faad7ecc58a",
                   "allocationId": "eipalloc-0429268f30c4a34f7",
                   "ipAddress": "203.0.113.2"
               }
           ],
           "isPublic": true
       }
   }
   ```
Note the `associationId`. You need it to disassociate the address.

1. Repeat the command for each Elastic IP address.

You have associated the Elastic IP addresses with the public HCX VLAN subnet. Next, check the environment before you connect HCX. See [Prepare for HCX migration](#hcx-prepare).

#### Disassociate an Elastic IP address
<a name="evs-env-hcx-internet-access-disassociate-eip"></a>

You can disassociate Elastic IP addresses from the public HCX VLAN subnet after deployment.

**Important**  
Disassociating an Elastic IP address can cause a loss of internet connectivity for the HCX appliances that use the public HCX VLAN subnet.

To disassociate an address in the Amazon EVS console or with the AWS CLI, choose the tab for the method that you use.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**.

1. Select the environment.

1. On the **Networks and connectivity** tab, select the HCX public VLAN.

1. Choose **Disassociate EIP from VLAN**.

1. Select the Elastic IP addresses to disassociate.

1. Choose **Disassociate EIPs**.

1. On the **EIP associations** tab, check that the addresses are no longer listed.

1. Run the `disassociate-eip-from-vlan` command with these values:  
 `environment-id`   
The ID of your environment.  
 `vlan-name`   
Must be `hcx`.  
 `association-id`   
The ID of the association to remove.  

   ```bash
   aws evs disassociate-eip-from-vlan \
     --environment-id <environment-id> \
     --vlan-name "hcx" \
     --association-id "eipassoc-09e966faad7ecc58a"
   ```

1. In the output, check that the association is no longer in the `eipAssociations` array.

You have disassociated the Elastic IP addresses from the public HCX VLAN subnet.

To release the Elastic IP addresses and delete the IPAM resources when you no longer need them, see [Clean up network resources](#cleanup).

### Optimize and verify HCX migration
<a name="hcx-optimize-verify"></a>

After you prepare your Amazon Elastic VMware Service environment, you can turn on optional VMware Hybrid Cloud Extension (HCX) features that improve migration traffic, and then verify HCX connectivity. You do these steps in HCX and NSX, using the Broadcom documentation for each feature.

#### Before you begin
<a name="hcx-optimize-verify-prereqs"></a>
+ You have completed the checks and created the HCX uplink port group. For instructions, see [Prepare for HCX migration](#hcx-prepare).

#### (Optional) Set up HCX WAN Optimization
<a name="hcx-wan-optimization"></a>

If your HCX version includes the HCX WAN Optimization service (HCX-WO), you can use it to improve migration performance over private lines or the internet. HCX-WO applies WAN optimization techniques such as data reduction and WAN path conditioning. It is deployed with the HCX Interconnect (HCX-IX) appliance, which replicates data between your on-premises environment and the Amazon EVS environment.

HCX-WO helps most on deployments that can’t dedicate 10 Gbit paths to migrations. On 10 Gbit, low-latency deployments, it might not improve migration performance.

To use HCX-WO with Amazon EVS, use a distributed port group on the HCX VLAN subnet. Use the port group that you created in [Step 5. Create a distributed port group with the HCX uplink VLAN ID](#hcx-create-port-group).

#### (Optional) Turn on HCX Mobility Optimized Networking
<a name="hcx-mon"></a>

HCX Mobility Optimized Networking (MON) is a feature of the HCX Network Extension service. MON-enabled network extensions route traffic for migrated VMs within your Amazon EVS environment, instead of on a long round-trip path through the source gateway. This applies when you stretch Layer 2 networks. MON is available for all Amazon EVS deployments.

Before you turn on HCX MON, do the following:
+ Read the restrictions in [Restrictions and Limitations for Network Extension](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-10/vmware-hcx-user-guide-4-10/extending-networks-with-vmware-hcx/about-vmware-hcx-network-extension/restrictions-and-limitations-for-network-extension.html) and [Restrictions and Limitations for Mobility Optimized Networking Topologies](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-10/vmware-hcx-user-guide-4-10/extending-networks-with-vmware-hcx/hcx-network-extension-with-mobility-optimized-networking/limitations-for-mobility-optimized-networking.html) in the Broadcom documentation.
+ In NSX, configure route redistribution for the destination network CIDR. For more information, see [Configure BGP and Route Redistribution](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/nsxt-dc/3-2/administration-guide/nsx-cloud-overview/nsx-t-supported-features-for-nsx-cloud/service-insertion-for-your-public-cloud/task-set-up-bgp-on-the-vpn-tunnel.html) in the Broadcom documentation.

In HCX, turn on MON for the network extension by following [Configuring Mobility Optimized Networking](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-10/vmware-hcx-user-guide-4-10/extending-networks-with-vmware-hcx/hcx-network-extension-with-mobility-optimized-networking/configuring-hcx-mobility-optimized-networking.html) in the Broadcom documentation.

#### Verify HCX connectivity
<a name="hcx-step-10"></a>

VMware HCX includes diagnostic tools that test connectivity. To run them, follow [VMware HCX Troubleshooting](https://techdocs.broadcom.com/us/en/vmware-cis/hcx/vmware-hcx/4-10/vmware-hcx-user-guide-4-10/vmware-hcx-troubleshooting.html) in the Broadcom documentation.

If the checks fail, review the HCX VLAN subnet, network ACL, and route table checks in [Prepare for HCX migration](#hcx-prepare). For internet connectivity, also check the Elastic IP address associations in [Configure HCX public internet connectivity](#evs-env-hcx-internet-access). For other network issues, see [Troubleshoot networking](#troubleshoot-networking).

You have verified HCX connectivity, and you can start migrating workloads.

## Use VCF features on Amazon EVS
<a name="operate-vcf-features"></a>

For VMware Cloud Foundation (VCF) features, follow the Broadcom documentation. For some features, Amazon Elastic VMware Service adds steps or conditions of its own.

### Check whether a feature is supported
<a name="operate-vcf-features-support"></a>

Before you use a VCF feature, check its row in [VCF feature support on Amazon EVS](#vcf-feature-support). Each row gives the support status, the VCF versions, the instance types or storage model, any Amazon EVS conditions, and a link to the Broadcom documentation. Support can differ by VCF version, so check the row for the version that your environment runs.

### When a feature has its own page
<a name="operate-vcf-features-when"></a>

A feature gets a page in this section when one or more of the following is true:
+ Amazon EVS adds three or more steps of its own that the Broadcom documentation doesn’t give, such as console, API, CLI, IAM, network, or DNS actions.
+ Amazon EVS adds an API, CLI command, or AWS CloudFormation resource for the feature.
+ The Amazon EVS conditions for the feature need more than a short note, or a decision table.
+ A problem that happens only on Amazon EVS needs a troubleshooting entry.

Otherwise, the feature has only its row on the feature support page, and you follow the Broadcom documentation. When a feature has a page, its row links to the page. The page doesn’t repeat the support status, so the feature support page is the one place to check it.

Features that you decide on before deployment are in [Plan](#plan-overview). Settings that you choose when you deploy are in [Deploy](#deploy-overview). The following pages cover features that you use after deployment.

### Feature pages
<a name="operate-vcf-features-pages"></a>

The following feature has its own page.
+  [Set up NSX Federation](#nsx-federation) 

**Topics**

### Set up NSX Federation
<a name="nsx-federation"></a>

With VMware NSX Federation, you can manage networking and security for several NSX sites from one Global Manager. You can use NSX Federation with Amazon Elastic VMware Service for disaster recovery and multi-site networking across AWS Regions, Availability Zones, and on-premises sites.

For the supported VMware Cloud Foundation (VCF) versions and conditions, see [VCF feature support on Amazon EVS](#vcf-feature-support). For an example architecture, see [Amazon EVS with NSX Federation for disaster recovery and multi-site networking](https://aws.amazon.com/blogs/migration-and-modernization/amazon-evs-with-nsx-federation-for-disaster-recovery-and-multi-site-networking/) on the AWS blog.

#### How NSX Federation fits an Amazon EVS environment
<a name="nsx-federation-design"></a>

Keep the following in mind when you design for NSX Federation.
+ An Amazon EVS environment runs in one Availability Zone. Its VLAN subnets sit in that Availability Zone. You can’t stretch the underlay network across environments. Federation connects separate environments or sites at the NSX layer. For more information, see [VPC and subnets](#plan-network-supported-vpc).
+ Amazon EVS provides expansion VLAN subnets for VCF functions such as NSX Federation. You choose their CIDR blocks when you create the environment, and you can’t add or resize VLAN subnets later. For sizing, see [Plan your VPC and VLAN subnets](#plan-network-sizing).
+ For stateful failover, NSX Federation uses Active/Standby Tier-0 gateways.
+ To add a site, you register it with the Global Manager.

#### Before you begin
<a name="nsx-federation-prereqs"></a>
+ You have an Amazon EVS environment for each Amazon EVS site, with expansion VLAN subnets sized for Federation.
+ Your sites can reach each other. For the connectivity options of an Amazon EVS VPC, see [Supported network configurations](#plan-network-supported).
+ You have read the NSX Federation requirements in the Broadcom documentation for your NSX version. For NSX 4.2, start with the [VMware NSX Installation Guide](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide.html).

#### Set up NSX Federation
<a name="nsx-federation-procedure"></a>

To set up Federation in NSX, follow the NSX Federation steps in the Broadcom documentation for the NSX version in your environment.

# Security in Amazon Elastic VMware Service
<a name="security"></a>

Cloud security at AWS is the highest priority. As an AWS customer, you benefit from a data center and network architecture that is built to meet the requirements of the most security-sensitive organizations.

Security is a shared responsibility between AWS and you. The [shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/) describes this as security of the cloud and security in the cloud:
+  **Security of the cloud**: AWS is responsible for protecting the infrastructure that runs AWS services in the AWS Cloud. AWS also provides you with services that you can use securely. Third-party auditors regularly test and verify the effectiveness of our security as part of the [AWS Compliance Programs](https://aws.amazon.com/compliance/programs/). To learn about the compliance programs that apply to Amazon Elastic VMware Service (Amazon EVS), see [AWS services in Scope by Compliance Program](https://aws.amazon.com/compliance/services-in-scope/).
+  **Security in the cloud**: Your responsibility is determined by the AWS service that you use. You are also responsible for other factors including the sensitivity of your data, your company’s requirements, and applicable laws and regulations.

With Amazon EVS, you are also responsible for the VMware Cloud Foundation (VCF) software that runs in your environment, including its configuration and credentials. What Amazon EVS does for you differs by deployment mode. For the split of responsibilities between AWS, Broadcom, and you in each mode, see [Shared responsibility and support](#plan-shared-responsibility).

The procedures in [Deploy in Self-deployed mode](#getting-started-self-deployed) deploy VCF 9.x. Pages in [Deploy in EVS-deployed mode](#evs-deployed-overview) cover VCF 5.2.x. A page with no line and outside those sections covers both.

Use the following topics to configure Amazon EVS to meet your security and compliance objectives, and to monitor and secure your Amazon EVS resources with other AWS services.

**Topics**
+ [Data protection for Amazon EVS](#data-protection)
+ [Identity and access management for Amazon Elastic VMware Service](#security-iam)
+ [Resilience for Amazon EVS](#disaster-recovery-resiliency)
+ [Compliance validation for Amazon EVS](#compliance-validation)
+ [Infrastructure security for Amazon EVS](#infrastructure-security)

## Data protection for Amazon EVS
<a name="data-protection"></a>

The [AWS shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/) applies to data protection in Amazon Elastic VMware Service. As described in this model, AWS is responsible for protecting the global infrastructure that runs all of the AWS Cloud. You are responsible for maintaining control over your content that is hosted on this infrastructure, including the VMware Cloud Foundation (VCF) components. You are also responsible for the security configuration and management tasks for the AWS services that you use. For more information about data privacy, see the [Data Privacy FAQ](https://aws.amazon.com/compliance/data-privacy-faq). For information about data protection in Europe, see the [AWS Shared Responsibility Model and GDPR blog post](https://aws.amazon.com/blogs/security/the-aws-shared-responsibility-model-and-gdpr) on the * AWS Security Blog*.

For data protection purposes, we recommend that you protect AWS account credentials and set up individual users with AWS IAM Identity Center or AWS Identity and Access Management. That way, each user is given only the permissions necessary to fulfill their job duties. We also recommend that you secure your data in the following ways:
+ Use multi-factor authentication (MFA) with each account.
+ Use SSL/TLS to communicate with AWS resources. We require TLS 1.2 and recommend TLS 1.3.
+ Set up API and user activity logging with AWS CloudTrail. For information about using CloudTrail trails to capture AWS activities, see [Working with CloudTrail trails](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-trails.html) in the * AWS CloudTrail User Guide*.
+ Use AWS encryption solutions, along with all default security controls within AWS services.
+ Use advanced managed security services such as Amazon Macie, which assists in discovering and securing sensitive data that is stored in Amazon S3.
+ If you require FIPS 140-3 validated cryptographic modules when accessing AWS through the AWS CLI or an API, use a FIPS endpoint. For more information about the available FIPS endpoints, see [Federal Information Processing Standard (FIPS) 140-3](https://aws.amazon.com/compliance/fips/).

Amazon EVS doesn’t log user activity for non-AWS components, such as activity inside your VCF environment. VCF logs this activity in VMware consoles such as the vSphere Client and NSX Manager. For centralized VCF logging in VCF 5.2.x, you can configure a VCF monitoring solution, such as VMware Aria Operations or VMware Tanzu Observability. For more information, see [VMware Cloud Foundation with VMware Tanzu](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/working-with-workload-management-admin.html) and [VMware Aria Suite Lifecycle in VMware Cloud Foundation mode](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/vrealize-suite-lifecycle-manager-in-cloud-foundation-admin.html) in the Broadcom documentation.

Don’t put confidential or sensitive information, such as your customers' email addresses, into tags or free-form text fields such as a **Name** field. This includes when you work with Amazon EVS or other AWS services using the console, API, AWS CLI, or AWS SDKs. Any data that you enter into tags or free-form text fields used for names might be used for billing or diagnostic logs. Don’t include credentials in a URL that you provide to an external server.

### Encryption at rest
<a name="encryption-at-rest"></a>

Amazon EVS hosts are Amazon EC2 bare metal instances. How your data is encrypted at rest depends on where it is stored: the Amazon EBS boot volume, the NVMe instance store volumes, or external storage.

#### Amazon EBS boot volume
<a name="encryption-at-rest-boot-volume"></a>

Amazon EVS instances use an Amazon EBS boot volume. The boot volume contains the operating system and other necessary files for the Amazon EC2 instance to boot and run. The boot volume is not encrypted, and Amazon EVS does not support boot volume encryption. The boot volume does not contain user data from your virtual machines.

#### Hosts with local NVMe storage
<a name="encryption-at-rest-instance-store-volume"></a>

Amazon EVS instance types with local storage come with NVMe SSD instance store volumes, which are part of the instance’s hardware. On these hosts, Amazon EVS uses the NVMe instance store volumes as the disks for vSAN datastores. The vSAN datastore holds your management and workload virtual machines after you deploy your Amazon EVS environment.

The data on NVMe instance store volumes is encrypted using an XTS-AES-256 cipher, implemented on a hardware module on the instance. The keys that encrypt data written to locally attached NVMe storage devices are unique to each customer and each volume. For more information, see [Encryption at rest](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/data-protection.html#encryption-rest) in the * Amazon EC2 User Guide*.

After you deploy VCF, you can enable vSAN data-at-rest encryption for all data stored in the vSAN datastore, for individual virtual machines (VMs), or for individual files within VMs. This granular control is useful when some VMs require encryption and others don’t, or when specific disks or files within a VM need to be encrypted. For more information, see [How vSAN Data-At-Rest Encryption Works](https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration/using-encryption-in-a-vsan-cluster-1/vsan-data-at-rest-encryption/how-vsan-data-at-rest-encryption-works.html) in the Broadcom documentation.

#### Diskless hosts
<a name="encryption-at-rest-diskless"></a>

This section applies to Self-deployed environments that run VCF 9.x.

Diskless hosts have no instance store volumes. For the instance types that are diskless, see [Supported VCF versions by instance type](#current-vcf-versions). Like other Amazon EVS hosts, they boot from an Amazon EBS boot volume. This volume isn’t encrypted and doesn’t contain user data from your virtual machines.

Your virtual machines are stored on external storage, and the encryption at rest of that data depends on the storage service that you use. For more information, see [External storage options for diskless hosts](#external-storage-diskless) and the documentation of your storage service.

### Encryption in transit
<a name="encryption-in-transit"></a>

Amazon EVS does not encrypt your in-transit traffic by default. To encrypt the data in transit that traverses Amazon EVS, you can use application layer encryption with a protocol like Transport Layer Security (TLS). To learn about Amazon EC2 instance traffic encryption, see [Encryption in transit](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/data-protection.html#encryption-transit) in the * Amazon EC2 User Guide*.

**Note**  
Nitro network encryption does not apply to the Amazon EC2 instances that Amazon EVS deploys. Amazon EVS does not support in-transit encryption of inter-host traffic.

#### In-transit encryption options for on-premises connectivity
<a name="encryption-in-transit-on-prem"></a>

To encrypt traffic between your on-premises data center and Amazon EVS, you can combine AWS Direct Connect and AWS Site-to-Site VPN with AWS Transit Gateway. This combination provides an IPsec-encrypted private connection. It also reduces network costs, increases bandwidth throughput, and provides a more consistent network experience than internet-based VPN connections. For more information, see [Private IP AWS Site-to-Site VPN with AWS Direct Connect](https://docs.aws.amazon.com/vpn/latest/s2svpn/private-ip-dx.html) in the * AWS Site-to-Site VPN User Guide*.

**Note**  
Not every connection type to the underlay VPC is supported. Amazon EVS does support IPsec VPN termination on the NSX Edge Tier-0 or Tier-1 gateway. For more information, see [Add an NSX IPSec VPN Service](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-1/administration-guide/virtual-private-network-vpn/adding-nsx-vpn-services/add-an-ipsec-vpn-service.html) in the Broadcom documentation. For the network connections that Amazon EVS supports, see [Supported network configurations](#plan-network-supported).

MAC Security (MACsec) is an IEEE standard that provides data confidentiality, data integrity, and data origin authenticity. You can use AWS Direct Connect connections that support MACsec to encrypt your data from your corporate data center to the AWS Direct Connect location. For more information, see [MAC Security in AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/MACsec.html) in the * AWS Direct Connect User Guide*.

#### Encryption in transit for VMware network data
<a name="vcf-encryption-in-transit"></a>

After you deploy VCF, you have multiple options to enforce data in transit encryption at the VCF layer:
+ With the VMware vDefend Distributed Firewall, you can implement fine-grained network segmentation and enforce TLS/SSL encryption between virtual machines. For more information, see [Configure Security Settings for Distributed Firewall by Using the User Interface](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/security-and-compliance-configuration-for-vmware-cloud-foundation-5-2/securing-nsx-t-data-center/security-configurations-for-further-evaluation/configure-security-settings.html) in the Broadcom documentation.
+ vSAN data-in-transit encryption encrypts all data and metadata between hosts in your vSAN cluster. For more information, see [vSAN Data-In-Transit Encryption](https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration/using-encryption-in-a-vsan-cluster-1/vsan-data-in-transit-encryption.html) in the Broadcom documentation.
+ Encrypted vSphere vMotion secures the confidentiality, integrity, and authenticity of data that is transferred with vSphere vMotion. For more information, see [What is Encrypted vSphere vMotion](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security-8-0/virtual-machine-encryption/encrypted-vsphere-vmotion.html) in the Broadcom documentation.

### Key and secret management
<a name="key-management"></a>

During environment deployment, Amazon EVS uses AWS Secrets Manager to create, encrypt, and store secrets. The secrets contain the VCF credentials needed to install and access the VCF management appliances, and the ESX root password. Amazon EVS also deletes the secrets that it manages on your behalf when you delete the environment. For more information, see [What’s in a Secrets Manager secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/whats-in-a-secret.html) in the * AWS Secrets Manager User Guide*.

Secrets Manager uses envelope encryption with AWS KMS keys and data keys to protect each secret value. Secrets Manager uses its default AWS managed key unless you specify a customer managed key when you create the environment. For more information, see [Secret encryption and decryption in AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/security-encryption.html) in the * AWS Secrets Manager User Guide*.

**Note**  
There are additional usage charges for customer managed keys. The default AWS managed key is provided at no cost. For more information, see [Pricing](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html#asm_pricing) in the * AWS Secrets Manager User Guide*.

Amazon EVS does not rotate secrets on your behalf and does not provide managed rotation of secrets. You are responsible for rotating the secrets associated with your environment.

We strongly recommend that you rotate your secrets as soon as the environment is created, and that you rotate them on a regular schedule after that. For more information about rotating Secrets Manager secrets, see [Rotation by Lambda function](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotate-secrets_lambda.html) in the * AWS Secrets Manager User Guide*. For more information about VCF password management, see [Password Management](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/manage-passwords-admin.html) in the Broadcom documentation. For the secrets that Amazon EVS uses with connectors, see [Manage secrets](#evs-env-secret-rotation).

**Important**  
Amazon EVS doesn’t synchronize credentials between Secrets Manager and your VCF software after deployment. If you use Secrets Manager after deployment, you must keep the credentials in Secrets Manager and SDDC Manager in sync. If you don’t, VCF passwords can expire and you can lose access to VCF software.

**Note**  
There are costs to using a Lambda function for Secrets Manager secret rotation. For more information, see [Pricing](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html#asm_pricing) in the * AWS Secrets Manager User Guide*.

### Internetwork traffic privacy
<a name="internetwork-traffic-privacy"></a>

Amazon EVS uses an Amazon VPC that you provide to create boundaries between resources in the Amazon EVS environment. The VPC also controls traffic between those resources, your on-premises network, and the internet. For more information about Amazon VPC security, see [Ensure internetwork traffic privacy in Amazon VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Security.html) in the * Amazon VPC User Guide*.

For how Amazon EVS isolates VLAN subnets and how to control their traffic, see [Infrastructure security for Amazon EVS](#infrastructure-security).

If you are an NSX administrator, you can configure the following NSX features to secure network traffic:
+ VMware vDefend Gateway Firewall secures the network perimeter and protects against external threats (north-south traffic). For more information, see [Add a Gateway Firewall Policy and Rule](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/nsxt-dc/3-2/administration-guide/security/gateway-firewall/add-a-gateway-firewall-policy-and-rule.html) in the Broadcom documentation.
+ vDefend Distributed Firewall protects against attacks that originate from within an internal network (east-west traffic). For more information, see [Add a Distributed Firewall](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/nsxt-dc/3-2/administration-guide/security/distributed-firewall/add-a-distributed-firewall.html) in the Broadcom documentation.

For how Amazon EVS isolates its own service traffic, see [Infrastructure security for Amazon EVS](#infrastructure-security).

## Identity and access management for Amazon Elastic VMware Service
<a name="security-iam"></a>

 AWS Identity and Access Management (IAM) is an AWS service that helps an administrator securely control access to AWS resources. IAM administrators control who can be *authenticated* (signed in) and *authorized* (have permissions) to use Amazon Elastic VMware Service (Amazon EVS) resources. IAM is an AWS service that you can use with no additional charge.

For the IAM role that your administrators use to deploy Amazon EVS, see [Create an IAM role for Amazon EVS administrators](#setup-iam-role). For account-level guardrails such as service control policies, see [Account permissions and guardrails](#plan-account-guardrails).

### Audience
<a name="security-iam-audience"></a>

How you use AWS Identity and Access Management (IAM) differs, depending on the work you do in Amazon EVS.

 **Service user**: If you use the Amazon EVS service to do your job, then your administrator provides you with the credentials and permissions that you need. As you use more Amazon EVS features to do your work, you might need additional permissions. Understanding how access is managed can help you request the right permissions from your administrator. If you can’t access a feature in Amazon EVS, see [Troubleshoot Amazon EVS identity and access](#security-iam-troubleshoot).

 **Service administrator**: If you’re in charge of Amazon EVS resources at your company, you probably have full access to Amazon EVS. It’s your job to determine which Amazon EVS features and resources your service users can access. You must then submit requests to your IAM administrator to change the permissions of your service users. To learn more about how your company can use IAM with Amazon EVS, see [How Amazon EVS works with IAM](#security_iam_service-with-iam).

 ** IAM administrator**: If you’re an IAM administrator, you might want to learn details about how you can write policies to manage access to Amazon EVS. To view example Amazon EVS identity-based policies that you can use in IAM, see [Amazon EVS identity-based policy examples](#security-iam-id-based-policy-examples).

### Authenticating with identities
<a name="security-iam-authentication"></a>

Authentication is how you sign in to AWS using your identity credentials. You must be *authenticated* (signed in to AWS) as the AWS account root user, as an IAM user, or by assuming an IAM role.

You can sign in to AWS as a federated identity by using credentials provided through an identity source. AWS IAM Identity Center (IAM Identity Center) users, your company’s single sign-on authentication, and your Google or Facebook credentials are examples of federated identities. When you sign in as a federated identity, your administrator previously set up identity federation using IAM roles. When you access AWS by using federation, you are indirectly assuming a role.

Depending on the type of user you are, you can sign in to the AWS Management Console or the AWS access portal. For more information about signing in to AWS, see [How to sign in to your AWS account](https://docs.aws.amazon.com/signin/latest/userguide/how-to-sign-in.html) in the * AWS Sign-In User Guide*.

If you access AWS programmatically, AWS provides a software development kit (SDK) and a command line interface (CLI) to cryptographically sign your requests using your credentials. If you don’t use AWS tools, you must sign requests yourself. For more information about using the recommended method to sign requests yourself, see [Signature Version 4 signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) in the * AWS General Reference*.

Regardless of the authentication method that you use, you might also be required to provide additional security information. For example, AWS recommends that you use multi-factor authentication (MFA) to increase the security of your account. To learn more, see [Multi-factor authentication](https://docs.aws.amazon.com/singlesignon/latest/userguide/enable-mfa.html) in the * AWS IAM Identity Center User Guide* and [Using multi-factor authentication (MFA) in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.html) in the *IAM User Guide*.

#### AWS account root user
<a name="security-iam-authentication-rootuser"></a>

When you first create an AWS account, you begin with a single sign-in identity that has complete access to all AWS services and resources in the account. This identity is called the AWS account root user and is accessed by signing in with the email address and password that you used to create the account. We strongly recommend that you don’t use the root user for your everyday tasks. Safeguard your root user credentials and use them to perform the tasks that only the root user can perform. For the complete list of tasks that require you to sign in as the root user, see [Tasks that require root user credentials](https://docs.aws.amazon.com/accounts/latest/reference/root-user-tasks.html) in the *Account Management Reference Guide*.

#### Federated identity
<a name="security-iam-authentication-federateduser"></a>

As a best practice, require human users, including users that require administrator access, to use federation with an identity provider to access AWS services by using temporary credentials.

A federated identity is a user from your enterprise user directory, a web identity provider, the AWS Directory Service, the Identity Center directory, or any user that accesses AWS services by using credentials provided through an identity source. When federated identities access AWS accounts, they assume roles, and the roles provide temporary credentials.

For centralized access management, we recommend that you use AWS IAM Identity Center. You can create users and groups in IAM Identity Center, or you can connect and synchronize to a set of users and groups in your own identity source for use across all your AWS accounts and applications. For information about IAM Identity Center, see [What is IAM Identity Center?](https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html) in the * AWS IAM Identity Center User Guide*.

#### IAM users and groups
<a name="security-iam-authentication-iamuser"></a>

An * [IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) * is an identity within your AWS account that has specific permissions for a single person or application. Where possible, we recommend relying on temporary credentials instead of creating IAM users who have long-term credentials such as passwords and access keys. However, if you have specific use cases that require long-term credentials with IAM users, we recommend that you rotate access keys. For more information, see [Rotate access keys regularly for use cases that require long-term credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#rotate-credentials) in the *IAM User Guide*.

An [IAM group](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html) is an identity that specifies a collection of IAM users. You can’t sign in as a group. You can use groups to specify permissions for multiple users at a time. Groups make permissions easier to manage for large sets of users. For example, you could have a group named *IAMAdmins* and give that group permissions to administer IAM resources.

Users are different from roles. A user is uniquely associated with one person or application, but a role is intended to be assumable by anyone who needs it. Users have permanent long-term credentials, but roles provide temporary credentials. To learn more, see [When to create an IAM user (instead of a role)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html#id_which-to-choose) in the *IAM User Guide*.

#### IAM roles
<a name="security-iam-authentication-iamrole"></a>

An * [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) * is an identity within your AWS account that has specific permissions. It is similar to an IAM user, but is not associated with a specific person. You can temporarily assume an IAM role in the AWS Management Console by [switching roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-console.html). You can assume a role by calling an AWS CLI or AWS API operation or by using a custom URL. For more information about methods for using roles, see [Using IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) in the *IAM User Guide*.

 IAM roles with temporary credentials are useful in the following situations:
+  **Federated user access**: To assign permissions to a federated identity, you create a role and define permissions for the role. When a federated identity authenticates, the identity is associated with the role and is granted the permissions that are defined by the role. For information about roles for federation, see [Creating a role for a third-party Identity Provider](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp.html) in the *IAM User Guide*. If you use IAM Identity Center, you configure a permission set. To control what your identities can access after they authenticate, IAM Identity Center correlates the permission set to a role in IAM. For information about permission sets, see [Permission sets](https://docs.aws.amazon.com/singlesignon/latest/userguide/permissionsetsconcept.html) in the * AWS IAM Identity Center User Guide*.
+  **Temporary IAM user permissions**: An IAM user can assume an IAM role to temporarily take on different permissions for a specific task.
+  **Cross-account access**: You can use an IAM role to allow someone (a trusted principal) in a different account to access resources in your account. Roles are the primary way to grant cross-account access. However, with some AWS services, you can attach a policy directly to a resource (instead of using a role as a proxy). To learn the difference between roles and resource-based policies for cross-account access, see [How IAM roles differ from resource-based policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_compare-resource-policies.html) in the *IAM User Guide*.
+  **Cross-service access**: Some AWS services use features in other AWS services. A service can do this with the permissions of the calling principal, with a service role that it assumes on your behalf, or with a service-linked role that the service owns. When an action in one service triggers an action in another, you must have permissions for both actions. For more information, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*. For the service-linked role that Amazon EVS uses, see [Using service-linked roles for Amazon EVS](#using-service-linked-roles).
+  **Applications running on Amazon EC2 **: You can use an IAM role to manage temporary credentials for applications that run on an Amazon EC2 instance, instead of storing access keys on the instance. You attach an instance profile that contains the role to the instance. For more information, see [Using an IAM role to grant permissions to applications running on Amazon EC2 instances](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html) in the *IAM User Guide*.

To learn whether to use IAM roles, see [When to create an IAM role (instead of a user)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html#id_which-to-choose_role) in the *IAM User Guide*.

### Managing access using policies
<a name="security-iam-access-manage"></a>

You control access in AWS by creating policies and attaching them to AWS identities or resources. A policy is an object in AWS that, when associated with an identity or resource, defines their permissions. AWS evaluates these policies when a principal (user, root user, or role session) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents. For more information about the structure and contents of JSON policy documents, see [Overview of JSON policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#access_policies-json) in the *IAM User Guide*.

Administrators can use AWS JSON policies to specify who has access to what. That is, which **principal** can perform **actions** on what **resources**, and under what **conditions**.

Every IAM entity (user or role) starts with no permissions. By default, users can do nothing, not even change their own password. To give a user permission to do something, an administrator must attach a permissions policy to a user. Or the administrator can add the user to a group that has the intended permissions. When an administrator gives permissions to a group, all users in that group are granted those permissions.

 IAM policies define permissions for an action regardless of the method that you use to perform the operation. For example, suppose that you have a policy that allows the `iam:GetRole` action. A user with that policy can get role information from the AWS Management Console, the AWS CLI, or the AWS API.

#### Identity-based policies
<a name="security-iam-access-manage-id-based-policies"></a>

Identity-based policies are JSON permissions policy documents that you can attach to an identity, such as an IAM user, role, or group. These policies control what actions users and roles can perform, on which resources, and under what conditions. To learn how to create an identity-based policy, see [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) in the *IAM User Guide*.

Identity-based policies can be further categorized as *inline policies* or *managed policies*. Inline policies are embedded directly into a single user, group, or role. Managed policies are standalone policies that you can attach to multiple users, groups, and roles in your AWS account. Managed policies include AWS managed policies and customer managed policies. To learn how to choose between a managed policy or an inline policy, see [Choosing between managed policies and inline policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#choosing-managed-or-inline) in the *IAM User Guide*.

#### Resource-based policies
<a name="security-iam-access-manage-resource-based-policies"></a>

Resource-based policies are JSON policy documents that you attach to a resource such as an Amazon S3 bucket. Service administrators can use these policies to define what actions a specified principal (account member, user, or role) can perform on that resource and under what conditions. Resource-based policies are inline policies. There are no managed resource-based policies.

#### Access control lists (ACLs)
<a name="security-iam-access-manage-acl"></a>

Access control lists (ACLs) control which principals can access a resource, without the JSON policy document format. To learn more about ACLs, see [Access Control List (ACL) overview](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html) in the *Amazon Simple Storage Service Developer Guide*.

#### Other policy types
<a name="security-iam-access-manage-other-policies"></a>

 AWS supports additional, less-common policy types. These policy types can set the maximum permissions granted to you by the more common policy types.
+  **Permissions boundaries**: A permissions boundary is an advanced feature in which you set the maximum permissions that an identity-based policy can grant to an IAM entity (IAM user or role). You can set a permissions boundary for an entity. The resulting permissions are the intersection of the entity’s identity-based policies and its permissions boundaries. Resource-based policies that specify the user or role in the `Principal` field are not limited by the permissions boundary. An explicit deny in any of these policies overrides the allow. For more information about permissions boundaries, see [Permissions boundaries for IAM entities](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html) in the *IAM User Guide*.
+  **Service control policies (SCPs)**: SCPs are JSON policies that specify the maximum permissions for an organization or organizational unit (OU) in AWS Organizations. AWS Organizations is a service for grouping and centrally managing multiple AWS accounts that your business owns. If you enable all features in an organization, then you can apply service control policies (SCPs) to any or all of your accounts. The SCP limits permissions for entities in member accounts, including each AWS account root user. For more information about Organizations and SCPs, see [How SCPs work](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_about-scps.html) in the * AWS Organizations User Guide*.
+  **Session policies**: Session policies are advanced policies that you pass as a parameter when you programmatically create a temporary session for a role or federated user. The resulting session’s permissions are the intersection of the user or role’s identity-based policies and the session policies. Permissions can also come from a resource-based policy. An explicit deny in any of these policies overrides the allow. For more information, see [Session policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) in the *IAM User Guide*.

#### Multiple policy types
<a name="security-iam-access-manage-multiple-policies"></a>

When multiple types of policies apply to a request, the resulting permissions are more complicated to understand. To learn how AWS determines whether to allow a request when multiple policy types are involved, see [Policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html) in the *IAM User Guide*.

### Topics in this section
<a name="security-iam-topics"></a>

Use the following topics to learn how Amazon EVS works with IAM. They include example policies, the AWS managed policy, and the service-linked role.

If you have an access problem, see [Troubleshoot Amazon EVS identity and access](#security-iam-troubleshoot).

**Topics**

### How Amazon EVS works with IAM
<a name="security_iam_service-with-iam"></a>

Before you use IAM to manage access to Amazon Elastic VMware Service, learn what IAM features are available to use with Amazon EVS. The following table lists each IAM feature in the first column and whether Amazon EVS supports it in the second column.


|  IAM feature | Amazon EVS support | 
| --- | --- | 
|  [Identity-based policies for Amazon EVS](#security_iam_service-with-iam-id-based-policies)  | Yes | 
|  [Resource-based policies within Amazon EVS](#security_iam_service-with-iam-resource-based-policies)  | No | 
|  [Policy actions for Amazon EVS](#security_iam_service-with-iam-id-based-policies-actions)  | Yes | 
|  [Policy resources for Amazon EVS](#security_iam_service-with-iam-id-based-policies-resources)  | Partial | 
|  [Policy condition keys for Amazon EVS](#security_iam_service-with-iam-id-based-policies-conditionkeys)  | Yes | 
|  [Access control lists (ACLs) in Amazon EVS](#security_iam_service-with-iam-acls)  | No | 
|  [Attribute-based access control (ABAC) with Amazon EVS](#security_iam_service-with-iam-tags)  | Yes | 
|  [Using temporary credentials with Amazon EVS](#security_iam_service-with-iam-roles-tempcreds)  | Yes | 
|  [Forward access sessions for Amazon EVS](#security_iam_service-with-iam-principal-permissions)  | Yes | 
|  [Service roles for Amazon EVS](#security_iam_service-with-iam-roles-service)  | No | 
|  [Service-linked roles for Amazon EVS](#security_iam_service-with-iam-roles-service-linked)  | Yes | 

To get a high-level view of how Amazon EVS and other AWS services work with IAM, see [AWS services that work with IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) in the *IAM User Guide*.

#### Identity-based policies for Amazon EVS
<a name="security_iam_service-with-iam-id-based-policies"></a>

 **Supports identity-based policies:** Yes

Identity-based policies are JSON permissions policy documents that you can attach to an identity, such as an IAM user, group of users, or role. These policies control what actions users and roles can perform, on which resources, and under what conditions. To learn how to create an identity-based policy, see [Define custom IAM permissions with customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) in the *IAM User Guide*.

With IAM identity-based policies, you can specify allowed or denied actions and resources as well as the conditions under which actions are allowed or denied. You can’t specify the principal in an identity-based policy because it applies to the user or role to which it is attached. To learn about all of the elements that you use in a JSON policy, see [IAM JSON policy elements reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html) in the *IAM User Guide*.

##### Identity-based policy examples for Amazon EVS
<a name="security_iam_service-with-iam-id-based-policies-examples"></a>

To view examples of Amazon EVS identity-based policies, see [Amazon EVS identity-based policy examples](#security-iam-id-based-policy-examples).

##### Resource-based policies within Amazon EVS
<a name="security_iam_service-with-iam-resource-based-policies"></a>

 **Supports resource-based policies:** No

Resource-based policies are JSON policy documents that you attach to a resource. Examples of resource-based policies are IAM role trust policies and Amazon S3 bucket policies. In services that support resource-based policies, service administrators can use them to control access to a specific resource. For the resource where the policy is attached, the policy defines what actions a specified principal can perform on that resource and under what conditions. You must [specify a principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html) in a resource-based policy. Principals can include accounts, users, roles, federated users, or AWS services.

To enable cross-account access, you can specify an entire account or IAM entities in another account as the principal in a resource-based policy. Adding a cross-account principal to a resource-based policy is only half of establishing the trust relationship. When the principal and the resource are in different AWS accounts, an IAM administrator in the trusted account must also grant the principal entity (user or role) permission to access the resource. They grant permission by attaching an identity-based policy to the entity. However, if a resource-based policy grants access to a principal in the same account, no additional identity-based policy is required. For more information, see [Cross account resource access in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies-cross-account-resource-access.html) in the *IAM User Guide*.

##### Policy actions for Amazon EVS
<a name="security_iam_service-with-iam-id-based-policies-actions"></a>

 **Supports policy actions:** Yes

Administrators can use AWS JSON policies to specify who has access to what. That is, which **principal** can perform **actions** on what **resources**, and under what **conditions**.

The `Action` element of an IAM identity-based policy describes the specific action or actions that the policy allows or denies. Policy actions usually have the same name as the associated AWS API operation. The action is used in a policy to grant permissions to perform the associated operation.

Policy actions in Amazon EVS use the following prefix before the action: `evs:`. For example, to grant someone permission to create an environment with the Amazon EVS `CreateEnvironment` API operation, you include the `evs:CreateEnvironment` action in their policy. Policy statements must include either an `Action` or `NotAction` element. Amazon EVS defines its own set of actions that describe tasks that you can perform with this service.

To specify multiple actions in a single statement, separate them with commas as follows:

```json
"Action": [
      "evs:action1",
      "evs:action2"
]
```

You can specify multiple actions using wildcards (\*). For example, to specify all actions that begin with the word `List`, include the following action:

```json
"Action": "evs:List*"
```

To see a list of Amazon EVS actions, see [Actions defined by Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html#list_evs-actions-as-permissions) in the *Service Authorization Reference*.

##### Policy resources for Amazon EVS
<a name="security_iam_service-with-iam-id-based-policies-resources"></a>

 **Supports policy resources:** Partial

Some Amazon EVS actions, such as list operations, don’t support resource-level permissions.

Administrators can use AWS JSON policies to specify who has access to what. That is, which **principal** can perform **actions** on what **resources**, and under what **conditions**.

The `Resource` JSON policy element specifies the object or objects to which the action applies. Statements must include either a `Resource` or a `NotResource` element. As a best practice, specify a resource using its Amazon Resource Name (ARN). You can do this for actions that support a specific resource type, known as *resource-level permissions*.

For actions that don’t support resource-level permissions, such as listing operations, use a wildcard (\*) to indicate that the statement applies to all resources.

```json
"Resource": "*"
```

To see a list of Amazon EVS resource types and their ARNs, see [Resources defined by Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html#list_evs-resources-for-iam-policies) in the *Service Authorization Reference*. To learn with which actions you can specify the ARN of each resource, see [Actions defined by Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html#list_evs-actions-as-permissions).

Some Amazon EVS API actions support multiple resources. For example, multiple environments can be referenced when calling the `ListEnvironments` API action. To specify multiple resources in a single statement, separate the ARNs with commas.

```json
"Resource": [
      "arn:aws:evs:region-code:111122223333:environment/my-environment-1",
      "arn:aws:evs:region-code:111122223333:environment/my-environment-2"
]
```

For example, the Amazon EVS environment resource has the following ARN:

```text
arn:${Partition}:evs:${Region}:${Account}:environment/${EnvironmentId}
```

To specify the environments `my-environment-1` and `my-environment-2` in your statement, use the following example ARNs. The examples use the `aws` partition. In other partitions, replace `aws` with the name of your partition.

```json
"Resource": [
         "arn:aws:evs:region-code:111122223333:environment/my-environment-1",
         "arn:aws:evs:region-code:111122223333:environment/my-environment-2"
]
```

To specify all environments that belong to a specific account, use the wildcard (\*):

```json
"Resource": "arn:aws:evs:region-code:111122223333:environment/*"
```

##### Policy condition keys for Amazon EVS
<a name="security_iam_service-with-iam-id-based-policies-conditionkeys"></a>

 **Supports service-specific policy condition keys:** Yes

Administrators can use AWS JSON policies to specify who has access to what. That is, which **principal** can perform **actions** on what **resources**, and under what **conditions**.

The `Condition` element (or `Condition` block) lets you specify conditions in which a statement is in effect. The `Condition` element is optional. You can create conditional expressions that use [condition operators](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html), such as equals or less than, to match the condition in the policy with values in the request.

If you specify multiple `Condition` elements in a statement, or multiple keys in a single `Condition` element, AWS evaluates them using a logical `AND` operation. If you specify multiple values for a single condition key, AWS evaluates the condition using a logical `OR` operation. All of the conditions must be met before the statement’s permissions are granted.

You can also use placeholder variables when you specify conditions. For example, you can grant an IAM user permission to access a resource only if it is tagged with their IAM user name. For more information, see [IAM policy elements: variables and tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html) in the *IAM User Guide*.

Amazon EVS defines its own set of condition keys and also supports using some global condition keys. To see all AWS global condition keys, see [AWS global condition context keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html) in the *IAM User Guide*.

All Amazon EC2 actions support the `aws:RequestedRegion` and `ec2:Region` condition keys. For more information, see [Example: Restricting access to a specific region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ExamplePolicies_EC2.html#iam-example-region).

To see a list of Amazon EVS condition keys, see [Condition keys for Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html#list_evs-policy-keys) in the *Service Authorization Reference*. To learn with which actions and resources you can use a condition key, see [Actions defined by Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html#list_evs-actions-as-permissions).

#### Access control lists (ACLs) in Amazon EVS
<a name="security_iam_service-with-iam-acls"></a>

 **Supports ACLs:** No

Access control lists (ACLs) control which principals (account members, users, or roles) have permissions to access a resource. ACLs are similar to resource-based policies, although they do not use the JSON policy document format.

#### Attribute-based access control (ABAC) with Amazon EVS
<a name="security_iam_service-with-iam-tags"></a>

 **Supports ABAC (tags in policies):** Yes

Attribute-based access control (ABAC) is an authorization strategy that defines permissions based on attributes. In AWS, these attributes are called tags. You can attach tags to IAM entities (users or roles) and to many AWS resources. Tagging entities and resources is the first step of ABAC. Then you design ABAC policies to allow operations when the principal’s tag matches the tag on the resource that they are trying to access.

ABAC is helpful in environments that are growing rapidly and helps with situations where policy management becomes cumbersome.

You can attach tags to Amazon EVS resources or pass tags in a request to Amazon EVS. To control access based on tags, you provide tag information in the [condition element](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html) of a policy using the `aws:ResourceTag/<key-name>`, `aws:RequestTag/<key-name>`, or `aws:TagKeys` condition keys. For more information about which actions that you can use tags in condition keys with, see [Actions defined by Amazon Elastic VMware Service](https://docs.aws.amazon.com/service-authorization/latest/reference/list_evs.html#list_evs-actions-as-permissions) in the *Service Authorization Reference*.

#### Using temporary credentials with Amazon EVS
<a name="security_iam_service-with-iam-roles-tempcreds"></a>

 **Supports temporary credentials:** Yes

Some AWS services don’t work when you sign in using temporary credentials. For additional information, including which AWS services work with temporary credentials, see [AWS services that work with IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) in the *IAM User Guide*.

You are using temporary credentials if you sign in to the AWS Management Console using any method except a user name and password. For example, when you access AWS using your company’s single sign-on (SSO) link, that process automatically creates temporary credentials. You also automatically create temporary credentials when you sign in to the console as a user and then switch roles. For more information about switching roles, see [Switch from a user to an IAM role (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-console.html) in the *IAM User Guide*.

You can manually create temporary credentials using the AWS CLI or AWS API. You can then use those temporary credentials to access AWS. AWS recommends that you dynamically generate temporary credentials instead of using long-term access keys. For more information, see [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html).

#### Forward access sessions for Amazon EVS
<a name="security_iam_service-with-iam-principal-permissions"></a>

 **Supports forward access sessions (FAS):** Yes

When you use an IAM user or role to perform actions in AWS, you are considered a principal. When you use some services, you might perform an action that then initiates another action in a different service. FAS uses the permissions of the principal calling an AWS service, combined with the requesting AWS service, to make requests to downstream services. FAS requests are only made when a service receives a request that requires interactions with other AWS services or resources to complete. In this case, you must have permissions to perform both actions. For policy details when making FAS requests, see [Forward access sessions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_forward_access_sessions.html).

#### Service roles for Amazon EVS
<a name="security_iam_service-with-iam-roles-service"></a>

 **Supports service roles:** No

A service role is an IAM role that a service assumes to perform actions on your behalf. An IAM administrator can create, modify, and delete a service role from within IAM. For more information, see [Create a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*.

#### Service-linked roles for Amazon EVS
<a name="security_iam_service-with-iam-roles-service-linked"></a>

 **Supports service-linked roles:** Yes

A service-linked role is a type of service role that is linked to an AWS service. The service can assume the role to perform an action on your behalf. Service-linked roles appear in your AWS account and are owned by the service. An IAM administrator can view, but not edit the permissions for service-linked roles.

For details about creating or managing Amazon EVS service-linked roles, see [Using service-linked roles for Amazon EVS](#using-service-linked-roles).

### Amazon EVS identity-based policy examples
<a name="security-iam-id-based-policy-examples"></a>

By default, IAM users and roles don’t have permission to create or modify Amazon EVS resources. They also can’t perform tasks using the AWS Management Console, AWS CLI, or AWS API. An IAM administrator must create IAM policies that grant users and roles permission to perform specific API operations on the specified resources they need. The administrator must then attach those policies to the IAM users or groups that require those permissions.

To learn how to create an IAM identity-based policy using these example JSON policy documents, see [Creating policies using the JSON editor](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html#access_policies_create-json-editor) in the *IAM User Guide*.

#### Policy best practices
<a name="security_iam_service-with-iam-policy-best-practices"></a>

Identity-based policies determine whether someone can create, access, or delete Amazon Elastic VMware Service resources in your account. These actions can incur costs for your AWS account. When you create or edit identity-based policies, follow these guidelines and recommendations:
+  **Get started with AWS managed policies and move toward least-privilege permissions**: To get started granting permissions to your users and workloads, use the AWS managed policies that grant permissions for many common use cases. They are available in your AWS account. We recommend that you reduce permissions further by defining AWS customer managed policies that are specific to your use cases. For more information, see [AWS managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) or [AWS managed policies for job functions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html) in the *IAM User Guide*.
+  **Apply least-privilege permissions**: When you set permissions with IAM policies, grant only the permissions required to perform a task. You do this by defining the actions that can be taken on specific resources under specific conditions, also known as *least-privilege permissions*. For more information about using IAM to apply permissions, see [Policies and permissions in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) in the *IAM User Guide*.
+  **Use conditions in IAM policies to further restrict access**: You can add a condition to your policies to limit access to actions and resources. For example, you can write a policy condition to specify that all requests must be sent using SSL. You can also use conditions to grant access to service actions if they are used through a specific AWS service, such as CloudFormation. For more information, see [IAM JSON policy elements: condition](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html) in the *IAM User Guide*.
+  **Use IAM Access Analyzer to validate your IAM policies to ensure secure and functional permissions**: IAM Access Analyzer validates new and existing policies so that the policies adhere to the IAM policy language (JSON) and IAM best practices. IAM Access Analyzer provides more than 100 policy checks and actionable recommendations to help you author secure and functional policies. For more information, see [IAM Access Analyzer policy validation](https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-validation.html) in the *IAM User Guide*.
+  **Require multi-factor authentication (MFA)**: If you have a scenario that requires IAM users or root users in your account, turn on MFA for additional security. To require MFA when API operations are called, add MFA conditions to your policies. For more information, see [Configuring MFA-protected API access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_configure-api-require.html) in the *IAM User Guide*.

#### Using the Amazon EVS console
<a name="security-iam-id-based-policy-examples-console"></a>

To access the Amazon EVS console, an IAM principal must have a minimum set of permissions. These permissions must allow the principal to list and view details about the Amazon EVS resources in your AWS account. If you create an identity-based policy that is more restrictive than the minimum required permissions, the console won’t function as intended for principals with that policy attached to them.

To ensure that your IAM principals can still use the Amazon EVS console, create a policy with your own unique name, such as `AmazonEVSAdminPolicy`. Attach the policy to the principals. For more information, see [Adding permissions to a user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) in the *IAM User Guide*.

The following policy grants the minimum permissions that principals need to use the Amazon EVS console.

```json
{
    "Version":"2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "evs:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "EVSServiceLinkedRole",
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/evs.amazonaws.com/AWSServiceRoleForEVS",
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "evs.amazonaws.com"
                }
            }
        }
    ]
}
```

You don’t need to allow minimum console permissions for users that are making calls only to the AWS CLI or the AWS API. Instead, allow access to only the actions that match the API operation that you’re trying to perform.

#### Allow users to view their own permissions
<a name="security-iam-id-based-policy-examples-view-own-permissions"></a>

This example shows how you might create a policy that allows IAM users to view the inline and managed policies that are attached to their user identity. This policy includes permissions to complete this action on the console or programmatically using the AWS CLI or AWS API.

```json
{
    "Version":"2012-10-17",		 	 	 
    "Statement": [
        {
            "Sid": "ViewOwnUserInfo",
            "Effect": "Allow",
            "Action": [
                "iam:GetUserPolicy",
                "iam:ListGroupsForUser",
                "iam:ListAttachedUserPolicies",
                "iam:ListUserPolicies",
                "iam:GetUser"
            ],
            "Resource": ["arn:aws:iam::*:user/${aws:username}"]
        },
        {
            "Sid": "NavigateInConsole",
            "Effect": "Allow",
            "Action": [
                "iam:GetGroupPolicy",
                "iam:GetPolicyVersion",
                "iam:GetPolicy",
                "iam:ListAttachedGroupPolicies",
                "iam:ListGroupPolicies",
                "iam:ListPolicyVersions",
                "iam:ListPolicies",
                "iam:ListUsers"
            ],
            "Resource": "*"
        }
    ]
}
```

#### Create and manage an Amazon EVS environment
<a name="security-iam-id-based-policy-examples-create-env"></a>

This example policy includes the permissions required to create and delete an Amazon EVS environment, and to add or delete hosts after the environment has been created.

Replace the Region in the policy with the Region where you want to create the environment. If your account already has the `AWSServiceRoleForAmazonEVS` role, you can remove the `iam:CreateServiceLinkedRole` action from the policy. If you’ve ever created an Amazon EVS environment in your account, a role with these permissions already exists, unless you deleted it. For more information about the role, see [Using service-linked roles for Amazon EVS](#using-service-linked-roles).

```json
{
    "Version":"2012-10-17",		 	 	 
    "Statement": [
        {
            "Sid": "ReadOnlyDescribeActions",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVpcs",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeHosts",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeAddresses",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeSubnets",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInstances",
                "ec2:DescribeRouteServers",
                "ec2:DescribeRouteServerEndpoints",
                "ec2:DescribeRouteServerPeers",
                "ec2:DescribePlacementGroups",
                "ec2:DescribeVolumes",
                "ec2:DescribeSecurityGroups",
                "support:DescribeServices",
                "support:DescribeSupportLevel",
                "servicequotas:GetServiceQuota",
                "servicequotas:ListServiceQuotas"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ModifyNetworkInterfaceStatement",
            "Effect": "Allow",
            "Action": [
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "arn:aws:ec2:*:*:network-interface/*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "ModifyNetworkInterfaceStatementForSubnetAssociation",
            "Effect": "Allow",
            "Action": [
                "ec2:ModifyNetworkInterfaceAttribute"
            ],
            "Resource": "arn:aws:ec2:*:*:subnet/*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "CreateNetworkInterfaceWithTag",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateNetworkInterface"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "CreateNetworkInterfaceAdditionalResources",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateNetworkInterface"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:subnet/*",
                "arn:aws:ec2:*:*:security-group/*"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "TagOnCreateEC2Resources",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*",
                "arn:aws:ec2:*:*:instance/*",
                "arn:aws:ec2:*:*:volume/*",
                "arn:aws:ec2:*:*:subnet/*"
            ],
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": [
                        "CreateNetworkInterface",
                        "RunInstances",
                        "CreateSubnet",
                        "CreateVolume"
                    ]
                },
                "Null": {
                    "aws:RequestTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "DetachNetworkInterface",
            "Effect": "Allow",
            "Action": [
                "ec2:DetachNetworkInterface"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*",
                "arn:aws:ec2:*:*:instance/*"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "RunInstancesWithTag",
            "Effect": "Allow",
            "Action": [
                "ec2:RunInstances"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:instance/*",
                "arn:aws:ec2:*:*:volume/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "RunInstancesWithTagResource",
            "Effect": "Allow",
            "Action": [
                "ec2:RunInstances"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:subnet/*",
                "arn:aws:ec2:*:*:network-interface/*"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "RunInstancesWithoutTag",
            "Effect": "Allow",
            "Action": [
                "ec2:RunInstances"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:image/*",
                "arn:aws:ec2:*:*:security-group/*",
                "arn:aws:ec2:*:*:key-pair/*",
                "arn:aws:ec2:*:*:placement-group/*"
            ]
        },
        {
            "Sid": "TerminateInstancesWithTag",
            "Effect": "Allow",
            "Action": [
                "ec2:TerminateInstances",
                "ec2:ModifyInstanceAttribute"
            ],
            "Resource": "arn:aws:ec2:*:*:instance/*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "CreateSubnetWithTag",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSubnet"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:subnet/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "CreateSubnetWithoutTagForExistingVPC",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSubnet"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:vpc/*"
            ]
        },
        {
            "Sid": "DeleteSubnetWithTag",
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteSubnet"
            ],
            "Resource": "arn:aws:ec2:*:*:subnet/*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "VolumeDeletion",
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteVolume"
            ],
            "Resource": "arn:aws:ec2:*:*:volume/*",
             "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "VolumeDetachment",
            "Effect": "Allow",
            "Action": [
                "ec2:DetachVolume"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:instance/*",
                "arn:aws:ec2:*:*:volume/*"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "RouteServerAccess",
            "Effect": "Allow",
            "Action": [
                "ec2:GetRouteServerAssociations"
            ],
            "Resource": "arn:aws:ec2:*:*:route-server/*"

        },
        {
            "Sid": "EVSServiceLinkedRole",
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/evs.amazonaws.com/AWSServiceRoleForEVS",
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "evs.amazonaws.com"
                }
            }
        },
        {
            "Sid": "SecretsManagerCreateWithTag",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:*",
            "Condition": {
                "StringEquals": {
                    "aws:RequestTag/AmazonEVSManaged": "true"
                },
                "ForAllValues:StringEquals": {
                    "aws:TagKeys": [
                        "AmazonEVSManaged"
                    ]
                }
            }
        },
        {
            "Sid": "SecretsManagerTagging",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:TagResource"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:*",
            "Condition": {
                "StringEquals": {
                    "aws:RequestTag/AmazonEVSManaged": "true",
                    "aws:ResourceTag/AmazonEVSManaged": "true"
                },
                "ForAllValues:StringEquals": {
                    "aws:TagKeys": [
                        "AmazonEVSManaged"
                    ]
                }
            }
        },
        {
            "Sid": "SecretsManagerOps",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:DeleteSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:UpdateSecret"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/AmazonEVSManaged": "false"
                }
            }
        },
        {
            "Sid": "SecretsManagerRandomPassword",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetRandomPassword"
            ],
            "Resource": "*"
        },
        {
            "Sid": "EVSPermissions",
            "Effect": "Allow",
            "Action": [
                "evs:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "KMSKeyAccessInConsole",
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:*:*:key/*"
        },

        {
            "Sid": "KMSKeyAliasAccess",
            "Effect": "Allow",
            "Action": [
                "kms:ListAliases"
            ],
            "Resource": "*"
        }
    ]
}
```

#### Get and list Amazon EVS environments, hosts, and VLANs
<a name="security-iam-id-based-policy-examples-list-env"></a>

This example policy includes the minimum permissions required for an administrator to get and list all Amazon EVS environments, hosts, and VLANs within a given account in a specific AWS Region.

```json
{
  "Version":"2012-10-17",		 	 	 
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "evs:Get*",
        "evs:List*"
      ],
      "Resource": "*"
    }
  ]
}
```

#### Permissions for an Amazon EVS organizational unit
<a name="security-iam-id-based-policy-examples-ou"></a>

If you use AWS Organizations, service control policies (SCPs) set the maximum permissions for the accounts in an organizational unit (OU). An SCP that denies an action overrides the identity-based policies in this topic. Make sure that the SCPs on the OU allow the actions that the example policies on this page grant. These include Amazon EVS actions and actions in other AWS services. For account-level guardrails, see [Account permissions and guardrails](#plan-account-guardrails).

### AWS managed policies for Amazon EVS
<a name="security-iam-awsmanpol"></a>

An AWS managed policy is a standalone policy that is created and administered by AWS. AWS managed policies are designed to provide permissions for many common use cases so that you can start assigning permissions to users, groups, and roles.

 AWS managed policies might not grant least-privilege permissions for your use cases because they’re available to all AWS customers. We recommend that you reduce permissions further by defining [customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#customer-managed-policies) that are specific to your use cases.

You can’t change the permissions defined in AWS managed policies. If AWS updates the permissions defined in an AWS managed policy, the update affects all principal identities (users, groups, and roles) that the policy is attached to. AWS is most likely to update an AWS managed policy when a new AWS service is launched or new API operations become available for existing services. For more information, see [AWS managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) in the * IAM User Guide*.

#### AWS managed policy: AmazonEVSServiceRolePolicy
<a name="security-iam-awsmanpol-amazonevsservicerolepolicy"></a>

You can’t attach `AmazonEVSServiceRolePolicy` to your IAM entities. This policy is attached to a service-linked role that allows Amazon Elastic VMware Service to perform actions on your behalf. For more information, see [Using service-linked roles for Amazon EVS](#using-service-linked-roles). When you create an environment using an IAM principal that has the `iam:CreateServiceLinkedRole` permission, the `AWSServiceRoleForAmazonEVS` service-linked role is automatically created for you with this policy attached to it.

This policy allows the `AWSServiceRoleForAmazonEVS` service-linked role to call AWS services on your behalf.

The following table lists the permissions in this policy and what Amazon EVS does with each of them.


| Permission scope | What Amazon EVS does | 
| --- | --- | 
|  `ec2`  | Amazon EVS discovers VPC networking components, including subnets and VPCs. Amazon EVS creates, modifies, tags, and deletes elastic network interfaces. These interfaces keep a persistent connection between Amazon EVS and the VMware Cloud Foundation (VCF) SDDC Manager appliance in your VPC subnet. Amazon EVS requires this connectivity to deploy, manage, and monitor the VCF deployment. | 
|  `ec2`  | Amazon EVS deletes the Amazon EC2 instances that it creates when you make an Amazon EVS host deletion request. Amazon EVS describes and modifies Amazon EC2 instance attributes so that it can disable default Amazon EC2 instance termination and stop protection if host deletion needs it. | 
|  `ec2`  | Amazon EVS manages EBS volumes for Cloud Builder installation and cleanup. During environment creation, Amazon EVS installs Cloud Builder onto one of the hosts that it deploys to perform VCF configuration changes. After completion, Amazon EVS removes Cloud Builder by detaching and deleting the EBS volume that Cloud Builder is stored on. | 
|  `ec2`  | Amazon EVS deletes Amazon EVS VLAN subnets on your behalf if you request environment deletion. | 
|  `secretsmanager` (delete) | Amazon EVS deletes the VCF passwords that it creates and stores in AWS Secrets Manager during environment creation. Amazon EVS deletes all secrets that it creates in your account if environment creation fails, or if you request environment deletion. | 
|  `secretsmanager` (read) | Amazon EVS retrieves vCenter credentials from AWS Secrets Manager when you configure a vCenter connector by providing a secret ARN. The permission is scoped with the resource tag condition `EvsAccess=true`, so Amazon EVS only accesses secrets that are explicitly tagged for Amazon EVS vCenter access. | 
|  `kms`  | Amazon EVS decrypts secrets and describes KMS keys when vCenter credentials stored in Secrets Manager are encrypted with KMS keys. The permission is scoped with the resource tag condition `EvsAccess=true`, so Amazon EVS only accesses KMS keys that are explicitly tagged for vCenter access. | 
|  `cloudwatch`  | Amazon EVS publishes AWS usage metrics to CloudWatch for Amazon EVS resources that have quotas. | 

To view more details about the policy, including the latest version of the JSON policy document, see [AmazonEVSServiceRolePolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEVSServiceRolePolicy.html) in the * AWS Managed Policy Reference Guide*.

#### Amazon EVS updates to AWS managed policies
<a name="security-iam-awsmanpol-account-updates"></a>

View details about updates to AWS managed policies for Amazon EVS since this service began tracking these changes. For automatic alerts about changes to this page, subscribe to the RSS feed on the [Document history for the Amazon Elastic VMware Service User Guide](#doc-history) page. The following table lists each change in the first column, a description in the second column, and the date in the third column.


| Change | Description | Date | 
| --- | --- | --- | 
| AmazonEVSServiceRolePolicy: Policy updated | Amazon EVS updated the policy to allow the service to retrieve vCenter credentials from AWS Secrets Manager and decrypt secrets encrypted with KMS keys. To learn more, see [AWS managed policy: AmazonEVSServiceRolePolicy](#security-iam-awsmanpol-amazonevsservicerolepolicy). | March 23, 2026 | 
| AmazonEVSServiceRolePolicy: Policy updated | Amazon EVS updated the policy to add resource management permissions, including Amazon EC2 instance management, EBS volume operations, and AWS Secrets Manager integration. To learn more, see [AWS managed policy: AmazonEVSServiceRolePolicy](#security-iam-awsmanpol-amazonevsservicerolepolicy). | August 14, 2025 | 
| AmazonEVSServiceRolePolicy: Policy updated | Amazon EVS updated the policy to allow the service to delete Amazon EVS VLAN subnets, and to publish Amazon EVS usage metrics to CloudWatch. To learn more, see [AWS managed policy: AmazonEVSServiceRolePolicy](#security-iam-awsmanpol-amazonevsservicerolepolicy). | July 14, 2025 | 
| AmazonEVSServiceRolePolicy: New policy added | Amazon EVS added a new policy that allows the service to connect to a VPC subnet in your account. Amazon EVS requires this connection to work. To learn more, see [AWS managed policy: AmazonEVSServiceRolePolicy](#security-iam-awsmanpol-amazonevsservicerolepolicy). | June 09, 2025 | 
| Amazon EVS started tracking changes | Amazon EVS started tracking changes for its AWS managed policies. | June 09, 2025 | 

### Using service-linked roles for Amazon EVS
<a name="using-service-linked-roles"></a>

Amazon Elastic VMware Service uses AWS Identity and Access Management (IAM) [service-linked roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-service-linked-role). A service-linked role is a unique type of IAM role that is linked directly to Amazon EVS. Service-linked roles are predefined by Amazon EVS and include all the permissions that the service requires to call other AWS services on your behalf.

A service-linked role makes setting up Amazon EVS easier because you don’t have to manually add the necessary permissions. Amazon EVS defines the permissions of its service-linked roles, and unless defined otherwise, only Amazon EVS can assume its roles. The defined permissions include the trust policy and the permissions policy, and that permissions policy can’t be attached to any other IAM entity.

You can delete a service-linked role only after first deleting its related resources. This protects your Amazon EVS resources because you can’t inadvertently remove permission to access the resources.

For information about other services that support service-linked roles, see [AWS services that work with IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) and look for the services that have **Yes** in the **Service-linked role** column. Choose a **Yes** with a link to view the service-linked role documentation for that service.

#### Service-linked role permissions for Amazon EVS
<a name="service-linked-role-permissions"></a>

Amazon EVS uses the service-linked role named `AWSServiceRoleForAmazonEVS`. The role allows Amazon EVS to manage environments in your account. The attached policy allows the role to manage Amazon EVS elastic network interfaces, Amazon EVS hosts, Amazon EVS VLAN subnets, EBS volumes, and CloudWatch metrics. It also allows the role to discover VPCs and subnets, delete and read AWS Secrets Manager secrets, and describe KMS keys.

The `AWSServiceRoleForAmazonEVS` service-linked role trusts the following services to assume the role:
+  `evs.amazonaws.com` 

The role permissions policy allows Amazon EVS to complete the following actions on the specified resources:
+  [AmazonEVSServiceRolePolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEVSServiceRolePolicy.html) 

For a description of each permission in the policy, see [AWS managed policy: AmazonEVSServiceRolePolicy](#security-iam-awsmanpol-amazonevsservicerolepolicy).

You must configure permissions to allow an IAM entity (such as a user, group, or role) to create, edit, or delete a service-linked role. For more information, see [Service-linked role permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#service-linked-role-permissions) in the *IAM User Guide*.

#### Creating a service-linked role for Amazon EVS
<a name="creating-evs-slr"></a>

You don’t need to manually create a service-linked role. When you create an environment in the AWS Management Console, the AWS CLI, or the AWS API, Amazon EVS creates the service-linked role for you.

If you delete this service-linked role, and then need to create it again, you can use the same process to recreate the role in your account. When you create an environment, Amazon EVS creates the service-linked role for you again.

#### Editing a service-linked role for Amazon EVS
<a name="editing-evs-slr"></a>

Amazon EVS does not allow you to edit the `AWSServiceRoleForAmazonEVS` service-linked role. After you create a service-linked role, you can’t change the name of the role because various entities might reference the role. However, you can edit the description of the role using IAM. For more information, see [Editing a service-linked role](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#edit-service-linked-role) in the *IAM User Guide*.

#### Deleting a service-linked role for Amazon EVS
<a name="deleting-evs-slr"></a>

If you no longer need to use a feature or service that requires a service-linked role, we recommend that you delete that role. That way you don’t have an unused entity that is not actively monitored or maintained. However, you must clean up your service-linked role before you can manually delete it.

##### Cleaning up a service-linked role
<a name="evs-slr-cleanup"></a>

Before you can use IAM to delete a service-linked role, you must first delete any resources used by the role. To delete an Amazon EVS environment with its hosts, see [Clean up a Self-deployed environment](#self-deployed-cleanup) for Self-deployed environments, or [Delete an EVS-deployed environment](#getting-started-cleanup-env-hosts) for EVS-deployed environments.

**Note**  
If the Amazon EVS service is using the role when you try to delete the resources, then the deletion might fail. If that happens, wait for a few minutes and try the operation again.

##### Manually delete the service-linked role
<a name="evs-slr-manual-delete"></a>

Use the IAM console, the AWS CLI, or the AWS API to delete the `AWSServiceRoleForAmazonEVS` service-linked role. For more information, see [Deleting a service-linked role](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#delete-service-linked-role) in the *IAM User Guide*.

#### Supported Regions for Amazon EVS service-linked roles
<a name="evs-slr-regions"></a>

Amazon EVS supports using service-linked roles in all of the Regions where the service is available. For the list of Regions, see [AWS Regions for Amazon EVS](#regions-reference). For more information, see [Amazon Elastic VMware Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/evs.html) in the * AWS General Reference*.

## Resilience for Amazon EVS
<a name="disaster-recovery-resiliency"></a>

The AWS global infrastructure is built around AWS Regions and Availability Zones. AWS Regions provide multiple physically separated and isolated Availability Zones, which are connected through low-latency, high-throughput, and highly redundant networking. With Availability Zones, you can design and operate applications and databases that automatically fail over between zones without interruption. Availability Zones are more highly available, fault tolerant, and scalable than traditional single or multiple data center infrastructures. For more information about AWS Regions and Availability Zones, see [AWS Global Infrastructure](https://aws.amazon.com/about-aws/global-infrastructure).

To choose a recovery approach for your workloads, see [Plan for resilience and recovery](#plan-resilience).

### Amazon EVS infrastructure resilience
<a name="disaster-recovery-resiliency-infra"></a>

An Amazon Elastic VMware Service environment runs in a single AWS Availability Zone. Amazon EVS supports Single-AZ deployments only. To help keep your Single-AZ infrastructure available, Amazon EVS offers the following features:
+ Amazon EVS supports partition placement groups and cluster placement groups for Amazon EVS hosts. For more information, see [Placement strategies](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-strategies.html) in the * Amazon EC2 User Guide*.
+ In EVS-deployed environments that run VMware Cloud Foundation (VCF) 5.2.x, Amazon EVS deploys an Active/Standby NSX Edge cluster with two NSX Edge nodes, as VCF requires. The NSX Edge nodes run on different hosts, which allows quick failover in the rare event that an NSX Edge node fails. In Self-deployed environments that run VCF 9.x, you deploy and configure the NSX Edge cluster yourself. For design guidance, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design).

A partition placement group spreads your Amazon EC2 instances across logical partitions. Instances in one partition don’t share underlying hardware with instances in other partitions. This strategy helps reduce the likelihood of correlated hardware failures for large distributed workloads. A cluster placement group places your Amazon EC2 instances within the same physical rack for low latency.

### Host count and availability
<a name="disaster-recovery-resiliency-hosts"></a>

In EVS-deployed environments (VCF 5.2.x), Amazon EVS deploys a minimal environment of four ESX hosts, which VCF requires. You can add hosts after deployment. This VMware design requirement ensures proper vSAN quorum and maintains availability during maintenance operations and host failures. For more information, see [vSphere Cluster Design for VMware Cloud Foundation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vcf-design-5-2/vsphere-design-for-vmware-cloud-foundation/vcf-vsphere-cluster-design.html) in the Broadcom documentation.

In Self-deployed environments that run VCF 9.x, you add enough hosts for your VCF topology. For the minimum host counts of each topology, see the Broadcom documentation for your VCF version. For host limits and sizing guidance in both modes, see [Size your environment](#plan-sizing).

### VMware component resilience
<a name="disaster-recovery-resiliency-app"></a>

You are responsible for configuring the VMware components that run on Amazon EVS for the high availability of your virtual machines (VMs) and for workload resiliency. You can configure the following VCF resiliency features:
+ vSphere Replication provides host-based, asynchronous replication of your VMs for disaster recovery and workload migration. For more information, see [How vSphere Replication Works](https://techdocs.broadcom.com/us/en/vmware-cis/live-recovery/vsphere-replication/9-0-3/using-vsphere-replication/about-vmware-vsphere-replication/how-vsphere-replication-works.html) in the Broadcom documentation.
+ vSphere HA provides automatic failover for VMs when a host fails. For more information, see [High Availability Design for vCenter Server for VMware Cloud Foundation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vcf-design-5-2/vsphere-design-for-vmware-cloud-foundation/vcf-vcenter-server-design.html#GUID-B4380728-190C-428E-83AE-3F7667506948-en) in the Broadcom documentation.
+ With vSAN data protection, you can quickly recover VMs from operational failures or ransomware attacks by using native snapshots stored locally on the vSAN cluster. For more information, see [Using vSAN Data Protection](https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration/expanding-and-managing-a-vsan-cluster/using-vsan-data-protection.html) in the Broadcom documentation.
+ vSAN Failures to Tolerate (FTT) is a vSAN setting that determines how many host failures a VM can withstand before it becomes inaccessible. This setting defines the level of redundancy for your VMs within the vSAN cluster. For more information, see [Tolerate Additional Failures with Fault Domain in vSAN Cluster](https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration/expanding-and-managing-a-vsan-cluster/managing-fault-domains-in-vsan-clusters/tolerate-additional-failures-with-fault-domains-in-vsan-cluster.html) in the Broadcom documentation.

The vSAN features apply only to clusters that use vSAN storage. Diskless hosts, which need VCF 9.x, have no instance store volumes and use external principal storage. The resilience of their VM data depends on the storage service. For more information, see [External storage options for diskless hosts](#external-storage-diskless).

**Note**  
Amazon EVS doesn’t support vSphere Fault Tolerance (FT). Workload VMs on Amazon EVS run on NSX overlay networks, and Broadcom doesn’t support FT on VMs that use NSX port groups or on NSX Manager and NSX Edge nodes. For more information, see [article 317806](https://knowledge.broadcom.com/external/article?articleNumber=317806) in the Broadcom Knowledge Base.

For the support status of VCF features on Amazon EVS, including the [vSphere Fault Tolerance](#feat-vsphere-fault-tolerance) row, see [VCF feature support on Amazon EVS](#vcf-feature-support).

## Compliance validation for Amazon EVS
<a name="compliance-validation"></a>

Third-party auditors assess the security and compliance of AWS services as part of AWS compliance programs. To find out whether Amazon Elastic VMware Service (Amazon EVS) is in scope for a compliance program, see [AWS services in Scope by Compliance Program](https://aws.amazon.com/compliance/services-in-scope/). Then choose the program that you need. For general information, see [AWS Compliance Programs](https://aws.amazon.com/compliance/programs/).

You can download third-party audit reports using AWS Artifact. For more information, see [Downloading reports in AWS Artifact](https://docs.aws.amazon.com/artifact/latest/ug/downloading-documents.html) in the * AWS Artifact User Guide*.

### Your compliance responsibility
<a name="compliance-validation-responsibility"></a>

Your compliance responsibility when you use Amazon EVS is determined by the sensitivity of your data, your company’s compliance objectives, and applicable laws and regulations. An AWS compliance program covers the AWS side of the service. You are responsible for the compliance of the VMware Cloud Foundation (VCF) software, the virtual machines, and the data that you run in your Amazon EVS environment. For how responsibilities are split in each deployment mode, see [Shared responsibility and support](#plan-shared-responsibility).

### Compliance resources
<a name="compliance-validation-resources"></a>

 AWS provides the following resources to help with compliance:
+  [Security and Compliance Quick Start Guides](https://aws.amazon.com/quickstart/?awsf.quickstart-homepage-filter=categories%23security-identity-compliance). These deployment guides discuss architectural considerations and provide steps for deploying security-focused and compliance-focused baseline environments on AWS.
+  [Architecting for HIPAA Security and Compliance Whitepaper](https://d0.awsstatic.com/whitepapers/compliance/AWS_HIPAA_Compliance_Whitepaper.pdf). This whitepaper describes how companies can use AWS to create HIPAA-compliant applications.
+  [AWS Compliance Resources](https://aws.amazon.com/compliance/resources/). This collection of workbooks and guides might apply to your industry and location.
+  [Evaluating Resources with Rules](https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config.html) in the * AWS Config Developer Guide*. AWS Config assesses how well your resource configurations comply with internal practices, industry guidelines, and regulations.
+  [AWS Security Hub](https://docs.aws.amazon.com/securityhub/latest/userguide/what-is-securityhub.html). This AWS service provides a comprehensive view of your security state within AWS that helps you check your compliance with security industry standards and best practices.

## Infrastructure security for Amazon EVS
<a name="infrastructure-security"></a>

As a managed service, Amazon Elastic VMware Service (Amazon EVS) is protected by the AWS global network security procedures. These procedures are described in the [Amazon Web Services: Overview of Security Processes](https://d0.awsstatic.com/whitepapers/Security/AWS_Security_Whitepaper.pdf) whitepaper.

### Access to the Amazon EVS API
<a name="infrastructure-security-api-access"></a>

You use AWS published API calls to access Amazon EVS through the network. Clients must support Transport Layer Security (TLS). We require TLS 1.2 and recommend TLS 1.3. Clients must also support cipher suites with perfect forward secrecy (PFS), such as Ephemeral Diffie-Hellman (DHE) or Elliptic Curve Ephemeral Diffie-Hellman (ECDHE). Most modern systems such as Java 7 and later support these modes.

Requests must be signed by using an access key ID and a secret access key that is associated with an IAM principal. You can also use the [AWS Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) (AWS STS) to generate temporary security credentials to sign requests. To control who can call Amazon EVS API operations, see [Identity and access management for Amazon Elastic VMware Service](#security-iam).

### How Amazon EVS reaches your environment
<a name="infrastructure-security-service-access"></a>

Amazon EVS reaches the VMware Cloud Foundation (VCF) deployment in your account through the service access subnet. The service access subnet is a standard VPC subnet that you choose when you create the environment. Amazon EVS provisions elastic network interfaces into this subnet to provide management connectivity to VCF appliances and ESX hosts. Amazon EVS needs this connectivity to deploy, manage, and monitor the VCF deployment. For more information, see [Service access subnet](#concepts-service-access-subnet).

The service-linked role `AWSServiceRoleForAmazonEVS` grants Amazon EVS the permissions to create and manage these network interfaces. For the actions that the role allows, see [AWS managed policies for Amazon EVS](#security-iam-awsmanpol) and [Using service-linked roles for Amazon EVS](#using-service-linked-roles).

### Network isolation in your VPC
<a name="infrastructure-security-network-isolation"></a>

Your Amazon EVS environment runs in an Amazon VPC that you own. The VPC creates boundaries between the resources in the environment and controls traffic between them, your on-premises network and the internet.

By default, Amazon EVS creates private VLAN subnets during environment creation that deny direct internet access. To add another layer of security, you can create a custom network access control list (network ACL) with rules that further restrict internet connectivity.

**Important**  
 Amazon EC2 security group rules are not enforced on Amazon EVS elastic network interfaces that are attached to VLAN subnets. To control traffic to and from VLAN subnets, you must use a network ACL. For the steps, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

Inside the VCF environment, you can segment and filter VM traffic with NSX features such as the VMware vDefend Gateway Firewall and Distributed Firewall. For details and links, see [Internetwork traffic privacy](#internetwork-traffic-privacy). For the network connections that Amazon EVS supports, see [Supported network configurations](#plan-network-supported).

# Troubleshooting
<a name="troubleshooting"></a>

When something doesn’t work as expected in your Amazon Elastic VMware Service environment, find the symptom in the following table. Then go to the page that covers it. If a problem remains, see [Get help from AWS Support and Broadcom](#get-help).

The procedures in [Deploy in Self-deployed mode](#getting-started-self-deployed) deploy VCF 9.x. Pages in [Deploy in EVS-deployed mode](#evs-deployed-overview) cover VCF 5.2.x. A page with no line and outside those sections covers both.

## Problems by area and symptom
<a name="troubleshooting-by-symptom"></a>

The following table lists common problems. The columns show the area, the symptom, the VMware Cloud Foundation (VCF) version that it applies to, and the page that covers it. For how VCF versions relate to the deployment modes, see [Differences between deployment modes](#deployment-mode-differences).


| Area | Symptom | VCF version | Where to look | 
| --- | --- | --- | --- | 
| Deployment |  `create-environment` returns a `ValidationException`, or a host stays in `CREATING`  | 9.x |  [Environment, host, and connector symptoms](#troubleshoot-sd-env-host)  | 
| Deployment |  `create-environment-host` fails with an ESX version error, or `create-environment-connector` fails with a Secrets Manager access error | 9.x |  [`create-environment-host` fails with an ESX version error](#troubleshoot-sd-esx-version) and [`create-environment-connector` fails with a Secrets Manager access error](#troubleshoot-sd-connector-secret)  | 
| Deployment | The VCF Installer validation or VCF bringup fails | 9.x |  [VCF Installer and bringup symptoms](#troubleshoot-sd-installer)  | 
| Deployment | BGP sessions between the NSX Edge nodes and Amazon VPC Route Server don’t come up or flap | 9.x |  [NSX Edge and BGP symptoms](#troubleshoot-sd-nsx) and [Step 5: Configure NSX networking](#self-deployed-nsx-edge)  | 
| Diskless hosts | The NFS datastore doesn’t mount, or the VCF Installer rejects the specification or reports low capacity | 9.x |  [Diskless host symptoms](#troubleshoot-diskless)  | 
| Diskless hosts | Creating an FSx for OpenZFS volume fails with a `BadRequest` error, the datastore doesn’t show the new size, or the VCF Installer shows an inventory error after the first start | 9.x |  [Creating an FSx for OpenZFS volume fails with a `BadRequest` error](#troubleshoot-diskless-volume-tag), [The datastore doesn’t show the new size after a file system increase](#troubleshoot-diskless-size-refresh), and [The VCF Installer shows an inventory error after the first start](#troubleshoot-diskless-installer-inventory)  | 
| Status checks | The **Environment status** shows **Failed** for reachability, host count, key re-use, or key coverage | 5.2.x |  [Troubleshoot environment status checks](#troubleshoot-status-checks)  | 
| Connectors | A connector reachability check stays `FAILED`  | All |  [The connector is `ACTIVE`, but its reachability check stays `FAILED`](#troubleshoot-sd-connector-reachability) and [Troubleshoot environment status checks](#troubleshoot-status-checks)  | 
| Hosts | Host creation fails with a vCPU limit error | All |  [Host creation fails with a vCPU limit error](#troubleshoot-vcpu-limit)  | 
| Hosts | A vSphere HA isolation address message, a vSAN upgrade precheck failure, or an incompatible cluster image | All |  [Troubleshoot hosts](#troubleshoot-hosts)  | 
| Hosts | You receive a vSAN capacity warning in AWS Health | All |  [vSAN capacity warning received in AWS Health](#troubleshoot-vsan-capacity)  | 
| Hosts | SDDC Manager fails host validation during host commissioning | 5.2.x |  [SDDC Manager fails VCF host validation during host commissioning](#troubleshoot-sddc-failure-host-commission)  | 
| Hosts | Hosts can’t mount an FSx for ONTAP NFS datastore | All |  [Hosts can’t mount an FSx for ONTAP NFS datastore](#troubleshoot-nfs-ontap)  | 
| Hosts | You can’t reach an ESX host over the network | All |  [You can’t reach an ESX host over the network](#access-ec2-serial-console)  | 
| Networking | Amazon EVS can’t reach DNS servers or SDDC Manager | All |  [Amazon EVS can’t reach DNS servers or SDDC Manager](#troubleshoot-dns-reachability)  | 
| Networking | Route server routes don’t propagate, or the deployment fails because of the route server configuration | All |  [Routes don’t propagate or BGP sessions don’t come up](#troubleshoot-rs-routes) and [The deployment fails because of the VPC Route Server configuration](#troubleshoot-rs-deploy-fails)  | 
| Networking | NSX connectivity fails, or on-premises systems can’t reach the environment | All |  [NSX connectivity fails after the environment is deployed](#troubleshoot-rt-association) and [On-premises systems can’t reach the environment](#troubleshoot-tgw-routes)  | 
| Networking | Security group rules have no effect on VLAN subnet traffic | All |  [Security group rules have no effect on VLAN subnet traffic](#troubleshoot-nacl)  | 
| Networking | You can’t reach the VCF appliances after you add a distributed firewall rule | All |  [You can’t reach the VCF appliances after you add a distributed firewall rule](#troubleshoot-dfw-lockout)  | 
| Windows Server entitlements | An entitlement is `CREATE_FAILED`, `AT_RISK`, or `ENTITLEMENT_REMOVED`  | All |  [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements)  | 
| Identity and access | An IAM permission error when you use Amazon EVS | All |  [Troubleshoot Amazon EVS identity and access](#security-iam-troubleshoot)  | 

## Troubleshooting topics
<a name="troubleshooting-topics"></a>

The following pages cover troubleshooting and support.
+  [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed) 
+  [Troubleshoot environment status checks](#troubleshoot-status-checks) 
+  [Troubleshoot hosts](#troubleshoot-hosts) 
+  [Troubleshoot networking](#troubleshoot-networking) 
+  [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements) 
+  [Troubleshoot Amazon EVS identity and access](#security-iam-troubleshoot) 
+  [Get help from AWS Support and Broadcom](#get-help) 

**Topics**

## Troubleshoot Self-deployed deployments
<a name="troubleshoot-self-deployed"></a>

Use the following sections to fix problems in Self-deployed environments that run VMware Cloud Foundation (VCF) 9.x, from environment creation through VCF bringup and NSX Edge configuration. Some symptoms are specific to diskless hosts. For status checks, host operations, and VPC networking, see [Troubleshoot environment status checks](#troubleshoot-status-checks), [Troubleshoot hosts](#troubleshoot-hosts), and [Troubleshoot networking](#troubleshoot-networking).

In Self-deployed mode, Amazon Elastic VMware Service doesn’t manage the VCF installation.

### Environment, host, and connector symptoms
<a name="troubleshoot-sd-env-host"></a>

These symptoms occur when you create an environment, add hosts, or create connectors.

#### `create-environment` returns a `ValidationException` that names a parameter
<a name="troubleshoot-sd-validation-exception"></a>

The error names a parameter such as `licenseInfo`, `hosts`, `vcfHostnames`, `siteId`, or `connectivityInfo`.

 **Cause** 

These parameters aren’t supported when `vcfVersion` is `SELF_DEPLOYED`.

 **Solution** 

Remove the parameter that the error names from your request, and run the command again. For the request, see [Step 1: Create the environment](#self-deployed-create-env).

#### The environment stays in the `CREATING` state longer than expected
<a name="troubleshoot-sd-env-creating"></a>

 **Cause** 

The environment creation didn’t complete, and the cause isn’t visible in your account.

 **Solution** 

Open a support case with AWS Support, and include the environment ID.

#### `create-environment-host` fails with an ESX version error
<a name="troubleshoot-sd-esx-version"></a>

 **Cause** 

The ESX version string in the request is wrong, or your account doesn’t have access to the requested version.

 **Solution** 

1. Get the version strings that are available to you with `aws evs get-versions --region <region-code>`. Replace `<region-code>` with the Region where you created the environment.

1. Run `create-environment-host` again with a version string from the output.

For more information, see [Supported VCF versions and instance types](#versions-provided) and [Step 3: Add hosts](#self-deployed-add-hosts).

#### A host stays in the `CREATING` state or moves to `CREATE_FAILED`
<a name="troubleshoot-sd-host-stuck"></a>

A host usually reaches the `CREATED` state in about 9 minutes.

 **Cause** 

A likely cause is a missing or incorrect DNS record for the host.

 **Solution** 

1. Verify that a forward (A) record and a reverse (PTR) record exist for the host.

1. From an instance in the VPC, query each DNS server in your VPC DHCP option set for the forward and the reverse record. For example, run `nslookup <host-fqdn> <dns-server-ip>` and `nslookup <host-ip-address> <dns-server-ip>`.

1. Make sure that the IP address in the A record is inside the host management (`vmkManagement`) VLAN subnet, and that the FQDN is 62 characters or fewer.

1. If you use Route 53, make sure that the record changes are in sync.

1. Check the host state again after you fix a record. Don’t run `create-environment-host` again for the same host name while the host exists.

After you fix the records, the host reaches the `CREATED` state. If the records resolve correctly and the host stays in the same state, contact AWS Support and include the environment ID and the host name. For the DNS requirements, see [Step 2: Create DNS records](#self-deployed-host-dns-records).

#### `create-environment-connector` fails with a Secrets Manager access error
<a name="troubleshoot-sd-connector-secret"></a>

 **Cause** 

Amazon EVS can’t read the AWS Secrets Manager secret, because the secret or its AWS KMS key doesn’t carry the required tag.

 **Solution** 

Confirm that the secret and its AWS KMS encryption key are both tagged with `EvsAccess=true`, and create the connector again. For more information, see [Step 6: Create connectors](#self-deployed-create-connectors).

#### The connector is `ACTIVE`, but its reachability check stays `FAILED`
<a name="troubleshoot-sd-connector-reachability"></a>

Connector creation doesn’t validate reachability or credentials. Within 10 minutes after the connector reaches `ACTIVE`, the reachability check changes from `UNKNOWN` to `PASSED` or `FAILED`.

 **Cause** 

Amazon EVS can’t resolve the appliance FQDN, can’t reach the appliance, or can’t sign in with the stored credentials.

 **Solution** 

1. Confirm that the appliance FQDN resolves from the service access subnet through the DNS servers in your DHCP option set.

1. Confirm that the credentials in the Secrets Manager secret are current. If they changed, update the secret. For more information, see [Update a connector](#evs-env-update-connector).

1. Confirm that your network ACL rules allow traffic between the service access subnet and the appliance.

For more information, see [Monitor environment status](#evs-env-status-check) and [Troubleshoot networking](#troubleshoot-networking).

### VCF Installer and bringup symptoms
<a name="troubleshoot-sd-installer"></a>

These symptoms occur when you validate the deployment specification or run VCF bringup.

#### The gateway ping validation fails
<a name="troubleshoot-sd-gateway-ping"></a>

 **Cause** 

Amazon EVS VLAN subnet gateways don’t respond to ICMP ping from outside the subnet. The check fails even when routing is correct.

 **Solution** 

In a specification file, set `skipGatewayPingValidation` to `true`. In the VCF Installer wizard, acknowledge the failed gateway ping check and continue. For more information, see [VCF Installer settings for Amazon EVS](#vcf-installer-settings).

#### The ESX host thumbprint validation fails
<a name="troubleshoot-sd-thumbprint"></a>

 **Cause** 

The specification doesn’t contain the correct thumbprints for the hosts.

 **Solution** 

In a specification file, set `skipEsxThumbprintValidation` to `true`, or get the correct thumbprints for all hosts and enter them in the specification. In the wizard, thumbprint validation is mandatory. Review and accept the thumbprint at the hosts step to continue. For more information, see [VCF Installer settings for Amazon EVS](#vcf-installer-settings).

#### VCF bringup fails
<a name="troubleshoot-sd-bringup-fails"></a>

 **Cause** 

A setting in the deployment specification doesn’t match the Amazon EVS network, or a VCF component fails.

 **Solution** 

1. Compare your specification with the settings in [VCF Installer settings for Amazon EVS](#vcf-installer-settings). When you run bringup through the VCF Installer wizard, the wizard shows validation problems more clearly than the API.

1. Compare your specification with an example that the [spec-generator tool](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy/EVS-Deployment-Orchestrator/spec-generator) generates, in the `Deploy` folder of the [Solutions for Amazon EVS](https://github.com/aws/solutions-for-amazon-evs) repository on GitHub.

1. If bringup still fails, contact AWS Support. For VCF-specific requests, you can also contact Broadcom with your Broadcom support entitlement.

For the bringup procedure, see [Step 4b (local storage): Run VCF bringup](#sd-install-bringup). On diskless hosts, validation reports NTP warnings that you can acknowledge. For the list, see [Validate and deploy](#getting-started-diskless-deploy).

### NSX Edge and BGP symptoms
<a name="troubleshoot-sd-nsx"></a>

To check BGP peering, see [Verify BGP peering](#sd-nsx-verify).

#### The BGP session stays in the `Active` state
<a name="troubleshoot-sd-bgp-active"></a>

 **Cause** 

A firewall or a network ACL blocks TCP port 179.

 **Solution** 

Verify that your network ACL allows TCP 179 between the Edge uplink IP addresses and the route server endpoint IP addresses. Also verify that the security group of the route server endpoints allows inbound TCP 179. For more information, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge).

#### The BGP session flaps repeatedly
<a name="troubleshoot-sd-bgp-flaps"></a>

 **Cause** 

The MTU doesn’t match along the uplink path.

 **Solution** 

Make sure that the MTU is consistent along the NSX uplink path. The NSX uplink network uses MTU 1500. The overlay (TEP) networks use MTU 8500. For more information, see [Step 5: Configure NSX networking](#self-deployed-nsx-edge).

#### NSX overlay routes don’t appear in the VPC route table
<a name="troubleshoot-sd-bgp-routes"></a>

 **Cause** 

Route server propagation isn’t turned on for the route table, or the route table has no explicit subnet association.

 **Solution** 

Turn on route server propagation for the target route table, and make sure that the route table has at least one explicit subnet association. For more information, see [Troubleshoot networking](#troubleshoot-networking).

#### Only one BGP session is up
<a name="troubleshoot-sd-bgp-one-sided"></a>

 **Cause** 

One Edge node can’t reach both route server endpoints.

 **Solution** 

Verify that both Edge nodes can reach both route server endpoints. We recommend four route server peers in a full mesh, so that routes keep propagating when one endpoint is in maintenance. For more information, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

### Diskless host symptoms
<a name="troubleshoot-diskless"></a>

These symptoms apply to environments with diskless hosts, which use NFS principal storage. Diskless hosts are available only with VCF 9.x. For the deployment procedure, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

#### A diskless host doesn’t finish creation
<a name="troubleshoot-diskless-host-stuck"></a>

The host stays in the `CREATING` state or moves to `CREATE_FAILED`. The cause and the solution are the same as for hosts with local storage. For the steps, see [A host stays in the `CREATING` state or moves to `CREATE_FAILED`](#troubleshoot-sd-host-stuck).

#### The VCF Installer rejects the deployment specification
<a name="troubleshoot-diskless-spec"></a>

The VCF Installer validation fails for an environment with NFS principal storage and no vSAN.

 **Cause** 

The specification is missing an entry that an NFS principal datastore needs, or it uses a wrong value.

 **Solution** 

Check the error against the following table. The columns show the result that the VCF Installer reports and the solution.


| Result | Solution | 
| --- | --- | 
|  `MISSING_NET_SPEC_ENTRY` or `MISSING_DVS_SPEC_NET_ENTRY`  | Add an `NFS` network to `networkSpecs`, and add `NFS` to the `networks` list of the distributed switch. Declare it on the host management subnet. | 
|  `NOT_ENOUGH_IPS` or `MISSING_INCLUSION_RANGE`  | Add an `includeIpAddressRanges` entry to the `NFS` network. | 
| Release version is not compatible with the supported versions | Use one of the version strings from the list in the error message, and not a build number. | 
| Datastore not found on the host | Mount the principal datastore on the first host before you validate, and use the same server and path as in the specification. See [The NFS datastore doesn’t mount](#troubleshoot-diskless-mount). | 
| Capacity warning for the datastore | See [The VCF Installer reports insufficient datastore capacity](#troubleshoot-diskless-capacity). | 

For the specification, see [Create the deployment specification](#getting-started-diskless-spec). For the warnings that you can acknowledge, see [Validate and deploy](#getting-started-diskless-deploy).

#### The NFS datastore doesn’t mount
<a name="troubleshoot-diskless-mount"></a>

Mounting the Amazon FSx for OpenZFS datastore on an ESX host fails. For example, the Host Client or the host log reports that the mount request was denied.

 **Cause** 

A mount request that the server denies usually means that the host management subnet is missing from the NFS export list. A security group or network ACL that blocks NFS traffic also stops the mount.

 **Solution** 

1. Make sure that the NFS export list of the volume includes the CIDR block of the host management VLAN subnet, with the options `rw,crossmnt,no_root_squash`.

1. Make sure that the security group of the file system allows the NFS ports from the host management VLAN subnet.

1. If you use a network ACL for the VLAN subnets, make sure that it allows the NFS traffic in both directions.

1. Mount the datastore through the host management VMkernel adapter (`vmk0`).

1. Make sure that the NFS server is the endpoint IP address of the file system and the path is `/fsx/<volume-name>`.

For more information, see [Security group rules](#fsx-openzfs-security-group) and [article 446573](https://knowledge.broadcom.com/external/article/446573) in the Broadcom knowledge base.

#### The VCF Installer reports insufficient datastore capacity
<a name="troubleshoot-diskless-capacity"></a>

Validation reports that the datastore capacity is less than the required capacity, or recommends a larger capacity to stay under 80 percent utilization.

 **Cause** 

The datastore size that the ESX host reports is the capacity of the file system, or the quota of the volume if you set one.

 **Solution** 

1. Increase the capacity of the file system, and then raise the quota of the volume if you set one.

1. If you haven’t deployed VCF yet, unmount the datastore from the host and mount it again, so that the host reads the new size.

For the steps and for the limits after VCF is deployed, see [Increase the capacity](#fsx-openzfs-grow). For sizing guidance, see [Size the file system](#fsx-openzfs-sizing).

#### The datastore doesn’t show the new size after a file system increase
<a name="troubleshoot-diskless-size-refresh"></a>

The ESX host or the VCF Installer still shows the old datastore size. A storage rescan doesn’t change it.

 **Cause** 

A volume doesn’t use the new capacity of the file system until you raise its quota. Before VCF is deployed, the host also keeps the size that it read at mount time.

 **Solution** 

Use the solution for insufficient capacity. For the steps, see [The VCF Installer reports insufficient datastore capacity](#troubleshoot-diskless-capacity).

#### Creating an FSx for OpenZFS volume fails with a `BadRequest` error
<a name="troubleshoot-diskless-volume-tag"></a>

The `aws fsx create-volume` command fails. The error says that a tag in the request has the same key as a tag on the parent file system but a different value.

 **Cause** 

The file system copies its tags to volumes, and the file system and the volume have `Name` tags with different values.

 **Solution** 

Turn off the option to copy tags to volumes on the file system, and run the command again. The change causes no interruption. For the command, see [Create the file system](#fsx-openzfs-create-fs).

#### The VCF Installer shows an inventory error after the first start
<a name="troubleshoot-diskless-installer-inventory"></a>

You can sign in right after the VCF Installer appliance starts, but the inventory page shows an error. The list of releases is empty until you connect the Broadcom online depot.

 **Cause** 

The VCF Installer has no release information until it connects to the Broadcom online depot.

 **Solution** 

Connect the Broadcom online depot, as described in [Connect the software depot](#getting-started-diskless-depot). The inventory error might remain after you connect the Broadcom online depot, and it might last more than 30 minutes. The error doesn’t block the connection to the Broadcom online depot, the bundle downloads, or the validation of the specification. Continue with the next step without waiting for the page to clear.

If these steps don’t resolve the problem, see [Get help from AWS Support and Broadcom](#get-help) for who to contact and what to include in a case.

## Troubleshoot environment status checks
<a name="troubleshoot-status-checks"></a>

Amazon Elastic VMware Service runs status checks on your environment and reports the result in the environment status. This page applies to both deployment modes.

### Environment status checks
<a name="troubleshoot-env-status"></a>

The environment status combines four checks. They validate the connection to SDDC Manager, the host count, and the VMware Cloud Foundation (VCF) license key. The SDDC Manager connector is the management connector for VCF 5.2.x. In Self-deployed environments that run VCF 9.x, Amazon EVS connects to VCF through the VCF Operations connector and runs a reachability check on each connector. The columns show the check and what it verifies.


| Check | What it verifies | 
| --- | --- | 
| Reachability | Amazon EVS has a persistent connection to SDDC Manager. | 
| Host count | The environment meets the minimum host count of your VCF deployment topology. | 
| Key re-use | The VCF license key isn’t in use by another Amazon EVS environment. | 
| Key coverage | The VCF license key that is assigned to vCenter Server covers the vCPU cores and vSAN storage capacity of all hosts. | 
| Connector reachability | Amazon EVS can reach and sign in to the appliance of each connector. | 

For a connector whose reachability check stays `FAILED` in a Self-deployed environment, see [The connector is `ACTIVE`, but its reachability check stays `FAILED`](#troubleshoot-sd-connector-reachability). For a vCenter connector that affects Windows Server entitlements, see [Troubleshoot Windows Server entitlements](#troubleshoot-entitlements).

#### Review the status check results
<a name="view-env-status"></a>

To review the status check results, do the following:

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. In the navigation pane, choose **Environments**, and then select your environment.

1. Choose the **Details** tab to see an overview of the environment.

1. Check the **Environment status**. Hover over the field to see the result of each environment status check.

For how to read the status with the AWS CLI, see [Monitor environment status](#evs-env-status-check).

### The reachability check failed
<a name="troubleshoot-reachability"></a>

 **Cause** 

Amazon EVS can’t reach SDDC Manager. When this check fails, Amazon EVS can’t validate the environment status, and you can’t add hosts to the environment. A reachability failure also makes the key re-use and key coverage checks fail, and the host count check returns **Unknown**.

 **Solution** 

1. Make sure that your certificates are valid and unexpired. You can use the SDDC Manager UI or the vSphere Client to manage certificates in a VCF environment. After deployment, we recommend that you replace all certificates of the VCF management domain. For more information, see [Managing Certificates in VMware Cloud Foundation](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/certificate-management-admin.html) in the Broadcom documentation.

1. Fix the reachability of your DNS servers, your DNS records, and your firewall rules, as described in [Amazon EVS can’t reach DNS servers or SDDC Manager](#troubleshoot-dns-reachability).

### The host count check failed
<a name="troubleshoot-host-count"></a>

 **Cause** 

The environment has fewer hosts than your VCF deployment topology requires.

 **Solution** 

Add hosts so that your environment meets the minimum requirement. For the minimum host counts by VCF version and deployment topology, see the Broadcom documentation for [VCF 9.x](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1.html) or [VCF 5.2.x](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2.html). To add a host, see [Add a host (VCF 5.x)](#evs-env-create-host).

### The key re-use check failed
<a name="troubleshoot-key-reuse"></a>

The request to create the environment returns an error that says the environment couldn’t be created.

 **Cause** 

You can use a VCF license key for only one Amazon EVS environment. The check fails if the environment creation request contains a VCF license key that another environment already uses.

 **Solution** 

Review your license settings in SDDC Manager, and replace any license key that is already in use with an unused key.

**Important**  
Manage VCF solution and vSAN license keys in the SDDC Manager user interface. Amazon EVS requires valid VCF solution and vSAN license keys in SDDC Manager to function. Assign the keys to your hosts and vSAN cluster in the vSphere Client. Make sure that the keys appear on the licensing screen of SDDC Manager.

For more information, see [VCF licensing](#vcf-license-mgmt).

### The key coverage check failed
<a name="troubleshoot-key-coverage"></a>

The request to create the environment returns an error that says the environment couldn’t be created.

 **Cause** 

The cause is one of the following:
+ The VCF license keys aren’t assigned to vCenter Server. You must assign a license key to vCenter Server before its evaluation period expires or before the assigned license key expires.
+ The VCF license keys don’t cover the vCPU cores and vSAN storage capacity that your hosts need. The requirements for the VCF solution key (including the minimum core count) and the vSAN license key (including the minimum vSAN capacity) depend on the instance type.

 **Solution** 

1. Review the license key assignments in SDDC Manager, and assign a license key to vCenter Server if none is assigned.

1. If the license keys don’t cover your hosts, add vSAN license keys in SDDC Manager until they cover your hosts. For the thresholds for your configuration, see [VCF licensing](#vcf-license-mgmt).

Manage license keys in the SDDC Manager user interface, as described in [The key re-use check failed](#troubleshoot-key-reuse).

If these steps don’t resolve the problem, see [Get help from AWS Support and Broadcom](#get-help) for who to contact and what to include in a case.

## Troubleshoot hosts
<a name="troubleshoot-hosts"></a>

Host problems might occur when you create, add, upgrade, or connect to ESX hosts in an Amazon Elastic VMware Service environment. This page applies to both deployment modes. For host creation problems during a Self-deployed deployment, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

### Host creation fails with a vCPU limit error
<a name="troubleshoot-vcpu-limit"></a>

When you create an environment or add a host, the host creation fails with an error such as `VcpuLimitExceeded`.

 **Cause** 

Your Amazon EC2 Running On-Demand Standard Instances quota is measured in vCPUs, and it doesn’t cover the vCPUs of all the EC2 instances that you use. Host creation also fails when the host count per Amazon EVS environment quota is lower than the number of hosts that you create.

 **Solution** 

1. Check both quotas, and compare them with the hosts that you plan to run. For the quotas and the vCPUs of each instance type, see [Service quotas for Amazon EVS](#service-quotas-evs) and [Supported VCF versions and instance types](#versions-provided).

1. Request an increase for the quota that is too low. For more information, see [Request an increase](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html#request-increase) in the *Amazon EC2 User Guide*.

1. Create the host again after the increase is approved.

### vSphere HA agent on this host could not reach isolation address
<a name="troubleshoot-vsphere-ha-agent-ipv6"></a>

In the vSphere Client, with the ESX host selected, you see the message "vSphere HA agent on this host could not reach isolation address <IPv6 address>". The host can show a yellow warning status while the cluster stays healthy.

 **Cause** 

The vSphere HA agent on the host can’t reach the default IPv6 isolation address that vSphere HA uses for heartbeat checks. The message occurs because Amazon EVS doesn’t support IPv6.

 **Solution** 

No action is needed. The message doesn’t indicate a problem, and the absence of IPv6 support doesn’t affect the core functions of vSphere HA.

### vSAN upgrade prechecks fail for the ESX host cluster
<a name="troubleshoot-vsan-precheck-fail"></a>

This section applies to hosts with local storage, which use vSAN.

When you upgrade the ESX host cluster with SDDC Manager, vSAN disk-related prechecks can fail.

 **Cause** 

Amazon EVS uses vSAN Express Storage Architecture (ESA), and these upgrade prechecks don’t apply to vSAN ESA.

 **Solution** 

Follow the guidance in [article 369423](https://knowledge.broadcom.com/external/article/369423/upgrade-prechecks-fails-with-esxi-cluste.html) in the Broadcom knowledge base. For how to update and upgrade your environment, see [Update and upgrade](#evs-lifecycle-mgmt).

### vSAN capacity warning received in AWS Health
<a name="troubleshoot-vsan-capacity"></a>

This section applies to hosts with local storage, which use vSAN.

You received an AWS Health notification with the event type `AWS_EVS_VSAN_CAPACITY_WARNING`. You can view the notification in the [AWS Health Dashboard](https://docs.aws.amazon.com/health/latest/ug/aws-health-account-views.html). To receive these notifications automatically, use Amazon EventBridge. For more information, see [Monitoring events in AWS Health with Amazon EventBridge](https://docs.aws.amazon.com/health/latest/ug/cloudwatch-events-health.html).

Amazon EVS checks the vSAN utilization of each cluster once an hour. Amazon EVS sends this notification when a cluster reaches 70% utilization or more, or when it can’t read a cluster’s capacity.

Amazon EVS closes the notification automatically after the issue is resolved. These warnings appear only in AWS Health and don’t affect your environment status or connector check results.

Act on this notification promptly. vSAN needs free capacity to rebuild data after a host or device failure and to complete operations such as rebalancing and storage policy changes. If vSAN can’t rebuild data in a timely manner, you run the risk of cascading VM failures. A vSAN capacity warning might indicate one of the following issues:
+ A cluster’s vSAN utilization is 70% or more. To reduce utilization, do one or more of the following:
  + In the vSphere Client, choose your cluster, and then choose **Monitor**, **vSAN**, **Capacity** to see what is using the space.
  + Delete virtual machines and snapshots that you no longer need.
  + Provision additional external storage, such as [Amazon FSx for NetApp ONTAP](#fsx-ontap), and move virtual machines to it.
  + Add hosts to your environment to increase capacity. For more information, see [Add a host (VCF 5.x)](#evs-env-create-host).
+ The credentials in your environment connector don’t have permission to read cluster capacity. Amazon EVS reports a permission problem when the appliance denies the request, or when the appliance returns no clusters because the account can’t see them. If this is the issue, verify the access of the account that the connector uses. If you create a new user or API key, update the credentials in AWS Secrets Manager as described in [Update a connector](#evs-env-update-connector).
  + For an SDDC Manager connector, verify that the API key belongs to a service user with the `VIEWER` role.
  + For a VCF Operations connector, verify that the user has a read-only role, such as `ReadOnly`, with a scope that includes your clusters, such as `All Objects`.
+ Amazon EVS can’t read a cluster’s capacity. Amazon EVS can reach the appliance and has permission, but the appliance doesn’t return valid capacity values for a cluster. If this is the issue, in SDDC Manager or VCF Operations, verify that the cluster is listed and reports its vSAN capacity.

If you are still unable to resolve the issue after following this guidance, contact AWS Support for further assistance.

### Adding a host fails because the cluster image is incompatible
<a name="troubleshoot-cluster-image"></a>

Adding a host to your environment fails with an error that the new host isn’t compatible with your cluster image.

<a name="troubleshoot-add-host-failure-cluster-image"></a> **Cause** 

A new host has the latest available version of the Amazon EVS custom vendor add-on. If the other hosts in your cluster use an older add-on version, the new host doesn’t match the cluster image.

 **Solution** 

Use vSphere Lifecycle Manager (vLCM) to extract the latest add-on version from the new host, and update the cluster image.

**Important**  
Updating the cluster hosts to the latest add-on version requires host reboots.

 **In the vSphere Client** 

1. Go to the **Hosts and Clusters** inventory in vCenter Server.

1. To extract the add-on from the new host, create a temporary empty cluster.

1. Under **Basics**, select **Import image from an existing host in the vCenter Inventory**, and create the cluster. Leave all other settings at their defaults.

1. After the temporary cluster is created with the extracted image, delete the temporary cluster. The add-on is now available in your vLCM depot.

1. Go to your environment cluster, and choose the **Updates** tab.

1. Edit your cluster image, and change the add-on version to the version that you extracted.

1. Choose **Save**.

 **In SDDC Manager** 

1. Retry the failed add host task. The task remediates your cluster hosts and updates all of them to the latest add-on version.

For where to get the add-on, see [Use the Amazon EVS custom add-on depot](#addon-depot).

### SDDC Manager fails VCF host validation during host commissioning
<a name="troubleshoot-sddc-failure-host-commission"></a>

This section applies to EVS-deployed environments that run VMware Cloud Foundation (VCF) 5.2.x.

SDDC Manager fails during VCF host validation in the commission hosts step.

 **Cause** 

You updated the ESX version of your cluster after the Amazon EVS environment was deployed, and the new host runs a different ESX version.

 **Solution** 

Use vLCM to upgrade ESX on the new host to the ESX version of your cluster.

These steps require that you add the host to vCenter Server temporarily, outside of SDDC Manager.

**Warning**  
If you use vLCM for any operation other than an ESX upgrade on this host, the host might become unusable. If that happens, you must delete the host and create a new Amazon EVS host.

 **In the vSphere Client** 

1. Go to the **Hosts and Clusters** inventory in vCenter Server.

1. Add the host temporarily to your virtual data center, and select **manage host with an image**. You remove the host in a later step. For more information, see [How to Add a Host to Your vSphere Data Center or Folder](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/organizing-your-inventory/add-a-host-to-a-folder-or-a-data-center.html) in the Broadcom documentation.

1. On the **Updates** tab of the host, edit the host image to match the ESX version of your cluster, and upgrade the host.

1. After the upgrade completes, remove the host from your vCenter Server inventory. For more information, see [How to Remove an ESX Host from Your vCenter Server Instance](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-and-host-management/managing-hosts-in-the-vsphere-client/remove-a-host-from-vcenter-server.html) in the Broadcom documentation.

 **In SDDC Manager** 

1. Commission the host. For more information, see [Commission Hosts](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/map-for-administering-vcf-5-2/host-management-admin/commission-hosts-admin.html) in the Broadcom documentation.

1. After the host is commissioned, add the host to your cluster.

For the full procedure, see [Add a host (VCF 5.x)](#evs-env-create-host).

### Hosts can’t mount an FSx for ONTAP NFS datastore
<a name="troubleshoot-nfs-ontap"></a>

 **Cause** 

A security group rule or a route between the service access subnet and the VMkernel management VLAN subnet is missing.

 **Solution** 

Check the security group rules of the Amazon FSx for NetApp ONTAP file system. Also check the routes between the service access subnet and the VMkernel management VLAN subnet. For the configuration, see [Configure FSx for ONTAP as an NFS datastore](#config-fsx-nfs-datastore). For diskless hosts (VCF 9.x) with FSx for OpenZFS, see [The NFS datastore doesn’t mount](#troubleshoot-diskless-mount).

### You can’t reach an ESX host over the network
<a name="access-ec2-serial-console"></a>

A host has a boot problem, a network configuration problem, or another issue, and you can’t reach the host over the network.

 **Cause** 

The host has lost its network configuration or doesn’t boot correctly. Amazon EVS disables remote SSH by default.

 **Solution** 

Use the Amazon EC2 serial console. By default, Amazon EVS enables the ESX Shell on newly deployed hosts, so that you can access the serial port of the EC2 instance through the EC2 serial console. The serial console doesn’t require your instance to have network connectivity. You enter commands as if your keyboard and monitor were attached directly to the serial port of the instance.

**Note**  
The EC2 serial console is the only mechanism that Amazon EVS supports to access the Direct Console User Interface (DCUI) of an ESX host locally.

**Warning**  
Don’t use the Amazon EC2 console to stop, start, or terminate Amazon EVS hosts. On hosts with local storage, these actions cause the loss of the vSAN data on the instance store volumes. To create, replace, or delete a host, use the Amazon EVS console, the AWS CLI, or the Amazon EVS API. For more information, see [Manage hosts](#host-management).

Before you begin, complete the prerequisites for the EC2 serial console. For more information, see [Prerequisites for the EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console-prerequisites.html) in the *Amazon EC2 User Guide*.

To connect to the serial console, do the following.

1. Grant serial console access at the account level, and configure IAM policies that grant access to your users. For more information, see [Configure access to the EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-access-to-serial-console.html) in the *Amazon EC2 User Guide*.

1. Make sure that the instance state is `running`. You can’t connect to the serial console when the instance is in the `pending`, `stopping`, `stopped`, `shutting-down`, or `terminated` state. For more information, see [Amazon EC2 instance state change](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html) in the *Amazon EC2 User Guide*.

1. Connect to the serial console with the EC2 console or the AWS CLI. For more information, see [Connect to the EC2 Serial Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-to-serial-console.html) in the *Amazon EC2 User Guide*.

To use the remote ESX Shell over SSH instead, see [Remote ESX Shell Access with SSH](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-sdks-tools/8-0/getting-started-with-esxcli/running-host-management-commands-in-the-esxi-shell/remote-esxi-shell-access-with-ssh.html) in the Broadcom documentation.

If these steps don’t resolve the problem, see [Get help from AWS Support and Broadcom](#get-help) for who to contact and what to include in a case.

## Troubleshoot networking
<a name="troubleshoot-networking"></a>

Network problems in an Amazon Elastic VMware Service environment might involve DNS, Amazon VPC Route Server, route tables, network ACLs, or the NSX distributed firewall. These solutions apply to both deployment modes. For BGP symptoms between the NSX Edge nodes and VPC Route Server in a Self-deployed environment, see [NSX Edge and BGP symptoms](#troubleshoot-sd-nsx).

### Amazon EVS can’t reach DNS servers or SDDC Manager
<a name="troubleshoot-dns-reachability"></a>

Amazon EVS needs a persistent connection to SDDC Manager and to the DNS servers in your VPC DHCP option set. When the connection fails, the reachability status check or a connector reachability check fails. Amazon EVS can then no longer validate the environment status, and you can lose environment access.

 **Cause** 

The DNS servers aren’t reachable from the service access subnet, a DNS record is missing or wrong, or a network ACL blocks the traffic.

 **Solution** 

1. Make sure that your DNS servers are reachable from the service access subnet.

1. Make sure that the DNS records are valid, and that no duplicate host names or IP addresses exist.

1. Make sure that your network ACL rules allow TCP and UDP access to the DNS servers, and HTTPS and SSH access to the host management and management VM VLAN subnets.

1. If you use custom DNS domain names in a private hosted zone in Route 53, make sure that the `enableDnsHostnames` and `enableDnsSupport` attributes of the VPC are both `true`. For more information, see [DNS attributes for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#vpc-dns-support).

1. Don’t use both Route 53 and a custom DNS server in the DHCP option set, because the combination can cause unexpected behavior.

For the DNS settings, see [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp). For the status check, see [The reachability check failed](#troubleshoot-reachability).

### The deployment fails because of the VPC Route Server configuration
<a name="troubleshoot-rs-deploy-fails"></a>

 **Cause** 

The route server configuration doesn’t meet the Amazon EVS requirements.

 **Solution** 

Check that your configuration meets each of the following requirements, and deploy again:
+ At least two route server endpoints are in the service access subnet.
+ The route server peer ASN matches the NSX Edge peer ASN of the Tier-0 gateway.
+ Each route server peer uses a unique, unused IP address from the NSX uplink VLAN CIDR block that you specify when you create the environment.
+ Every route table that the route server propagates to has at least one explicit subnet association.

For the setup, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

### Routes don’t propagate or BGP sessions don’t come up
<a name="troubleshoot-rs-routes"></a>

After deployment, NSX routes don’t appear in your VPC route tables, or the BGP sessions with the route server stay down.

 **Cause** 

A propagated route table has no explicit subnet association, the peer ASNs don’t match, or two peers use the same IP address. BGP route advertisement fails if a propagated route table has no explicit subnet association.

 **Solution** 

1. Verify that each propagated route table has an explicit subnet association.

1. Check that the peer ASN on the route server peers matches the ASN of the NSX Tier-0 gateway.

1. Confirm that each route server peer IP address is unique.

1. Review the route propagation status in your route tables.

1. Use VPC Route Server peer logging to monitor BGP session health and find connection problems. For more information, see [Route server peer logging](https://docs.aws.amazon.com/vpc/latest/userguide/route-server-peer-logging.html) in the * Amazon VPC User Guide*.

Amazon EVS supports only the default BGP keepalive mechanism for peer liveness detection. Amazon EVS doesn’t support multi-hop Bidirectional Forwarding Detection (BFD). For the route server setup, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

### NSX connectivity fails after the environment is deployed
<a name="troubleshoot-rt-association"></a>

 AWS resources can’t reach virtual machines on NSX network segments.

 **Cause** 

The Amazon EVS VLAN subnets are implicitly associated with the VPC main route table, and you haven’t associated them explicitly with a route table. NSX connectivity fails if the VLAN subnets aren’t explicitly associated with a VPC route table.

 **Solution** 

Explicitly associate each Amazon EVS VLAN subnet with a route table in your VPC. A public HCX VLAN subnet needs a public route table. For the steps, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).

### On-premises systems can’t reach the environment
<a name="troubleshoot-tgw-routes"></a>

 **Cause** 

The transit gateway route tables don’t contain the VPC CIDR blocks that the Amazon EVS environment created. Transit Gateway doesn’t learn NSX overlay routes from VPC Route Server. Amazon EVS doesn’t support connectivity through an Direct Connect private virtual interface, or through an AWS Site-to-Site VPN connection that ends directly in the underlay VPC.

 **Solution** 

1. Update the transit gateway route tables with the VPC CIDR blocks that the environment created. Add static routes for the NSX overlay networks.

1. If you use Direct Connect, update your allowed prefixes so that the Direct Connect gateway sends and receives the updated routes.

For the steps, see [Set up VPC Route Server and on-premises connectivity](#network-route-server).

### Security group rules have no effect on VLAN subnet traffic
<a name="troubleshoot-nacl"></a>

 **Cause** 

Amazon EC2 security groups don’t function on elastic network interfaces that are attached to Amazon EVS VLAN subnets. Until you change the association, the VLAN subnets use the default network ACL of the VPC.

 **Solution** 

1. Find the network ACL that a VLAN subnet uses. In the following command, replace `<subnet-id>` with the ID of the VLAN subnet.

   ```bash
   aws ec2 describe-network-acls \
     --filters Name=association.subnet-id,Values=<subnet-id> \
     --query 'NetworkAcls[0].NetworkAclId' \
     --output text
   ```

   The command returns the ID of the network ACL.

1. Make sure that the network ACL allows the traffic that Amazon EVS requires. If it doesn’t, the environment deployment fails, and an existing environment can have a degraded compliance status.

1. Associate your network ACL with the VLAN subnets, if it isn’t associated.

For the required traffic and the association steps, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

### You can’t reach the VCF appliances after you add a distributed firewall rule
<a name="troubleshoot-dfw-lockout"></a>

You added an NSX distributed firewall rule, such as a rule that denies all traffic, and you can no longer reach your management appliances.

 **Cause** 

The distributed firewall rule applies to the traffic that you use to reach the appliances.

 **Solution** 

You configure and own the NSX distributed firewall rules in your environment. For how distributed firewall rules work, see [Add a Distributed Firewall](https://techdocs.broadcom.com/us/en/vmware-cis/nsx/nsxt-dc/3-2/administration-guide/security/distributed-firewall/add-a-distributed-firewall.html) in the Broadcom documentation. For help with NSX configuration, you can contact Broadcom with your Broadcom support entitlement, or contact AWS Support.

If these steps don’t resolve the problem, see [Get help from AWS Support and Broadcom](#get-help) for who to contact and what to include in a case.

## Troubleshoot Windows Server entitlements
<a name="troubleshoot-entitlements"></a>

A Windows Server entitlement in an Amazon Elastic VMware Service environment might fail to create, enter the `AT_RISK` status, or be removed. This page applies to both deployment modes. For how entitlements work, see [Windows Server entitlements](#entitlements).

### Entitlement status is `CREATE_FAILED`
<a name="create-entitlement-failure-reasons"></a>

When you create an entitlement, Amazon EVS validates the configuration of each virtual machine (VM). If a VM fails validation, its entitlement status is set to `CREATE_FAILED` with error details. After you fix the underlying issue for that VM, you must create the entitlement again. For the procedure, see [Create Windows Server entitlements](#create-entitlements).

**Note**  
Amazon EVS deletes entitlements in the `CREATE_FAILED` state after 90 days.

Read the error details of the entitlement, and then find the cause in the following sections.

#### VM not found
<a name="create-entitlement-failure-vm-not-found"></a>

The VM ID (Managed Object ID) that you specified isn’t in the vCenter inventory that the connector targets.

 **Cause** 

The VM ID is incorrect, or the VM doesn’t exist in the vCenter Server appliance that the connector points to.

 **Solution** 

1. Verify that the VM ID is correct. To find the Managed Object ID of a VM, sign in to vCenter Server and select the VM. The ID appears in the browser address bar as part of the URL.

1. Confirm that the VM exists in the vCenter Server that the connector targets.

1. Create the entitlement again.

#### VM is disconnected, orphaned, or inaccessible
<a name="create-entitlement-failure-vm-not-connected"></a>

The VM is in a disconnected, orphaned, or inaccessible state in vCenter Server.

 **Cause** 

An underlying host or network issue typically causes this state.

 **Solution** 

1. Check the state of the VM in vCenter Server.

1. Resolve the host or network issue so that the VM is no longer disconnected, orphaned, or inaccessible.

1. Create the entitlement again.

#### Unsupported guest operating system
<a name="create-entitlement-failure-unsupported-os"></a>

The guest operating system in the VM settings in vCenter Server must be Windows Server 2016 or later. Validation fails if the guest operating system is any of the following:
+ Not Windows.
+ A Windows client operating system, such as Windows 10 or Windows 11.
+ A Windows Server version earlier than 2016.

 **Cause** 

The guest operating system in the VM settings isn’t a supported Windows Server version.

 **Solution** 

1. Verify the guest operating system in the VM settings in vCenter Server.

1. Change the VM settings to a supported Windows Server version (2016 or later).

1. Create the entitlement again.

### Entitlement status is `AT_RISK`
<a name="troubleshoot-entitlement-at-risk"></a>

An entitlement enters the `AT_RISK` status when the reachability check of its vCenter connector fails. The failure affects the entitlements of all Windows Server VMs under the affected vCenter Server appliance.

You have 8 hours from the time the vCenter reachability check starts failing to restore the connection:
+ If you restore the connection within the 8-hour grace period, Amazon EVS recovers all events in that period and keeps your Windows Server VM entitlements with no further action.
+ If you don’t restore the connection within the 8-hour grace period, Amazon EVS removes the entitlements and changes their status to `ENTITLEMENT_REMOVED`. Amazon EVS also sends you a notification through the AWS Health Dashboard. Windows Server license usage tracking stops as of the time the connection was lost.

 **Cause** 

Amazon EVS can’t reach or sign in to the vCenter Server appliance through the connector.

 **Solution** 

1. Verify that the connector state is `ACTIVE` and that its reachability check status is `FAILED`.

1. Verify that the appliance credentials stored in AWS Secrets Manager are current and correct. If the credentials have changed, update them. For more information, see [Rotate vCenter connector credentials](#rotate-vcenter-credentials).

1. Fix the DNS records of the appliance FQDN, the reachability of your DNS servers, and your firewall rules, as described in [Amazon EVS can’t reach DNS servers or SDDC Manager](#troubleshoot-dns-reachability).

1. Make sure that the appliance is running and accessible.

### Entitlement status is `ENTITLEMENT_REMOVED`
<a name="troubleshoot-entitlement-removed"></a>

The `ENTITLEMENT_REMOVED` status means that Amazon EVS has removed the entitlement for the VM and stopped tracking Windows Server license usage for it.

**Note**  
Amazon EVS deletes entitlements that remain in the `ENTITLEMENT_REMOVED` status for 90 days.

 **Cause** 

The status can result from one of the following:
+ The vCenter connector reachability failure lasted longer than the 8-hour grace period, so the entitlement moved from `AT_RISK` to `ENTITLEMENT_REMOVED`. For more information, see [Entitlement status is `AT_RISK`](#troubleshoot-entitlement-at-risk).
+ The VM is disconnected, orphaned, or inaccessible, which typically indicates an underlying host or network issue.
+ The VM is no longer in the vCenter inventory.
+ The guest operating system of the VM was changed to an unsupported version. Amazon EVS supports Windows Server 2016 or later.

 **Solution** 

Check the error details of the entitlement to find the specific cause. Then, based on the cause, do the following:

1. If the vCenter reachability failure exceeded the grace period, restore the vCenter Server connection.

1. If the VM is disconnected, orphaned or inaccessible, check the connection state of the VM in vCenter Server. Resolve the underlying issue so that the VM returns to a compliant state. A VM can become disconnected if its host loses its connection to vCenter Server and vSphere HA fails to recover the VM on another host. A VM can become orphaned because of storage connectivity issues or interrupted migrations.

1. If the VM is no longer in the vCenter inventory, restore the VM in vCenter Server.

1. If the VM has an unsupported guest operating system, update the guest operating system to a supported version.

After you resolve the underlying issue in your VMware Cloud Foundation (VCF) environment, Amazon EVS validates the affected entitlements and restores them automatically shortly afterward. Entitlements resume from the time when Amazon EVS validates that the VMs are compliant again.

If these steps don’t resolve the problem, see [Get help from AWS Support and Broadcom](#get-help) for who to contact and what to include in a case.

## Troubleshoot Amazon EVS identity and access
<a name="security-iam-troubleshoot"></a>

Use the following information to help you diagnose and fix common issues that you might encounter when working with Amazon Elastic VMware Service and IAM. For other problems, see [Troubleshooting](#troubleshooting).

### I receive an AccessDeniedException
<a name="security-iam-troubleshoot-access-denied"></a>

You receive an error like the following when you call an Amazon EVS API operation.

```text
An error occurred (AccessDeniedException) when calling the CreateEnvironment operation:
User: arn:aws:iam::111122223333:user/user-name is not authorized to perform:
evs:CreateEnvironment on resource: arn:aws:evs:region-code:111122223333:environment/my-env
```

 **Cause**: The IAM principal credentials that you’re using don’t have the required permissions to make that call. In the preceding example message, the IAM user doesn’t have permissions to call the Amazon EVS `CreateEnvironment` API operation.

 **Solution**: Ask your IAM administrator to grant the principal the permissions for the operation. To provide Amazon EVS administrator permissions to an IAM principal, see [Amazon EVS identity-based policy examples](#security-iam-id-based-policy-examples). For the role that Amazon EVS administrators use, see [Create an IAM role for Amazon EVS administrators](#setup-iam-role).

For more general information about IAM, see [Control access to AWS resources using policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_controlling.html) in the *IAM User Guide*.

### I want to give other accounts access to my Amazon EVS resources
<a name="security-iam-troubleshoot-cross-account-access"></a>

You can create a role that users in other accounts or people outside of your organization can use to access your resources. You can specify who is trusted to assume the role. For services that support resource-based policies or access control lists (ACLs), you can use those policies to grant people access to your resources. Amazon EVS does not support resource-based policies or ACLs, so use a role.

To learn more, see the following:
+ To learn whether Amazon EVS supports these features, see [How Amazon EVS works with IAM](#security_iam_service-with-iam).
+ To learn how to provide access to your resources across AWS accounts that you own, see [Providing access to an IAM user in another AWS account that you own](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_aws-accounts.html) in the *IAM User Guide*.
+ To learn how to provide access to your resources to third-party AWS accounts, see [Providing access to AWS accounts owned by third parties](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) in the *IAM User Guide*.
+ To learn how to provide access through identity federation, see [Providing Access to Externally Authenticated Users (Identity Federation)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_federated-users.html) in the *IAM User Guide*.
+ To learn the difference between using roles and resource-based policies for cross-account access, see [How IAM roles differ from resource-based policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_compare-resource-policies.html) in the *IAM User Guide*.

## Get help from AWS Support and Broadcom
<a name="get-help"></a>

Contact AWS Support or Broadcom when the troubleshooting pages in this chapter don’t resolve your problem.

### Broadcom and AWS Support guidance
<a name="broadcom-aws-support-guidance"></a>

 AWS provides support for Amazon Elastic VMware Service and its associated infrastructure services, including VMware Cloud Foundation (VCF). For VCF configuration guidance, or for issues with other VMware products such as VCF Operations, HCX or NSX, you can also contact Broadcom with your Broadcom support entitlement. For more information, see the [Broadcom Support Portal](https://support.broadcom.com/).

Amazon EVS requires an AWS Business, AWS Enterprise On-Ramp, or AWS Enterprise Support plan. For more information, see [Compare AWS Support Plans](https://aws.amazon.com/premiumsupport/plans).

### Who to contact by problem and mode
<a name="get-help-by-problem"></a>

The following table shows who to contact for each type of problem. The columns show the problem area, the contact for Self-deployed environments that run VCF 9.x and the contact for EVS-deployed environments (VCF 5.2.x).


| Problem area | VCF 9.x (Self-deployed) | VCF 5.2.x (EVS-deployed) | 
| --- | --- | --- | 
| Amazon EVS environments, hosts, connectors, entitlements, the Amazon EVS API and console, VLAN subnets, VPC networking, Amazon VPC Route Server, and the bare metal EC2 hosts |  AWS Support |  AWS Support | 
| VCF installation | You install VCF, and Amazon EVS doesn’t manage the installation. Contact AWS Support for any Amazon EVS issue. For VCF-specific requests, you can also contact Broadcom with your Broadcom support entitlement. | Amazon EVS does the initial VCF installation. Contact AWS Support. | 
| VCF configuration, and VMware products such as VCF Operations, HCX, or NSX |  AWS Support. You can also contact Broadcom with your Broadcom support entitlement. |  AWS Support. You can also contact Broadcom with your Broadcom support entitlement. | 
| VCF subscriptions and license keys | Broadcom, through the Broadcom Support Portal. | Broadcom, through the Broadcom Support Portal. | 

For the division of responsibilities between you and AWS in each mode, see [Shared responsibility and support](#plan-shared-responsibility).

### Open a case with AWS Support
<a name="get-help-open-case"></a>

Open the case in the AWS Support Center. For the steps, see [Creating support cases and case management](https://docs.aws.amazon.com/awssupport/latest/user/case-management.html) in the * AWS Support User Guide*.

**Important**  
Don’t include passwords, secret values, or other credentials in a case.

Include the following information, so that AWS Support can find your resources:
+ The AWS Region and the environment ID
+ The host name and host ID, for a problem with a host
+ The connector ID, for a problem with a connector
+ The VM ID and the error details of the entitlement, for a problem with a Windows Server entitlement
+ The deployment mode and the VCF version of the environment
+ The command or console action that failed, the full error message, and the time when the problem started
+ The troubleshooting steps that you have already tried

### Open a case with Broadcom
<a name="get-help-broadcom-case"></a>

To open a case with Broadcom, sign in to the [Broadcom Support Portal](https://support.broadcom.com/) with the account that holds your Broadcom support entitlement. For the information that Broadcom needs, see the Broadcom Support Portal.

# Reference
<a name="reference-overview"></a>

The Plan, Deploy, and Operate chapters link to these lookup tables instead of repeating their rows. Start with [Supported VCF versions and instance types](#versions-provided) and [VCF feature support on Amazon EVS](#vcf-feature-support), because most design choices depend on them.

The procedures in [Deploy in Self-deployed mode](#getting-started-self-deployed) deploy VCF 9.x. Pages in [Deploy in EVS-deployed mode](#evs-deployed-overview) cover VCF 5.2.x. A page with no line and outside those sections covers both.

Use the following pages to look up each topic.
+  [Supported VCF versions and instance types](#versions-provided) lists the VCF versions, ESX builds, and Amazon EC2 instance types that Amazon EVS supports.
+  [VCF feature support on Amazon EVS](#vcf-feature-support) shows whether Amazon EVS supports a VCF feature, and under which conditions.
+  [Service quotas for Amazon EVS](#service-quotas-evs) gives the Amazon EVS quota values and the Amazon EC2 quotas to check before you add hosts.
+  [AWS Regions for Amazon EVS](#regions-reference) lists the AWS Regions where Amazon EVS is available.
+  [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference) describes the VLAN subnets that Amazon EVS creates, with their size rules and MTU values.
+  [VCF Installer settings for Amazon EVS](#vcf-installer-settings) collects the Amazon EVS settings that you enter in the VCF Installer for VCF 9.x.
+  [Use the AWS CLI, SDKs, and API with Amazon EVS](#evs-api-cli-reference) links the AWS CLI Command Reference, the API Reference, and the AWS CloudFormation resource types.

**Topics**
+ [Supported VCF versions and instance types](#versions-provided)
+ [VCF feature support on Amazon EVS](#vcf-feature-support)
+ [Service quotas for Amazon EVS](#service-quotas-evs)
+ [AWS Regions for Amazon EVS](#regions-reference)
+ [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference)
+ [VCF Installer settings for Amazon EVS](#vcf-installer-settings)
+ [Use the AWS CLI, SDKs, and API with Amazon EVS](#evs-api-cli-reference)

## Supported VCF versions and instance types
<a name="versions-provided"></a>

Amazon Elastic VMware Service supports the following VMware Cloud Foundation (VCF) versions and ESX builds on each Amazon EC2 instance type, for both deployment modes. Each row is an instance type. The VCF 5.2.x column covers EVS-deployed mode, which installs only VCF 5.2.x. The VCF 9.0 and VCF 9.1 columns cover the VCF 9.x versions that you deploy in Self-deployed mode, which is the only mode that supports VCF 9.x. You can also deploy VCF 5.x in Self-deployed mode. The last column lists the ESX builds, including the ESXi 8.0 builds for VCF 5.x.

### Supported VCF versions by instance type
<a name="current-vcf-versions"></a>

The following table shows the supported VCF versions and ESX builds for each instance type.


| Instance type | Storage | vCPUs and memory | VCF 5.2.x | VCF 9.0 | VCF 9.1 | ESX builds you can choose when you add a host | 
| --- | --- | --- | --- | --- | --- | --- | 
|  `i4i.metal`  | Local NVMe | 128 vCPUs, 1,024 GiB | 5.2.2, 5.2.1 (Restricted) | Available | Available | ESXi 8.0 U3b to 8.0 U3k, and ESX 9.0.2.0 to 9.1.1.0 | 
|  `i7i.metal-24xl`  | Local NVMe | 96 vCPUs, 768 GiB | 5.2.2, 5.2.1 (Restricted) | Available | Available | ESXi 8.0 U3b to 8.0 U3k, and ESX 9.0.2.0 to 9.1.1.0 | 
|  `i7i.metal-48xl`  | Local NVMe | 192 vCPUs, 1,536 GiB | 5.2.2 | Available | Available | ESXi 8.0 U3b to 8.0 U3k, and ESX 9.0.2.0 to 9.1.1.0 | 
|  `r7i.metal-24xl`  | Diskless | 96 vCPUs, 768 GiB | Not available | Available | Available | ESX 9.0.2.0 to 9.1.1.0 | 
|  `r7i.metal-48xl`  | Diskless | 192 vCPUs, 1,536 GiB | Not available | Available | Available | ESX 9.0.2.0 to 9.1.1.0 | 

All hosts in a VCF cluster must use the same instance type.

Instance types with the Diskless storage type have no local NVMe storage, so they can’t use vSAN on local disks and require external storage for the principal datastore. They support only VCF 9.x. For more information, see [Deploy in Self-deployed mode](#getting-started-self-deployed), [External storage options for diskless hosts](#external-storage-diskless), and [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

For the network bandwidth and other hardware details of each instance type, see [Amazon EC2 instance types](https://aws.amazon.com/ec2/instance-types/). For the AWS Regions where Amazon EVS is available, see [Amazon Elastic VMware Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/evs.html) in the * AWS General Reference*.

### Column key
<a name="column-key"></a>
+  **VCF 5.2.x** is deployed in EVS-deployed mode, where Amazon EVS installs VCF. The cells name the releases that you can choose. Release 5.2.2 is the API value `VCF-5.2.2` with the default ESX version `ESXi-8.0U3g-24859861`. Release 5.2.1 is the API value `VCF-5.2.1` with the default ESX version `ESXi-8.0U3b-24280767`.
+  **VCF 9.0** and **VCF 9.1**, which includes VCF 9.1.1, are deployed in Self-deployed mode. Both use the API value `SELF_DEPLOYED`. Amazon EVS creates the VLAN subnets, and you add the hosts and install VCF 9.x yourself. When you add a host without `--esx-version`, Amazon EVS uses the default ESX version for Self-deployed mode, which `get-versions` reports as `defaultEsxVersion`. For the steps, see [Deploy in Self-deployed mode](#getting-started-self-deployed).
+  **Available** means that you can choose the version. **Restricted** means that you must request access first. For more information, see [Request access to restricted VCF versions](#version-restrictions). **Not available** means that the instance type doesn’t support the version.
+  **ESX builds** are the builds that you can choose when you add a host of that instance type. The exact build strings are in the `instanceTypeEsxVersions` field of the `get-versions` response.

### Check what is available to you
<a name="check-provided-versions"></a>

The VCF versions and ESX builds available to you can differ from the table, for example when a version is restricted. The Amazon EVS console shows the VCF versions in the create environment wizard, and the ESX versions when you add a host. You can also use the AWS CLI to view the VCF versions, ESX versions, and EC2 instance types. If a version shows the Restricted status, see [Request access to restricted VCF versions](#version-restrictions) to request access.

To check the versions in the Amazon EVS console or with the AWS CLI, do the following.

**Example**  

1. Open the [Amazon EVS console](https://console.aws.amazon.com/evs).

1. On the navigation menu, choose **Environments**.

1. Do one of the following:

   To check VCF versions, do the following.

   1. Choose **Create Environment**.

   1. Under **Validate Amazon EVS requirements**, choose your VCF version. The status shows whether the version is available or restricted for your account.

   To check ESX versions, do the following.

   1. Choose an existing environment.

   1. Choose **Create host**.

   1. Select an instance type to see the available ESX versions.
Run the following command to retrieve information about VCF and ESX versions. Replace `<region-code>` with the code of the Region that you want to check.  

```bash
aws evs get-versions --region <region-code>
```
The following is example output.  

```json
{
    "vcfVersions": [
        {
            "vcfVersion": "VCF-5.2.1",
            "status": "RESTRICTED",
            "defaultEsxVersion": "ESXi-8.0U3b-24280767",
            "instanceTypes": [
                "i4i.metal",
                "i7i.metal-24xl"
            ]
        },
        {
            "vcfVersion": "VCF-5.2.2",
            "status": "AVAILABLE",
            "defaultEsxVersion": "ESXi-8.0U3g-24859861",
            "instanceTypes": [
                "i4i.metal",
                "i7i.metal-24xl",
                "i7i.metal-48xl"
            ]
        },
        {
            "vcfVersion": "SELF_DEPLOYED",
            "status": "AVAILABLE",
            "defaultEsxVersion": "ESXi-9.0.2.0.25148076",
            "instanceTypes": [
                "i4i.metal",
                "i7i.metal-24xl",
                "i7i.metal-48xl",
                "r7i.metal-24xl",
                "r7i.metal-48xl"
            ]
        }
    ],
    "instanceTypeEsxVersions": [
        {
            "instanceType": "i4i.metal",
            "esxVersions": [
                "ESXi-8.0U3b-24280767",
                "ESXi-8.0U3g-24859861",
                "ESXi-9.0.2.0.25148076",
                "ESXi-9.0.2.0100.25595025",
                "ESXi-9.1.0.0100.25433460",
                "ESXi-9.1.0.0200.25557999",
                "ESXi-9.1.1.0.25714478",
                "ESXi-8.0U3i-25205845",
                "ESXi-8.0U3j-25429389",
                "ESXi-8.0U3k-25595708"
            ]
        },
        {
            "instanceType": "i7i.metal-24xl",
            "esxVersions": [
                "ESXi-8.0U3b-24280767",
                "ESXi-8.0U3g-24859861",
                "ESXi-9.0.2.0.25148076",
                "ESXi-9.0.2.0100.25595025",
                "ESXi-9.1.0.0100.25433460",
                "ESXi-9.1.0.0200.25557999",
                "ESXi-9.1.1.0.25714478",
                "ESXi-8.0U3i-25205845",
                "ESXi-8.0U3j-25429389",
                "ESXi-8.0U3k-25595708"
            ]
        },
        {
            "instanceType": "i7i.metal-48xl",
            "esxVersions": [
                "ESXi-8.0U3b-24280767",
                "ESXi-8.0U3g-24859861",
                "ESXi-9.0.2.0.25148076",
                "ESXi-9.0.2.0100.25595025",
                "ESXi-9.1.0.0100.25433460",
                "ESXi-9.1.0.0200.25557999",
                "ESXi-9.1.1.0.25714478",
                "ESXi-8.0U3i-25205845",
                "ESXi-8.0U3j-25429389",
                "ESXi-8.0U3k-25595708"
            ]
        },
        {
            "instanceType": "r7i.metal-24xl",
            "esxVersions": [
                "ESXi-9.0.2.0.25148076",
                "ESXi-9.0.2.0100.25595025",
                "ESXi-9.1.0.0100.25433460",
                "ESXi-9.1.0.0200.25557999",
                "ESXi-9.1.1.0.25714478"
            ]
        },
        {
            "instanceType": "r7i.metal-48xl",
            "esxVersions": [
                "ESXi-9.0.2.0.25148076",
                "ESXi-9.0.2.0100.25595025",
                "ESXi-9.1.0.0100.25433460",
                "ESXi-9.1.0.0200.25557999",
                "ESXi-9.1.1.0.25714478"
            ]
        }
    ]
}
```

### ESX version considerations
<a name="esx-versions"></a>

Each VCF version has a default ESX version based on the Broadcom VCF Bill of Materials (BOM). When you create an environment, you can’t choose a specific ESX version. Amazon EVS applies the default ESX version for the selected VCF version.

When you add a host to your environment, you can select an available ESX version for the instance type that you choose. If you don’t specify one, Amazon EVS uses the default ESX version for the VCF version of your environment.

After you add a host, you can upgrade its ESX version only with vCenter Lifecycle Manager.

**Note**  
Amazon EVS doesn’t provide every VCF and ESX version that Broadcom releases. For software interoperability information, see the [Broadcom Interoperability Matrix](https://interopmatrix.broadcom.com/Interoperability?col=1). For hardware compatibility of AWS EC2 instances, see the [Broadcom Compatibility Guide](https://compatibilityguide.broadcom.com/).

### Request access to restricted VCF versions
<a name="version-restrictions"></a>

To use a VCF version with the `RESTRICTED` status, [contact AWS Support](https://support.console.aws.amazon.com/support/home#/case/create) with the following information:
+ Your AWS account ID
+ The AWS Region
+ The specific VCF version that you need
+ Your use case and business justification (for example, security and compliance, or a compatibility dependency)

 AWS Support reviews your request and either approves it or asks for more information. After approval, the version status changes to `AVAILABLE` in the Amazon EVS console and in the `get-versions` response.

## VCF feature support on Amazon EVS
<a name="vcf-feature-support"></a>

Check whether Amazon Elastic VMware Service supports a VMware Cloud Foundation (VCF) feature before you plan a design that depends on it.

Each table covers one area. For the VCF versions and EC2 instance types that Amazon EVS provides, see [Supported VCF versions and instance types](#versions-provided).

### Status values
<a name="feat-status-values"></a>

The **Status** column uses the following values.
+  **Supported** means that the feature works on every VCF version and instance type in the row, with no Amazon EVS-specific condition beyond the row.
+  **Supported with conditions** means that the feature works only under the conditions in the **Instance types** and **Notes for Amazon EVS** cells.
+  **Not supported** means that Amazon EVS doesn’t support the feature, or that Broadcom doesn’t support it on this hardware or environment.

### Compute and storage
<a name="feat-compute-storage"></a>

The following table lists the compute and storage features and whether Amazon EVS supports each one.


| Feature | Status | VCF versions | Instance types | Notes for Amazon EVS | Broadcom documentation | Amazon EVS page | 
| --- | --- | --- | --- | --- | --- | --- | 
|  vSAN Express Storage Architecture (ESA) | Supported with conditions | All | Hosts with local NVMe | Requires local NVMe instance storage. Set failures to tolerate (FTT) to at least 1 when you run VCF bringup. Diskless hosts, which need VCF 9.x, have no local NVMe and use external principal storage instead. | None |  [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf)  | 
|  vSAN Original Storage Architecture (OSA) | Not supported | All | All | Amazon EVS clusters use vSAN ESA. Use vSAN ESA for vSAN storage. | None | None | 

### Availability and recovery
<a name="feat-availability-recovery"></a>

The following table lists the availability and recovery features and whether Amazon EVS supports each one.


| Feature | Status | VCF versions | Instance types | Notes for Amazon EVS | Broadcom documentation | Amazon EVS page | 
| --- | --- | --- | --- | --- | --- | --- | 
|  vSphere Fault Tolerance | Not supported | All | All | Broadcom doesn’t support Fault Tolerance for VMs on NSX port groups, or for NSX Managers and Edge nodes. Workload VMs on Amazon EVS run on NSX overlay networks. |  [Broadcom knowledge base article 317806](https://knowledge.broadcom.com/external/article?articleNumber=317806)  | None | 

## Service quotas for Amazon EVS
<a name="service-quotas-evs"></a>

Your AWS account has default quotas, also called limits, for each AWS service, and most quotas are specific to a Region.

Amazon Elastic VMware Service is integrated with Service Quotas, an AWS service that you can use to view and manage your quotas from a central location. For more information, see [What Is Service Quotas?](https://docs.aws.amazon.com/servicequotas/latest/userguide/intro.html) in the *Service Quotas User Guide*.

### Amazon EVS quotas
<a name="evs-quota-values"></a>

The following table shows the Amazon EVS quotas. The columns show the quota name as it appears in the Service Quotas console, the default value, whether you can request an increase, and what the quota limits.


| Name | Default | Adjustable | Description | 
| --- | --- | --- | --- | 
| Environment count per AWS account | 3 in each supported Region | Yes | The maximum number of Amazon EVS environments that you can create in this account in the current Region. | 
| Host count per EVS environment | 5 in each supported Region | Yes | The maximum number of hosts that you can provision in a single Amazon EVS environment. | 

The default values are the initial quotas that AWS sets. Your applied quota values can differ. For the authoritative values, see [Amazon EVS quotas](https://docs.aws.amazon.com/general/latest/gr/evs.html#limits_evs) in the * AWS General Reference*.

### Quotas to check before you add hosts
<a name="evs-quotas-check-first"></a>

Check the following quotas before you create an environment or add a host. If the host count quota or the vCPU quota is too low, the host creation fails.
+  **Host count per EVS environment.** This quota must cover every host that you plan to create in the environment. Include the temporary host that you add when you replace a failed host. For more information, see [Replace a failed host (Self-deployed)](#host-replace).
+  **EC2 Running On-Demand Standard Instances.** This Amazon EC2 quota is measured in vCPUs. It must cover the vCPUs of all the EC2 instances that you use on Amazon EVS. For example, each `i4i.metal` instance uses 128 vCPUs. For the vCPUs of each instance type, see [Supported VCF versions and instance types](#versions-provided).
+  **EC2 Dedicated Hosts.** If you plan to use EC2 Dedicated Hosts, this quota must cover the number of Dedicated Hosts that you intend to use in the Region.
+  **IPAM netmask length.** If you configure HCX internet connectivity, your IPAM quota for the Amazon-provided contiguous public IPv4 CIDR block netmask length has a default value of /28, which is enough. You don’t need to request an increase. For more information, see [Quotas for your IPAM](https://docs.aws.amazon.com/vpc/latest/ipam/quotas-ipam.html) in the *Amazon VPC IP Address Manager User Guide*.

For information about increasing EC2 service quotas, see [Request an increase](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html#request-increase) in the *Amazon EC2 User Guide*.

**Note**  
Amazon CloudWatch collects AWS usage metrics for Amazon EVS resources that have quotas (environments and hosts). For more information, see [CloudWatch Usage Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Usage-Metrics.html) in the *Amazon CloudWatch User Guide*.

### View and request quotas in the AWS Management Console
<a name="view-service-quotas-console"></a>

To view and request quotas in the Service Quotas console, do the following.

1. Open the [Service Quotas console](https://console.aws.amazon.com/servicequotas/home/services/evs/quotas).

1. In the navigation pane, choose ** AWS services**.

1. From the ** AWS services** list, search for and select **Amazon Elastic VMware Service**.

1. Choose **View quotas**.

   In the **Service quotas** list, you can see the service quota name, the applied value (if it’s available), the AWS default quota, and whether the quota value is adjustable.

1. To view more information about a service quota, such as the description, choose the quota name.

1. (Optional) To request a quota increase, do the following.

   1. Select the quota that you want to increase.

   1. Choose **Request increase at account level**.

   1. Enter or select the required information, and then choose **Request**.

For more information, see [Requesting a quota increase](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html) in the *Service Quotas User Guide*.

### View and request quotas with the AWS CLI
<a name="view-service-quotas-cli"></a>

Run the following command to view the default Amazon EVS quotas.

```bash
aws service-quotas list-aws-default-service-quotas \
    --query 'Quotas[*].{Adjustable:Adjustable,Name:QuotaName,Value:Value,Code:QuotaCode}' \
    --service-code evs \
    --output table
```

The returned value is the number of Amazon EVS environments or hosts that you can create in this account in the current AWS Region.

To request an increase, use the [request-service-quota-increase](https://docs.aws.amazon.com/cli/latest/reference/service-quotas/request-service-quota-increase.html) command in the * AWS CLI Command Reference*. For the other Service Quotas commands, see [service-quotas](https://docs.aws.amazon.com/cli/latest/reference/service-quotas/index.html) in the * AWS CLI Command Reference*.

## AWS Regions for Amazon EVS
<a name="regions-reference"></a>

You create an Amazon Elastic VMware Service environment and its hosts in one Region and one Availability Zone, so check the Region before you plan your network.

The authoritative list of Amazon EVS Regions and service endpoints is [Amazon Elastic VMware Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/evs.html) in the * AWS General Reference*. That page is updated when Amazon EVS becomes available in a new Region.

### Regions where Amazon EVS is available
<a name="regions-list"></a>

The following table lists Regions where Amazon EVS is available. The columns show the Region name, the Region code that you use with the AWS CLI and API, and notes that apply to Amazon EVS in that Region.


| Region name | Region code | Notes | 
| --- | --- | --- | 
| US East (N. Virginia) |  `us-east-1`  | None | 
| US East (Ohio) |  `us-east-2`  | None | 
| US West (N. California) |  `us-west-1`  | None | 
| US West (Oregon) |  `us-west-2`  | None | 
| Asia Pacific (Hyderabad) |  `ap-south-2`  | None | 
| Asia Pacific (Malaysia) |  `ap-southeast-5`  | None | 
| Asia Pacific (Mumbai) |  `ap-south-1`  | None | 
| Asia Pacific (Osaka) |  `ap-northeast-3`  | None | 
| Asia Pacific (Seoul) |  `ap-northeast-2`  | None | 
| Asia Pacific (Singapore) |  `ap-southeast-1`  | None | 
| Asia Pacific (Sydney) |  `ap-southeast-2`  | None | 
| Asia Pacific (Taipei) |  `ap-east-2`  | None | 
| Asia Pacific (Tokyo) |  `ap-northeast-1`  | None | 
| Canada (Central) |  `ca-central-1`  | None | 
| Canada West (Calgary) |  `ca-west-1`  | None | 
| Europe (Frankfurt) |  `eu-central-1`  | None | 
| Europe (Ireland) |  `eu-west-1`  | None | 
| Europe (London) |  `eu-west-2`  | None | 
| Europe (Milan) |  `eu-south-1`  | None | 
| Europe (Paris) |  `eu-west-3`  | None | 
| Europe (Spain) |  `eu-south-2`  | None | 
| Europe (Stockholm) |  `eu-north-1`  | None | 
| Europe (Zurich) |  `eu-central-2`  | None | 
| Israel (Tel Aviv) |  `il-central-1`  | None | 
| Mexico (Central) |  `mx-central-1`  | None | 
| South America (São Paulo) |  `sa-east-1`  | None | 

Amazon EVS isn’t available in the AWS GovCloud (US) Regions. If you need Amazon EVS in a Region that isn’t listed, contact your AWS account team.

### Instance types by Region
<a name="regions-instance-types"></a>

For the Amazon EC2 instance types that Amazon EVS supports, see [Supported VCF versions and instance types](#versions-provided). The r7i instance types are available in every Region where Amazon EVS is available.

To check which instance types and VMware Cloud Foundation (VCF) versions are available to your account in a Region, run `aws evs get-versions` in that Region. For more information, see [Check what is available to you](#check-provided-versions).

## VLAN subnets, VLAN IDs, and IP addresses
<a name="vlan-subnet-reference"></a>

Amazon Elastic VMware Service creates the VLAN subnets for your environment from the CIDR blocks that you provide. For a short explanation of what a VLAN subnet is, see [Concepts and components of Amazon EVS](#concepts). For how to size the VPC and choose CIDR blocks, see [Plan your VPC and VLAN subnets](#plan-network-sizing).

### Rules for all VLAN subnets
<a name="vlan-subnet-rules"></a>

An Amazon EVS VLAN subnet is an Amazon VPC subnet that Amazon EVS manages. Each VLAN subnet has a VLAN tag that segments its traffic from the other VLAN subnets.

**Important**  
Amazon EVS creates all VLAN subnets when you create the environment, from the CIDR blocks that you provide. You can’t add VLAN subnets or change their CIDR blocks after you create the environment. Size each CIDR block for the maximum number of hosts and appliances that you expect over the life of the environment.

The following rules apply to the CIDR block of every VLAN subnet.
+ The netmask must be from /28 to /24.
+ The CIDR block must be inside a CIDR block that’s associated with the VPC. The block can be the primary CIDR block or a secondary CIDR block.
+ The CIDR block must not overlap with any other subnet in the VPC, including the other VLAN subnets and the service access subnet.
+ The vMotion, vSAN, and VTEP VLAN subnets must be the same size as the host management VLAN subnet.

We recommend /24 for each VLAN subnet, the largest size that Amazon EVS allows, because a larger subnet leaves room for more hosts and appliances.

When Amazon EVS creates the VLAN subnets, it associates them implicitly with the main route table of the VPC. After deployment, you must associate each VLAN subnet explicitly with a route table. We recommend a custom route table. For the steps, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt). For more information, see [Associate VLAN subnets with a route table](#evs-env-config-custom-rt).

**Important**  
Amazon EC2 security group rules aren’t enforced on the Amazon EVS elastic network interfaces that are attached to VLAN subnets. To control traffic to and from VLAN subnets, use a network access control list. For more information, see [Create a network ACL for VLAN subnet traffic](#evs-env-nacl-cong).

### VLAN subnet types
<a name="vlan-subnet-types"></a>

Amazon EVS creates 10 VLAN subnets of nine types. The Expansion type has two subnets. The following table lists them. The columns show the subnet type, the key that you use for its CIDR block in the `initialVlans` parameter of `create-environment`, and what the subnet carries.


| VLAN subnet |  `initialVlans` key | Used for | 
| --- | --- | --- | 
|  Host management |  `vmkManagement`  | ESX host management. The host management VMkernel interface connects to this subnet, which separates management traffic from user traffic. Plan one IP address for each host. | 
|  Management VM |  `vmManagement`  | VMware Cloud Foundation (VCF) management appliances, such as vCenter Server, NSX Manager, NSX Edge nodes, and SDDC Manager or VCF Operations. | 
|  vMotion |  `vMotion`  | vMotion traffic that moves virtual machines between hosts. | 
|  vSAN |  `vSan`  | vSAN storage traffic, separated from other network traffic. | 
|  VTEP (host overlay) |  `vTep`  | NSX tunnel endpoints (TEPs) on the ESX hosts, which encapsulate and decapsulate overlay (Geneve) traffic. The host VTEP network uses two IP addresses for each host. | 
|  Edge VTEP (Edge overlay) |  `edgeVTep`  | NSX Edge overlay tunnel endpoints, for overlay traffic between the NSX Edge nodes and the ESX hosts. | 
|  NSX uplink |  `nsxUplink`  | The Tier-0 gateway uplinks on the NSX Edge nodes, which connect the NSX overlay networks to the rest of your VPC and to external networks. | 
|  HCX uplink |  `hcx`  | Communication between the HCX Interconnect (HCX-IX) and HCX Network Extension (HCX-NE) appliances, and the HCX service mesh uplink. For HCX over the internet, this subnet uses a public CIDR block. For more information, see [Plan HCX connectivity](#plan-hcx-connectivity). | 
|  Expansion (two subnets) |  `expansionVlan1`, `expansionVlan2`  | Space for other VCF-supported functions, such as NSX Federation. | 

### MTU by network function
<a name="vlan-subnet-mtu"></a>

The following table lists the MTU that you configure in VCF for each VLAN subnet. The columns show the VLAN subnet, with its function name in parentheses where the guide gives one, and its MTU.


| VLAN subnet | MTU (bytes) | 
| --- | --- | 
| Host management (`vmkManagement`) | 1500 | 
| Management VM (`vmManagement`) | 1500 | 
| vMotion | 8500 | 
| vSAN | 8500 | 
| VTEP (host overlay, `vTep`) | 8500 | 
| Edge VTEP (Edge overlay, `edgeVTep`) | 8500 | 
| NSX uplink (`nsxUplink`) | 1500 | 

**Important**  
Configure jumbo frames (MTU 8500) on the vMotion, vSAN, and overlay (TEP) networks. The management and uplink networks use MTU 1500. The MTU must be the same across the network path, or vSAN and overlay traffic fails.

### VLAN IDs
<a name="vlan-subnet-ids"></a>

Amazon EVS assigns a VLAN ID to each VLAN subnet when it creates the environment. You use these VLAN IDs when you configure the distributed switch, port groups, uplink profiles, and host networking in VCF.

To find the VLAN IDs for your environment, use one of the following methods.
+ Open the Amazon EVS console. In the navigation pane, choose **Environments**, choose your environment, and then choose the **Networks and connectivity** tab.
+ Run the `list-environment-vlans` command, and match each VLAN on its `functionName` value, for example `vmManagement` or `edgeVTep`.

  In the following command, replace `<environment-id>` with the ID of your environment. Replace `<region-code>` with the Region where you created the environment.

  ```bash
  aws evs list-environment-vlans \
      --environment-id <environment-id> \
      --region <region-code>
  ```

  The response lists each VLAN subnet with its `vlanId`, `cidr`, `functionName`, `subnetId`, and `vlanState`. For the full response, see [ListEnvironmentVlans](https://docs.aws.amazon.com/evs/latest/APIReference/API_ListEnvironmentVlans.html) in the *Amazon EVS API Reference*.

### Reserved IP addresses
<a name="vlan-subnet-reserved-addresses"></a>

VLAN subnets are VPC subnets, so the standard Amazon VPC rules for reserved addresses apply. The first four IP addresses and the last IP address in each VLAN subnet CIDR block aren’t available for your hosts or appliances. For example, a /24 VLAN subnet has 251 addresses that you can use. For more information, see [Subnet CIDR blocks](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-sizing.html) in the *Amazon VPC User Guide*.

The gateway address of a VLAN subnet doesn’t respond to ICMP ping from outside the subnet. A gateway reachability check can fail even when routing is correct.

### MTU and throughput on the network path
<a name="vlan-subnet-mtu-throughput"></a>

Traffic that leaves the VPC is limited by the MTU of each AWS network component on the path. The following table lists the maximum MTU of each component.


| Network component | Maximum MTU (bytes) | 
| --- | --- | 
| Amazon EC2 instances in a VPC | 9001 | 
|  AWS Transit Gateway | 8500 | 
|  Direct Connect transit virtual interface | 8500 | 

All north-south traffic between the NSX overlay and the VPC goes through the active NSX Edge node. An undersized Edge node can limit throughput. For more information, see [Plan NSX Edge and VPC Route Server design](#plan-nsx-design).

## VCF Installer settings for Amazon EVS
<a name="vcf-installer-settings"></a>

This page applies to Self-deployed environments that run VCF 9.x. For how EVS-deployed environments (VCF 5.2.x) differ, see [Differences between deployment modes](#deployment-mode-differences).

When you install VMware Cloud Foundation (VCF) 9.x, you enter some settings in the VCF Installer that are specific to Amazon Elastic VMware Service. Both installation procedures in the Self-deployed path use these settings, one for hosts with local storage and one for diskless hosts. For the procedures, see [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf) and [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

The rest of the installation follows the standard Broadcom VCF process. For the installation mechanics, see the [Broadcom documentation for VMware Cloud Foundation 9.0](https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html).

### Network settings
<a name="installer-settings-network"></a>

Amazon EVS assigns a VLAN ID to each network function in your environment. You use these VLAN IDs and the MTU values when you configure the distributed switch, port groups, and host networking during installation. For the VLAN subnets, how to find their VLAN IDs, and the MTU of each network function, see [VLAN subnets, VLAN IDs, and IP addresses](#vlan-subnet-reference).

The following table lists the network settings that Amazon EVS requires in the VCF Installer. The columns show the setting, the value to use, and where you set it.


| Setting | Value | Where you set it | 
| --- | --- | --- | 
|  `VM Network` port group on the host that runs the VCF Installer | The VLAN ID of the VM management (`vmManagement`) network | VMware Host Client, before you deploy the VCF Installer appliance | 
|  `Management` port group on that host | Untagged, with VLAN ID `0`  | VMware Host Client | 
| VCF Installer appliance IP address | The SDDC Manager address from your DNS plan, because the appliance becomes SDDC Manager during bringup | VCF Installer OVA deployment | 
| Uplink teaming for distributed switch port groups | A failover teaming policy (an active uplink with a standby uplink), not a load-balancing policy | Deployment specification | 
| NSX teaming policy |  `FAILOVER_ORDER`  | Deployment specification | 

You don’t need to turn on SSH on the hosts. The VCF Installer migrates host networking to a distributed switch during bringup.

### Time (NTP) settings
<a name="installer-settings-ntp"></a>

You enter NTP servers in two places, when you deploy the VCF Installer OVA and in the deployment specification. You also set an NTP server in the DHCP option set of your VPC. For more information, see [Configure DNS and NTP with a DHCP option set](#network-dhcp-dns-ntp).

Amazon EVS hosts use the link-local Amazon Time Sync Service address `169.254.169.123` as their NTP source. For diskless hosts, the specification example uses `time.aws.com`, and validation reports NTP warnings that you can acknowledge. For the list of warnings, see [Validate and deploy](#getting-started-diskless-deploy).

### Passwords
<a name="installer-settings-passwords"></a>

During installation, you set passwords for the VCF Installer appliance and the VCF management appliances. Each appliance enforces its own password complexity rules, which VCF defines. If an appliance rejects a password, the validation error states the rule that the password must meet.

When you deploy the VCF Installer OVA, the root user password and the local user password must be at least 15 characters long. They must contain an uppercase letter, a lowercase letter, a digit, and one of the special characters `! % @ $ ^ # ? *`.

The deployment specification also needs the ESX root password of each host. Amazon EVS stores the ESX root password of each host in AWS Secrets Manager. For more information, see [Get secrets from AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets.html) in the * AWS Secrets Manager User Guide*.

### Broadcom online depot settings
<a name="installer-settings-depot"></a>

The VCF Installer downloads the VCF software from the Broadcom online depot. You connect it to the Broadcom online depot with a credential from your Broadcom account, and your Broadcom account must have a valid VCF entitlement.

The credential and the screens differ between VCF 9.0 and VCF 9.1. For the steps, see [Step 4a (local storage): Prepare the VCF Installer](#self-deployed-install-vcf) for hosts with local storage or [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless) for diskless hosts.

The Broadcom online depot requires outbound internet access from the VCF Installer. The NAT gateway in your network foundation provides this access. For more information, see [Create a VPC with subnets and route tables](#network-create-vpc).

The Amazon EVS custom add-on depot is separate from the Broadcom online depot. It provides the Amazon EVS add-on for ESX cluster images, and you don’t enter it in the VCF Installer during installation. For more information, see [Use the Amazon EVS custom add-on depot](#addon-depot).

### Management cluster settings
<a name="installer-settings-cluster"></a>

When you configure the management cluster in the deployment specification, apply the following Amazon EVS-specific settings.

vSAN, for hosts with local storage  
Use vSAN Express Storage Architecture (ESA) with failures to tolerate (FTT) set to at least 1. Amazon EVS doesn’t support vSAN Original Storage Architecture (OSA). For more information, see [VCF feature support on Amazon EVS](#vcf-feature-support).

Principal storage, for diskless hosts  
Diskless hosts have no local NVMe storage and don’t use vSAN. They use a datastore on external storage as principal storage. For more information, see [Step 4a (diskless): Prepare diskless hosts and the VCF Installer](#getting-started-diskless).

EVC mode  
Set the cluster Enhanced vMotion Compatibility (EVC) mode to match your instance type. Use `INTEL_SAPPHIRERAPIDS` for `i7i.metal-24xl` and `i7i.metal-48xl`. For `i4i.metal`, don’t set an EVC mode, because setting `INTEL_ICELAKE` can cause bringup to fail on VCF 9.1. For `r7i.metal-24xl` diskless hosts, the hosts report Intel Sapphire Rapids as their highest EVC mode. The VCF Installer doesn’t apply the EVC mode to a diskless management cluster, so you set it after deployment. For more information, see [Verify the result](#getting-started-diskless-verify).

### Bringup validation settings
<a name="installer-settings-validation"></a>

Some standard VCF Installer validation checks don’t apply to the Amazon EVS network. For example, Amazon EVS VLAN subnet gateways don’t respond to ICMP ping from outside the subnet, so the gateway ping check fails even when routing is correct. How you handle these checks depends on whether you run bringup with a specification file or with the VCF Installer wizard. The following table lists the checks. The columns show the check and what to do in a specification file and in the wizard.


| Check | Specification file | Wizard | 
| --- | --- | --- | 
| Gateway ping validation | Set `skipGatewayPingValidation` to `true`. | Acknowledge the failed gateway ping check and continue. | 
| ESX host thumbprint validation | Set `skipEsxThumbprintValidation` to `true`, or get the correct thumbprints for all hosts and enter them in the specification. | Thumbprint validation is mandatory. Review and accept the thumbprint at the hosts step to continue. | 
| vSAN ESA certified disk validation (hosts with local storage) | Set `skipVsanEsaCertifiedDiskValidation` to `true`. The Amazon EC2 instance store NVMe disks aren’t on the vSAN ESA certified disk list, so this check can fail. | If the wizard reports a failed disk check that you can’t acknowledge, run bringup with a specification file. | 

When you run bringup through the VCF Installer wizard, use the wizard to find and correct errors in the specification. The wizard shows validation problems more clearly than the API, whose errors are less descriptive.

To generate an example bringup specification, use the [spec-generator tool](https://github.com/aws/solutions-for-amazon-evs/tree/main/Deploy/EVS-Deployment-Orchestrator/spec-generator) in the `Deploy` folder of the [Solutions for Amazon EVS](https://github.com/aws/solutions-for-amazon-evs) repository on GitHub.

### Support boundaries
<a name="installer-settings-support"></a>

Amazon EVS doesn’t support running ESX outside of a full VCF deployment. VMware workload virtual machines must be deployed onto NSX overlay networks.

**Important**  
In Self-deployed mode, Amazon EVS doesn’t manage the VCF installation. For VCF-specific requests, you can use your active VCF subscription entitlements to contact Broadcom through the Broadcom Support Portal. For more information, see [Troubleshoot Self-deployed deployments](#troubleshoot-self-deployed).

## Use the AWS CLI, SDKs, and API with Amazon EVS
<a name="evs-api-cli-reference"></a>

The AWS CLI, API, and AWS CloudFormation references for Amazon Elastic VMware Service give the exact command, operation, or resource property that a procedure in this guide uses.

The following table lists the references. The columns show the reference, what it documents, and where to find it.


| Reference | What it documents | Link | 
| --- | --- | --- | 
|  AWS CLI Command Reference | The `aws evs` commands, with their options and example output. |  [evs commands in the AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/evs/index.html)  | 
| Amazon EVS API Reference | The API operations, data types, parameters, and errors. The AWS SDKs use the same operations. |  [Amazon EVS API Reference](https://docs.aws.amazon.com/evs/latest/APIReference/Welcome.html)  | 
|  AWS SDKs | The language-specific SDKs that call the Amazon EVS API, and the settings that they share. |  [AWS SDKs and Tools Reference Guide](https://docs.aws.amazon.com/sdkref/latest/guide/overview.html)  | 
|  AWS CloudFormation resource types | The `AWS::EVS::Environment` resource type, with its properties and template examples. |  [Amazon Elastic VMware Service resource type reference in the AWS CloudFormation Template Reference](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/AWS_EVS.html)  | 
| Service endpoints and quotas | The Regional endpoints, including FIPS and dual-stack endpoints, and the default quotas. |  [Amazon Elastic VMware Service endpoints and quotas in the AWS General Reference](https://docs.aws.amazon.com/general/latest/gr/evs.html)  | 

### Set up the AWS CLI
<a name="api-cli-setup"></a>

To run the `aws evs` commands in this guide, install and configure the AWS CLI. For more information, see [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html) in the * AWS Command Line Interface User Guide*.

### Related topics
<a name="api-cli-related"></a>

For related information, see the following topics.
+ To create an environment with a CloudFormation template, see [Create Amazon EVS resources with AWS CloudFormation](#creating-resources-with-cloudformation).
+ To record Amazon EVS API calls, see [Log API calls with CloudTrail](#logging-using-cloudtrail).
+ To control who can call the Amazon EVS API, see [Identity and access management for Amazon Elastic VMware Service](#security-iam).

# Document history for the Amazon Elastic VMware Service User Guide
<a name="doc-history"></a>

The following table describes the documentation releases for Amazon Elastic VMware Service.

| Change | Description | Date | 
| --- |--- |--- |
| [EVS Deployment Orchestrator supports r7i instance types and VCF 9.1.1](https://docs.aws.amazon.com/evs/latest/userguide/deploy-with-edo.html) | The EVS Deployment Orchestrator (EDO) now deploys the `r7i.metal-24xl` and `r7i.metal-48xl` instance types, using Amazon FSx for NetApp ONTAP as the principal storage, and supports VCF 9.1.1 on every instance type. | November 16, 2026 | 
| [Combined the supported configurations into one table](https://docs.aws.amazon.com/evs/latest/userguide/versions-provided.html) | Combined the supported VCF versions, ESX builds, and Amazon EC2 instance types for both deployment modes into a single table. Each row is an instance type, with one column for VCF 5.2.x in EVS-deployed mode and one column for each VCF 9.x release in Self-deployed mode. | November 4, 2026 | 
| [Added the diskless host deployment path for Self-deployed mode](https://docs.aws.amazon.com/evs/latest/userguide/getting-started-diskless.html) | Added steps to deploy VCF 9.x in Self-deployed mode on diskless hosts, such as the `r7i.metal-24xl` and `r7i.metal-48xl` instance types. The steps cover external principal storage with Amazon FSx for OpenZFS, the VCF Installer, and the deployment specification. Diskless hosts are available only in Self-deployed mode with VCF 9.x. | November 4, 2026 | 
| [Added the Troubleshooting chapter](https://docs.aws.amazon.com/evs/latest/userguide/troubleshooting.html) | Added a Troubleshooting chapter that lists common problems by area and symptom and links to the page that covers each one. It includes pages for Self-deployed deployments, environment status checks, hosts, networking, and Windows Server entitlements. | November 4, 2026 | 
| [Added the Reference chapter](https://docs.aws.amazon.com/evs/latest/userguide/reference-overview.html) | Added a Reference chapter that holds the lookup tables that the other chapters share. It covers supported VMware Cloud Foundation (VCF) versions and instance types, VCF feature support, service quotas, AWS Regions, VLAN subnets, and VCF Installer settings. | November 4, 2026 | 
| [vSAN capacity warnings in AWS Health](https://docs.aws.amazon.com/evs/latest/userguide/troubleshoot-hosts.html#troubleshoot-vsan-capacity) | You now receive an AWS Health notification when a vSAN cluster in your environment reaches the utilization warning threshold of 70%. This notification alerts you when a cluster runs low on the free capacity that vSAN needs to rebuild data. If vSAN can’t rebuild data in a timely manner, you run the risk of cascading VM failures. You also receive a notification when Amazon EVS can’t determine the vSAN capacity of a cluster. | November 2, 2026 | 
| [Restructured the Windows Server entitlements documentation](https://docs.aws.amazon.com/evs/latest/userguide/entitlements.html) | Restructured the Windows Server entitlements documentation into a dedicated section with expanded topics covering setup, maintenance, and troubleshooting. | September 11, 2026 | 
| [Added support for the `i7i.metal-48xl` instance type](https://docs.aws.amazon.com/evs/latest/userguide/versions-provided.html) | Amazon EVS now supports the `i7i.metal-48xl` instance type for use with VCF environments.<br />This bare-metal instance type is available across all VCF versions. | August 27, 2026 | 
| [Added the Amazon EVS custom add-on depot topic](https://docs.aws.amazon.com/evs/latest/userguide/addon-depot.html) | Added documentation for accessing the Amazon EVS custom add-on depot. Call the `GetDepotUrl` API action to get the URL of the Amazon EVS custom add-on depot. Add the URL to vSphere Lifecycle Manager (vLCM) as a download source, and then sync and install the Amazon EVS add-on. | May 21, 2026 | 
| [Added support for 32 hosts in each environment](https://docs.aws.amazon.com/evs/latest/userguide/evs-env-create-host.html) | Amazon EVS now supports up to 32 hosts in each Amazon EVS environment. | May 18, 2026 | 
| [Updated Setting up guide for Self-deployed mode](https://docs.aws.amazon.com/evs/latest/userguide/setting-up.html) | Updated the Setting up guide and the deployment prerequisite checklist to identify which setup tasks apply only when Amazon EVS installs VCF (VCF 5.2.x). In Self-deployed mode, VCF license keys and VCF management appliance DNS records are not required at environment creation time. | May 6, 2026 | 
| [Updated Getting started guide for Self-deployed mode](https://docs.aws.amazon.com/evs/latest/userguide/getting-started.html) | Updated the Getting started guide to clarify that it covers environment creation with VCF 5.2.x (where Amazon EVS installs VCF), and to reference the new Self-deployed mode guide. | May 6, 2026 | 
| [Added Self-deployed mode getting started guide](https://docs.aws.amazon.com/evs/latest/userguide/getting-started-self-deployed.html) | Amazon EVS now supports Self-deployed mode, in which Amazon EVS provisions the AWS infrastructure for your environment and you install VCF yourself. Self-deployed mode supports VCF 9.0 and 9.1 and gives you direct control over VCF installation.<br />The new Self-deployed mode guide covers prerequisites, environment creation with `vcfVersion=SELF_DEPLOYED`, adding hosts, installing VCF, and creating Operations Manager and SDDC Manager connectors. | May 6, 2026 | 
| [Added support for the `i7i.metal-24xl` instance type](https://docs.aws.amazon.com/evs/latest/userguide/versions-provided.html) | Amazon EVS now supports the `i7i.metal-24xl` instance type for use with VCF environments.<br />This bare-metal instance type is available across all VCF versions. | April 27, 2026 | 
| [Added environment connector and entitlement user guides](https://docs.aws.amazon.com/evs/latest/userguide/evs-env-create-connector.html) | Added user guides for managing Amazon EVS environment connectors and entitlements. Connectors establish a persistent connection between Amazon EVS and a VCF appliance, enabling features such as Windows License Included. With entitlements, you can enable or remove Windows Server license coverage for virtual machines in your Amazon EVS environment. | April 20, 2026 | 
| [Updated AmazonEVSServiceRolePolicy to retrieve vCenter credentials](https://docs.aws.amazon.com/evs/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-amazonevsservicerolepolicy) | Amazon EVS has updated the managed policy `AmazonEVSServiceRolePolicy` to allow the service to retrieve vCenter credentials from AWS Secrets Manager and decrypt secrets encrypted with customer-managed KMS keys. | March 23, 2026 | 
| [Release support for VCF-5.2.2](https://docs.aws.amazon.com/evs/latest/userguide/versions-provided.html#current-vcf-versions) | Amazon EVS now supports VCF 5.2.2. You can specify supported combinations of VCF and ESX software versions when you create environments and add hosts. | January 20, 2026 | 
| [Amazon EVS released in seven more AWS Regions](https://docs.aws.amazon.com/general/latest/gr/evs.html) | Amazon EVS was released in the US West (N. California), Asia Pacific (Hyderabad), Asia Pacific (Malaysia), Canada West (Calgary), Europe (Milan), Mexico (Central), and South America (São Paulo) Regions. | December 15, 2025 | 
| [Amazon EVS released in four more AWS Regions](https://docs.aws.amazon.com/general/latest/gr/evs.html) | Amazon EVS was released in the Asia Pacific (Mumbai), Asia Pacific (Sydney), Canada (Central), and Europe (Paris) Regions. | November 6, 2025 | 
| [Amazon EVS released in Asia Pacific (Singapore) and Europe (London) Regions](https://docs.aws.amazon.com/general/latest/gr/evs.html) | Amazon EVS was released in the Asia Pacific (Singapore) and Europe (London) Regions. | September 30, 2025 | 
| [Amazon EVS supports HCX migration over the public internet](https://docs.aws.amazon.com/general/latest/gr/evs.html) | You can now securely migrate and stretch your layer 2 networks from your on-premises data centers to Amazon EVS environments over the public internet. | September 18, 2025 | 
| [Updated AmazonEVSServiceRolePolicy to add resource management permissions](https://docs.aws.amazon.com/evs/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-amazonevsservicerolepolicy) | Amazon EVS has updated the managed policy `AmazonEVSServiceRolePolicy` to add permissions for managing Amazon EC2 instances and Amazon EBS volumes, and for using AWS Secrets Manager. For more information, see [Amazon EVS updates to AWS managed policies](https://docs.aws.amazon.com/evs/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-account-updates). | August 14, 2025 | 
| [Updated AmazonEVSServiceRolePolicy](https://docs.aws.amazon.com/evs/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-amazonevsservicerolepolicy) | Updated the AWS managed policy `AmazonEVSServiceRolePolicy`. | August 4, 2025 | 
| [Released the environment count per AWS account quota](https://docs.aws.amazon.com/evs/latest/userguide/service-quotas-evs.html) | Amazon EVS added a quota for the number of environments in each account and Region.<br />The environment count per AWS account quota represents the maximum number of Amazon EVS environments that can be created in a given account and Region. | July 8, 2025 | 
| [Amazon EVS released in the Europe (Ireland) Region](https://docs.aws.amazon.com/general/latest/gr/evs.html) | Amazon EVS was released in the Europe (Ireland) Region. | June 18, 2025 | 
| [Released AmazonEVSServiceRolePolicy](https://docs.aws.amazon.com/evs/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-amazonevsservicerolepolicy) | Amazon EVS released the AWS managed policy `AmazonEVSServiceRolePolicy`. | June 9, 2025 | 
| [Initial User Guide release](https://docs.aws.amazon.com/evs/latest/userguide) | The Amazon Elastic VMware Service User Guide was released.<br />The Amazon EVS User Guide describes Amazon EVS concepts and explains how to use the features in the console and with the AWS CLI. | June 9, 2025 | 