# Failback to on-premises environment ## Using the Failback Client Failback replication allows you to replicate data from AWS back to your original source server. To initiate this process, the Failback Client is booted directly on the source server that will receive the replicated data. **Before you begin** Before starting failback replication, ensure you have completed the following: + Meet the [failback prerequisites](#failback-performing-prerequesites). + [Generate failback AWS credentials](#failback-performing-credentials). **Monitoring failback progress** Once failback replication is underway, you can track its progress in the AWS Elastic Disaster Recovery Console on the **Recovery instances** page. [Learn more about the Recovery instances page](recovery-instances.md#managing-recovery-instances). ### Failback prerequisites Prior to performing a failback, ensure that you meet all [replication network requirements](preparing-environments.md) and these failback-specific requirements: + Ensure that the volumes on the server you are failing back to are the same size, or larger, than the Recovery instance. + The Failback Client must be able to communicate with the Recovery instance on TCP 1500, this can be done either via a private route (VPN/DX) or a public route (public IP assigned to the recovery instance) + TCP Port 1500 inbound and TCP Port 443 outbound must be open on the recovery instance for the pairing to succeed. + You must allow traffic to S3 from the server you are failing back to. + The server on which the Failback Client is run must have at least 4 GB of dedicated RAM. + The recovery instance used as a source for failback must have permissions to access the DRS service via API calls. This is done using instance profile for the underlying EC2 instance. The instance profile must include the AWSElasticDisasterRecoveryRecoveryInstancePolicy in addition to any other policy you require the EC2 instance to have. By default, the launch settings that DRS creates for source servers already have an instance profile defined that includes that policy and that instance profile will be used when launching a Recovery Instance. + Be sure to deactivate secure boot on the server on which the Failback Client is run. + Ensure the hardware clock on the server on which the Failback Client is run is set to UTC rather than Local Time. ### Failback AWS credentials In order to perform a failback with the Elastic Disaster Recovery Failback Client, you must first generate the required AWS credentials. You can create temporary credentials with AWS Security Token Service. These credentials are only used during Failback Client initialization. You will need to enter your credentials into the Failback Client when prompted. #### Generating temporary failback credentials In order to generate the temporary credentials required to install the AWS Elastic Disaster Recovery Failback Client, take these steps: 1. [Create a new IAM Role ](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) with the **AWSElasticDisasterRecoveryFailbackInstallationPolicy** policy. 1. Request temporary security credentials [via AWS STS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) using the [AssumeRole API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). For example: ``` aws sts assume-role \ --role-arn arn:aws:iam:::role/ \ --role-session-name drs-failback-session ``` This command returns temporary credentials consisting of an **AccessKeyId**, **SecretAccessKey**, and **SessionToken**. 1. When prompted by the Failback Client, enter: + **AWS Access Key ID** – the `AccessKeyId` value + **AWS Secret Access Key** – the `SecretAccessKey` value + **AWS Session Token** – the `SessionToken` value + **AWS Region** – the Region where your Recovery Instance resides **Note** Temporary credentials expire after a default session duration of 1 hour. Ensure you complete the Failback Client initialization before they expire. Learn more about creating a role to delegate permissions to an AWS service [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html). Attach this policy to the role: **AWSElasticDisasterRecoveryFailbackInstallationPolicy**. ### Failback Client detailed walkthrough Once you are ready to perform a failback to your original source servers or to different servers, take these steps: **Note** Replication from the source instance to the source server (in the target AWS Region) will continue when you perform failback on a test machine. 1. Complete the recovery [ as described above](failback-preparing-failover.md). 1. Configure your failback replication settings on the recovery instances you want to fail back. [Learn more about failback replication settings.](recovery-instances-details.md#recovery-instances-details-failback-replication-settings) 1. Download the AWS Elastic Disaster Recovery Failback Client ISO (aws-failback-livecd-64bit.iso) from the S3 bucket that corresponds to the AWS Region in which your recovery instances are located. 1. Direct download link: Failback Client ISO: ` https://aws-elastic-disaster-recovery-{REGION}.s3.{REGION}.amazonaws.com/latest/failback_livecd/aws-failback-livecd-64bit.iso ` 1. Failback Client ISO hash link: ` https://aws-elastic-disaster-recovery-hashes-{REGION}.s3.{REGION}.amazonaws.com/latest/failback_livecd/aws-failback-livecd-64bit.iso.sha512 ` 1. Boot the Failback Client ISO on the server you want to fail back to. This can be the original source server that is paired with the recovery instance, or a different server. **Important** Ensure that the server you are failing back to has the same number of volumes or more than the Recovery Instance and that the volume sizes are equal to or larger than the ones on the recovery instance. **Note** When performing a recovery **for a Linux server**, you must boot the Failback Client with BIOS boot mode. When performing a recovery **for a Windows server**, you must boot the Failback Client with the same boot mode (BIOS or UEFI) as the Windows source server. 1. If you plan on using a static IP for the Failback Client, run the following once the Failback Client ISO boots: `IPADDR="enter IPv4 address" NETMASK="subnet mask" GATEWAY="default gateway" DNS="DNS server IP address" CONFIG_NETWORK=1 /usr/bin/start.sh ` For example, `IPADDR="192.168.10.20" NETMASK="255.255.255.0" GATEWAY="192.168.10.1" DNS="192.168.10.10" CONFIG_NETWORK=1 /usr/bin/start.sh ` 1. Enter your AWS credentials, including your **AWS Access Key ID** and **AWS Secret Access Key** that you created for Failback Client installation, the **AWS Session Token** (if you are using temporary credentials – users who are not using temporary credentials can leave this field blank), and the **AWS Region** in which your Recovery instance resides. You can attach the Elastic Disaster Recovery Failback Client credentials policy to a user or create a role and attach the policy to that role to obtain temporary credentials. [Learn more about Elastic Disaster Recovery credentials.](#failback-performing-credentials) ![\[AWS credentials input fields for Failback Client, including Access Key ID, Secret Key, and Region.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-credentials.png) 1. Enter the custom endpoint or press Enter to use the default endpoint. You should enter a custom endpoint if you want to use a VPC Endpoint (PrivateLink). ![\[Text input field for entering a custom endpoint or leaving blank for default.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-credentials2.png) 1. If you are failing back to the original source machine, the Failback Client will automatically choose the correct corresponding recovery instance. ![\[Command line output showing automated instance detection matching Recovery instance with Failback Client.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient1.png) 1. If the Failback Client is unable to automatically map the instance, then you will be prompted to select the recovery instance to fail back from. The Failback Client displays a list with all recovery instances. Select the correct recovery instance by either entering the numerical choice from the list that corresponds to the correct recovery instance or by typing in the full recovery instance ID. ![\[Command line interface showing manual selection of a recovery instance for failback.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient3.png) **Note** The Failback Client will only display recovery instances whose volume sizes are equal to or smaller than the volume sizes of the server you’re failing back to. If the recovery instance has volume sizes that are larger than that of the server you are failing back to, then these Recovery instances will not be displayed. 1. If you are failing back to the original source server, then the Failback Client will attempt to automatically map the volumes of the instance. ![\[Output showing local and remote devices with 8.0 GB storage capacity each.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient2.png) 1. If the Failback Client is unable to automatically map the volumes, you will need to manually enter a local block device (example /dev/sdg) to replicate to from the remote block device. Enter the `EXCLUDE` command to specifically exclude Recovery Instance volumes from replication. Optionally, you can also enter the complete volume mapping in the same CSV or JSON format used by --device-mapping Failback Client argument. For example: `ALL="/dev/nvme2n1=/dev/sda,/dev/nvme0n1=EXCLUDE, . . ."`. The full volume mapping should be provided as single CSV or JSON line in the format of --device-mapping Failback Client argument. [Learn more about using --device-mapping program argument](#failback-failover-program-arg-device-mapping) ![\[Terminal output showing manual volume mapping process with successful result.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient4.png) **Important** The local volumes must be the same in size or larger than the recovery instance volumes. The valid special case is when original local volume has fractional GiB size (e.g. 9.75 GiB). Then the recovery instance volume size will be larger because of rounding to nearest GiB (e.g. 10 GiB). 1. The Failback Client will verify connectivity between the recovery instance and AWS Elastic Disaster Recovery. ![\[Command line interface showing successful connectivity establishment to Dirrus service.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient5.png) 1. The Failback Client will download the replication software from a public S3 bucket onto the source server. ![\[Terminal output showing successful download of AWS Replication Software.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient6.png) **Important** You must allow traffic to S3 from the source server for this step to succeed. 1. The Failback Client will configure the replication software. ![\[Console output showing AWS Replication Software configuration completed successfully.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient7.png) 1. The Failback Client will pair with the AWS Replication Agent running on the recovery instance and will establish a connection. ![\[Console output showing successful pairing and connection establishment between Failback Client and AWS Replication Agent.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient8.png) **Important** TCP Port 1500 inbound must be open on the recovery instance for the pairing to succeed. 1. Data replication will begin. ![\[Terminal window showing text "Connection established. Replication in progress..."\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drs-failback-failbackclient9.png) You can monitor data replication progress on the **Recovery instances** page in the AWS Elastic Disaster Recovery Console. 1. Once data replication has been completed, the Recovery instance on the **Recovery instances** page will show the **Ready** status under the **Failback state** column and the **Healthy** status under the **Data replication status** column. 1. Once all of the recovery instances you are planning to fail back show the statuses above, select each Instance and choose **Failback**. This will stop data replication and will start the conversion process. This will finalize the failback process and create a replica of each recovery instance on the corresponding source server. Select one or more recovery instances that are in the **Ready** state and click **Failback** to continue the failback process after performing a failback with the Elastic Disaster Recovery Failback Client. This action will stop data replication and will start the conversion process. This will finalize the failback process and will create a replica of each recovery instance on the corresponding source server. When the **Continue with failback for X instances** dialog appears, click **Failback**. This action will create a Job, which you can follow on the **Recovery job history** page. [Learn more about the recovery job history page.](recovery-job.md) 1. Once the failback is complete, the Failback Client will show that the failback has been completed successfully. You can reboot the server and check that it has the needed data, before proceeding. **Note** The server client iso should not be in the boot order when you want to recover into the original OS. 1. You can opt to either terminate, delete, or disconnect the Recovery instance. [Learn more about each action.](monitoring-recovery-instances.md#recovery-instances-actions) ### Failback Client program arguments The arguments supported by Failback Client LiveCD process are: + --aws-access-key-id AWS\$1ACCESS\$1KEY\$1ID + --aws-secret-access-key AWS\$1SECRET\$1ACCESS\$1KEY + --aws-session-token AWS\$1SESSION\$1TOKEN + --region REGION + --endpoint ENDPOINT + --default-endpoint + --recovery-instance-id RECOVERY\$1INSTANCE\$1ID + --dm-value-format \$1dev-name,by-path,by-id,by-uuid,all-strict\$1 + --device-mapping DEVICE\$1MAPPING] [--no-prompt + --log-console + --log-file LOG\$1FILE All arguments are optional. #### [--device-mapping DEVICE\$1MAPPING] `--device-mapping` argument will skip mapping auto-detection and manual mapping and use the mapping provided in this parameter. There are three formats supported: 1. Classic CE format of key-value CSV string as one line. You may use either ":" or "=" as CSV fields separator which is more suitable for Windows drive letters. Examples are: ``` recovery_device1=local_device1,recovery_device2=local_device2,recovery_device3=EXCLUDE, . . . ``` ``` recovery_device1:local_device1,recovery_device2:local_device2, . . . ``` 1. JSON format: ``` '{"/dev/xvdb":"/dev/sdb","/dev/xvdc":"/dev/sdc","recovery_device3":"local_device3"}' ``` 1. JSON list DRS API format: ``` '[{"recoveryInstanceDeviceName": "recovery_device1","failbackClientDeviceName": "local_device1"},{"recoveryInstanceDeviceName" . . .: }]' ``` No matter which format you choose, you need to provide either valid Failback Client device name or EXCLUDE for each Recovery Instance device. #### [dm-value-format DM\$1VALUE\$1FORMAT] `--dm-value-format` allows to use Failback Client persistent block devices identifiers in --device-mapping argument. Such persistent identifiers will always refer to the same block devices after Failback Client reboot. Possible --dm-value-format choices are: 1. "dev-name" - default format for using /dev/sda, /dev/xvda, /dev/nvme3n1 etc 1. "by-path" - from ls -l /dev/disk/by-id/ e.g. pci-0000:00:10.0-scsi-0:0:3:0, pci-0000:00:1e.0-nvme-1, pci-0000:02:01.0-ata-1, xen-vbd-768 etc 1. "by-id" - from ls -l /dev/disk/by-id/ e.g. device serial numbers 1. "by-uuid" - UUIDs from ls -l /dev/disk/by-uuid/ 1. "all-strict" - all of the above mixed We will use the example of SCSI identifiers from the command output below: ``` # root@ubuntu:~# ls -l /dev/disk/by-path/ total 0 lrwxrwxrwx 1 root root 9 Jun 27 12:25 pci-0000:00:10.0-scsi-0:0:0:0 -> ./../sda lrwxrwxrwx 1 root root 10 Jun 27 12:25 pci-0000:00:10.0-scsi-0:0:0:0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 9 Jun 27 12:25 pci-0000:00:10.0-scsi-0:0:1:0 -> ../../sdb lrwxrwxrwx 1 root root 9 Jun 27 12:25 pci-0000:00:10.0-scsi-0:0:2:0 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 27 12:25 pci-0000:00:10.0-scsi-0:0:3:0 -> ../../sdd ``` To use block device SCSI identifiers like 'pci-0000:00:10.0-scsi-0:0:0:0' you need to add to command line:`--dm-value-format by-path` The examples of valid --device-mapping for `--dm-value-format by-path` are: ``` /dev/nvme2n1=pci-0000:00:10.0-scsi-0:0:0:0,/dev/nvme0n1=pci-0000:00:10.0-scsi-0:0:1:0,/dev/nvme3n1=pci-0000:00:10.0-scsi-0:0:2:0... ``` ``` '{"/dev/nvme2n1":"pci-0000:00:10.0-scsi-0:0:0:0","/dev/nvme0n1":"pci-0000:00:10.0-scsi-0:0:1:0","/dev/nvme3n1":"pci-0000:00:10.0-scsi-0:0:2:0", . . .}' ``` No matter which format you choose, you need to provide either valid Failback Client device name or EXCLUDE for each Recovery Instance device. # Performing a failback with the DRS Mass Failback Automation Client DRS allows you to perform a scalable failback for vCenter with the DRS Mass Failback Automation Client (DRSFA Client). This allows you to perform a one-click or custom failback for multiple vCenter machines at once. **Note**  The DRSFA client only works with vCenters source servers. **Note**  The DRSFA client was only tested on vCenter versions 6.7 and 7.0. ## DRSFA prerequisites These are the prerequisites for performing failback automation with the DRSFA client: 1. Ensure that you meet all of the [network requirements](preparing-environments.md). 1. Ensure that you have [initialized DRS](getting-started-initializing.md). 1. Each server that is being failed back must have at least 3 GB of RAM. 1. Each server that is being failed back must have the hardware clock set to UTC rather than Local Time. 1. The recovery instance used as a source for failback must have permissions to access AWS Elastic Disaster Recovery via API calls. This is done using instance profile for the underlying EC2 instance. The instance profile must include the AWSElasticDisasterRecoveryRecoveryInstancePolicy in addition to any other policy you require the EC2 instance to have. By default, the launch settings that DRS creates for source servers already have an instance profile defined that includes that policy and that instance profile will be used when launching a Recovery Instance. 1. Inbound port TCP 1500 must be open on the Recovery instance in AWS. 1. The server on which the DRSFA client is run needs to be able to communicate with your vCenter environment. 1. The server on which the DRSFA client is run must have at least 4 GB of RAM. 1. The server on which the DRSFA client is run must run Python 3.9.4 with pip installed (other versions of Python will not work). **Note** The installation procedure shown below uses Ubuntu 20.04 running Python 3.9.4. 1. The server on which the DRSFA client is run requires these tools for DRSFA Client installation. The installer will attempt to install them if they are not already present: build-essential curl genisoimage git libbz2-dev libffi-dev liblzma-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libssl-dev llvm make tk-dev unzip wget xz-utils zlib1g-dev 1. To see the list of python libraries required for the DRSFA Client to run, see the requirements.txt file (https://drsfa-us-west-2.s3.us-west-2.amazonaws.com/requirements.txt). These libraries will be installed automatically by DRSFA Client. 1. The vCenter source servers must have two CD ROM devices with IDE controllers attached to run the DRSFA client - one for the DRS Failback Client and one for the drs\$1failback\$1automation\$1seed.iso **Note** If no attached CD ROM devices are found, the DRSFA client will attempt to add the CD ROM devices. 1. The DRS Failback Client must be uploaded to your vCenter Datastore. 1. We recommend using the latest version of the DRS Failback Client. Download the [latest version of the DRS Failback Client](failback-performing.md#failback-performing-performing)and upload it to your vCenter datastore. 1. We recommend running SHA512 checksum verification on the DRS Failback Client prior to using it with the DRSFA client. You can verify the checksum at this address: ` https://aws-elastic-disaster-recovery-hashes-{REGION}.s3.amazonaws.com/latest/failback_livecd/aws-failback-livecd-64bit.iso.sha512 ` 1. We recommend running SHA512 checksum verification on the drs\$1failback\$1automation\$1seed.iso file prior to using it with the DRSFA client. 1. The DRSFA client does not require root privileges. We recommend low privileges for running the client. 1. You need to have these vCenter API credentials and permissions: ‘Virtual machine’ : [ ‘Change Settings’, ‘Guest operation queries’, ‘Guest operation program execution’, ‘Connect devices’, ‘Power off’, ‘Power on’. ‘Add or remove device’, ‘Configure CD media] ‘Datastore’: [‘Browse datastore’] 1. vCenter credentials should only be constrained to the VMs you plan to failback. 1. You should be able to fail back all of the Recovery instances in a single AWS Region simultaneously with the aid of the DRSFA Client as long as your vCenter hardware supports the failback load. ### Security best practices These are security best practices for using the DRSFA Client: 1. Follow the least privilege principle and set the appropriate permissions on the folder where the JSON generated by the client will be stored. 1. Ensure that you are always using the latest version of the DRSFA Client. The client will automatically check and verify that you are using the latest version upon startup. 1. You should not provide any additional permissions to the DRSFA Client other than the ones listed in the prerequisites. 1. Ensure that you follow the [ AWS recommended password policy ](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html) when setting the password for the VM that hosts the DRS Failback Client when generating the drs\$1failback\$1automation\$1seed.iso file. 1. Ensure that you manually verify the DRSFA client hashes when automatic hash verification is not performed. The hash verification hint is shown when the DRSFA client is installed. 1. Ensure that only trusted administrators have access to the vCenter environment. The DRSFA Client will consider the customer executing scripts and every person with access to the datastore as a single trust entity 1. We suggest performing a hash verification on the DRS Failback Client and the drs\$1failback\$1automation\$1seed.iso file before proceeding. The hash is exported to the `drs_failback_automation_seed.iso.sha512` file once the seed iso is created. 1. We suggest using low level privilege when running the DRSFA client. 1. We suggest following the least privilege principle and setting the appropriate permissions on the folder where the Failback Client and seed.iso files will be stored. 1. The vCenter credentials used should only have permissions to the VMs involved in the failback attempt. ## Installing the DRSFA Client Prior to running the DRSFA Client, you must first install it. Installing the client is a one-time operation. The DRSFA client was fully tested on Ubuntu 20.04 and an installation script for this version is provided. Use this vanilla AMI or public ISO to run the client locally in your vCenter environment. Follow the [Create your EC2 resources and launch your EC2 instance](https://docs.aws.amazon.com/efs/latest/ug/gs-step-one-create-ec2-resources.html) guidelines as per the EC2 documentation. When asked to select an AMI, select the option below instead of the Amazon Linux 2 AMI and then proceed according to the documentation. Use this AMI from EC2: Ubuntu Server 20.04 LTS (HVM), SSD Volume Type: ![\[Ubuntu Server 20.04 LTS (HVM) option with SSD volume type and virtualization details.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/failback-drsfa-1.png) Download the Ubuntu Server 20.04 LTS server install image ISO from the [Ubuntu download site](https://releases.ubuntu.com/20.04.3/ubuntu-20.04.3-live-server-amd64.iso?_ga=2.226405082.1942739102.1640275732-1020059774.1638346768). Once your VM instance is set up and ready, connect to the Ubuntu instance and run command prompt and download the DRSFA client using this command: `wget https://drsfa-us-west-2.s3.us-west-2.amazonaws.com/drs_failback_automation_installer.sh` ![\[Terminal output showing successful download of a DRS failback automation installer script.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa9.png) **Note** You should verify the hash of the installer after running the installation command: `https://drsfa-hashes-us-west-2.s3.us-west-2.amazonaws.com/drs_failback_automation_installer.sh.sha512` Use this command to execute the installation script: `bash drs_failback_automation_installer.sh` ![\[Terminal output showing HTTP request, file saving, and installation of DRS Mass Failback Automation.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa10.png) ![\[Terminal window showing ls command output with three drs_failback_automation files listed.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa11.png) **Note** This command may ask for a sudo password if you use the Ubuntu ISO. Enter the password but **do not** run this command as sudo. `source ~/.profile ` ![\[Terminal window showing command to source the .profile file.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa12.png) The DRSFA client has a one-time installation. The DRSFA client will be installed in the `drs_failback_automation_client` directory. Once you've successfully run the command above and installed the client, you can delete the DRSFA client installer from your server by running this command: `rm drs_failback_automation_installer.sh` ![\[Terminal commands showing removal of an installer file and listing remaining files.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa13.png) Once installation is complete, you will need to set up a password for the VM on which the DRSFA client is run. This is done by generating a seed.iso file that you must upload to your Datastore. Run these commands to generate the seed.iso file: `bash drs_failback_automation_seed_creator.sh` You will be prompted to enter a password. Ensure that you enter a unique password that follows the [AWS recommended password policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html). ![\[Terminal window showing HTTP request, file saving, and password prompt for generating an ISO file.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa14.png) Two files will be generated, the `drs_failback_automation_seed.iso` file and the `drs_failback_automation_seed.iso.sha512` hash. Upload the seed.iso file to the same Datastore where the DRS Failback Client ISO file is stored. ![\[Terminal output showing DRS failback automation files including seed ISO and hash.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa15.png) Once the `drs_failback_automation_seed.iso` file is generated, you can run this command to delete the seed creator: `rm drs_failback_automation_seed_creator.sh` ![\[Terminal command removing a file and listing directory contents showing remaining files.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa17.png) Once you have completed the initial installation, you can generate the required credentials and run the DRSFA client. ## Generating IAM credentials and configuring Cloudwatch logging In order to run the DRSFA Client, you must first generate the required AWS credentials. **Important** Temporary credentials have many advantages. You don't need to rotate them or revoke them when they're no longer needed, and they cannot be reused after they expire. You can specify for how long the credentials are valid, up to a maximum limit. Because they provide enhanced security, using temporary credentials is considered best practice and the recommended option. ### Temporary credentials To create temporary credentials: 1. [Create a new IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) with the [ AWSElasticDisasterRecoveryFailbackInstallationPolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSElasticDisasterRecoveryFailbackInstallationPolicy.html) policy. 1. Request temporary security credentials [via AWS STS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) using the [AssumeRole API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Once your credentials are generated, you should create a logGroup for CloudWatch logging named **DRS\$1Mass\$1Failback\$1Automation**. If this log group is not created or if it's created with the wrong name, the DRSFA client will still work, but logs will not be sent to CloudWatch. Learn more about working with log groups in the [ Amazon CloudWatch Logs documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html). ## Running the DRSFA client Once you have installed the DRSFA client, you can run it by following these instructions: `cd` into the `drs_failback_automation_client` directory and enter these parameters in a single line or setting the environment variables one by one, replace the defaults with your specific parameters and paths followed by the `python drs_failback_automation_init.pyc` command and press enter. ![\[Terminal commands showing directory navigation and file listing in a Linux environment.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa18.png) + AWS\$1REGION=XXXXX – The AWS Region in which your Recovery instances are located. + AWS\$1ACCESS\$1KEY=XXXXX – The AWS Access Key you generated for the DRSFA client. + AWS\$1SECRET\$1ACCESS\$1KEY=XXXXXX – The AWS Secret Access Key you generated for the DRSFA client. + AWS\$1SESSION\$1TOKEN=XXXXXX – (Optional) The AWS Session Token you generated for the DRSFA client. + DRS\$1FAILBACK\$1CLIENT\$1PASSWORD = XXXXXX – The custom password you set for the Failback Client in the drs\$1failback\$1automation\$1seed.iso file. + VCENTER\$1HOST=XX.XX.XXX.XXX – The IP address of the vCenter Host. + VCENTER\$1PORT=XXX – The vCenter Port (usually 443) + VCENTER\$1USER=sample@vsphere.local – The vCenter username + VCENTER\$1PASSWORD=samplepassword – The vCenter password + VCENTER\$1DATASTORE=DatastoreX – The Datastore within vCenter where the Failback Client ISO file (aws-failback-livecd-64bit.iso) and seed.iso file (drs\$1failback\$1automation\$1seed.iso) are stored. + VCENTER\$1FAILBACK\$1CLIENT\$1PATH='samplepath/aws-failback-livecd-64bit.iso' – Failback Client ISO path in the Datastore. + VCENTER\$1SEED\$1ISO\$1PATH='samplepath/drs\$1failback\$1automation\$1seed.iso' – The seed.iso file path in the Datastore. Enter all of the parameters in a single line or enter the environmental variables individually one by one. Once you have entered your parameters, enter the `python drs_failback_automation_init.pyc` command and press enter. The full parameters and command should look like this example: `AWS_REGION=XXXX AWS_ACCESS_KEY=XXXX AWS_SECRET_ACCESS_KEY=XXXX DRS_FAILBACK_CLIENT_PASSWORD=XXXX VCENTER_HOST=XXXX VCENTER_PORT=XXXX VCENTER_USER=XXXX VCENTER_PASSWORD=XXXX VCENTER_DATASTORE=XXXX VCENTER_FAILBACK_CLIENT_PATH=XXXX VCENTER_SEED_ISO_PATH=XXXX python drs_failback_automation_init.pyc ` ![\[Terminal output showing successful update of DRS Mass Failback Automation Client.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa52.png) **Note** SSL verification is active by default. If you want to deactivate SSL verification, then add this parameter: DISABLE\$1SSL\$1VERIFICATION=true By default, the DRSFA client initiates a failback for 10 servers at once (if failing back more than 10 servers). To change the default value, use the THREAD\$1POOL\$1SIZE parameter. ## One-click failback Once the client has connected successfully and finished verification, select the **One-Click Failback** option under **What would you like to do?** ![\[CLI menu for DRS Mass Failback Automation with options numbered 1 to 6.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa20.png) Enter a custom prefix for the results output for this failback operation. This file is saved in the `/drs_failback_automation_client/results/Failback` directory. ![\[Text input field for entering a custom prefix for failback operation results output.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa31.png) If failback replication has already been started for some of the Recovery instances, the console prompts you to decide if you want to skip the instances that are already in failback or restart replication for those instances. ![\[Console prompt asking whether to restart machines, with options to skip or restart all instances.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa22.png) The DRSFA client will list the Recovery instances that are currently present in your AWS Account. The client will then prompt you **Would you like to continue? **. Enter **Y** to continue. ![\[Command prompt showing Recovery instances to be failed back and a confirmation prompt.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa23.png) The client will initiate failback. You can see the failback progress on the **Recovery instances** page in the DRS Console. ![\[Console output showing server replication progress over time in the eu-west-1 region.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa24.png) Once the failback has been completed, the DRSFA client displays the results of the failback, including the number of servers for which replication has successfully been initiated and the number of servers for which the failback operation failed. The full results of the failback will be exported as a JSON file to the failback client folder path under the `/drs_failback_automation_client/results/Failback` folder with the custom prefix you set, the AWS account ID, the AWS Region, and a timestamp. The JSON file displays: + The AWS ID of the Recovery instance + The status of the failback (succeeded, skipped, or failed) + A message (which provides the cause for failure in the case of failure) + The vCenter VM UUID ![\[JSON output showing replication status as "succeeded" with progress message for two items.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa53.png) If failback failed for any of your machines, you can troubleshoot the failure by looking at the machine configuration `failback_hosts_settings.json` file in the same folder. ![\[JSON configuration file showing network settings with static IP and automatic device mapping.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa54.png) Here, you can see the exact configurations of the failed machines. You can then fix any problems and use the custom failback flow explained below to fail back these specific machines. ## Custom failback The custom failback option gives you more control and flexibility over the failback process. When utilizing the custom failback option, you will first create a failback configuration file, in which you can edit specific settings for each individual machine, and you will then use this file to perform a failback in a flow that is similar to that of the one-step failback. ### Generating the configuration file To use the custom failback option, you can either create a custom configuration JSON file or generate a default failback configuration file through the client. To generate a default failback configuration file, once the client has connected successfully and finished verification, select the **Generate a default failback configuration file** option under **What would you like to do?** ![\[CLI menu showing options for DRS Mass Failback Automation, with cursor on option 3.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa26.png) Enter a custom prefix for the configuration file name. The configuration file will be created as a JSON file in the `/drs_failback_automation_client/` `Configurations` /folder with the name: "\$1prefix\$1\$1\$1account\$1id\$1\$1\$1region\$1.json" ![\[Command line interface showing custom prefix input and default configuration file creation.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa27.png) You can edit any of the fields in the file in order to correctly configure it. The file displays these fields for each machine. You can edit every field to have absolute control over your failback configuration for each machine. Ensure to save your changes. + NETMASK + VCENTER\$1MACHINE\$1UUID + PROXY + DNS + CONFIG\$1NETWORK + IPADDR + GATEWAY + SOURCE\$1SERVER\$1ID + DEVICE\$1MAPPING **Note** The `CONFIG_NETWORK` value should be set to "DHCP" if you are using DHCP. The value should be set to "STATIC" if you want to manually configure the network settings. If CONFIG\$1NETWORK is set to "DHCP", then the `DNS, IPADDR, GATEWAY, NETMASK`, and `PROXY` parameters are ignored but should not be deleted. If you are using a proxy server, leave the `PROXY` field as an empty string, do not remove it. If a source server does not have an attached recovery instance, the file will still be generated, but the **SOURCE SERVER ID** field will be empty. You can edit any of the fields in the file in order to correctly configure it. The file displays these fields for each machine. You can edit every field to have absolute control over your failback configuration for each machine. Ensure to save your changes. ### Custom device mapping parameter Custom "DEVICE\$1MAPPING" field is passed to the LiveCD failback process as --device-mapping argument. [Learn more about using --device-mapping program argument](failback-performing.md#failback-failover-program-arg-device-mapping) There are three formats supported: 1. Classic CE format of key-value CSV string as one line. You may use either ":" or "=" as CSV fields separator which is more suitable for Windows drive letters. Examples are: ``` "DEVICE_MAPPING": "recovery_device1=local_device1,recovery_device2=local_device2,recovery_device3=EXCLUDE" ``` ``` "DEVICE_MAPPING": "recovery_device1:local_device1,recovery_device2:local_device2" ``` 1. JSON format: ``` "DEVICE_MAPPING": { "/dev/xvdb":"/dev/sdb", "/dev/xvdc":"/dev/sdc", "recovery_device3":"local_device3" } ``` 1. JSON list DRS API format: ``` [ { "recoveryInstanceDeviceName": "recovery_device1", "failbackClientDeviceName": "local_device1" }, { "recoveryInstanceDeviceName": "recovery_device2", "failbackClientDeviceName": "local_device2" } ] ``` No matter which format you choose, you need to provide either valid Failback Client device name or EXCLUDE for each Recovery Instance device. ### Performing the custom failback Once you are done editing your configuration file, rerun the DRSFA client and select the **Perform a Custom Failback** option. ![\[CLI menu for DRS Mass Failback Automation with 6 numbered options.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa28.png) Select your configuration file. You can either define a custom path or select the default path that's automatically displayed by the client. ![\[CLI menu for DRS Mass Failback Automation with options and configuration file selection.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa29.png) ![\[DRS Mass Failback Automation CLI menu with options for failback operations and configuration.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa30.png) Enter a custom prefix for the results output for this failback operation. This file is saved in the `/drs_failback_automation_client/Results/Failback` directory. ![\[Text input field for entering a custom prefix for failback operation results output.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa31.png) If failback replication has already been started for some of the recovery instances, the console prompts you to decide if you want to skip the instances that are already in failback or restart replication for those instances. ![\[Console prompt asking whether to restart machines, with options to skip or restart all instances.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa22.png) The Client will identify the recovery instances that will be failed back to their original VMs and list them. The client will then prompt you whether you would like to continue. Choose **Y** to continue. ![\[Command prompt showing Recovery instances to be failed back and a confirmation prompt.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa23.png) The Client will initiate failback. You can see the failback progress on the **Recovery instances** page in the AWS DRS Console. ![\[Console output showing server replication progress over time in the eu-west-1 region.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa24.png) Once the failback has been completed, the DRSFA client displays the results of the failback, including the number of servers for which replication has successfully been initiated and the number of servers for which the failback operation failed. The full results of the failback will be exported as a JSON file to the failback client folder path under the `/drs_failback_automation_client/Results/Failback` folder with the custom prefix you set, the AWS account ID, the AWS Region, and a timestamp. The JSON file displays: + The AWS ID of the Recovery instance + The status of the failback (succeeded, skipped, or failed) + A message (which provides the cause for failure in the case of failure) + The vCenter VM UUID + The vCenter UUID of the original source server ![\[JSON output showing replication status as "succeeded" with progress message for two items.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa53.png) If failback failed for any of your machines, you can troubleshoot the failure by looking at the machine configuration `failback_hosts_settings.json` file in the same folder. ![\[JSON configuration file showing network settings with static IP and automatic device mapping.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa54.png) Here, you can see the exact configurations of the failed machines. You can then fix any problems and use the custom failback flow explained below to fail back these specific machines. ## Find servers in vCenter Select the **Find servers in vCenter** option to find machines in vCenter. This makes it easier to discover the disks/volumes of your machines for custom failback. ![\[CLI menu showing options for DRS Mass Failback Automation, with "Find servers in vCenter" highlighted.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa32.png) Enter a name to filter or press Enter to see all results. Choose **Yes** to print your results. ![\[Command-line interface showing options for failback operations and VM search results.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa33.png) The results will be exported to the `Results/VMFinder` folder in the DRSFA client folder. The results will be named after the vCenter IP and the time stamp. `{vcenter_host}_{ts}.txt` These are displayed for each server: + Name + UUID + Disk and volume info ![\[Virtual machine details showing Windows 2019 20GB with disk information and specifications.\]](http://docs.aws.amazon.com/drs/latest/userguide/images/drsfa34.png) ## Upgrading the DRSFA Client Most of DRSFA components are upgraded automatically upon execution. However, in certain scenarios, you will see a message informing you that you need to upgrade the DRSFA Client manually. To complete the upgrade, take these steps: 1. Change directory (cd) into the directory where the installation originally took place. 1. Download the DRSFA installer: `wget https://drsfa-us-west-2.s3.us-west-2.amazonaws.com/drs_failback_automation_installer.sh` **Note** You should verify the hash of the installer after running the installation command: `https://drsfa-hashes-us-west-2.s3.us-west-2.amazonaws.com/drs_failback_automation_installer.sh.sha512` 1. Run the installer. `bash drs_failback_automation_installer.sh` 1. Remove the installer. `rm drs_failback_automation_installer.sh` ## Troubleshooting + To troubleshoot the DRSFA Client, review the `drs_failback_automation.log` file that is generated in the `/drs_failback_automation_client/` folder on the server from which the client is run. + To find the log for a specific server, open the VM, and find the `drs_failback_automation.log` and `failback.log` file, which can be used for troubleshooting. ## Using the failback client to perform a failback to the original source server When using the failback client, you can fail back to the original source server or a different source server using AWS Elastic Disaster Recovery. To ensure that the original source server has not been deleted and still exists, check its status in the AWS DRS console. Source servers that have been deleted or no longer exist will show as having **Lag** and being **Stalled**. **Note** After failing back to the original source server, you don't need to reinstall the DRS agent to start replication back to AWS. If the original source server is healthy and you decide to fail back to it, it will undergo a rescan until it reaches the **Ready** status. You can tell whether you are failing back to the original or a new source server in the recovery instance details view under **Failback status**.