Creating EBS volume backups with AMIs and EBS snapshots
AWS provides a wealth of options for creating and managing AMIs and snapshots. You can use the approach that meets your needs. A common issue that many customers face is managing the snapshot lifecycle and clearly aligning snapshots by purpose, retention policy, etc. Without proper tagging, there is a risk that snapshots might be deleted accidentally or as part of an automated cleanup process. You might also end up paying for obsolete snapshots that are retained because there is no clear understanding whether they are still needed.
Preparing an EBS volume before creating a snapshot or AMI
Before you take a snapshot or create an AMI, make the necessary preparations to your EBS volume. Creating an AMI results in a new snapshot for each EBS volume that is attached to the instance, so these preparations also apply to AMIs.
You can take a snapshot of an attached EBS volume that is in use by a powered-on EC2 instance. However, snapshots capture only data that has been written to your EBS volume at the time the snapshot command is issued. This might exclude any data that has been cached by applications or the operating system. A best practice is to have the system in a state where it is not performing any I/O. Ideally, the machine isn’t accepting traffic and is in a stopped state, but this is rare as 24/7 IT operations become the norm. If you can flush any data from system memory to the disk being used by your applications and pause any file writes to the volume long enough to take a snapshot, your snapshot should be complete.
To make a clean backup, you must quiesce the database or file system. The way in which you do this depends on your database or file system.
The process for a database is as follows:
-
If possible, put the database into hot backup mode.
-
Run the Amazon EBS snapshot commands.
-
Take the database out of hot backup mode or, if using a read replica, terminate the read replica instance.
The process for a file system is similar, but it depends on the capabilities of
the operating system or file system. For example, XFS is a file system that can
flush its data for a consistent backup. For more information, see xfs_freeze
However, if you can't flush or pause all file writes to the volume, do the following:
-
Unmount the volume from the operating system.
-
Issue the snapshot command.
-
Remount the volume to achieve a consistent and complete snapshot. You can remount and use your volume while the snapshot status is pending.
The snapshot process continues in the background and snapshot creation is fast and captures a point in time. The volumes that you’re backing up are unmounted for only a matter of seconds. You can schedule a small backup window where an outage is expected and handled by clients gracefully.
When you create a snapshot for an EBS volume that serves as a root device, stop the instance before you take the snapshot. Windows provides the Volume Shadow Copy Service (VSS) to help create application-consistent snapshots. AWS provides a Systems Manager document that you can run to take image-level backups of VSS-aware applications. The snapshots include data from pending transactions between these applications and the disk. You don’t have to shut down your instances or disconnect them when you back up all attached volumes. For more information, see the AWS documentation.
Note
If you are creating a Windows AMI so that you can deploy another similar
instance, use EC2Config or EC2Launch
Creating EBS volume snapshots manually from the console
Create snapshots of the appropriate volumes or the entire instance before you make any major changes that have not been fully tested on the instance. For example, you might want to create a snapshot before you upgrade or patch application or system software on your instance.
You can create a snapshot manually from the console. On the Amazon EC2 console, on the Elastic Block Store Volumes page, select the volume that you want to back up. Then on the Actions menu, choose Create Snapshot. You can search for volumes that are attached to a specific instance by entering the instance ID in the filter box.
Enter a description and add the appropriate tags. Add a Name tag to
make it easier to find the volume later. Add any other appropriate tags based on
your tagging strategy.
Creating AMIs
An AMI provides the information that is required to launch an instance. The AMI includes the root volume and snapshots of the EBS volumes attached to the instance when the image was created. You can’t launch new instances from EBS snapshots alone; you must launch new instances from an AMI.
When you create an AMI, it is created in the account and Region that you are using. The AMI creation process creates Amazon EBS snapshots for each volume attached to the instance, and the AMI refers to these Amazon EBS snapshots. These snapshots reside in Amazon S3 and are highly durable.
After you create an AMI of your EC2 instance, you can use the AMI to re-create the instance or launch more copies of the instance. You can also copy AMIs from one Region to another for application migration or DR.
An AMI must be created from an EC2 instance unless you are migrating a virtual machine, such as a VMWARE virtual machine, to AWS. To create an AMI from the Amazon EC2 console, select the instance, choose Actions, choose Image, and then choose Create Image.
Amazon Data Lifecycle Manager
To automate the creation, retention, and deletion of Amazon EBS snapshots, you can use Amazon Data Lifecycle Manager. Automating snapshot management helps you to do the following:
-
Protect valuable data by enforcing a regular backup schedule.
-
Retain backups as required by auditors or internal compliance.
-
Reduce storage costs by deleting outdated backups.
Using Amazon Data Lifecycle Manager, you can automate the snapshot management process for EC2 instances (and their attached EBS volumes) or separate EBS volumes. It supports options such as cross-Region copy, so you can copy snapshots automatically to other AWS Regions. Copying snapshots to alternative Regions is one approach to support DR efforts and restore options in an alternative Region. You can also use Amazon Data Lifecycle Manager to create a snapshot lifecycle policy that supports fast snapshot restore.
Amazon Data Lifecycle Manager is an included feature of Amazon EC2 and Amazon EBS. There is no charge for Amazon Data Lifecycle Manager.
AWS Backup
AWS Backup is unique from Amazon Data Lifecycle Manager because you can create a backup plan that includes resources across multiple AWS services. You can coordinate your backup to cover the resources you are using together rather than coordinating the backups of the resources individually.
AWS Backup also includes the concept of backup vaults, which can restrict access to the recovery points for your completed backups. Restore operations can be initiated from AWS Backup rather than proceeding to each individual resource and restoring the backup created. AWS Backup also includes a host of additional features, such as audit management and reporting. For more information, see the Backup and recovery using AWS Backup section of this guide.
Performing multi-volume backups
If you want to back up the data on the EBS volumes in a RAID array using snapshots, the snapshots must be consistent. This is because the snapshots of these volumes are created independently. Restoring EBS volumes in a RAID array from snapshots that are out of sync degrades the integrity of the array.
To create a consistent set of snapshots for your RAID array, use the CreateSnapshots API operation, or log in to the Amazon EC2 console and choose Elastic Block Store, Snapshots, Create Snapshot.
Snapshots of instances that have multiple volumes attached in a RAID configuration are taken as a multi-volume snapshot, collectively. Multi-volume snapshots provide point-in-time, data-coordinated, and crash-consistent snapshots across multiple EBS volumes attached to an EC2 instance. You do not have to stop your instance to coordinate between volumes to achieve consistency because snapshots are automatically taken across multiple EBS volumes. After the snapshot for the volumes is initiated (usually a second or two), the file system can continue its operations.
After the snapshots are created, each snapshot is treated as an individual snapshot. You can perform all snapshot operations, such as restore, delete, and cross-Region and account copy, as you would with a single-volume snapshot. You can also tag your multi-volume snapshots as you would a single-volume snapshot. We recommend that you tag your multi-volume snapshots to manage them collectively during restore, copy, or retention. For more information, see the AWS documentation.
You can also perform these backups from a logical volume manager or a file
system–level backup. In these cases, using a traditional backup agent enables
the data to be backed up over the network. A number of agent-based backup solutions
are available on the internet and in the AWS Marketplace
An alternative approach is to create a replica of the primary system volumes that exist on a single large volume. This simplifies the backup process, because only one large volume must be backed up, and the backup does not take place on the primary system. However, first determine whether the single volume can perform sufficiently during the backup and whether the maximum volume size is appropriate for the application.
Protecting your Amazon EC2 backups
It is important to consider the security of your backups and to prevent accidental or malicious deletion of your backups. You can use a number of approaches collectively to accomplish this. To prevent the loss of your critical backups due to a security breach, we recommend that you copy your backups to another AWS account. If you have multiple AWS accounts, you can designate a separate account as your archive account to which all the other accounts can copy backups. For example, you can accomplish this with a cross-account backup in AWS Backup.
Your disaster recovery plan might also require you to be able to reproduce EC2 instances in another AWS Region in case of a regional failure. You can support this goal by copying your backups to another Region within the same account. This can provide an additional layer of accidental deletion protection as well as support disaster recovery (DR) objectives. AWS Backup provides support for cross-Region backups.
Consider blocking IAM permissions to the ec2:DeleteSnapshot and ec2:DeregisterImage actions. Instead, you can let your retention policies and methods manage the lifecycle of EBS snapshots and Amazon EC2 AMIs. Blocking delete actions is one way to implement a write-once, read-many (WORM) strategy for your EBS snapshots. You can also use AWS Backup Vault Lock, which provides support for EBS snapshots and other AWS resources.
Additionally, consider blocking the ability for users to share AMIs and EBS snapshots by blocking the ec2:ModifyImageAttribute and ec2:ModifySnapshotAttribute IAM actions. This will prevent your AMIs and snapshots from being shared with AWS accounts that are external to your organization. If you are using AWS Backup, limit users from performing similar operations on backup vaults. For more information, see the Backup and recovery using AWS Backup section of this guide.
Amazon EC2 includes a Recycle Bin feature that can help you restore accidentally deleted EBS snapshots. If you allow your users to delete snapshots, turn this feature on so that needed snapshots aren’t permanently deleted. Users should be particularly careful about deleting multiple snapshots, because the Amazon EC2 console allows you to select multiple snapshots and delete them in one operation. Additionally, be careful when you use cleanup scripts and automation so that you don’t unintentionally delete snapshots you need. The Recycle Bin feature helps provide protection from these types of situations.
Archiving EBS snapshots
Archiving your EBS snapshots can be a cost-effective method for keeping a copy of a volume for reference purposes that you don’t intend to restore for 90 or more days. This can be a good intermediate step before permanently deleting all related snapshots for an EBS volume. For example, you might consider archiving snapshots as an end-of-lifecycle step for EBS volumes that are no longer used. Archiving rather than deleting can also be a more cost effective method of deletion retention instead of using the Recycle Bin.
Automating snapshot and AMI creation with Systems Manager, the AWS CLI, and the AWS SDKs
Your backup approach might require operations before and after a snapshot or AMI is created. For example, you might need to stop and start services to quiesce the file system. Or you might need to stop and start your instance during AMI creation. You might also need to create backups of multiple components in your architecture collectively, each with its own pre-creation and post-creation steps.
You can reduce your maintenance window times for your backups by automating your process and verifying that your backup process is consistently applied. To automate your custom pre-creation and post-creation operations, script your backup process by using the AWS CLI and the SDK.
Your automation can be defined in a Systems Manager runbook that can be run on demand or during a Systems Manager maintenance window. You can grant your users access to run Systems Manager runbooks without the need to grant them permissions to Amazon EC2 disruptive commands. This can also help you verify that your backup process and tags are applied consistently by your users. You can use the AWS-CreateSnapshot and AWS-CreateImage runbooks for creating snapshots and AMIs, or you can grant other users permissions to use them. Systems Manager also includes the AWS-UpdateLinuxAmi and AWS-UpdateWindowsAmi runbooks to automate the AMI patching and AMI creation.
You can also use the AWS CLI and AWS Tools for Windows PowerShell
You can use the AWS CLI to create new AMIs. You can use the aws ec2 register-image command to create a new image for your EC2 instance. To automate the shutdown, image creation, and restart of your instances, combine this command with the aws ec2 stop-instances and aws ec2 start-instances commands.
