Putting the pieces together: EBS volume attachment

To create a better picture of how many of the pieces of the Nitro System operate together, let’s review what happens when a customer makes a public EC2 API call that changes the running state of their EC2 instance on a Nitro System. In particular, we’ll look at the case where a customer attaches an existing encrypted EBS volume to a running instance.

In the first step, the customer uses the AWS Command Line Interface (AWS CLI), AWS SDK, or AWS Management Console to invoke the AttachVolume command, targeting the instance. After validating that the customer’s IAM identity is authenticated and authorized to make the AttachVolume command, the API call is processed by a set of microservices inside the EC2 and EBS control planes. In the end, the control plane services call a defined set of encrypted, authenticated network APIs provided by the Nitro Controller with the information required to allocate the resources needed to attach the volume. Multiple services are involved in this operation, and each microservice has separated duties that limit the scope of access to the Nitro Controller APIs.

The EC2 control plane allocates the PCIe device resources of the Nitro Card for EBS that are required to service reads and writes to the logical EBS volume (either a NVMe virtual function for a virtualized instance or NVMe physical function for a bare metal instance). The EBS control plane provides the information needed to connect to the EBS servers that house the encrypted volume data over the network, and also an encrypted copy of the volume’s data key that is stored as volume metadata. The encrypted data key is protected by an AWS KMS key present only inside the AWS Key Management Service (AWS KMS); therefore, as part of the process for attaching the volume, the encrypted key must be sent to AWS KMS for decryption.

Assuming that the customer IAM identity that made the AttachVolume command is also authorized to make a Decrypt command in AWS KMS under the expected AWS KMS key, the encrypted volume’s data key will be decrypted. The Nitro System's access to this operation is protected by AWS KMS Grants and by IAM Forward Access Sessions. (Refer to this explanation of IAM Forward Access Sessions in the context of Elastic Load Balancing, AWS Certificate Manager and AWS Key Management Service in a presentation by Colm MacCárthaigh, VP and Distinguished Engineer at AWS.)

Together, these mechanisms cryptographically ensure that the Nitro System is granted access to use a customer's AWS KMS managed key only when the customer has recently authorized and authenticated this access. The Nitro System is not granted use of AWS KMS-managed keys on an ad-hoc basis or absent a recent customer authorization.

After being decrypted inside AWS KMS, and before being sent to the Nitro Controller using an encrypted Transport Layer Security (TLS) network connection, AWS KMS encrypts the data key again using a public key that serves as the cryptographic digital identity for the individual production Nitro host server. This public key was sent along with the data key for the encrypted volume by the EBS control plane to AWS KMS. Therefore, in addition to the entire message being encrypted on the wire by TLS, the data key is also asymmetrically encrypted within the message—double-encrypted on the wire. Only the Nitro Card of that specific production host with the specific customer’s compute environment has the private key necessary to decrypt the encrypted data key. Once locally decrypted, that plaintext data key is stored only in volatile memory on that single Nitro Card during the time the volume is attached and in use.

Now the Nitro EBS Card is ready to present the EBS volume to the EC2 instance through a PCIe attachment of an NVMe interface. When the host is configured to use the Nitro Hypervisor, the Nitro Controller sends a message over the PCIe interface to instruct the Nitro Hypervisor to assign the NVMe virtual function for that EBS volume to the appropriate EC2 instance. The hypervisor then sends a virtual hardware hot-plug event to the VM to alert the customer-provided system software that a new NVMe block device is available. In the case of a bare-metal instance, the Nitro Card for EBS signals a PCIe hot plug event directly to the server processor, and the customer-provided system software running on the processor handles the PCIe hot-plug event of the NVMe device as it would on any other server.

At this point, the customer instance operating system running either as a virtualized guest or a bare metal instance interacts with a NVMe device presented by the Nitro Card for EBS over the PCIe interface. This interaction occurs either as an SR-IOV function in the case of virtualized EC2 instances, or a PCIe physical function in the case of bare metal EC2 instances. The NVMe commands sent over the PCIe interface are handled by firmware running on the Nitro Card for EBS, which in turn interacts with the EBS service via the Nitro SoC’s integrated network interface. And, as previously noted, the Nitro EBS Card is also able to offload the cryptographic operations for AES-256 XTS-encrypted EBS volumes so that every single block of customer data is fully encrypted before leaving the Nitro Card with no performance impact. A customer may also choose to use an encrypting filesystem at the operating system level so that all customer data is fully encrypted before it is written or transmitted to the Nitro Card for EBS. This approach also establishes an additional layer of encryption for EBS data both on the wire and in the EBS storage system.