Menu
Amazon Elastic Compute Cloud
User Guide for Windows Instances

Troubleshooting PV Drivers

This topic describes solutions to common issues that you might encounter with Amazon EC2 PV drivers.

Windows Server 2012 R2 loses network and storage connectivity after an instance reboot

Windows Server 2012 R2 Amazon Machine Images (AMIs) made available before September 10, 2014 can lose network and storage connectivity after an instance reboot. The error in the AWS Management Console system log states: “Difficulty detecting PV driver details for Console Output.” The connectivity loss is caused by the Windows Server 2012 R2 Plug and Play Cleanup feature. This features scans for and disables inactive system devices every 30 days. The feature incorrectly identifies the EC2 network device as inactive and removes it from the system. When this happens, the instance loses network connectivity after a reboot.

For systems that you suspect could be affected by this issue, you can download and run an in-place driver upgrade. If you are unable to perform the in-place driver upgrade, you can run a helper script. The script determines if your instance is affected. If it is affected, and the Amazon EC2 network device has not been removed, the script disables the Plug and Play Cleanup scan. If the Amazon EC2 network device has been removed, the script repairs the device, disables the Plug and Play Cleanup scan, and allows your instance to reboot with network connectivity enabled.

In this section

Choose How You Want to Fix This Problem

There are two methods for restoring network and storage connectivity to an instance affected by this issue. Choose one of the following methods:

Method Prerequisites Procedure Overview
Method 1 - Enhanced networking Enhanced networking is only available in a virtual private cloud (VPC) which requires a C3 instance type. If the server does not currently use the C3 instance type, then you must temporarily change it. Enhanced networking is not available for ec2-classic. You change the server instance type to a C3 instance. Enhanced networking then enables you to connect to the affected instance and fix the problem. After you fix the problem, you change the instance back to the original instance type. This method is typically faster than Method 2 and less likely to result in user error. You will incur additional charges as long as the C3 instance is running.
Method 2 - Registry configuration Ability to create or access a second server. Ability to change Registry settings. You detach the root volume from the affected instance, attach it to a different instance, connect, and make changes in the Registry. You will incur additional charges as long as the additional server is running. This method is slower than Method 1, but this method has worked in situations where Method 1 failed to resolve the problem.

Method 1 - Enhanced Networking

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Instances.

  3. Locate the affected instance. Open the context (right-click) menu for the instance, choose Instance State, and then choose Stop.

    Warning

    When you stop an instance, the data on any instance store volumes is erased. Therefore, if you have any data on instance store volumes that you want to keep, be sure to back it up to persistent storage.

  4. After the instance is stopped create a backup. Open the context (right-click) menu for the instance, choose Image, and then choose Create Image.

  5. Change the instance type to any C3 instance type.

  6. Start the instance.

  7. Connect to the instance using Remote Desktop and then download the AWS PV Drivers Upgrade package to the instance.

  8. Extract the contents of the folder and run AWSPVDriverSetup.msi.

    After running the MSI, the instance automatically reboots and then upgrades the drivers. The instance will not be available for up to 15 minutes.

  9. After the upgrade is complete and the instance passes both health checks in the Amazon EC2 console, connect to the instance using Remote Desktop and verify that the new drivers were installed. In Device Manager, under Storage Controllers, locate AWS PV Storage Host Adapter. Verify that the driver version is the same as the latest version listed in the Driver Version History table. For more information, see AWS PV Driver Version History.

  10. Stop the instance and change the instance back to its original instance type.

  11. Start the instance and resume normal use.

Method 2 - Registry configuration

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Instances.

  3. Locate the affected instance. Open the context (right-click) menu for the instance, choose Instance State, and then choose Stop.

    Warning

    When you stop an instance, the data on any instance store volumes is erased. Therefore, if you have any data on instance store volumes that you want to keep, be sure to back it up to persistent storage.

  4. Choose Launch Instance and create a temporary Windows Server 2008 or Windows Server 2012 instance in the same Availability Zone as the affected instance. Do not create a Windows Server 2012 R2 instance.

    Important

    If you do not create the instance in the same Availability Zone as the affected instance you will not be able to attach the root volume of the affected instance to the new instance.

  5. In the navigation pane, choose Volumes.

  6. Locate the root volume of the affected instance. Detach the volume and attach it to the temporary instance you created earlier. Attach it with the default device name (xvdf).

  7. Use Remote Desktop to connect to the temporary instance, and then use the Disk Management utility to make the volume available for use.

  8. On the temporary instance, open the Run dialog box, type regedit, and press Enter.

  9. In the Registry Editor navigation pane, choose HKEY_Local_Machine, and then from the File menu choose Load Hive.

  10. In the Load Hive dialog box, navigate to Affected Volume\Windows\System32\config\System and type a temporary name in the Key Name dialog box. For example, enter OldSys.

  11. In the navigation pane of the Registry Editor, locate the following keys:

    HKEY_LOCAL_MACHINE\your_temporary_key_name\ControlSet001\Control\Class\4d36e97d-e325-11ce-bfc1-08002be10318

    HKEY_LOCAL_MACHINE\your_temporary_key_name\ControlSet001\Control\Class\4d36e96a-e325-11ce-bfc1-08002be10318

  12. For each key, double-click UpperFilters, enter a value of XENFILT, and then click OK.

    
                                Registry key for affected volume.
  13. Locate the following key:

    HKEY_LOCAL_MACHINE\your_temporary_key_name\ControlSet001\Services\XENBUS\Parameters

  14. Create a new string (REG_SZ) with the name ActiveDevice and the following value:

    PCI\VEN_5853&DEV_0001&SUBSYS_00015853&REV_01

  15. Locate the following key:

    HKEY_LOCAL_MACHINE\your_temporary_key_name\ControlSet001\Services\XENBUS

  16. Change the Count from 0 to 1.

  17. Locate and delete the following keys:

    HKEY_LOCAL_MACHINE\your_temporary_key_name\ControlSet001\Services\xenvbd\StartOverride

    HKEY_LOCAL_MACHINE \your_temporary_key_name\ControlSet001\Services\xenfilt\StartOverride

  18. In the Registry Editor navigation pane, choose the temporary key that you created when you first opened the Registry Editor.

  19. From the File menu, choose Unload Hive.

  20. In the Disk Management Utility, choose the drive you attached earlier, open the context (right-click) menu, and choose Offline.

  21. In the Amazon EC2 console, detach the affected volume from the temporary instance and reattach it to your Windows Server 2012 R2 instance with the device name /dev/sda1. You must specify this device name to designate the volume as a root volume.

  22. Start the instance.

  23. Connect to the instance using Remote Desktop and then download the AWS PV Drivers Upgrade package to the instance.

  24. Extract the contents of the folder and run AWSPVDriverSetup.msi.

    After running the MSI, the instance automatically reboots and then upgrades the drivers. The instance will not be available for up to 15 minutes.

  25. After the upgrade is complete and the instance passes both health checks in the Amazon EC2 console, connect to the instance using Remote Desktop and verify that the new drivers were installed. In Device Manager, under Storage Controllers, locate AWS PV Storage Host Adapter. Verify that the driver version is the same as the latest version listed in the Driver Version History table. For more information, see AWS PV Driver Version History.

  26. Delete or stop the temporary instance you created in this procedure.

Run the Remediation Script

If you are unable to perform an in-place driver upgrade or migrate to a newer instance you can run the remediation script to fix the problems caused by the Plug and Play Cleanup task.

To run the remediation script

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Instances.

  3. Choose the instance for which you want to run the remediation script. Open the context (right-click) menu for the instance, choose Instance State, and then choose Stop.

    Warning

    When you stop an instance, the data on any instance store volumes is erased. Therefore, if you have any data on instance store volumes that you want to keep, be sure to back it up to persistent storage.

  4. After the instance is stopped create a backup. Open the context (right-click) menu for the instance, choose Image, and then choose Create Image.

  5. Open the context (right-click) menu for the instance, choose Instance State, and then choose Start.

  6. Connect to the instance by using Remote Desktop and then download the RemediateDriverIssue.zip folder to the instance.

  7. Extract the contents of the folder.

  8. Run the remediation script according to the instructions in the Readme.txt file. The file is located in the folder where you extracted RemediateDriverIssue.zip.

TCP Offloading

By default, TCP offloading is enabled for the Citrix PV drivers in Windows AMIs. If you encounter transport-level errors or packet transmission errors (as visible on the Windows Performance Monitor)—for example, when you're running certain SQL workloads—you may need to disable this feature.

Important

Disabling TCP offloading may reduce the network performance of your instance.

You do not need to perform this procedure on instances running AWS PV or Intel network drivers.

To disable TCP offloading for Windows Server 2012 and 2008

  1. Connect to your instance and log in as the local administrator.

  2. If you're using Windows Server 2012, press Ctrl+Esc to access the Start screen, and then click Control Panel. If you're using Windows Server 2008, click Start and select Control Panel.

  3. Click Network and Internet, then Network and Sharing Center.

  4. Click Change adapter settings.

  5. Right-click Citrix PV Ethernet Adapter #0 and select Properties.

    
                            Local area connection properties
  6. In the Local Area Connection Properties dialog box, click Configure to open the Citrix PV Ethernet Adapter #0 Properties dialog box.

  7. On the Advanced tab, disable each of the following properties by selecting them in the Property list, and selecting Disabled from the Value list:

    • IPv4 Checksum Offload

    • Large Receive Offload (IPv4)

    • Large Send Offload Version 2 (IPv4)

    • TCP Checksum Offload (IPv4)

    • UDP Checksum Offload (IPv4)

    
                            Citrix PV Ethernet adapter properties
  8. Click OK.

  9. Run the following commands from a Command Prompt window.

    Copy
    netsh int ip set global taskoffload=disabled netsh int tcp set global chimney=disabled netsh int tcp set global rss=disabled netsh int tcp set global netdma=disabled
  10. Reboot the instance.

Time Synchronization

Prior to the release of the 2013.02.13 Windows AMI, the Citrix Xen guest agent could set the system time incorrectly. This can cause your DHCP lease to expire. If you have issues connecting to your instance, you might need to update the agent.

To determine whether you have the updated Citrix Xen guest agent, check whether the C:\Program Files\Citrix\XenGuestAgent.exe file is from March 2013. If the date on this file is earlier than that, update the Citrix Xen guest agent service. For more information, see Upgrade Your Citrix Xen Guest Agent Service.