Walkthrough: Update application dependencies, patch an instance, and perform an application-specific health check - AWS Systems Manager

Walkthrough: Update application dependencies, patch an instance, and perform an application-specific health check

In many cases, a managed instance must be rebooted after it has been patched with the latest software update. However, rebooting an instance in production without safeguards in place can cause several problems, such as invoking alarms, recording incorrect metric data, and interrupting data synchronizations.

This walkthrough demonstrates how to avoid problems like these by using the AWS Systems Manager document (SSM document) AWS-RunPatchBaselineWithHooks to achieve a complex, multi-step patching operation that accomplishes the following:

  1. Prevent new connections to the application

  2. Install operating system updates

  3. Update the package dependencies of the application

  4. Restart the system

  5. Perform an application-specific health check

For this example, we have set up our infrastructure this way:

  • The virtual machines targeted are registered as managed instances with Systems Manager.

  • Iptables is used as a local firewall.

  • The application hosted on the instances is running on port 443.

  • The application hosted on the instances is a nodeJS application.

  • The application hosted on the instances is managed by the pm2 process manager.

  • The application already has a specified health check endpoint.

  • The application’s health check endpoint requires no end user authentication. The endpoint allows for a health check that meets the organization’s requirements in establishing availability. (In your environments, it may be enough to simply ascertain that the nodeJS application is running and able to listen for requests. In other cases, you might want to also verify that a connection to the caching layer or database layer has already been established.)

The examples in this walkthrough are for demonstration purposes only and not meant to be implemented as-is into production environments. Also, keep in mind that the lifecycle hooks feature of Patch Manager, a capability of Systems Manager, with the AWS-RunPatchBaselineWithHooks document can support numerous other scenarios. Here are several examples.

  • Stop a metrics reporting agent before patching and restarting it after the instance reboots.

  • Detach the instance from a CRM or PCS cluster before patching and reattach after the instance reboots.

  • Update third-party software (for example, Java, Tomcat, Adobe applications, and so on) on Windows Server machines after operating system (OS) updates are applied, but before the instance reboots.

To update application dependencies, patch an instance, and perform an application-specific health check

  1. Create an SSM document for your preinstallation script with the following contents and name it NodeJSAppPrePatch. Replace your_application with the name of your application.

    This script immediately blocks new incoming requests and provides five seconds for already active ones to complete before beginning the patching operation. For the sleep option, specify a number of seconds greater than it usually takes for incoming requests to complete.

    # exit on error set -e # set up rule to block incoming traffic iptables -I INPUT -j DROP -p tcp --syn --destination-port 443 || exit 1 # wait for current connections to end. Set timeout appropriate to your application's latency sleep 5 # Stop your application pm2 stop your_application

    For information about creating SSM documents, see Creating SSM documents.

  2. Create another SSM document with the following content for your postinstall script to update your application dependencies and name it NodeJSAppPostPatch. Replace /your/application/path with the path to your application.

    cd /your/application/path npm update # you can use npm-check-updates if you want to upgrade major versions
  3. Create another SSM document with the following content for your onExit script to bring your application back up and perform a health check. Name this SSM document NodeJSAppOnExitPatch. Replace your_application with the name of your application.

    # exit on error set -e # restart nodeJs application pm2 start your_application # sleep while your application starts and to allow for a crash sleep 10 # check with pm2 to see if your application is running pm2 pid your_application # re-enable incoming connections iptables -D INPUT -j DROP -p tcp --syn --destination-port # perform health check /usr/bin/curl -m 10 -vk -A "" http://localhost:443/health-check || exit 1
  4. Create an association in State Manager, a capability of AWS Systems Manager, to issue the operation by performing the following steps:

    1. Open the AWS Systems Manager console at https://console.aws.amazon.com/systems-manager/.

    2. In the navigation pane, choose State Manager, and then choose Create association.

    3. For Name, provide a name to help identify the purpose of the association.

    4. In the Document list, choose AWS-RunPatchBaselineWithHooks.

    5. For Operation, choose Install.

    6. (Optional) For Snapshot Id, provide a GUID that you generate to help speed up the operation and ensure consistency. The GUID value can be as simple as 00000000-0000-0000-0000-111122223333.

    7. For Pre Install Hook Doc Name, enter NodeJSAppPrePatch.

    8. For Post Install Hook Doc Name, enter NodeJSAppPostPatch.

    9. For On ExitHook Doc Name,enter NodeJSAppOnExitPatch.

  5. For Targets, identify your instances by specifying instance tags, choosing instances manually, choosing a resource group, or choosing all instances.

  6. For Specify schedule, specify how often to run the association. For instance patching, once per week is a common cadence.

  7. In the Rate control section, choose options to control how the association runs on multiple instances. Ensure that only a portion of instances are updated at a time. Otherwise, all or most of your fleet could be taken offline at once. For more information about using rate controls, see About targets and rate controls in State Manager associations.

  8. (Optional) For Output options, to save the command output to a file, select the Enable writing output to S3 box. Enter the bucket and prefix (folder) names in the boxes.

    Note

    The S3 permissions that grant the ability to write the data to an S3 bucket are those of the instance profile assigned to the instance, not those of the IAM user performing this task. For more information, see Create an IAM instance profile for Systems Manager. In addition, if the specified S3 bucket is in a different AWS account, verify that the instance profile associated with the instance has the necessary permissions to write to that bucket.

  9. Choose Create Association.