Menu
AWS CodeDeploy
User Guide (API Version 2014-10-06)

Troubleshoot Deployment Issues

Note

The causes of many deployment failures can be identified by reviewing the log files created during the deployment process. For simplicity, we recommend using Amazon CloudWatch Logs to centrally monitor log files instead of viewing them instance by instance. For information, see View AWS CodeDeploy Logs in CloudWatch Logs Console.

Deployment fails with the message “Validation of PKCS7 signed message failed”

This error message indicates the instance is running a version of the AWS CodeDeploy agent that supports only the SHA-1 hash algorithm. Support for the SHA-2 hash algorithm was introduced in version 1.0.1.854 of the AWS CodeDeploy agent, released in November 2015. Effective October 17, 2016, deployments will fail if a version of the AWS CodeDeploy agent earlier than 1.0.1.854 is installed. For more information, see AWS to Switch to SHA256 Hash Algorithm for SSL Certificates, NOTICE: Retiring AWS CodeDeploy host agents older than version 1.0.1.85, and Update the AWS CodeDeploy Agent.

Troubleshooting a failed AllowTraffic lifecycle event with no error reported in the deployment logs

In some cases, a blue/green deployment fails during the AllowTraffic lifecycle event, but no cause for the failure is indicated in the deployment logs.

This failure is typically due to the health checks being configured incorrectly in Elastic Load Balancing for the Classic Load Balancer used to manage traffic for the deployment group.

To resolve the issue, review and correct any errors in the the health check configuration for the load balancer.

For information, see Configure Health Checks in the Classic Load Balancer Guide and ConfigureHealthCheck in the Elastic Load Balancing API Reference version 2012-06-01.

Troubleshooting a failed ApplicationStop deployment lifecycle event

A deployment can fail during the ApplicationStop deployment lifecycle event for one of the following reasons:

  • The AWS CodeDeploy agent finds the deployment-group-id_last_successful_install file in the correct location, but the location listed in the deployment-group-id_last_successful_install file does not exist.

    On Amazon Linux, Ubuntu Server, and RHEL instances, this file must exist in /opt/codedeploy-agent/deployment-root/deployment-instructions.

    On Windows Server instances, the file must be stored in the C:\ProgramData\Amazon\CodeDeploy\deployment-instructions folder.

  • In the location listed in the deployment-group-id_last_successful_install file, either the AppSpec file is invalid or the scripts do not run successfully.

  • The script contains an error that cannot be corrected, so it will never run successfully.

Use the AWS CodeDeploy console to investigate why a deployment might have failed during this event. On the event details page for the deployment, in the ApplicationStop row, choose View logs. Alternatively, use the AWS CLI to call the get-deployment-instance command.

Note

The causes of many deployment failures can be identified by reviewing the log files created during the deployment process. For simplicity, we recommend using Amazon CloudWatch Logs to centrally monitor log files instead of viewing them instance by instance. For information, see View AWS CodeDeploy Logs in CloudWatch Logs Console.

You must use the AWS CLI, not the AWS CodeDeploy console, to recover from a deployment that failed during the ApplicationStop deployment lifecycle event. Call the create-deployment command, set the --ignore-application-stop-failures option, and deploy the application revision again. The deployment will continue even if the ApplicationStop deployment lifecycle event fails again.

Troubleshooting a failed DownloadBundle deployment lifecycle event with "UnknownError: not opened for reading"

If you are trying to deploy an application revision from Amazon S3, and the deployment fails during the DownloadBundle deployment lifecycle event with the "UnknownError: not opened for reading" error:

  • There was internal Amazon S3 service error. Deploy the application revision again.

  • The IAM instance profile on your Amazon EC2 instance does not have permissions to access the application revision in Amazon S3. For information about Amazon S3 bucket policies, see Push a Revision and Deployment Prerequisites.

  • The instances to which you will deploy are associated with one region (for example, US West (Oregon)), but the Amazon S3 bucket that contains the application revision is associated with another region (for example, US East (N. Virginia)). Make sure the application revision is in an Amazon S3 bucket associated with the same region as the instances.

On the event details page for the deployment, in the Download bundle row, choose View logs. Alternatively, use the AWS CLI to call the get-deployment-instance command. If this error occurred, there should be an error in the output with the error code "UnknownError" and the error message "not opened for reading."

To determine the reason for this error:

  1. Enable wire logging on at least one of the instances, and then deploy the application revision again.

  2. Examine the wire logging file to find the error. Common error messages for this issue include the phrase "access denied."

  3. After you have examined the log files, we recommend that you disable wire logging to reduce log file size and the amount of sensitive information that may appear in the output in plain text on the instance in the future.

To learn how to find the wire logging file and enable and disable wire logging, see :log_aws_wire: in Working with the AWS CodeDeploy Agent.

Windows PowerShell scripts fail to use the 64-bit version of Windows PowerShell by default

If a Windows PowerShell script running as part of a deployment relies on 64-bit functionality (for example, because it consumes more memory than a 32-bit application will allow or calls libraries that are offered only in a 64-bit version), the script may crash or otherwise not run as expected. This is because, by default, AWS CodeDeploy uses the 32-bit version of Windows PowerShell to run Windows PowerShell scripts that are part of an application revision.

Add code like the following to the beginning of any script that must run with the 64-bit version of Windows PowerShell:

Copy
# Are you running in 32-bit mode? # (\SysWOW64\ = 32-bit mode) if ($PSHOME -like "*SysWOW64*") { Write-Warning "Restarting this script under 64-bit Windows PowerShell." # Restart this script under 64-bit Windows PowerShell. # (\SysNative\ redirects to \System32\ for 64-bit mode) & (Join-Path ($PSHOME -replace "SysWOW64", "SysNative") powershell.exe) -File ` (Join-Path $PSScriptRoot $MyInvocation.MyCommand) @args # Exit 32-bit script. Exit $LastExitCode } # Was restart successful? Write-Warning "Hello from $PSHOME" Write-Warning " (\SysWOW64\ = 32-bit mode, \System32\ = 64-bit mode)" Write-Warning "Original arguments (if any): $args" # Your 64-bit script code follows here... # ...

Although the file path information in this code may seem counterintuitive, 32-bit Windows PowerShell uses a path like:

c:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe

64-bit Windows PowerShell uses a path like:

c:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

Long-running processes can cause deployments to fail

For deployments to Amazon Linux, Ubuntu Server, and RHEL instances, if you have a deployment script that starts a long-running process, AWS CodeDeploy may spend a long time waiting in the deployment lifecycle event and then fail the deployment. This is because if the process runs longer than the foreground and background processes in that event are expected to take, AWS CodeDeploy stops and fails the deployment, even if the process is still running as expected.

For example, an application revision contains two files in its root, after-install.sh and sleep.sh. Its AppSpec file contains the following instructions:

Copy
version: 0.0 os: linux files: - source: ./sleep.sh destination: /tmp hooks: AfterInstall: - location: after-install.sh timeout: 60

The after-install.sh file runs during the AfterInstall application lifecycle event. Here are its contents:

Copy
#!/bin/bash /tmp/sleep.sh

The sleep.sh file contains the following, which suspends program execution for three minutes (180 seconds), simulating some long-running process:

Copy
#!/bin/bash sleep 180

When after-install.sh calls sleep.sh, sleep.sh will start and keep running for three minutes (180 seconds), which is two minutes (120 seconds) past the time AWS CodeDeploy expects sleep.sh (and, by relation, after-install.sh) to stop running. After the timeout of one minute (60 seconds), AWS CodeDeploy stops and fails the deployment at the AfterInstall application lifecycle event, even though sleep.sh continues to run as expected. The following error is displayed:

Script at specified location: after-install.sh failed to complete in 60 seconds.

You cannot simply add an ampersand (&) in after-install.sh to run sleep.sh in the background.

Copy
#!/bin/bash # Do not do this. /tmp/sleep.sh &

Doing so can leave the deployment in a pending state for up to the default one-hour deployment lifecycle event timeout period, after which AWS CodeDeploy stops and fails the deployment at the AfterInstall application lifecycle event as before.

In after-install.sh, call sleep.sh as follows, which enables AWS CodeDeploy to continue after the process starts running:

Copy
#!/bin/bash /tmp/sleep.sh > /dev/null 2> /dev/null < /dev/null &

In the preceding call, sleep.sh is the name of the process you want to start running in the background, redirecting stdout, stderr, and stdin to /dev/null.