Troubleshooting - Innovation Sandbox on AWS

Troubleshooting

This section provides information about known issues and instructions to mitigate known errors. If these instructions do not resolve your issue, see the Contact AWS Support section to open an AWS Support case for this solution.

Application sign in error

If you receive an error titled "Application sign in error" when accessing the Innovation Sandbox on AWS Web UI, it is likely that the IAM Identity Center application has been misconfigured. Possible root causes include:

  • Incorrect Attribute mappings in the IAM Identity Center SAML 2.0 application

  • Incorrect Application ACS URL in the IAM Identity Center SAML 2.0 application

  • Incorrect webAppUrl in the AWS AppConfig global configuration

  • Incorrect idpSignInUrl in the AWS AppConfig global configuration

  • Missing primary email address in IAM Identity Center for the user attempting to login

Refer to the Post deployment configuration tasks section, and validate that the configurations in both IAM Identity Center and AWS AppConfig are correct.

Incorrect global configuration

If you receive an error with the message "Incorrect global configuration" when accessing the Innovation Sandbox on AWS Web UI, your global configuration in AWS AppConfig has not been updated properly and likely contains invalid fields or values. Review any validation errors included in the error message or logs in the Compute-ISBLogGroup log group and make the appropriate corrections to the global configuration. Refer to the Updating the global configuration section for information on how to update the global configuration.

Authentication failed, Invalid document signature

If you receive an error with the message "Invalid document signature" when accessing the Innovation Sandbox on AWS Web UI, the IDP Cert secret was either not updated or an incorrect value was provided. Refer back to the Update values in AWS Secrets Manager section to ensure that the value is correct.

Authentication failed, SAML assertion audience mismatch

If you receive an error with the message "SAML assertion audience mismatch" when accessing the Innovation Sandbox on AWS Web UI, the audience value provided in your IAM Identity Center SAML application configuration does not match the one configured in AWS AppConfig. These values must match for the solution authentication to work properly. Refer back to the Create a SAML 2.0 application and Updating the global config sections for information on how to update these values.

Investigating accounts in Quarantine state

Note

If the account clean-up mechanism fails to automatically delete resources at the end of an active lease, you might have accounts in a Quarantine state. We highly recommend investigating quarantined accounts as quickly as possible, as these accounts can incur costs for resources running inside these accounts.

When the Innovation Sandbox solution is unable to cleanup resources in a sandbox account, the account is moved to a Quarantine state and an email is sent to the solution administrators indicating that action should be taken to resolve the account’s quarantine status.

To resolve the quarantined status:

  1. Log in to the web UI as an Admin, and from the left, under Administration, choose Accounts.

  2. Verify the accounts in Quarantine status, and decide whether to clean up the account and return to the account pool, or to eject the account from the solution.

    • To clean up the account and return it to the account pool, choose the account, and under Actions, choose Retry cleanup.

    • To eject the account, choose the account, and under Actions, choose Eject account. This moves the account to the Exit OU, from which you can manually move it to your desired OU. For more information, refer to the Uninstall the solution section.

If the account is in quarantine because the retry cleanup failed, refer to the Resolving cleanup failures section.

Resolving cleanup failures

If the cleanup process fails to completely clean an account at the end of a lease, Innovation Sandbox will move the account into a Quarantine state, and email the Administrators notifying them of the issue.

To resolve an account that has failed cleanup:

  1. Log in to the web UI as an Admin, and from the left, under Administration, choose Accounts.

  2. Confirm the account that has failed the cleanup process. You will need this to view log information in the AWS Console.

  3. Log in to the AWS Console using the Hub account, and navigate to the CloudWatch > Logs Insights page.

  4. From the right pane, under Sample queries, choose the ISB group, and from the dropdown, choose the AccountCleanupLogs saved query, and choose Apply.

  5. In the query window, choose a time frame that includes when the account was last cleaned up (for example: last 3 days) and paste the 'Last Cleanup ReferenceID' into the indicated section.

  6. Choose Run query to see related events. The log information is displayed under the Logs tab.

  7. To manually handle any deletion failures in the affected account, navigate back to the Accounts page, and log in to the account using the Login to account option.

  8. After you have manually handled all errors, to restart the cleanup process in the web UI, choose the account and under Actions, choose Retry cleanup.

Viewing a specific Lease history

  1. Log in to the web UI as an Admin, and from the left, under Administration, choose Leases.

  2. Choose the lease name to view lease details.

  3. Copy the LeaseID from the Lease Summary page. You will need this to view lease history in the AWS Console.

  4. Log in to the AWS Console, and navigate to the CloudWatch > Logs Insights page.

  5. From the right pane, under Sample queries, choose the ISB group, and from the dropdown, choose LogQuery saved query and choose Apply.

  6. In the query window, choose the time frame to view logs for and paste the LeaseID into the indicated section.

  7. Choose Run query to view logs related to the LeaseID provided for the selected time frame. The log information is displayed under the Logs tab.

Viewing a specific User history

  1. Log in to the web UI as an Admin, and from the left, under Administration, choose Accounts.

  2. From the Accounts page, confirm the user email address you want to view history for. You will need this to view user/account history in the AWS Console.

  3. Log in to the AWS Console, and navigate to the CloudWatch > Logs Insights page.

  4. From the right pane, under Sample queries, choose the ISB group, and from the dropdown, choose LogQuery saved query and choose Apply.

  5. In the query window, choose the time frame to view logs for and paste the email address into the indicated section.

  6. Choose Run query to view logs related to the email address provided for the selected time frame. The log information is displayed under the Logs tab.

403 Permissions error

If you find an issue within the Identity Center:

  • Your session might have timed out. Refresh your browser to resolve this.

  • Maintenance mode is enabled and you are signed in using a Manager or User role. You will need to contact your Admin to disable maintenance mode in AWS AppConfig. For more information, refer to the Maintenance mode section.

Unexpected server errors

If you find unexpected server errors while using the web UI, you can trace the issue by using AWS X-Ray.

  1. Copy the X-Ray trace ID from the error:

    • When an unexpected error occurs in the web UI, a trace ID will be provided.

    • Or, for any error logs found in Amazon CloudWatch, expand the log to find the X-Ray trace ID for the operation.

  2. In the AWS Console, navigate to the AWS X-Ray page and paste the trace ID into the search box.

For more information, refer to the AWS X-Ray Traces, and AWS X-Ray Common Errors pages.