Testing using workflows in CodeCatalyst - Amazon CodeCatalyst

Amazon CodeCatalyst is in preview release and is subject to change.

Testing using workflows in CodeCatalyst

In CodeCatalyst, you can run tests as part of different workflow actions, such as build and test. These workflow actions can all generate test reports. A test action is a workflow action that produces test and code coverage reports, as well as software composition analysis (SCA) reports. These reports are displayed in the CodeCatalyst console.

Test report types

The CodeCatalyst test action supports the following types of reports.

Test reports

In CodeCatalyst, you can configure unit tests, integration tests, and system tests that run during builds. Then CodeCatalyst can create reports that contain the results of your tests.

You can use a test report to help troubleshoot problems with your tests. If you have many test reports from multiple builds, you can use your test reports to view failure rates to help you optimize your builds.

You can use the following test report file formats:

  • Cucumber JSON (.json)

  • JUnit XML (.xml)

  • NUnit XML (.xml)

  • NUnit3 XML (.xml)

  • TestNG XML (.xml)

  • Visual Studio TRX (.trx | .xml)

Code coverage reports

In CodeCatalyst, you can generate code coverage reports for your tests. CodeCatalyst provides the following code coverage metrics:

Line coverage

Measures how many statements your tests cover. A statement is a single instruction, not including comments.

line coverage = (total lines covered)/(total number of lines)

Branch coverage

Measures how many branches your tests cover out of every possible branch of a control structure such as an if or case statement.

branch coverage = (total branches covered)/(total number of branches)

The following code coverage report file formats are supported:

  • JaCoCo XML (.xml)

  • SimpleCov JSON (generated by simplecov, not simplecov-json, .json)

  • Clover XML (version 3, .xml)

  • Cobertura XML (.xml)

  • LCOV (.info)

Software Composition Analysis (SCA) reports

In CodeCatalyst, you can use software composition analysis (SCA) tools to analyze components of your application and check for known security vulnerabilities. You can discover and parse SARIF reports that detail vulnerabilities with varying severities and ways to fix them. Valid severity values, from most to least severe, are: CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL.

The following SCA report file formats are supported:

  • SARIF (.sarif | .json)