Testing with workflows - Amazon CodeCatalyst
Quality report types

Testing with workflows

In CodeCatalyst, you can run tests as part of different workflow actions, such as build and test. These workflow actions can all generate quality reports. A test action is a workflow action that produces test, code coverage, software composition analysis, and static analysis reports. These reports are displayed in the CodeCatalyst console.

Topics

Quality report types

The Amazon CodeCatalyst test action supports the following types of quality reports. For an example on how to format these reports in your YAML, see Quality reports YAML example.

Test reports

In CodeCatalyst, you can configure unit tests, integration tests, and system tests that run during builds. Then CodeCatalyst can create reports that contain the results of your tests.

You can use a test report to help troubleshoot problems with your tests. If you have many test reports from multiple builds, you can use your test reports to view failure rates to help you optimize your builds.

You can use the following test report file formats:

  • Cucumber JSON (.json)

  • JUnit XML (.xml)

  • NUnit XML (.xml)

  • NUnit3 XML (.xml)

  • TestNG XML (.xml)

  • Visual Studio TRX (.trx, .xml)

Code coverage reports

In CodeCatalyst, you can generate code coverage reports for your tests. CodeCatalyst provides the following code coverage metrics:

Line coverage

Measures how many statements your tests cover. A statement is a single instruction, not including comments.

line coverage = (total lines covered)/(total number of lines)

Branch coverage

Measures how many branches your tests cover out of every possible branch of a control structure such as an if or case statement.

branch coverage = (total branches covered)/(total number of branches)

The following code coverage report file formats are supported:

  • JaCoCo XML (.xml)

  • SimpleCov JSON (generated by simplecov, not simplecov-json, .json)

  • Clover XML (version 3, .xml)

  • Cobertura XML (.xml)

  • LCOV (.info)

Software composition analysis reports

In CodeCatalyst, you can use software composition analysis (SCA) tools to analyze components of your application and check for known security vulnerabilities. You can discover and parse SARIF reports that detail vulnerabilities with varying severities and ways to fix them. Valid severity values, from most to least severe, are: CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL.

The following SCA report file formats are supported:

  • SARIF (.sarif, .json)

Static analysis reports

You can use static analysis (SA) reports to identify source-level code defects. In CodeCatalyst, you can generate SA reports to help resolve issues in your code before you deploy it. These issues include bugs, security vulnerabilities, quality issues, and other vulnerabilities. Valid severity values, from most to least severe, are: CRITICAL, HIGH, MEDIUM, LOW, and INFORMATIONAL.

CodeCatalyst provides the following SA metrics:

Bugs

Identifies a number of possible bugs found in your source code. These bugs can include issues regarding memory safety. The following is an example of a bug.

// The while loop will inadvertently index into array x out-of-bounds
int x[64];
while (int n = 0; n <= 64; n++) {
  x[n] = 0;
}
Security vulnerabilities

Identifies a number of possible security vulnerabilities found in your source code. These security vulnerabilities can include issues such as storing your secret tokens in plaintext.

Quality issues

Identifies a number of possible quality issues found in your source code. These quality issues can include issues regarding style conventions. The following is an example of a quality issue.

// The function name doesn't adhere to the style convention of camelCase
int SUBTRACT(int x, int y) {
  return x-y
}
Other vulnerabilities

Identifies a number of possible other vulnerabilities found in your source code.

CodeCatalyst supports the following SA report file formats:

  • PyLint (.py)

  • ESLint (.js, .jsx, .ts, .tsx)

  • SARIF (.sarif, .json)