Amazon Comprehend
Developer Guide

Confusion Matrix

When a custom classifier model is trained, Amazon Comprehend creates a confusion matrix that provides metrics on how well the model performed in training. This enables you to assess how well the classifier will perform when run. This matrix shows a matrix of labels as predicted by the model compared to actual labels and is created using 10 to 20 percent of the documents submitted to test the trained model.

Consider the following example:

A B X Y <-(predicted label) A 1 2 0 4 B 0 3 0 1 X 0 0 1 0 Y 1 1 1 1 ^ | (actual label)

In this case, the model predicted the following:

  • One "A" label was correctly identified, two "A" labels were incorrectly identified as actually "B" labels, and four "A" labels were incorrectly identified as "Y" labels.

  • Three "B" labels were correctly identified, and one "B" label was incorrectly identified as a "Y" label.

  • One "X" was correctly identified.

  • One "Y" label was incorrectly identified as an "A" label, one was incorrectly identified as a "B" label, one was incorrectly identified as an "X" label, and one was correctly identified as a "Y" label.

In this matrix, the correctly identified labels are shown on the diagonal line (A:A, B:B, X:X, and Y:Y) so you can easily check the table for prediction errors because they will be represented as values outside this diagonal. In this case, you can see that the model correctly identifies "X" labels (although the sample is very small) and can correctly identify "B" labels 75% of the time. However, it incorrectly identifies "A" labels 86% of the time and correctly identifies "Y" labels at a rate no better than random chance.

The confusion matrix is presented in JSON format and for the above example is shown as the following:

{ "type": "multi_class", "confusion_matrix": [ [1, 2, 0,4], [0, 3, 0, 1], [0, 0, 1, 0], [1, 1, 1, 1]], "labels": ["A", "B", "X", "Y"], "all_labels": ["A", "B", "X", "Y"] }

Confusion Matrix Type

It is important to note that the type parameter in this example is multi_class. This type directly affects the format of the confusion matrix. For instance, when "type": "multi_class", as in the examples above, then the confusion matrix will show the numbers of predicted labels compared to actual labels. If other types are used, the confusion matrix will be presented in a different format.

Important

Currently, Amazon Comprehend only supports the multi_class type.

The CreateDocumentClassifier API

The confusion matrix is available when running the CreateDocumentClassifier API. When the operation is run, the confusion matrix is shown in the confusion_matrix.json file, located at s3://user-defined-path/unique-value/output/output.tar.gz where the user-defined-path is the S3Uri value of the OutputDataConfig parameter in the CreateDocumentClassifier operation.