Verifying your model with a trial detection task
If you want to verify or improve the quality of your model, you can run a trial detection task. A trial detection task detects anomalies in new images that you supply.
You can verify the detection results and add the verified images to your dataset. If you have separate training and test datasets, the verified images are added to the training dataset.
You can verify images from your local computer or images located in an Amazon S3 bucket. If you want to add verified images to the dataset, images located in an S3 bucket must be in the same S3 bucket as the images in your dataset.
Note
To run a trial detection task, ensure that your S3 bucket has versioning enabled. For more information, see Using versioning. The console bucket is created with versioning enabled.
By default your images are encrypted with a key that AWS owns and manages. You can also choose to use your own AWS Key Management Service (KMS) key. For more information, see AWS Key Management Service concepts.
Topics
Running a trial detection task
Perform the following steps to run a trial detection task.
To run a trial detection (console)
Open the Amazon Lookout for Vision console at https://console.aws.amazon.com/lookoutvision/
. Choose Get started.
In the left navigation pane, choose Projects.
In the projects view, choose the project that contains the version of the model that you want to view.
In the left navigation pane, under the project name, choose Trial detections.
In the trial detections view, choose the Run trial detection.
On the Run trial detection page, enter a name for your trial detection task in Task name.
In Choose model, choose the version of that model that you want to use.
Import the images according to the source of the images as follows:
If you are importing your source images from an Amazon S3 bucket, enter the S3 URI.
Tip
If you're using the Getting Started example images, use the extra_images folder. The Amazon S3 URI is
s3://.your bucket/circuitboard/extra_imagesIf you are uploading images from your computer, add the images after you choose Detect anomalies.
(Optional) If you want to use your own AWS KMS encryption key, do the following:
For Image data encryption, choose Customize encryption settings (advanced).
In encryption.aws_kms_key, enter the Amazon Resource Name (ARN) of your key, or choose an existing AWS KMS key. To create a new key, choose Create an AWS IMS key.
Choose Detect anomalies and then choose Run trial detection to start the trial detection task.
Check the current status in the trial detections view. The trial detection might take a while to complete.
Verifying trial detection results
Verifying the results of a trial detection can help you improve your model.
If the performance metrics are poor, improve your model by running a trial detection and then add verified images to the dataset (training dataset, if you have a separate datasets).
If the model's performance metrics are good, but the results of a trial detection are poor, you can improve your model by adding verified images to the dataset (training dataset). If you have a separate test dataset, consider adding more images to the test dataset.
After you add verified images to your dataset, retrain and re-evaluate your model. For more information, see Training your model.
To verify the results of a trial detection
Open the Amazon Lookout for Vision console at https://console.aws.amazon.com/lookoutvision/
. In the left navigation pane, choose Projects.
In the Projects page, choose the project that you want to use. The dashboard for your project is displayed.
In the left navigation pane, choose Trial detections.
Choose the trial detection that you want to verify.
On the trial detection page, choose Verify machine predictions.
Choose Select all images on this page.
If the predictions are correct, choose Verify as correct. Otherwise, choose Verify as incorrect. The prediction and prediction confidence score is shown under each image.
If you need to change the label for an image, do the following:
Choose Correct or Incorrect under the image.
If you can't determine the correct label for an image, magnify the image by choosing the image in the gallery.
Note
You can filter image labels by choosing the desired label, or label state, in the Filters section. You can sort by confidence score in the Sorting options section.
-
If your model is a segmentation model and the mask or anomaly label for an image is wrong, choose Anomalous area under the image and open the annotation tool. Update the segmentation information by doing Correcting segmentation labels with the annotation tool.
Repeat steps 7-10 on each page as necessary until all the images have been verified.
Choose Add verified images to dataset. If you have separate datasets, the images are added to the training dataset.
Retrain your model. For more information, see Training your model.
Correcting segmentation labels with the annotation tool
You use the annotation tool to segment an image by marking anomalous areas with a mask.
To correct the segmentation labels for an image with the annotation tool
Open the annotation tool by selecting anomalous area under an image in the dataset gallery.
If the anomaly label for a mask isn't correct, choose the mask and then choose the correct anomaly label under Anomaly labels. If necessary, choose Add anomaly label to add a new anomaly label.
If the mask isn't correct, choose a drawing tool at the bottom of the page and draw masks that tightly covers anomalous areas for the anomaly label. The following image is an example of a mask that tightly covers an anomaly.
The following is an example of a poor mask that doesn't tightly cover an anomaly.
If you have more images to correct, choose Next and repeat steps 2 and 3.
Choose Submit and close to finish updating images.
