Video frame labeling job reference - Amazon SageMaker AI

Video frame labeling job reference

Use this page to learn about the object detection and object tracking video frame labeling jobs. The information on this page applies to both of these built-in task types.

The video frame labeling job is unique because of the following:

  • You can either provide data objects that are ready to be annotated (video frames), or you can provide video files and have Ground Truth automatically extract video frames.

  • Workers have the ability to save work as they go.

  • You cannot use the Amazon Mechanical Turk workforce to complete your labeling tasks.

  • Ground Truth provides a worker UI, as well as assistive and basic labeling tools, to help workers complete your tasks. You do not need to provide a worker task template.

Use the following topics to learn more about video frame labeling jobs.

Input data

The video frame labeling job uses sequences of video frames. A single sequence is a series of images that have been extracted from a single video. You can either provide your own sequences of video frames, or have Ground Truth automatically extract video frame sequences from your video files. To learn more, see Provide Video Files.

Ground Truth uses sequence files to identify all images in a single sequence. All of the sequences that you want to include in a single labeling job are identified in an input manifest file. Each sequence is used to create a single worker task. You can automatically create sequence files and an input manifest file using Ground Truth automatic data setup. To learn more, see Set up Automated Video Frame Input Data.

To learn how to manually create sequence files and an input manifest file, see Create a Video Frame Input Manifest File.

Job completion times

Video and video frame labeling jobs can take workers hours to complete. You can set the total amount of time that workers can work on each task when you create a labeling job. The maximum time you can set for workers to work on tasks is 7 days. The default value is 3 days.

We strongly recommend that you create tasks that workers can complete within 12 hours. Workers must keep the worker UI open while working on a task. They can save work as they go and Ground Truth saves their work every 15 minutes.

When using the SageMaker AI CreateLabelingJob API operation, set the total time a task is available to workers in the TaskTimeLimitInSeconds parameter of HumanTaskConfig.

When you create a labeling job in the console, you can specify this time limit when you select your workforce type and your work team.

Task types

When you create a video object tracking or video object detection labeling job, you specify the type of annotation that you want workers to create while working on your labeling task. The annotation type determines the type of output data Ground Truth returns and defines the task type for your labeling job.

If you are creating a labeling job using the API operation CreateLabelingJob, you specify the task type using the label category configuration file parameter annotationType. To learn more, see Labeling category configuration file with label category and frame attributes reference.

The following task types are available for both video object tracking or video object detection labeling jobs:

  • Bounding box – Workers are provided with tools to create bounding box annotations. A bounding box is a box that a worker draws around an objects to identify the pixel-location and label of that object in the frame.

  • Polyline – Workers are provided with tools to create polyline annotations. A polyline is defined by the series of ordered x, y coordinates. Each point added to the polyline is connected to the previous point by a line. The polyline does not have to be closed (the start point and end point do not have to be the same) and there are no restrictions on the angles formed between lines.

  • Polygon – Workers are provided with tools to create polygon annotations. A polygon is a closed shape defined by a series of ordered x, y coordinates. Each point added to the polygon is connected to the previous point by a line and there are no restrictions on the angles formed between lines. Two lines (sides) of the polygon cannot cross. The start and end point of a polygon must be the same.

  • Keypoint – Workers are provided with tools to create keypoint annotations. A keypoint is a single point associated with an x, y coordinate in the video frame.

Workforces

When you create a video frame labeling job, you need to specify a work team to complete your annotation tasks. You can choose a work team from a private workforce of your own workers, or from a vendor workforce that you select in the AWS Marketplace. You cannot use the Amazon Mechanical Turk workforce for video frame labeling jobs.

To learn more about vendor workforces, see Subscribe to vendor workforces.

To learn how to create and manage a private workforce, see Private workforce.

Worker user interface (UI)

Ground Truth provides a worker user interface (UI), tools, and assistive labeling features to help workers complete your video labeling tasks. You can preview the worker UI when you create a labeling job in the console.

When you create a labeling job using the API operation CreateLabelingJob, you must provide an ARN provided by Ground Truth in the parameter HumanTaskUiArn to specify the worker UI for your task type. You can use HumanTaskUiArn with the SageMaker AI RenderUiTemplate API operation to preview the worker UI.

You provide worker instructions, labels, and optionally, attributes that workers can use to provide more information about labels and video frames. These attributes are referred to as label category attributes and frame attributes respectively. They are all displayed in the worker UI.

Label category and frame attributes

When you create a video object tracking or video object detection labeling job, you can add one or more label category attributes and frame attributes:

  • Label category attribute – A list of options (strings), a free form text box, or a numeric field associated with one or more labels. It is used by workers to provide metadata about a label.

  • Frame attribute – A list of options (strings), a free form text box, or a numeric field that appears on each video frame a worker is sent to annotate. It is used by workers to provide metadata about video frames.

Additionally, you can use label and frame attributes to have workers verify labels in a video frame label verification job.

Use the following sections to learn more about these attributes. To learn how to add label category and frame attributes to a labeling job, use the Create Labeling Job sections on the task type page of your choice.

Label category attributes

Add label category attributes to labels to give workers the ability to provide more information about the annotations they create. A label category attribute is added to an individual label, or to all labels. When a label category attribute is applied to all labels it is referred to as a global label category attribute.

For example, if you add the label category car, you might also want to capture additional data about your labeled cars, such as if they are occluded or the size of the car. You can capture this metadata using label category attributes. In this example, if you added the attribute occluded to the car label category, you can assign partial, completely, no to the occluded attribute and enable workers to select one of these options.

When you create a label verification job, you add labels category attributes to each label you want workers to verify.

Frame level attributes

Add frame attributes to give workers the ability to provide more information about individual video frames. Each frame attribute you add appears on all frames.

For example, you can add a number-frame attribute to have workers identify the number of objects they see in a particular frame.

In another example, you may want to provide a free-form text box to give workers the ability to provide an answer to a question.

When you create a label verification job, you can add one or more frame attributes to ask workers to provide feedback on all labels in a video frame.

Worker instructions

You can provide worker instructions to help your workers complete your video frame labeling tasks. You might want to cover the following topics when writing your instructions:

  • Best practices and things to avoid when annotating objects.

  • The label category attributes provided (for object detection and object tracking tasks) and how to use them.

  • How to save time while labeling by using keyboard shortcuts.

You can add your worker instructions using the SageMaker AI console while creating a labeling job. If you create a labeling job using the API operation CreateLabelingJob, you specify worker instructions in your label category configuration file.

In addition to your instructions, Ground Truth provides a link to help workers navigate and use the worker portal. View these instructions by selecting the task type on Worker Instructions.

Declining tasks

Workers are able to decline tasks.

Workers decline a task if the instructions are not clear, input data is not displaying correctly, or if they encounter some other issue with the task. If the number of workers per dataset object (NumberOfHumanWorkersPerDataObject) decline the task, the data object is marked as expired and will not be sent to additional workers.

Video frame job permission requirements

When you create a video frame labeling job, in addition to the permission requirements found in Assign IAM Permissions to Use Ground Truth, you must add a CORS policy to your S3 bucket that contains your input manifest file.

CORS permission policy for your S3 bucket

When you create a video frame labeling job, you specify buckets in S3 where your input data and manifest file are located and where your output data will be stored. These buckets may be the same. You must attach the following Cross-origin resource sharing (CORS) policy to your input and output buckets. If you use the Amazon S3 console to add the policy to your bucket, you must use the JSON format.

JSON

[ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "GET", "HEAD", "PUT" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [ "Access-Control-Allow-Origin" ], "MaxAgeSeconds": 3000 } ]

XML

<?xml version="1.0" encoding="UTF-8"?> <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <CORSRule> <AllowedOrigin>*</AllowedOrigin> <AllowedMethod>GET</AllowedMethod> <AllowedMethod>HEAD</AllowedMethod> <AllowedMethod>PUT</AllowedMethod> <MaxAgeSeconds>3000</MaxAgeSeconds> <ExposeHeader>Access-Control-Allow-Origin</ExposeHeader> <AllowedHeader>*</AllowedHeader> </CORSRule> </CORSConfiguration>

To learn how to add a CORS policy to an S3 bucket, see How do I add cross-domain resource sharing with CORS? in the Amazon Simple Storage Service User Guide.