Example: Identifying objects in video streams using SageMaker

This example demonstrates how to create a solution that uses SageMaker to identify when certain objects appear in an Amazon Kinesis video stream. SageMaker is the managed platform for developers and data scientists to build, train, and deploy machine learning models quickly and easily.

The example consists of a Docker container that includes the application functionality, and an AWS CloudFormation template that automates the deployment of the application's AWS resources.

The AWS CloudFormation template creates the following resources:

• An Amazon Elastic Container Service (Amazon ECS) cluster that uses the AWS Fargate compute engine that runs the library software.

• An Amazon DynamoDB table that maintains checkpoints and related state across workers that run on Fargate tasks.

• A Kinesis data stream that captures the inference outputs generated from SageMaker.

• An AWS Lambda function that parses the output from SageMaker.

• AWS Identity and Access Management (IAM) resources for providing access across services.

• Amazon CloudWatch resources for monitoring the application.

The application is compatible with any SageMaker endpoint that processes data. This example contains instructions for creating a SageMaker endpoint that uses a sample object identification algorithm template. You can modify or replace the algorithm based on your application's use cases and requirements.

Prerequisites

The example application has the following prerequisites:

• SageMaker
• Kinesis video stream
• Service-Linked role

SageMaker

This example requires a SageMaker notebook. For information about creating a notebook, see Creating a Notebook Instance in the Amazon SageMaker Developer Guide. Note the following when creating your notebook:

• Add the Amazon_JumpStart_Object_Detection.ipynb example (from the Introduction to Amazon Algorithms section in the SageMaker Examples tab of the Jupyter console) to the notebook.

• Create an Amazon Simple Storage Service (Amazon S3) bucket, and provide its name in the Prerequisites step when adding the example.

• After you create the notebook, choose Endpoint configuration on the SageMaker console, and make a note of the Endpoint name.

Kinesis video stream

This example requires one or more Kinesis Video Streams that have live video data. For information about creating a Kinesis video stream and sending data to it from a camera, see GStreamer. Make a note of your Kinesis video stream name.

Service-Linked role

This example requires that your account has a service-linked role for Fargate operation. New AWS accounts have this role enabled by default. If you see the following error when creating the application, you must enable the service-linked role:

Unable to assume the service linked role. Please verify that the ECS service linked role exists

To enable the service-linked role, run the following command:

aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

Creating the application

To create the sample application, you use AWS CloudFormation and the templates that are provided.

To use AWS CloudFormation to create the application

1. Sign in to the AWS Management Console and open the AWS CloudFormation console using one of the following links for your AWS Region. The link launches the correct stack for your Region:

   • Launch in Asia Pacific (Sydney) Region (ap-southeast-2)

   • Launch in Asia Pacific (Tokyo) Region (ap-northeast-1)

   • Launch in Europe (Frankfurt) Region (eu-central-1)

   • Launch in Europe (Ireland) Region (eu-west-1)

   • Launch in US East (N. Virginia) Region (us-east-1)

   • Launch in US West (Oregon) Region (us-west-2)

2. On the Create Stack page, provide the following values:

   • Give the stack a unique name (for example, username-KVS-SageMaker).

   • Provide the SageMaker endpoint name (not the endpoint ARN) that you created in the previous section.

   • Provide the name of your Kinesis video stream. If you have more than one Kinesis video stream, provide the stream names in quotation marks and separated by commas.

   • Keep the rest of the settings as they are.

   Choose Next.

3. On the Options page, keep the settings as they are.

4. Select the I acknowledge that AWS CloudFormation might create IAM resources check box. Choose Next.

AWS CloudFormation creates the application.

The following table lists several parameters used by the Docker container when you create a stack using this AWS CloudFormation template. Their values are predefined in the SSM resource in the template, but you can customize them as needed.

Resource name	Default value	Description
inferenceInterval	6	The sampling ratio for video frames that are sent to the SageMaker endpoint. Currently, we only support inferencing on I-Frames. The default value of 6 means that 1 out of every 6 I-Frames is sent to the SageMaker endpoint.
sageMakerTaskQueueSize	5000	The size of the queue that maintains the pending requests to the SageMaker endpoint. The size of the queue is affected by ‘inferenceInternval’ and ‘sageMakerTaskTimeoutInMilli’. If SageMaker inference takes longer, requests are buffered in this queue.
sageMakerTaskThreadPoolSize	20	Number of threads that's used to concurrently carry out SageMaker requests.
sageMakerTaskTimeoutInMilli	20000	The maximum duration accepted for a single request (or a retry request) that's sent to the SageMaker endpoint.
sageMakerTaskThreadPoolName	SageMakerThreadPool-%d	The name of the threadpool that's sending requests to the SageMaker endpoint.

To customize the values of these parameters, download the AWS CloudFormation template by choosing the template URL on the Create stack page, and then locate these parameters in the Params section of the template that looks like this:

Params:
    Type: AWS::SSM::Parameter
    Properties: 
      Name:
        Ref: AppName
      Description: "Configuration for SageMaker app"
      Type: String
      Value: 
        Fn::Sub: |
          {"streamNames":[${StreamNames}], "tagFilters":[${TagFilters}],"sageMakerEndpoint":"${SageMakerEndpoint}",
           "endPointAcceptContentType": "${EndPointAcceptContentType}",
           "kdsStreamName":"${Kds}","inferenceInterval":6,"sageMakerTaskQueueSize":5000,
           "sageMakerTaskThreadPoolSize":20,"sageMakerTaskTimeoutInMilli":20000,
           "sageMakerTaskThreadPoolName":"SageMakerThreadPool-%d"}                
            

Monitoring the application

The application created by the AWS CloudFormation template includes an Amazon CloudWatch dashboard and a CloudWatch log stream used to monitor application metrics and events.

Application dashboard

The application includes a CloudWatch dashboard for monitoring application metrics. To view the application dashboard, open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/ and choose Dashboards in the left navigation bar.

Choose the KVS-SageMaker-Driver-KvsSageMakerIntegration-aws-region dashboard. The dashboard shows the following information:

• Frame Metrics: The metrics for processing the video stream, sending frames to the SageMaker endpoint, and writing to the Kinesis data stream that connects the SageMaker notebook with the AWS Lambda function that processes SageMaker inference output results.

• IngestToProcessLatency: The time difference between when a video frame is ingested into the Kinesis Video Streams service and when the application receives the frame.

• Current Lease Total: The application is granted permissions to read from the Kinesis video stream using a lease. This metric shows the number of active leases. The application uses one lease per Kinesis video stream, and one lease for synchronization between streams.

• Lease Sync Metrics: The frequency and duration of permission lease synchronization.

• LeaseCount per Worker: The distribution of leases among the SageMaker worker threads.

• Number of Workers: The number of SageMaker workers processing streams. Each task in an Amazon ECS cluster has one worker running. One worker can process more than one stream.

• ECS Service Utilization: The usage metrics for the Amazon ECS cluster.

• KinesisDataStream: The usage metrics of the Kinesis data stream.

• SageMaker: The operations performed by the SageMaker notebook.

• Lambda: The number and duration of the Lambda function that processes the output from the SageMaker notebook.

If any of the information in these graphs indicates an operational issue (such as a value steadily increasing rather than being stable), see the following section about how to read the application logs to determine the issue.

CloudWatch Logs

The application includes two CloudWatch Logs:

Topics

• The application log
• The Lambda function log

The application log

You can use the application log to monitor application events and error conditions. You can also use this log if you need to contact product support with an issue.

To read the Application Log

1. Open the Amazon ECS console at https://console.aws.amazon.com/ecs.

2. Choose the KVS-Sagemaker-Driver cluster.

3. Choose the stack-name-SageMakerDriverService service in the Services tab.

4. Choose the Logs tab.

The application log shows events such as initialization, configuration, and lease activity.

The Lambda function log

You can use the Lambda function log to track successful object identifications.

To read the Lambda log

1. Open the AWS Lambda console at https://console.aws.amazon.com/lambda.

2. Choose the Lambda function for your application. The Lambda function name is in the following format:

   stack-name-LambdaFunction-A1B2C3D4E5F6G

3. Choose the Monitoring panel.

4. Choose View logs in CloudWatch.

The CloudWatch log for the application shows successful identifications of objects in the Kinesis video stream and other application events.

Extending the application

You can add custom functionality to your application by modifying the values that you provide in the AWS CloudFormation template window as follows:

• EndPointAcceptContentType: You can change this value if your SageMaker endpoint does not accept frames in JPG format. The following formats are supported:

  • image/jpeg

  • image/png

  • image/bmp

  • image/gif

  • application/x-image

• LambdaFunctionBucket, LambdaFunctionKey: The provided settings use an AWS Lambda function that processes the SageMaker output and writes it to CloudWatch Logs. If you want to send the SageMaker output elsewhere, you can provide your own Lambda function.

• Tag Filters: If you have streams that are tagged using the TagStream action, you can specify the tags of streams that you want to process. For example, if you have two streams that have the Location key with the values Front and Parking, you would filter to only use those streams using the following entry:

  {"key":"Location","values":["Front","Parking"]}

Cleaning up the application

After you've finished with the application that you created for this tutorial, we recommend that you delete any resources that you don't want to keep, to avoid incurring any ongoing charges.

1. SageMaker endpoint: If you created the SageMaker endpoint for this tutorial rather than using an existing endpoint, delete the endpoint. In the SageMaker control panel, choose Endpoint configurations. Choose the endpoint that you created, and choose Actions, Delete. Confirm the deletion.

2. SageMaker notebook: On the SageMaker console, choose Notebook instances. Choose the notebook that you created, and choose Actions, Stop. When the notebook shows that its Status is Stopped, choose Actions, Delete. Confirm the deletion.

   Note

   For more information about cleaning up SageMaker resources, see Clean up in the SageMaker developer guide.

3. SageMaker execution policy: On the IAM console, in the navigation pane, choose Policies. Choose the policy that you created for this tutorial. The name of the policy is similar to the following: AmazonSageMaker-ExecutionPolicy-timestamp.

   Choose Policy actions, Delete. Confirm the deletion.

4. SageMaker execution role: On the IAM console, in the navigation pane, choose Roles. Choose the role that you created for this tutorial. The name of the role is similar to the following: AmazonSageMaker-ExecutionRole-timestamp.

   Choose Delete role. Confirm the deletion.

5. AWS CloudFormation stack: On the AWS CloudFormation console, choose the stack that you created for this tutorial. Choose Actions, Delete Stack. Confirm the deletion.

6. Amazon S3 bucket: On the Amazon S3 console, choose the bucket that you created to store the SageMaker assets. Choose Delete. Enter the name of the bucket and choose Confirm to confirm deletion.

7. Kinesis video stream: On the Kinesis Video Streams console, choose the video stream that you created for the application. Choose Delete. Confirm the deletion.