Amazon Kinesis Video Streams
Developer Guide

Example: Identifying Objects in Video Streams Using Amazon SageMaker

This example demonstrates how to create a solution that uses Amazon SageMaker to identify when certain objects appear in an Amazon Kinesis video stream. Amazon SageMaker is the managed platform for developers and data scientists to build, train, and deploy machine learning models quickly and easily.

The example consists of a Docker container that includes the application functionality, and an AWS CloudFormation template that automates the deployment of the application's AWS resources.

The AWS CloudFormation template creates the following resources:

The application is compatible with any Amazon SageMaker endpoint that processes data. This example contains instructions for creating an Amazon SageMaker endpoint that uses a sample object identification algorithm template. You can modify or replace the algorithm based on your application's use cases and requirements.

Prerequisites

The example application has the following prerequisites:

Amazon SageMaker

This example requires an Amazon SageMaker notebook. For information about creating a notebook, see Creating a Notebook Instance in the Amazon SageMaker Developer Guide. Note the following when creating your notebook:

  • Add the object_detection_image_json_format.ipynb example (from the Introduction to Amazon Algorithms section in the SageMaker Examples tab of the Jupyter console) to the notebook.

  • Create an Amazon Simple Storage Service (Amazon S3) bucket, and provide its name in the Prerequisites step when adding the example.

  • After you create the notebook, choose Endpoint configuration on the Amazon SageMaker console, and make a note of the Endpoint name.

Kinesis Video Stream

This example requires one or more Kinesis video streams that have live video data. For information about creating a Kinesis video stream and sending data to it from a camera, see GStreamer. Make a note of your Kinesis video stream name.

Service-Linked Role

This example requires that your account have a service-linked role for AWS Fargate operation. New AWS accounts have this role enabled by default. If you see the following error when creating the application, you must enable the service-linked role:

Unable to assume the service linked role. Please verify that the ECS service linked role exists

To enable the service-linked role, run the following command:

aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

Creating the Application

To create the sample application, you use AWS CloudFormation and the templates that are provided.

To use AWS CloudFormation to create the application

  1. Sign in to the AWS Management Console and open the AWS CloudFormation console using one of the following links for your AWS Region. The link launches the correct stack for your Region:

  2. On the Create Stack page, provide the following values:

    • Give the stack a unique name (for example, username-KVS-SageMaker).

    • Provide the Amazon SageMaker endpoint name (not the endpoint ARN) that you created in the previous section.

    • Provide the name of your Kinesis video stream. If you have more than one Kinesis video stream, provide the stream names in quotation marks and separated by commas.

    • Keep the rest of the settings as they are.

    Choose Next.

  3. On the Options page, keep the settings as they are.

  4. Select the I acknowledge that AWS CloudFormation might create IAM resources check box. Choose Next.

AWS CloudFormation creates the application.

Monitoring the Application

The application created by the AWS CloudFormation template includes an Amazon CloudWatch dashboard and a CloudWatch log stream that you use to monitor application metrics and events.

Application Dashboard

The application includes a CloudWatch dashboard for monitoring application metrics. To view the application dashboard, open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/ and choose Dashboards in the left navigation bar.

Choose the KVS-SageMaker-Driver-KvsSageMakerIntegration-aws-region dashboard. The dashboard shows the following information:

  • Frame Metrics: Metrics for processing the video stream, sending frames to the Amazon SageMaker endpoint, and writing to the Kinesis data stream that connects the Amazon SageMaker notebook with the AWS Lambda function that processes Amazon SageMaker inference output results.

  • IngestToProcessLatency: The time difference between when a video frame is ingested into the Kinesis Video Streams service and when the application receives the frame.

  • Current Lease Total: The application is granted permissions to read from the Kinesis video stream using a lease. This metric shows the number of active leases. The application uses one lease per Kinesis video stream, and one lease for synchronization between streams.

  • Lease Sync Metrics: The frequency and duration of permission lease synchronization.

  • LeaseCount per Worker: The distribution of leases among the Amazon SageMaker worker threads.

  • Number of Workers: The number of Amazon SageMaker workers processing streams. Each task in an Amazon ECS cluster has one worker running. One worker can process more than one stream.

  • ECS Service Utilization: Usage metrics for the Amazon ECS cluster.

  • KinesisDataStream: Usage metrics of the Kinesis data stream.

  • SageMaker: Operations performed by the Amazon SageMaker notebook.

  • Lambda: Number and duration of the Lambda function that processes the output from the Amazon SageMaker notebook.

If any of the information in these graphs indicates an operational issue (such as a value steadily increasing rather than being stable), see the following section about how to read the application logs to determine the issue.

CloudWatch Logs

The application includes two CloudWatch logs:

The Application Log

You can use the application log to monitor application events and error conditions. This log is helpful if you need to contact product support with an issue.

To read the Application Log

  1. Open the Amazon ECS console at https://console.aws.amazon.com/ecs.

  2. Choose the KVS-Sagemaker-Driver cluster.

  3. Choose the stack-name-SageMakerDriverService service in the Services tab.

  4. Choose the Logs tab.

The application log shows events such as initialization, configuration, and lease activity.

The Lambda Function Log

You can use the Lambda function log to track successful object identifications.

To read the Lambda log

  1. Open the AWS Lambda console at https://console.aws.amazon.com/lambda.

  2. Choose the Lambda function for your application. The Lambda function name is in the following format:

    stack-name-LambdaFunction-A1B2C3D4E5F6G
  3. Choose the Monitoring panel.

  4. Choose View logs in CloudWatch.

The CloudWatch log for the application shows successful identifications of objects in the Kinesis video stream and other application events.

Extending the Application

You can add custom functionality to your application by modifying the values that you provide in the AWS CloudFormation template window as follows:

  • EndPointAcceptContentType: You can change this value if your Amazon SageMaker endpoint does not accept frames in JPG format. The following formats are supported:

    • image/jpeg

    • image/png

    • image/bmp

    • image/gif

    • application/x-image

  • LambdaFunctionBucket, LambdaFunctionKey: The provided settings use an AWS Lambda function that processes the Amazon SageMaker output and writes it to CloudWatch Logs. If you want to send the Amazon SageMaker output elsewhere, you can provide your own Lambda function.

  • Tag Filters: If you have streams that are tagged using the TagStream action, you can specify the tags of streams that you want to process. For example, if you have two streams that have the Location key with the values Front and Parking, you would filter to only use those streams using the following entry:

    {"key":"Location","values":["Front","Parking"]}

Cleaning up the Application

After you've finished with the application that you created for this tutorial, we recommend that you delete any resources that you don't want to keep, to avoid incurring any ongoing charges.

  1. Amazon SageMaker endpoint: If you created the Amazon SageMaker endpoint for this tutorial rather than using an existing endpoint, delete the endpoint. In the Amazon SageMaker control panel, choose Endpoint configurations. Choose the endpoint you created, and choose Actions, Delete. Confirm the deletion.

  2. Amazon SageMaker notebook: On the Amazon SageMaker console, choose Notebook instances. Choose the notebook that you created, and choose Actions, Stop. When the notebook shows that its Status is Stopped, choose Actions, Delete. Confirm the deletion.

    Note

    For more information on cleaning up Amazon SageMaker resources, see Clean up in the Amazon SageMaker developer guide.

  3. Amazon SageMaker execution policy: On the IAM console, in the navigation pane, choose Policies. Choose the policy that you created for this tutorial. The name of the policy is similar to the following: AmazonSageMaker-ExecutionPolicy-timestamp

    Choose Policy actions, Delete. Confirm the deletion.

  4. Amazon SageMaker execution role: On the IAM console, in the navigation pane, choose Roles. Choose the role that you created for this tutorial. The name of the role is similar to the following: AmazonSageMaker-ExecutionRole-timestamp

    Choose Delete role. Confirm the deletion.

  5. AWS CloudFormation stack: On the AWS CloudFormation console, choose the stack that you created for this tutorial. Choose Actions, Delete Stack. Confirm the deletion.

  6. Amazon S3 bucket: On the Amazon S3 console, choose the bucket that you created to store the Amazon SageMaker assets. Choose Delete. Enter the name of the bucket and choose Confirm to confirm deletion.

  7. Kinesis video stream: On the Kinesis Video Streams console, choose the video stream that you created for the application. Choose Delete. Confirm the deletion.