State machines - AWS Content Analysis

State machines

This solution uses state machines to orchestrate performing analysis on media. There is a state machine for both image and video inputs. Each state machine is composed of parallel states called stages containing analysis operations. Each stage runs sequentially. Analysis operations are sets of AWS Lambda functions that start, monitor, and collect results for specific types of analysis, for example, calling Amazon Rekognition APIs to detect labels in a video. Analysis operators are run within the state machine.

The MieCompleteWorkflow video analysis state machine consists of the following stages:

  • A defaultPrelimVideoStage2 input preparation stage that checks the format of the video and gathers structural information about the video package using MediaInfo. It also uses AWS MediaConvert to create a proxy encode and audio file that will be used as the input to downstream stages. The MediaConvert job also generates thumbnail images for the GUI.

  • A defaultVideoStage video analysis stage that analyzes the content of the video.

  • A defaultAudioStage audio analysis stage that analyzes the audio content of the video.

  • A defaultTextStage text analysis stage that analyzes the text from the audio stage using Amazon Translate to generate a translation and Amazon Comprehend to identify entities and key phrases from the text.

  • A defaultTextSynthesis text synthesis stage.

When you upload a video from the web application it initiates the appropriate state machine. A link to the AWS Step Function console for state machine runs of each job is available from the link in the Status column of the job in the Collection view of the UI (refer to Figure 13). In order to initiate state machine, the web application sends an analysis workflow run request to Amazon API Gateway where an AWS Lambda function validates the request and starts the analysis state machine. The web application uses a workflow status Amazon API Gateway request to periodically monitor the progress status and update the web application.


      Link to state machine execution result in AWS console
        from the sample application

Figure 13: Link to state machine run result in AWS Management Console from the sample application


      Video analysis workflow

Figure 14: Video analysis workflow


      Image analysis workflow

Figure 15: Image analysis workflow

Analysis operations

Analysis operations are sub-components of the MIE state machines. Each analysis operation in the analysis state machine has a consistent structure that uses a combination of Step Functions ASL logic and AWS Lambda functions to perform the following steps:

  • Check if the operation is activated and validate input: Each analysis operation can be activated or deactivated using the Configuration parameters that are passed in to the state machine from the run workflow API Gateway request. Each operation checks if it is activated or deactivated before running. If it is deactivated, it skips any subsequent steps.

  • Start analysis: Initiates the start of a specific type of analysis on the input.

  • Check analysis status, collect, and store analysis results in the data plane: Monitors the results of the analysis. When the analysis is complete, collects the results of the analysis and stores it in the data plane.

  • Handle errors: If an error occurs in any step of an analysis operation, the operation will handle the error using the logic described in the following analysis state machine error handling section.


        Example analysis operation state machine section
          (CelebrityRecognition)

Figure 16: Example analysis operation state machine section (CelebrityRecognition)

Analysis state machine error handling

The state machines have a built-in retry logic using the Catch and Retry AWS Step Functions to improve the resiliency of the workflow.

"Retry": [ { "ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException", "Lambda.Unknown", "MasExecutionError"], "IntervalSeconds": 2, "MaxAttempts": 2, "BackoffRate": 2 } ], "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "OPERATION_NAME Failed (STAGE_NAME)", "ResultPath": "$.Outputs" } ]

If the retries fail, the state machine calls the OperatorFailedLambda Lambda function to communicate that an error has occurred in the analysis. Other analysis operations in the same parallel state are allowed to complete and then the analysis state machine will stop running when the error is detected in the complete_stage_execution_lambda Lambda function that is run at the end of each parallel state of the analysis state machine.