Detecting labels in an image - Amazon Rekognition

Detecting labels in an image

You can use the DetectLabels operation to detect labels (objects and concepts) in an image and retrieve information about an image’s properties. Image properties include attributes like the color of the foreground and background and the image's sharpness, brightness, and contrast. You can retrieve just the labels in an image, just the properties of the image, or both. For an example, see Analyzing images stored in an Amazon S3 bucket.

The following examples use various AWS SDKs and the AWS CLI to call DetectLabels. For information about the DetectLabels operation response, see DetectLabels response.

To detect labels in an image
  1. If you haven't already:

    1. Create or update a user with AmazonRekognitionFullAccess and AmazonS3ReadOnlyAccess permissions. For more information, see Step 1: Set up an AWS account and create a User.

    2. Install and configure the AWS CLI and the AWS SDKs. For more information, see Step 2: Set up the AWS CLI and AWS SDKs.

  2. Upload an image that contains one or more objects—such as trees, houses, and boat—to your S3 bucket. The image must be in .jpg or .png format.

    For instructions, see Uploading Objects into Amazon S3 in the Amazon Simple Storage Service User Guide.

  3. Use the following examples to call the DetectLabels operation.

    Java

    This example displays a list of labels that were detected in the input image. Replace the values of bucket and photo with the names of the Amazon S3 bucket and image that you used in step 2.

    package com.amazonaws.samples; import java.util.List; import com.amazonaws.services.rekognition.model.BoundingBox; import com.amazonaws.services.rekognition.model.DetectLabelsRequest; import com.amazonaws.services.rekognition.model.DetectLabelsResult; import com.amazonaws.services.rekognition.model.Image; import com.amazonaws.services.rekognition.model.Instance; import com.amazonaws.services.rekognition.model.Label; import com.amazonaws.services.rekognition.model.Parent; import com.amazonaws.services.rekognition.model.S3Object; import com.amazonaws.services.rekognition.AmazonRekognition; import com.amazonaws.services.rekognition.AmazonRekognitionClientBuilder; import com.amazonaws.services.rekognition.model.AmazonRekognitionException; public class DetectLabels { public static void main(String[] args) throws Exception { String photo = "photo"; String bucket = "bucket"; AmazonRekognition rekognitionClient = AmazonRekognitionClientBuilder.defaultClient(); DetectLabelsRequest request = new DetectLabelsRequest() .withImage(new Image().withS3Object(new S3Object().withName(photo).withBucket(bucket))) .withMaxLabels(10).withMinConfidence(75F); try { DetectLabelsResult result = rekognitionClient.detectLabels(request); List<Label> labels = result.getLabels(); System.out.println("Detected labels for " + photo + "\n"); for (Label label : labels) { System.out.println("Label: " + label.getName()); System.out.println("Confidence: " + label.getConfidence().toString() + "\n"); List<Instance> instances = label.getInstances(); System.out.println("Instances of " + label.getName()); if (instances.isEmpty()) { System.out.println(" " + "None"); } else { for (Instance instance : instances) { System.out.println(" Confidence: " + instance.getConfidence().toString()); System.out.println(" Bounding box: " + instance.getBoundingBox().toString()); } } System.out.println("Parent labels for " + label.getName() + ":"); List<Parent> parents = label.getParents(); if (parents.isEmpty()) { System.out.println(" None"); } else { for (Parent parent : parents) { System.out.println(" " + parent.getName()); } } System.out.println("--------------------"); System.out.println(); } } catch (AmazonRekognitionException e) { e.printStackTrace(); } } }
    AWS CLI

    This example displays the JSON output from the detect-labels CLI operation. Replace the values of bucket and photo with the names of the Amazon S3 bucket and image that you used in Step 2. Replace the value of profile-name with the name of your developer profile.

    aws rekognition detect-labels --image '{ "S3Object": { "Bucket": "bucket-name", "Name": "file-name" } }' \ --features GENERAL_LABELS IMAGE_PROPERTIES \ --settings '{"ImageProperties": {"MaxDominantColors":1}, {"GeneralLabels":{"LabelInclusionFilters":["Cat"]}}}' \ --profile profile-name \ --region us-east-1

    If you are accessing the CLI on a Windows device, use double quotes instead of single quotes and escape the inner double quotes by backslash (i.e. \) to address any parser errors you may encounter. For an example, see the following:

    aws rekognition detect-labels --image "{\"S3Object\":{\"Bucket\":\"bucket-name\",\"Name\":\"file-name\"}}" --features GENERAL_LABELS IMAGE_PROPERTIES \ --settings "{\"GeneralLabels\":{\"LabelInclusionFilters\":[\"Car\"]}}" --profile profile-name --region us-east-1
    Python

    This example displays the labels that were detected in the input image. In the function main, replace the values of bucket and photo with the names of the Amazon S3 bucket and image that you used in Step 2. Replace the value of profile_name in the line that creates the Rekognition session with the name of your developer profile.

    #Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. #PDX-License-Identifier: MIT-0 (For details, see https://github.com/awsdocs/amazon-rekognition-developer-guide/blob/master/LICENSE-SAMPLECODE.) import boto3 def detect_labels(photo, bucket): session = boto3.Session(profile_name='profile-name') client = session.client('rekognition') response = client.detect_labels(Image={'S3Object':{'Bucket':bucket,'Name':photo}}, MaxLabels=10, # Uncomment to use image properties and filtration settings #Features=["GENERAL_LABELS", "IMAGE_PROPERTIES"], #Settings={"GeneralLabels": {"LabelInclusionFilters":["Cat"]}, # "ImageProperties": {"MaxDominantColors":10}} ) print('Detected labels for ' + photo) print() for label in response['Labels']: print("Label: " + label['Name']) print("Confidence: " + str(label['Confidence'])) print("Instances:") for instance in label['Instances']: print(" Bounding box") print(" Top: " + str(instance['BoundingBox']['Top'])) print(" Left: " + str(instance['BoundingBox']['Left'])) print(" Width: " + str(instance['BoundingBox']['Width'])) print(" Height: " + str(instance['BoundingBox']['Height'])) print(" Confidence: " + str(instance['Confidence'])) print() print("Parents:") for parent in label['Parents']: print(" " + parent['Name']) print("Aliases:") for alias in label['Aliases']: print(" " + alias['Name']) print("Categories:") for category in label['Categories']: print(" " + category['Name']) print("----------") print() if "ImageProperties" in str(response): print("Background:") print(response["ImageProperties"]["Background"]) print() print("Foreground:") print(response["ImageProperties"]["Foreground"]) print() print("Quality:") print(response["ImageProperties"]["Quality"]) print() return len(response['Labels']) def main(): photo = 'photo-name' bucket = 'bucket-name' label_count = detect_labels(photo, bucket) print("Labels detected: " + str(label_count)) if __name__ == "__main__": main()
    .NET

    This example displays a list of labels that were detected in the input image. Replace the values of bucket and photo with the names of the Amazon S3 bucket and image that you used in Step 2.

    //Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. //PDX-License-Identifier: MIT-0 (For details, see https://github.com/awsdocs/amazon-rekognition-developer-guide/blob/master/LICENSE-SAMPLECODE.) using System; using Amazon.Rekognition; using Amazon.Rekognition.Model; public class DetectLabels { public static void Example() { String photo = "input.jpg"; String bucket = "bucket"; AmazonRekognitionClient rekognitionClient = new AmazonRekognitionClient(); DetectLabelsRequest detectlabelsRequest = new DetectLabelsRequest() { Image = new Image() { S3Object = new S3Object() { Name = photo, Bucket = bucket }, }, MaxLabels = 10, MinConfidence = 75F }; try { DetectLabelsResponse detectLabelsResponse = rekognitionClient.DetectLabels(detectlabelsRequest); Console.WriteLine("Detected labels for " + photo); foreach (Label label in detectLabelsResponse.Labels) Console.WriteLine("{0}: {1}", label.Name, label.Confidence); } catch (Exception e) { Console.WriteLine(e.Message); } } }
    Ruby

    This example displays a list of labels that were detected in the input image. Replace the values of bucket and photo with the names of the Amazon S3 bucket and image that you used in Step 2.

    # Add to your Gemfile # gem 'aws-sdk-rekognition' require 'aws-sdk-rekognition' credentials = Aws::Credentials.new( ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'] ) bucket = 'bucket' # the bucket name without s3:// photo = 'photo' # the name of file client = Aws::Rekognition::Client.new credentials: credentials attrs = { image: { s3_object: { bucket: bucket, name: photo }, }, max_labels: 10 } response = client.detect_labels attrs puts "Detected labels for: #{photo}" response.labels.each do |label| puts "Label: #{label.name}" puts "Confidence: #{label.confidence}" puts "Instances:" label['instances'].each do |instance| box = instance['bounding_box'] puts " Bounding box:" puts " Top: #{box.top}" puts " Left: #{box.left}" puts " Width: #{box.width}" puts " Height: #{box.height}" puts " Confidence: #{instance.confidence}" end puts "Parents:" label.parents.each do |parent| puts " #{parent.name}" end puts "------------" puts "" end
    Node.js

    This example displays a list of labels that were detected in the input image. Replace the values of bucket and photo with the names of the Amazon S3 bucket and image that you used in Step 2. Replace the value of profile_name in the line that creates the Rekognition session with the name of your developer profile.

    If you are using TypeScript definitions, you may need to use import AWS from 'aws-sdk' instead of const AWS = require('aws-sdk'), in order to run the program with Node.js. You can consult the AWS SDK for Javascript for more details. Depending on how you have your configurations set up, you also may need to specify your region with AWS.config.update({region:region});.

    // Load the SDK var AWS = require('aws-sdk'); const bucket = 'bucket-name' // the bucketname without s3:// const photo = 'image-name' // the name of file var credentials = new AWS.SharedIniFileCredentials({profile: 'profile-name'}); AWS.config.credentials = credentials; AWS.config.update({region:'region-name'}); const client = new AWS.Rekognition(); const params = { Image: { S3Object: { Bucket: bucket, Name: photo }, }, MaxLabels: 10 } client.detectLabels(params, function(err, response) { if (err) { console.log(err, err.stack); // if an error occurred } else { console.log(`Detected labels for: ${photo}`) response.Labels.forEach(label => { console.log(`Label: ${label.Name}`) console.log(`Confidence: ${label.Confidence}`) console.log("Instances:") label.Instances.forEach(instance => { let box = instance.BoundingBox console.log(" Bounding box:") console.log(` Top: ${box.Top}`) console.log(` Left: ${box.Left}`) console.log(` Width: ${box.Width}`) console.log(` Height: ${box.Height}`) console.log(` Confidence: ${instance.Confidence}`) }) console.log("Parents:") label.Parents.forEach(parent => { console.log(` ${parent.Name}`) }) console.log("------------") console.log("") }) // for response.labels } // if });
    Java V2

    This code is taken from the AWS Documentation SDK examples GitHub repository. See the full example here.

    //snippet-start:[rekognition.java2.detect_labels.import] import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.rekognition.RekognitionClient; import software.amazon.awssdk.services.rekognition.model.Image; import software.amazon.awssdk.services.rekognition.model.DetectLabelsRequest; import software.amazon.awssdk.services.rekognition.model.DetectLabelsResponse; import software.amazon.awssdk.services.rekognition.model.Label; import software.amazon.awssdk.services.rekognition.model.RekognitionException; import software.amazon.awssdk.services.rekognition.model.S3Object; import java.util.List; /** * Before running this Java V2 code example, set up your development environment, including your credentials. * * For more information, see the following documentation topic: * * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html */ public class DetectLabels { public static void main(String[] args) { final String usage = "\n" + "Usage: " + " <bucket> <image>\n\n" + "Where:\n" + " bucket - The name of the Amazon S3 bucket that contains the image (for example, ,ImageBucket)." + " image - The name of the image located in the Amazon S3 bucket (for example, Lake.png). \n\n"; if (args.length != 2) { System.out.println(usage); System.exit(1); } String bucket = args[0]; String image = args[1]; Region region = Region.US_WEST_2; RekognitionClient rekClient = RekognitionClient.builder() .region(region) .credentialsProvider(ProfileCredentialsProvider.create("profile-name")) .build(); getLabelsfromImage(rekClient, bucket, image); rekClient.close(); } // snippet-start:[rekognition.java2.detect_labels_s3.main] public static void getLabelsfromImage(RekognitionClient rekClient, String bucket, String image) { try { S3Object s3Object = S3Object.builder() .bucket(bucket) .name(image) .build() ; Image myImage = Image.builder() .s3Object(s3Object) .build(); DetectLabelsRequest detectLabelsRequest = DetectLabelsRequest.builder() .image(myImage) .maxLabels(10) .build(); DetectLabelsResponse labelsResponse = rekClient.detectLabels(detectLabelsRequest); List<Label> labels = labelsResponse.labels(); System.out.println("Detected labels for the given photo"); for (Label label: labels) { System.out.println(label.name() + ": " + label.confidence().toString()); } } catch (RekognitionException e) { System.out.println(e.getMessage()); System.exit(1); } } // snippet-end:[rekognition.java2.detect_labels.main] }

DetectLabels operation request

The input to DetectLabel is an image. In this example JSON input, the source image is loaded from an Amazon S3 Bucket. MaxLabels is the maximum number of labels to return in the response. MinConfidence is the minimum confidence that Amazon Rekognition Image must have in the accuracy of the detected label for it to be returned in the response.

Features lets you specify one or more features of the image that you want returned, allowing you to select GENERAL_LABELS and IMAGE_PROPERTIES. Including GENERAL_LABELS will return the labels detected in the input image, while including IMAGE_PROPERTIES will allow you to access image color and quality.

Settings lets you filter the returned items for both the GENERAL_LABELS and IMAGE_PROPERTIES features. For labels you can use inclusive and exclusive filters. You can also filter by label specific, individual labels or by label category:

  • LabelInclusionFilters - Allows you to specify which labels you want included in the response.

  • LabelExclusionFilters - Allows you to specify which labels you want excluded from the response.

  • LabelCategoryInclusionFilters - Allows you to specify which label categories you want included in the response.

  • LabelCategoryExclusionFilters - Allows you to specify which label categories you want excluded from the response.

You can also combine inclusive and exclusive filters according to your needs, excluding some labels or categories and including others.

IMAGE_PROPERTIES refer to an image’s dominant colors and quality attributes such as sharpness, brightness, and contrast. When detecting IMAGE_PROPERTIES you can specify the maximum number of dominant colors to return (default is 10) by using the MaxDominantColors parameter.

{ "Image": { "S3Object": { "Bucket": "bucket", "Name": "input.jpg" } }, "MaxLabels": 10, "MinConfidence": 75, "Features": [ "GENERAL_LABELS", "IMAGE_PROPERTIES" ], "Settings": { "GeneralLabels": { "LabelInclusionFilters": [<Label(s)>], "LabelExclusionFilters": [<Label(s)>], "LabelCategoryInclusionFilters": [<Category Name(s)>], "LabelCategoryExclusionFilters": [<Category Name(s)>] }, "ImageProperties": { "MaxDominantColors":10 } } }

DetectLabels response

The response from DetectLabels is an array of labels detected in the image and the level of confidence by which they were detected.

The following is an example response from DetectLabels. The sample response below contains a variety of attributes returned for GENERAL_LABELS, including:

  • Name - The name of the detected label. In this example, the operation detected an object with the label Mobile Phone.

  • Confidence - Each label has an associated level of confidence. In this example, the confidence for the label was 99.36%.

  • Parents - The ancestor labels for a detected label. In this example, the label Mobile Phone has one parent label named Phone.

  • Aliases - Information about possible Aliases for the label. In this example, the Mobile Phone label has a possible alias of Cell Phone.

  • Categories - The label category that the detected label belongs to. In this example, it is Technology and Computing.

The response for common object labels includes bounding box information for the location of the label on the input image. For example, the Person label has an instances array containing two bounding boxes. These are the locations of two people detected in the image.

The response also includes attributes regarding IMAGE_PROPERTIES. The attributes presented by the IMAGE_PROPERTIES feature are:

  • Quality - Information about the Sharpness, Brightness, and Contrast of the input image, scored between 0 to 100. Quality is reported for the entire image and for the background and foreground of the image, if available. However, Contrast is only reported for the entire image while Sharpness and Brightness are also reported for Background and Foreground.

  • Dominant Color - An array of the dominant colors in the image. Each dominant color is described with a simplified color name, a CSS color palette, RGB values, and a hex code.

  • Foreground - Information about the dominant Colors, Sharpness and Brightness of the input image’s foreground.

  • Background - Information about the dominant Colors, Sharpness and Brightness of the input image’s background.

When GENERAL_LABELS and IMAGE_PROPERTIES are used together as input parameters, Amazon Rekognition Image will also return the dominant colors of objects with bounding boxes.

The field LabelModelVersion contains the version number of the detection model used by DetectLabels.

{ "Labels": [ { "Name": "Mobile Phone", "Parents": [ { "Name": "Phone" } ], "Aliases": [ { "Name": "Cell Phone" } ], "Categories": [ { "Name": "Technology and Computing" } ], "Confidence": 99.9364013671875, "Instances": [ { "BoundingBox": { "Width": 0.26779675483703613, "Height": 0.8562285900115967, "Left": 0.3604024350643158, "Top": 0.09245597571134567, } "Confidence": 99.9364013671875, "DominantColors": [ { "Red": 120, "Green": 137, "Blue": 132, "HexCode": "3A7432", "SimplifiedColor": "red", "CssColor": "fuscia", "PixelPercentage": 40.10 } ], } ] } ], "ImageProperties": { "Quality": { "Brightness": 40, "Sharpness": 40, "Contrast": 24, }, "DominantColors": [ { "Red": 120, "Green": 137, "Blue": 132, "HexCode": "3A7432", "SimplifiedColor": "red", "CssColor": "fuscia", "PixelPercentage": 40.10 } ], "Foreground": { "Quality": { "Brightness": 40, "Sharpness": 40, }, "DominantColors": [ { "Red": 200, "Green": 137, "Blue": 132, "HexCode": "3A7432", "CSSColor": "", "SimplifiedColor": "red", "PixelPercentage": 30.70 } ], } "Background": { "Quality": { "Brightness": 40, "Sharpness": 40, }, "DominantColors": [ { "Red": 200, "Green": 137, "Blue": 132, "HexCode": "3A7432", "CSSColor": "", "SimplifiedColor": "Red", "PixelPercentage": 10.20 } ], }, }, "LabelModelVersion": "3.0" }

Transforming the DetectLabels response

When using the DetectLabels API, you might need the response structure to mimic the older API response structure, where both primary labels and aliases were contained in the same list.

The following is an example of the current API response from DetectLabels:

"Labels": [ { "Name": "Mobile Phone", "Confidence": 99.99717712402344, "Instances": [], "Parents": [ { "Name": "Phone" } ], "Aliases": [ { "Name": "Cell Phone" } ] } ]

The following example shows the previous response from the DetectLabels API:

"Labels": [ { "Name": "Mobile Phone", "Confidence": 99.99717712402344, "Instances": [], "Parents": [ { "Name": "Phone" } ] }, { "Name": "Cell Phone", "Confidence": 99.99717712402344, "Instances": [], "Parents": [ { "Name": "Phone" } ] }, ]

If needed, you can transform the current response to follow the format of the older response. You can use the following sample code to transform the latest API response to the previous API response structure:

Python

The following code sample demonstrates how to transform the current response from the DetectLabels API. In the code sample below, you can replace the value of EXAMPLE_INFERENCE_OUTPUT with the output of a DetectLabels operation you have run.

from copy import deepcopy LABEL_KEY = "Labels" ALIASES_KEY = "Aliases" INSTANCE_KEY = "Instances" NAME_KEY = "Name" #Latest API response sample EXAMPLE_INFERENCE_OUTPUT = { "Labels": [ { "Name": "Mobile Phone", "Confidence": 97.530106, "Categories": [ { "Name": "Technology and Computing" } ], "Aliases": [ { "Name": "Cell Phone" } ], "Instances":[ { "BoundingBox":{ "Height":0.1549897, "Width":0.07747964, "Top":0.50858885, "Left":0.00018205095 }, "Confidence":98.401276 } ] }, { "Name": "Urban", "Confidence": 99.99982, "Categories": [ "Colors and Visual Composition" ] } ] } def expand_aliases(inferenceOutputsWithAliases): if LABEL_KEY in inferenceOutputsWithAliases: expandInferenceOutputs = [] for primaryLabelDict in inferenceOutputsWithAliases[LABEL_KEY]: if ALIASES_KEY in primaryLabelDict: for alias in primaryLabelDict[ALIASES_KEY]: aliasLabelDict = deepcopy(primaryLabelDict) aliasLabelDict[NAME_KEY] = alias[NAME_KEY] del aliasLabelDict[ALIASES_KEY] if INSTANCE_KEY in aliasLabelDict: del aliasLabelDict[INSTANCE_KEY] expandInferenceOutputs.append(aliasLabelDict) inferenceOutputsWithAliases[LABEL_KEY].extend(expandInferenceOutputs) return inferenceOutputsWithAliases if __name__ == "__main__": outputWithExpandAliases = expand_aliases(EXAMPLE_INFERENCE_OUTPUT) print(outputWithExpandAliases)

Below is an example of the transformed response:

#Output example after the transformation { "Labels": [ { "Name": "Mobile Phone", "Confidence": 97.530106, "Categories": [ { "Name": "Technology and Computing" } ], "Aliases": [ { "Name": "Cell Phone" } ], "Instances":[ { "BoundingBox":{ "Height":0.1549897, "Width":0.07747964, "Top":0.50858885, "Left":0.00018205095 }, "Confidence":98.401276 } ] }, { "Name": "Cell Phone", "Confidence": 97.530106, "Categories": [ { "Name": "Technology and Computing" } ], "Instances":[] }, { "Name": "Urban", "Confidence": 99.99982, "Categories": [ "Colors and Visual Composition" ] } ] }