Detecting labels in an image
You can use the DetectLabels
operation to detect labels (objects and concepts) in an image and retrieve information
about an image’s properties. Image properties include attributes like the color of the
foreground and background and the image's sharpness, brightness, and contrast. You can
retrieve just the labels in an image, just the properties of the image, or both. For an
example, see Analyzing images stored in an Amazon S3 bucket.
The following examples use various AWS SDKs and the AWS CLI to call
DetectLabels
. For information about the DetectLabels
operation response, see DetectLabels response.
To detect labels in an image
-
If you haven't already:
Create or update a user with AmazonRekognitionFullAccess
and
AmazonS3ReadOnlyAccess
permissions. For more
information, see Step 1: Set up an AWS account and create a
User.
Install and configure the AWS CLI and the AWS SDKs. For more information, see
Step 2: Set up the AWS CLI and AWS SDKs.
-
Upload an image that contains one or more objects—such as trees,
houses, and boat—to your S3 bucket. The image must be in
.jpg or .png format.
For instructions, see Uploading
Objects into Amazon S3 in the
Amazon Simple Storage Service User Guide.
-
Use the following examples to call the DetectLabels
operation.
- Java
-
This example displays a list of labels that were detected in the
input image. Replace the values of bucket
and
photo
with the names of the Amazon S3 bucket and image
that you used in step 2.
package com.amazonaws.samples;
import java.util.List;
import com.amazonaws.services.rekognition.model.BoundingBox;
import com.amazonaws.services.rekognition.model.DetectLabelsRequest;
import com.amazonaws.services.rekognition.model.DetectLabelsResult;
import com.amazonaws.services.rekognition.model.Image;
import com.amazonaws.services.rekognition.model.Instance;
import com.amazonaws.services.rekognition.model.Label;
import com.amazonaws.services.rekognition.model.Parent;
import com.amazonaws.services.rekognition.model.S3Object;
import com.amazonaws.services.rekognition.AmazonRekognition;
import com.amazonaws.services.rekognition.AmazonRekognitionClientBuilder;
import com.amazonaws.services.rekognition.model.AmazonRekognitionException;
public class DetectLabels {
public static void main(String[] args) throws Exception {
String photo = "photo";
String bucket = "bucket";
AmazonRekognition rekognitionClient = AmazonRekognitionClientBuilder.defaultClient();
DetectLabelsRequest request = new DetectLabelsRequest()
.withImage(new Image().withS3Object(new S3Object().withName(photo).withBucket(bucket)))
.withMaxLabels(10).withMinConfidence(75F);
try {
DetectLabelsResult result = rekognitionClient.detectLabels(request);
List<Label> labels = result.getLabels();
System.out.println("Detected labels for " + photo + "\n");
for (Label label : labels) {
System.out.println("Label: " + label.getName());
System.out.println("Confidence: " + label.getConfidence().toString() + "\n");
List<Instance> instances = label.getInstances();
System.out.println("Instances of " + label.getName());
if (instances.isEmpty()) {
System.out.println(" " + "None");
} else {
for (Instance instance : instances) {
System.out.println(" Confidence: " + instance.getConfidence().toString());
System.out.println(" Bounding box: " + instance.getBoundingBox().toString());
}
}
System.out.println("Parent labels for " + label.getName() + ":");
List<Parent> parents = label.getParents();
if (parents.isEmpty()) {
System.out.println(" None");
} else {
for (Parent parent : parents) {
System.out.println(" " + parent.getName());
}
}
System.out.println("--------------------");
System.out.println();
}
} catch (AmazonRekognitionException e) {
e.printStackTrace();
}
}
}
- AWS CLI
-
This example displays the JSON output from the
detect-labels
CLI operation. Replace the values of
bucket
and photo
with the names of the
Amazon S3 bucket and image that you used in Step 2. Replace the value of profile-name
with the name of your developer profile.
aws rekognition detect-labels --image '{ "S3Object": { "Bucket": "bucket-name", "Name": "file-name" } }' \
--features GENERAL_LABELS IMAGE_PROPERTIES \
--settings '{"ImageProperties": {"MaxDominantColors":1}, {"GeneralLabels":{"LabelInclusionFilters":["Cat"]}}}' \
--profile profile-name \
--region us-east-1
If you are accessing the CLI on a Windows device, use double
quotes instead of single quotes and escape the inner double quotes
by backslash (i.e. \) to address any parser errors you may
encounter. For an example, see the following:
aws rekognition detect-labels --image "{\"S3Object\":{\"Bucket\":\"bucket-name\",\"Name\":\"file-name\"}}" --features GENERAL_LABELS IMAGE_PROPERTIES \
--settings "{\"GeneralLabels\":{\"LabelInclusionFilters\":[\"Car\"]}}" --profile profile-name --region us-east-1
- Python
This example displays the labels that were detected in the input image. In the function main
,
replace the values of bucket
and photo
with the names of the
Amazon S3 bucket and image that you used in Step 2. Replace the value of profile_name
in the line that creates the Rekognition session with the name of your developer profile.
#Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#PDX-License-Identifier: MIT-0 (For details, see https://github.com/awsdocs/amazon-rekognition-developer-guide/blob/master/LICENSE-SAMPLECODE.)
import boto3
def detect_labels(photo, bucket):
session = boto3.Session(profile_name='profile-name')
client = session.client('rekognition')
response = client.detect_labels(Image={'S3Object':{'Bucket':bucket,'Name':photo}},
MaxLabels=10,
# Uncomment to use image properties and filtration settings
#Features=["GENERAL_LABELS", "IMAGE_PROPERTIES"],
#Settings={"GeneralLabels": {"LabelInclusionFilters":["Cat"]},
# "ImageProperties": {"MaxDominantColors":10}}
)
print('Detected labels for ' + photo)
print()
for label in response['Labels']:
print("Label: " + label['Name'])
print("Confidence: " + str(label['Confidence']))
print("Instances:")
for instance in label['Instances']:
print(" Bounding box")
print(" Top: " + str(instance['BoundingBox']['Top']))
print(" Left: " + str(instance['BoundingBox']['Left']))
print(" Width: " + str(instance['BoundingBox']['Width']))
print(" Height: " + str(instance['BoundingBox']['Height']))
print(" Confidence: " + str(instance['Confidence']))
print()
print("Parents:")
for parent in label['Parents']:
print(" " + parent['Name'])
print("Aliases:")
for alias in label['Aliases']:
print(" " + alias['Name'])
print("Categories:")
for category in label['Categories']:
print(" " + category['Name'])
print("----------")
print()
if "ImageProperties" in str(response):
print("Background:")
print(response["ImageProperties"]["Background"])
print()
print("Foreground:")
print(response["ImageProperties"]["Foreground"])
print()
print("Quality:")
print(response["ImageProperties"]["Quality"])
print()
return len(response['Labels'])
def main():
photo = 'photo-name'
bucket = 'bucket-name'
label_count = detect_labels(photo, bucket)
print("Labels detected: " + str(label_count))
if __name__ == "__main__":
main()
- .NET
-
This example displays a list of labels that were detected in the
input image. Replace the values of bucket
and
photo
with the names of the Amazon S3 bucket and image
that you used in Step 2.
//Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//PDX-License-Identifier: MIT-0 (For details, see https://github.com/awsdocs/amazon-rekognition-developer-guide/blob/master/LICENSE-SAMPLECODE.)
using System;
using Amazon.Rekognition;
using Amazon.Rekognition.Model;
public class DetectLabels
{
public static void Example()
{
String photo = "input.jpg";
String bucket = "bucket";
AmazonRekognitionClient rekognitionClient = new AmazonRekognitionClient();
DetectLabelsRequest detectlabelsRequest = new DetectLabelsRequest()
{
Image = new Image()
{
S3Object = new S3Object()
{
Name = photo,
Bucket = bucket
},
},
MaxLabels = 10,
MinConfidence = 75F
};
try
{
DetectLabelsResponse detectLabelsResponse = rekognitionClient.DetectLabels(detectlabelsRequest);
Console.WriteLine("Detected labels for " + photo);
foreach (Label label in detectLabelsResponse.Labels)
Console.WriteLine("{0}: {1}", label.Name, label.Confidence);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
}
- Ruby
-
This example displays a list of labels that were detected in the
input image. Replace the values of bucket
and
photo
with the names of the Amazon S3 bucket and image
that you used in Step 2.
# Add to your Gemfile
# gem 'aws-sdk-rekognition'
require 'aws-sdk-rekognition'
credentials = Aws::Credentials.new(
ENV['AWS_ACCESS_KEY_ID'],
ENV['AWS_SECRET_ACCESS_KEY']
)
bucket = 'bucket' # the bucket name without s3://
photo = 'photo' # the name of file
client = Aws::Rekognition::Client.new credentials: credentials
attrs = {
image: {
s3_object: {
bucket: bucket,
name: photo
},
},
max_labels: 10
}
response = client.detect_labels attrs
puts "Detected labels for: #{photo}"
response.labels.each do |label|
puts "Label: #{label.name}"
puts "Confidence: #{label.confidence}"
puts "Instances:"
label['instances'].each do |instance|
box = instance['bounding_box']
puts " Bounding box:"
puts " Top: #{box.top}"
puts " Left: #{box.left}"
puts " Width: #{box.width}"
puts " Height: #{box.height}"
puts " Confidence: #{instance.confidence}"
end
puts "Parents:"
label.parents.each do |parent|
puts " #{parent.name}"
end
puts "------------"
puts ""
end
- Node.js
-
This example displays a list of labels that were detected in the
input image. Replace the values of bucket
and
photo
with the names of the Amazon S3 bucket and image
that you used in Step 2. Replace the value of profile_name
in the line that creates the Rekognition session with the name of your developer profile.
If you are using TypeScript definitions, you may need to use
import AWS from 'aws-sdk'
instead of const
AWS = require('aws-sdk')
, in order to run the program
with Node.js. You can consult the AWS
SDK for Javascript for more details. Depending on how
you have your configurations set up, you also may need to specify
your region with
AWS.config.update({region:region
});
.
// Load the SDK
var AWS = require('aws-sdk');
const bucket = 'bucket-name' // the bucketname without s3://
const photo = 'image-name' // the name of file
var credentials = new AWS.SharedIniFileCredentials({profile: 'profile-name'});
AWS.config.credentials = credentials;
AWS.config.update({region:'region-name'});
const client = new AWS.Rekognition();
const params = {
Image: {
S3Object: {
Bucket: bucket,
Name: photo
},
},
MaxLabels: 10
}
client.detectLabels(params, function(err, response) {
if (err) {
console.log(err, err.stack); // if an error occurred
} else {
console.log(`Detected labels for: ${photo}`)
response.Labels.forEach(label => {
console.log(`Label: ${label.Name}`)
console.log(`Confidence: ${label.Confidence}`)
console.log("Instances:")
label.Instances.forEach(instance => {
let box = instance.BoundingBox
console.log(" Bounding box:")
console.log(` Top: ${box.Top}`)
console.log(` Left: ${box.Left}`)
console.log(` Width: ${box.Width}`)
console.log(` Height: ${box.Height}`)
console.log(` Confidence: ${instance.Confidence}`)
})
console.log("Parents:")
label.Parents.forEach(parent => {
console.log(` ${parent.Name}`)
})
console.log("------------")
console.log("")
}) // for response.labels
} // if
});
- Java V2
-
This code is taken from the AWS Documentation SDK examples
GitHub repository. See the full example here.
//snippet-start:[rekognition.java2.detect_labels.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.rekognition.RekognitionClient;
import software.amazon.awssdk.services.rekognition.model.Image;
import software.amazon.awssdk.services.rekognition.model.DetectLabelsRequest;
import software.amazon.awssdk.services.rekognition.model.DetectLabelsResponse;
import software.amazon.awssdk.services.rekognition.model.Label;
import software.amazon.awssdk.services.rekognition.model.RekognitionException;
import software.amazon.awssdk.services.rekognition.model.S3Object;
import java.util.List;
/**
* Before running this Java V2 code example, set up your development environment, including your credentials.
*
* For more information, see the following documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class DetectLabels {
public static void main(String[] args) {
final String usage = "\n" +
"Usage: " +
" <bucket> <image>\n\n" +
"Where:\n" +
" bucket - The name of the Amazon S3 bucket that contains the image (for example, ,ImageBucket)." +
" image - The name of the image located in the Amazon S3 bucket (for example, Lake.png). \n\n";
if (args.length != 2) {
System.out.println(usage);
System.exit(1);
}
String bucket = args[0];
String image = args[1];
Region region = Region.US_WEST_2;
RekognitionClient rekClient = RekognitionClient.builder()
.region(region)
.credentialsProvider(ProfileCredentialsProvider.create("profile-name"))
.build();
getLabelsfromImage(rekClient, bucket, image);
rekClient.close();
}
// snippet-start:[rekognition.java2.detect_labels_s3.main]
public static void getLabelsfromImage(RekognitionClient rekClient, String bucket, String image) {
try {
S3Object s3Object = S3Object.builder()
.bucket(bucket)
.name(image)
.build() ;
Image myImage = Image.builder()
.s3Object(s3Object)
.build();
DetectLabelsRequest detectLabelsRequest = DetectLabelsRequest.builder()
.image(myImage)
.maxLabels(10)
.build();
DetectLabelsResponse labelsResponse = rekClient.detectLabels(detectLabelsRequest);
List<Label> labels = labelsResponse.labels();
System.out.println("Detected labels for the given photo");
for (Label label: labels) {
System.out.println(label.name() + ": " + label.confidence().toString());
}
} catch (RekognitionException e) {
System.out.println(e.getMessage());
System.exit(1);
}
}
// snippet-end:[rekognition.java2.detect_labels.main]
}
DetectLabels operation request
The input to DetectLabel
is an image. In this example JSON input, the
source image is loaded from an Amazon S3 Bucket. MaxLabels
is the maximum
number of labels to return in the response. MinConfidence
is the
minimum confidence that Amazon Rekognition Image must have in the accuracy of the detected label for
it to be returned in the response.
Features lets you specify one or more features of the image that you want
returned, allowing you to select GENERAL_LABELS
and
IMAGE_PROPERTIES
. Including GENERAL_LABELS
will return
the labels detected in the input image, while including
IMAGE_PROPERTIES
will allow you to access image color and quality.
Settings lets you filter the returned items for both the
GENERAL_LABELS
and IMAGE_PROPERTIES
features. For
labels you can use inclusive and exclusive filters. You can also filter by label
specific, individual labels or by label category:
-
LabelInclusionFilters - Allows you to specify which labels you want
included in the response.
-
LabelExclusionFilters - Allows you to specify which labels you want
excluded from the response.
-
LabelCategoryInclusionFilters - Allows you to specify which label
categories you want included in the response.
-
LabelCategoryExclusionFilters - Allows you to specify which label
categories you want excluded from the response.
You can also combine inclusive and exclusive filters according to your needs,
excluding some labels or categories and including others.
IMAGE_PROPERTIES
refer to an image’s dominant colors and quality
attributes such as sharpness, brightness, and contrast. When detecting
IMAGE_PROPERTIES
you can specify the maximum number of dominant
colors to return (default is 10) by using the MaxDominantColors
parameter.
{
"Image": {
"S3Object": {
"Bucket": "bucket",
"Name": "input.jpg"
}
},
"MaxLabels": 10,
"MinConfidence": 75,
"Features": [ "GENERAL_LABELS", "IMAGE_PROPERTIES" ],
"Settings": {
"GeneralLabels": {
"LabelInclusionFilters": [<Label(s)>],
"LabelExclusionFilters": [<Label(s)>],
"LabelCategoryInclusionFilters": [<Category Name(s)>],
"LabelCategoryExclusionFilters": [<Category Name(s)>]
},
"ImageProperties": {
"MaxDominantColors":10
}
}
}
DetectLabels response
The response from DetectLabels
is an array of labels detected in the
image and the level of confidence by which they were detected.
The following is an example response from DetectLabels
. The sample
response below contains a variety of attributes returned for GENERAL_LABELS,
including:
-
Name - The name of the detected label. In this example, the operation
detected an object with the label Mobile Phone.
-
Confidence - Each label has an associated level of confidence. In this
example, the confidence for the label was 99.36%.
-
Parents - The ancestor labels for a detected label. In this example, the
label Mobile Phone has one parent label named Phone.
-
Aliases - Information about possible Aliases for the label. In this
example, the Mobile Phone label has a possible alias of Cell Phone.
-
Categories - The label category that the detected label belongs to. In
this example, it is Technology and Computing.
The response for common object labels includes bounding box information for the
location of the label on the input image. For example, the Person label has an
instances array containing two bounding boxes. These are the locations of two people
detected in the image.
The response also includes attributes regarding IMAGE_PROPERTIES. The attributes
presented by the IMAGE_PROPERTIES feature are:
-
Quality - Information about the Sharpness, Brightness, and Contrast of the
input image, scored between 0 to 100. Quality is reported for the entire
image and for the background and foreground of the image, if available.
However, Contrast is only reported for the entire image while Sharpness and
Brightness are also reported for Background and Foreground.
-
Dominant Color - An array of the dominant colors in the image. Each
dominant color is described with a simplified color name, a CSS color
palette, RGB values, and a hex code.
-
Foreground - Information about the dominant Colors, Sharpness and
Brightness of the input image’s foreground.
-
Background - Information about the dominant Colors, Sharpness and
Brightness of the input image’s background.
When GENERAL_LABELS and IMAGE_PROPERTIES are used together as input parameters,
Amazon Rekognition Image will also return the dominant colors of objects with bounding boxes.
The field LabelModelVersion
contains the version number of the
detection model used by DetectLabels
.
{
"Labels": [
{
"Name": "Mobile Phone",
"Parents": [
{
"Name": "Phone"
}
],
"Aliases": [
{
"Name": "Cell Phone"
}
],
"Categories": [
{
"Name": "Technology and Computing"
}
],
"Confidence": 99.9364013671875,
"Instances": [
{
"BoundingBox": {
"Width": 0.26779675483703613,
"Height": 0.8562285900115967,
"Left": 0.3604024350643158,
"Top": 0.09245597571134567,
}
"Confidence": 99.9364013671875,
"DominantColors": [
{
"Red": 120,
"Green": 137,
"Blue": 132,
"HexCode": "3A7432",
"SimplifiedColor": "red",
"CssColor": "fuscia",
"PixelPercentage": 40.10
}
],
}
]
}
],
"ImageProperties": {
"Quality": {
"Brightness": 40,
"Sharpness": 40,
"Contrast": 24,
},
"DominantColors": [
{
"Red": 120,
"Green": 137,
"Blue": 132,
"HexCode": "3A7432",
"SimplifiedColor": "red",
"CssColor": "fuscia",
"PixelPercentage": 40.10
}
],
"Foreground": {
"Quality": {
"Brightness": 40,
"Sharpness": 40,
},
"DominantColors": [
{
"Red": 200,
"Green": 137,
"Blue": 132,
"HexCode": "3A7432",
"CSSColor": "",
"SimplifiedColor": "red",
"PixelPercentage": 30.70
}
],
}
"Background": {
"Quality": {
"Brightness": 40,
"Sharpness": 40,
},
"DominantColors": [
{
"Red": 200,
"Green": 137,
"Blue": 132,
"HexCode": "3A7432",
"CSSColor": "",
"SimplifiedColor": "Red",
"PixelPercentage": 10.20
}
],
},
},
"LabelModelVersion": "3.0"
}
When using the DetectLabels API, you might need the response structure to mimic
the older API response structure, where both primary labels and aliases were
contained in the same list.
The following is an example of the current API response from DetectLabels:
"Labels": [
{
"Name": "Mobile Phone",
"Confidence": 99.99717712402344,
"Instances": [],
"Parents": [
{
"Name": "Phone"
}
],
"Aliases": [
{
"Name": "Cell Phone"
}
]
}
]
The following example shows the previous response from the DetectLabels API:
"Labels": [
{
"Name": "Mobile Phone",
"Confidence": 99.99717712402344,
"Instances": [],
"Parents": [
{
"Name": "Phone"
}
]
},
{
"Name": "Cell Phone",
"Confidence": 99.99717712402344,
"Instances": [],
"Parents": [
{
"Name": "Phone"
}
]
},
]
If needed, you can transform the current response to follow the format of the
older response. You can use the following sample code to transform the latest API
response to the previous API response structure:
- Python
-
The following code sample demonstrates how to transform the current
response from the DetectLabels API. In the code sample below, you can
replace the value of EXAMPLE_INFERENCE_OUTPUT
with the output of a DetectLabels operation you have run.
from copy import deepcopy
LABEL_KEY = "Labels"
ALIASES_KEY = "Aliases"
INSTANCE_KEY = "Instances"
NAME_KEY = "Name"
#Latest API response sample
EXAMPLE_INFERENCE_OUTPUT = {
"Labels": [
{
"Name": "Mobile Phone",
"Confidence": 97.530106,
"Categories": [
{
"Name": "Technology and Computing"
}
],
"Aliases": [
{
"Name": "Cell Phone"
}
],
"Instances":[
{
"BoundingBox":{
"Height":0.1549897,
"Width":0.07747964,
"Top":0.50858885,
"Left":0.00018205095
},
"Confidence":98.401276
}
]
},
{
"Name": "Urban",
"Confidence": 99.99982,
"Categories": [
"Colors and Visual Composition"
]
}
]
}
def expand_aliases(inferenceOutputsWithAliases):
if LABEL_KEY in inferenceOutputsWithAliases:
expandInferenceOutputs = []
for primaryLabelDict in inferenceOutputsWithAliases[LABEL_KEY]:
if ALIASES_KEY in primaryLabelDict:
for alias in primaryLabelDict[ALIASES_KEY]:
aliasLabelDict = deepcopy(primaryLabelDict)
aliasLabelDict[NAME_KEY] = alias[NAME_KEY]
del aliasLabelDict[ALIASES_KEY]
if INSTANCE_KEY in aliasLabelDict:
del aliasLabelDict[INSTANCE_KEY]
expandInferenceOutputs.append(aliasLabelDict)
inferenceOutputsWithAliases[LABEL_KEY].extend(expandInferenceOutputs)
return inferenceOutputsWithAliases
if __name__ == "__main__":
outputWithExpandAliases = expand_aliases(EXAMPLE_INFERENCE_OUTPUT)
print(outputWithExpandAliases)
Below is an example of the transformed response:
#Output example after the transformation
{
"Labels": [
{
"Name": "Mobile Phone",
"Confidence": 97.530106,
"Categories": [
{
"Name": "Technology and Computing"
}
],
"Aliases": [
{
"Name": "Cell Phone"
}
],
"Instances":[
{
"BoundingBox":{
"Height":0.1549897,
"Width":0.07747964,
"Top":0.50858885,
"Left":0.00018205095
},
"Confidence":98.401276
}
]
},
{
"Name": "Cell Phone",
"Confidence": 97.530106,
"Categories": [
{
"Name": "Technology and Computing"
}
],
"Instances":[]
},
{
"Name": "Urban",
"Confidence": 99.99982,
"Categories": [
"Colors and Visual Composition"
]
}
]
}