Data Sources - Amazon S3 - Amazon Macie

Data Sources - Amazon S3

The Amazon S3 Data Sources resource provides statistical data and other information about the Amazon Simple Storage Service (Amazon S3) buckets that Amazon Macie monitors and analyzes for your account. This includes a breakdown of each bucket's public access settings, and the size and number of objects that Macie can analyze to detect sensitive data in a bucket. This resource also indicates whether you configured any classification jobs to analyze objects in a bucket and, if you have, the date and time when one of those jobs most recently ran. The data is available for all the S3 buckets that Macie monitors and analyzes for your account.

Note that complete data is available for an S3 bucket only if Macie can retrieve and process information about the bucket. If a bucket's permissions settings or an error prevents Macie from retrieving and processing information about a bucket or a bucket's objects, statistical data and other information is limited to a subset of the bucket's properties, such as the bucket's name and the account ID for the AWS account that owns the bucket.

You can use the Amazon S3 Data Sources resource to retrieve (query) statistical data and other information about the settings and contents of one or more S3 buckets that Macie monitors and analyzes for your account. To customize and refine your query, you can use the supported parameters to specify whether and how to filter, sort, and paginate the query results.

URI

/datasources/s3

HTTP methods

POST

Operation ID: DescribeBuckets

Retrieves (queries) statistical data and other information about one or more S3 buckets that Amazon Macie monitors and analyzes.

Responses
Status code Response model Description
200 DescribeBucketsResponse

The request succeeded.

400 ValidationException

The request failed because it contains a syntax error.

402 ServiceQuotaExceededException

The request failed because fulfilling the request would exceed one or more service quotas for your account.

403 AccessDeniedException

The request was denied because you don't have sufficient access to the specified resource.

404 ResourceNotFoundException

The request failed because the specified resource wasn't found.

409 ConflictException

The request failed because it conflicts with the current state of the specified resource.

429 ThrottlingException

The request failed because you sent too many requests during a certain amount of time.

500 InternalServerException

The request failed due to an unknown internal server error, exception, or failure.

Schemas

Request bodies

{ "criteria": { }, "nextToken": "string", "maxResults": integer, "sortCriteria": { "orderBy": enum, "attributeName": "string" } }

Response bodies

{ "nextToken": "string", "buckets": [ { "bucketName": "string", "objectCount": integer, "sizeInBytes": integer, "serverSideEncryption": { "kmsMasterKeyId": "string", "type": enum }, "versioning": boolean, "classifiableObjectCount": integer, "allowsUnencryptedObjectUploads": enum, "publicAccess": { "effectivePermission": enum, "permissionConfiguration": { "accountLevelPermissions": { "blockPublicAccess": { "blockPublicPolicy": boolean, "restrictPublicBuckets": boolean, "blockPublicAcls": boolean, "ignorePublicAcls": boolean } }, "bucketLevelPermissions": { "accessControlList": { "allowsPublicReadAccess": boolean, "allowsPublicWriteAccess": boolean }, "blockPublicAccess": { "blockPublicPolicy": boolean, "restrictPublicBuckets": boolean, "blockPublicAcls": boolean, "ignorePublicAcls": boolean }, "bucketPolicy": { "allowsPublicReadAccess": boolean, "allowsPublicWriteAccess": boolean } } } }, "objectCountByEncryptionType": { "kmsManaged": integer, "s3Managed": integer, "customerManaged": integer, "unencrypted": integer, "unknown": integer }, "errorMessage": "string", "errorCode": enum, "classifiableSizeInBytes": integer, "tags": [ { "value": "string", "key": "string" } ], "unclassifiableObjectCount": { "total": integer, "storageClass": integer, "fileType": integer }, "accountId": "string", "lastUpdated": "string", "bucketArn": "string", "bucketCreatedAt": "string", "replicationDetails": { "replicationAccounts": [ "string" ], "replicatedExternally": boolean, "replicated": boolean }, "jobDetails": { "lastJobId": "string", "lastJobRunTime": "string", "isDefinedInJob": enum, "isMonitoredByJob": enum }, "sharedAccess": enum, "unclassifiableObjectSizeInBytes": { "total": integer, "storageClass": integer, "fileType": integer }, "region": "string", "sizeInBytesCompressed": integer } ] }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }

Properties

AccessControlList

Provides information about the permissions settings of the bucket-level access control list (ACL) for an S3 bucket.

Property Type Required Description
allowsPublicReadAccess

boolean

False

Specifies whether the ACL grants the general public with read access permissions for the bucket.

allowsPublicWriteAccess

boolean

False

Specifies whether the ACL grants the general public with write access permissions for the bucket.

AccessDeniedException

Provides information about an error that occurred due to insufficient access to a specified resource.

Property Type Required Description
message

string

False

The explanation of the error that occurred.

AccountLevelPermissions

Provides information about the account-level permissions settings that apply to an S3 bucket.

Property Type Required Description
blockPublicAccess

BlockPublicAccess

False

The block public access settings for the AWS account that owns the bucket.

BlockPublicAccess

Provides information about the block public access settings for an S3 bucket. These settings can apply to a bucket at the account level or bucket level. For detailed information about each setting, see Blocking public access to your Amazon S3 storage in the Amazon Simple Storage Service User Guide.

Property Type Required Description
blockPublicAcls

boolean

False

Specifies whether Amazon S3 blocks public access control lists (ACLs) for the bucket and objects in the bucket.

blockPublicPolicy

boolean

False

Specifies whether Amazon S3 blocks public bucket policies for the bucket.

ignorePublicAcls

boolean

False

Specifies whether Amazon S3 ignores public ACLs for the bucket and objects in the bucket.

restrictPublicBuckets

boolean

False

Specifies whether Amazon S3 restricts public bucket policies for the bucket.

BucketCriteria

Specifies, as a map, one or more property-based conditions that filter the results of a query for information about S3 buckets.

Property Type Required Description

*

object

False

BucketCriteriaAdditionalProperties

Specifies the operator to use in a property-based condition that filters the results of a query for information about S3 buckets.

Property Type Required Description
eq

Array of type string

False

The value for the property matches (equals) the specified value. If you specify multiple values, Amazon Macie uses OR logic to join the values.

gt

integer

Format: int64

False

The value for the property is greater than the specified value.

gte

integer

Format: int64

False

The value for the property is greater than or equal to the specified value.

lt

integer

Format: int64

False

The value for the property is less than the specified value.

lte

integer

Format: int64

False

The value for the property is less than or equal to the specified value.

neq

Array of type string

False

The value for the property doesn't match (doesn't equal) the specified value. If you specify multiple values, Amazon Macie uses OR logic to join the values.

prefix

string

False

The name of the bucket begins with the specified value.

BucketLevelPermissions

Provides information about the bucket-level permissions settings for an S3 bucket.

Property Type Required Description
accessControlList

AccessControlList

False

The permissions settings of the access control list (ACL) for the bucket. This value is null if an ACL hasn't been defined for the bucket.

blockPublicAccess

BlockPublicAccess

False

The block public access settings for the bucket.

bucketPolicy

BucketPolicy

False

The permissions settings of the bucket policy for the bucket. This value is null if a bucket policy hasn't been defined for the bucket.

BucketMetadata

Provides statistical data and other information about an S3 bucket that Amazon Macie monitors and analyzes for your account. If an error occurs when Macie attempts to retrieve and process information about the bucket or the bucket's objects, the value for the versioning property is false and the value for most other properties is null. Exceptions are accountId, bucketArn, bucketCreatedAt, bucketName, lastUpdated, and region. To identify the cause of the error, refer to the errorCode and errorMessage values.

Property Type Required Description
accountId

string

False

The unique identifier for the AWS account that owns the bucket.

allowsUnencryptedObjectUploads

string

Values: TRUE | FALSE | UNKNOWN

False

Specifies whether the bucket policy for the bucket requires server-side encryption of objects when objects are uploaded to the bucket. Possible values are:

  • FALSE - The bucket policy requires server-side encryption of new objects. PutObject requests must include the x-amz-server-side-encryption header and the value for that header must be AES256 or aws:kms.

  • TRUE - The bucket doesn't have a bucket policy or it has a bucket policy that doesn't require server-side encryption of new objects. If a bucket policy exists, it doesn't require PutObject requests to include the x-amz-server-side-encryption header and it doesn't require the value for that header to be AES256 or aws:kms.

  • UNKNOWN - Amazon Macie can't determine whether the bucket policy requires server-side encryption of new objects.

bucketArn

string

False

The Amazon Resource Name (ARN) of the bucket.

bucketCreatedAt

string

Format: date-time

False

The date and time, in UTC and extended ISO 8601 format, when the bucket was created.

bucketName

string

False

The name of the bucket.

classifiableObjectCount

integer

Format: int64

False

The total number of objects that Amazon Macie can analyze in the bucket. These objects use a supported storage class and have a file name extension for a supported file or storage format.

classifiableSizeInBytes

integer

Format: int64

False

The total storage size, in bytes, of the objects that Amazon Macie can analyze in the bucket. These objects use a supported storage class and have a file name extension for a supported file or storage format.

If versioning is enabled for the bucket, Macie calculates this value based on the size of the latest version of each applicable object in the bucket. This value doesn't reflect the storage size of all versions of each applicable object in the bucket.

errorCode

BucketMetadataErrorCode

False

Specifies the error code for an error that prevented Amazon Macie from retrieving and processing information about the bucket and the bucket's objects. If this value is ACCESS_DENIED, Macie doesn't have permission to retrieve the information. For example, the bucket has a restrictive bucket policy and Amazon S3 denied the request. If this value is null, Macie was able to retrieve and process the information.

errorMessage

string

False

A brief description of the error (errorCode) that prevented Amazon Macie from retrieving and processing information about the bucket and the bucket's objects. This value is null if Macie was able to retrieve and process the information.

jobDetails

JobDetails

False

Specifies whether any one-time or recurring classification jobs are configured to analyze data in the bucket, and, if so, the details of the job that ran most recently.

lastUpdated

string

Format: date-time

False

The date and time, in UTC and extended ISO 8601 format, when Amazon Macie most recently retrieved both bucket and object metadata from Amazon S3 for the bucket.

objectCount

integer

Format: int64

False

The total number of objects in the bucket.

objectCountByEncryptionType

ObjectCountByEncryptionType

False

The total number of objects that are in the bucket, grouped by server-side encryption type. This includes a grouping that reports the total number of objects that aren't encrypted or use client-side encryption.

publicAccess

BucketPublicAccess

False

Specifies whether the bucket is publicly accessible due to the combination of permissions settings that apply to the bucket, and provides information about those settings.

region

string

False

The AWS Region that hosts the bucket.

replicationDetails

ReplicationDetails

False

Specifies whether the bucket is configured to replicate one or more objects to buckets for other AWS accounts and, if so, which accounts.

serverSideEncryption

BucketServerSideEncryption

False

Specifies whether the bucket encrypts new objects by default and, if so, the type of server-side encryption that's used.

sharedAccess

string

Values: EXTERNAL | INTERNAL | NOT_SHARED | UNKNOWN

False

Specifies whether the bucket is shared with another AWS account. Possible values are:

  • EXTERNAL - The bucket is shared with an AWS account that isn't part of the same Amazon Macie organization.

  • INTERNAL - The bucket is shared with an AWS account that's part of the same Amazon Macie organization.

  • NOT_SHARED - The bucket isn't shared with other AWS accounts.

  • UNKNOWN - Amazon Macie wasn't able to evaluate the shared access settings for the bucket.

sizeInBytes

integer

Format: int64

False

The total storage size, in bytes, of the bucket.

If versioning is enabled for the bucket, Amazon Macie calculates this value based on the size of the latest version of each object in the bucket. This value doesn't reflect the storage size of all versions of each object in the bucket.

sizeInBytesCompressed

integer

Format: int64

False

The total storage size, in bytes, of the objects that are compressed (.gz, .gzip, .zip) files in the bucket.

If versioning is enabled for the bucket, Amazon Macie calculates this value based on the size of the latest version of each applicable object in the bucket. This value doesn't reflect the storage size of all versions of each applicable object in the bucket.

tags

Array of type KeyValuePair

False

An array that specifies the tags (keys and values) that are associated with the bucket.

unclassifiableObjectCount

ObjectLevelStatistics

False

The total number of objects that Amazon Macie can't analyze in the bucket. These objects don't use a supported storage class or don't have a file name extension for a supported file or storage format.

unclassifiableObjectSizeInBytes

ObjectLevelStatistics

False

The total storage size, in bytes, of the objects that Amazon Macie can't analyze in the bucket. These objects don't use a supported storage class or don't have a file name extension for a supported file or storage format.

versioning

boolean

False

Specifies whether versioning is enabled for the bucket.

BucketMetadataErrorCode

The error code for an error that prevented Amazon Macie from retrieving and processing information about an S3 bucket and the bucket's objects.

  • ACCESS_DENIED

BucketPermissionConfiguration

Provides information about the account-level and bucket-level permissions settings for an S3 bucket.

Property Type Required Description
accountLevelPermissions

AccountLevelPermissions

False

The account-level permissions settings that apply to the bucket.

bucketLevelPermissions

BucketLevelPermissions

False

The bucket-level permissions settings for the bucket.

BucketPolicy

Provides information about the permissions settings of the bucket policy for an S3 bucket.

Property Type Required Description
allowsPublicReadAccess

boolean

False

Specifies whether the bucket policy allows the general public to have read access to the bucket.

allowsPublicWriteAccess

boolean

False

Specifies whether the bucket policy allows the general public to have write access to the bucket.

BucketPublicAccess

Provides information about the permissions settings that determine whether an S3 bucket is publicly accessible.

Property Type Required Description
effectivePermission

string

Values: PUBLIC | NOT_PUBLIC | UNKNOWN

False

Specifies whether the bucket is publicly accessible due to the combination of permissions settings that apply to the bucket. Possible values are:

  • NOT_PUBLIC - The bucket isn't publicly accessible.

  • PUBLIC - The bucket is publicly accessible.

  • UNKNOWN - Amazon Macie can't determine whether the bucket is publicly accessible.

permissionConfiguration

BucketPermissionConfiguration

False

The account-level and bucket-level permissions settings for the bucket.

BucketServerSideEncryption

Provides information about the default server-side encryption settings for an S3 bucket. For detailed information about these settings, see Setting default server-side encryption behavior for Amazon S3 buckets in the Amazon Simple Storage Service User Guide.

Property Type Required Description
kmsMasterKeyId

string

False

The Amazon Resource Name (ARN) or unique identifier (key ID) for the AWS KMS key that's used by default to encrypt objects that are added to the bucket. This value is null if the bucket uses an Amazon S3 managed key to encrypt new objects or the bucket doesn't encrypt new objects by default.

type

string

Values: NONE | AES256 | aws:kms

False

The type of server-side encryption that's used by default when storing new objects in the bucket. Possible values are:

  • AES256 - New objects are encrypted with an Amazon S3 managed key. They use SSE-S3 encryption.

  • aws:kms - New objects are encrypted with an AWS KMS key (kmsMasterKeyId), either an AWS managed key or a customer managed key. They use SSE-KMS encryption.

  • NONE - New objects aren't encrypted by default. Default encryption is disabled for the bucket.

BucketSortCriteria

Specifies criteria for sorting the results of a query for information about S3 buckets.

Property Type Required Description
attributeName

string

False

The name of the bucket property to sort the results by. This value can be one of the following properties that Amazon Macie defines as bucket metadata: accountId, bucketName, classifiableObjectCount, classifiableSizeInBytes, objectCount, or sizeInBytes.

orderBy

string

Values: ASC | DESC

False

The sort order to apply to the results, based on the value specified by the attributeName property. Valid values are: ASC, sort the results in ascending order; and, DESC, sort the results in descending order.

ConflictException

Provides information about an error that occurred due to a versioning conflict for a specified resource.

Property Type Required Description
message

string

False

The explanation of the error that occurred.

DescribeBucketsRequest

Specifies criteria for filtering, sorting, and paginating the results of a query for statistical data and other information about S3 buckets.

Property Type Required Description
criteria

BucketCriteria

False

The criteria to use to filter the query results.

maxResults

integer

Format: int32

False

The maximum number of items to include in each page of the response. The default value is 50.

nextToken

string

False

The nextToken string that specifies which page of results to return in a paginated response.

sortCriteria

BucketSortCriteria

False

The criteria to use to sort the query results.

DescribeBucketsResponse

Provides the results of a query that retrieved statistical data and other information about one or more S3 buckets that Amazon Macie monitors and analyzes for your account.

Property Type Required Description
buckets

Array of type BucketMetadata

False

An array of objects, one for each bucket that meets the filter criteria specified in the request.

nextToken

string

False

The string to use in a subsequent request to get the next page of results in a paginated response. This value is null if there are no additional pages.

InternalServerException

Provides information about an error that occurred due to an unknown internal server error, exception, or failure.

Property Type Required Description
message

string

False

The explanation of the error that occurred.

JobDetails

Specifies whether any one-time or recurring classification jobs are configured to analyze data in an S3 bucket, and, if so, the details of the job that ran most recently.

Property Type Required Description
isDefinedInJob

string

Values: TRUE | FALSE | UNKNOWN

False

Specifies whether any one-time or recurring jobs are configured to analyze data in the bucket. Possible values are:

  • TRUE - The bucket is explicitly included in the bucket definition (S3BucketDefinitionForJob) for one or more jobs and at least one of those jobs has a status other than CANCELLED. Or the bucket matched the bucket criteria (S3BucketCriteriaForJob) for at least one job that previously ran.

  • FALSE - The bucket isn't explicitly included in the bucket definition (S3BucketDefinitionForJob) for any jobs, all the jobs that explicitly include the bucket in their bucket definitions have a status of CANCELLED, or the bucket didn't match the bucket criteria (S3BucketCriteriaForJob) for any jobs that previously ran.

  • UNKNOWN - An exception occurred when Amazon Macie attempted to retrieve job data for the bucket.

isMonitoredByJob

string

Values: TRUE | FALSE | UNKNOWN

False

Specifies whether any recurring jobs are configured to analyze data in the bucket. Possible values are:

  • TRUE - The bucket is explicitly included in the bucket definition (S3BucketDefinitionForJob) for one or more recurring jobs or the bucket matches the bucket criteria (S3BucketCriteriaForJob) for one or more recurring jobs. At least one of those jobs has a status other than CANCELLED.

  • FALSE - The bucket isn't explicitly included in the bucket definition (S3BucketDefinitionForJob) for any recurring jobs, the bucket doesn't match the bucket criteria (S3BucketCriteriaForJob) for any recurring jobs, or all the recurring jobs that are configured to analyze data in the bucket have a status of CANCELLED.

  • UNKNOWN - An exception occurred when Amazon Macie attempted to retrieve job data for the bucket.

lastJobId

string

False

The unique identifier for the job that ran most recently and is configured to analyze data in the bucket, either the latest run of a recurring job or the only run of a one-time job.

This value is typically null if the value for the isDefinedInJob property is FALSE or UNKNOWN.

lastJobRunTime

string

Format: date-time

False

The date and time, in UTC and extended ISO 8601 format, when the job (lastJobId) started. If the job is a recurring job, this value indicates when the most recent run started.

This value is typically null if the value for the isDefinedInJob property is FALSE or UNKNOWN.

KeyValuePair

Provides information about the tags that are associated with an S3 bucket or object. Each tag consists of a required tag key and an associated tag value.

Property Type Required Description
key

string

False

One part of a key-value pair that comprises a tag. A tag key is a general label that acts as a category for more specific tag values.

value

string

False

One part of a key-value pair that comprises a tag. A tag value acts as a descriptor for a tag key. A tag value can be an empty string.

ObjectCountByEncryptionType

Provides information about the number of objects that are in an S3 bucket and use certain types of server-side encryption, use client-side encryption, or aren't encrypted.

Property Type Required Description
customerManaged

integer

Format: int64

False

The total number of objects that are encrypted with a customer-provided key. The objects use customer-provided server-side encryption (SSE-C).

kmsManaged

integer

Format: int64

False

The total number of objects that are encrypted with an AWS KMS key, either an AWS managed key or a customer managed key. The objects use AWS KMS encryption (SSE-KMS).

s3Managed

integer

Format: int64

False

The total number of objects that are encrypted with an Amazon S3 managed key. The objects use Amazon S3 managed encryption (SSE-S3).

unencrypted

integer

Format: int64

False

The total number of objects that aren't encrypted or use client-side encryption.

unknown

integer

Format: int64

False

The total number of objects that Amazon Macie doesn't have current encryption metadata for. Macie can't provide current data about the encryption settings for these objects.

ObjectLevelStatistics

Provides information about the total storage size (in bytes) or number of objects that Amazon Macie can't analyze in one or more S3 buckets. In a BucketMetadata or MatchingBucket object, this data is for a specific bucket. In a GetBucketStatisticsResponse object, this data is aggregated for the buckets in the query results. If versioning is enabled for a bucket, total storage size values are based on the size of the latest version of each applicable object in the bucket.

Property Type Required Description
fileType

integer

Format: int64

False

The total storage size (in bytes) or number of objects that Amazon Macie can't analyze because the objects don't have a file name extension for a supported file or storage format.

storageClass

integer

Format: int64

False

The total storage size (in bytes) or number of objects that Amazon Macie can't analyze because the objects use an unsupported storage class.

total

integer

Format: int64

False

The total storage size (in bytes) or number of objects that Amazon Macie can't analyze because the objects use an unsupported storage class or don't have a file name extension for a supported file or storage format.

ReplicationDetails

Provides information about settings that define whether one or more objects in an S3 bucket are replicated to S3 buckets for other AWS accounts and, if so, which accounts.

Property Type Required Description
replicated

boolean

False

Specifies whether the bucket is configured to replicate one or more objects to any destination.

replicatedExternally

boolean

False

Specifies whether the bucket is configured to replicate one or more objects to an AWS account that isn't part of the same Amazon Macie organization.

replicationAccounts

Array of type string

False

An array of AWS account IDs, one for each AWS account that the bucket is configured to replicate one or more objects to.

ResourceNotFoundException

Provides information about an error that occurred because a specified resource wasn't found.

Property Type Required Description
message

string

False

The explanation of the error that occurred.

ServiceQuotaExceededException

Provides information about an error that occurred due to one or more service quotas for an account.

Property Type Required Description
message

string

False

The explanation of the error that occurred.

ThrottlingException

Provides information about an error that occurred because too many requests were sent during a certain amount of time.

Property Type Required Description
message

string

False

The explanation of the error that occurred.

ValidationException

Provides information about an error that occurred due to a syntax error in a request.

Property Type Required Description
message

string

False

The explanation of the error that occurred.

See also

For more information about using this API in one of the language-specific AWS SDKs and references, see the following:

DescribeBuckets