Data Sources - Amazon S3 - Amazon Macie

Data Sources - Amazon S3

The Amazon S3 Data Sources resource provides statistical data and other information about the Amazon Simple Storage Service (Amazon S3) buckets that Amazon Macie monitors and analyzes for your account. This includes a breakdown of each bucket's public access and encryption settings. It also includes details about the size and number of objects that Macie can analyze to detect sensitive data in a bucket, and whether and when that analysis most recently occurred. The data is available for all the S3 buckets that Macie monitors and analyzes for your account.

Note that complete data is available for an S3 bucket only if Macie can retrieve and process metadata from Amazon S3 for the bucket and the bucket's objects. If a bucket's permissions settings or an error prevents Macie from retrieving and processing information about a bucket or a bucket's objects, statistical data and other information is limited to a subset of the bucket's properties, such as the bucket's name and the account ID for the AWS account that owns the bucket.

You can use the Amazon S3 Data Sources resource to retrieve (query) statistical data and other information about the settings and contents of one or more S3 buckets that Macie monitors and analyzes for your account. To customize and refine your query, you can use the supported parameters to specify whether and how to filter, sort, and paginate the query results.

URI

/datasources/s3

HTTP methods

POST

Operation ID: DescribeBuckets

Retrieves (queries) statistical data and other information about one or more S3 buckets that Amazon Macie monitors and analyzes for an account.

Responses
Status codeResponse modelDescription
200DescribeBucketsResponse

The request succeeded.

400ValidationException

The request failed because the input doesn't satisfy the constraints specified by the service.

402ServiceQuotaExceededException

The request failed because fulfilling the request would exceed one or more service quotas for your account.

403AccessDeniedException

The request was denied because you don't have sufficient access to the specified resource.

404ResourceNotFoundException

The request failed because the specified resource wasn't found.

409ConflictException

The request failed because it conflicts with the current state of the specified resource.

429ThrottlingException

The request failed because you sent too many requests during a certain amount of time.

500InternalServerException

The request failed due to an unknown internal server error, exception, or failure.

Schemas

Request bodies

{ "criteria": { }, "maxResults": integer, "nextToken": "string", "sortCriteria": { "attributeName": "string", "orderBy": enum } }

Response bodies

{ "buckets": [ { "accountId": "string", "allowsUnencryptedObjectUploads": enum, "bucketArn": "string", "bucketCreatedAt": "string", "bucketName": "string", "classifiableObjectCount": integer, "classifiableSizeInBytes": integer, "errorCode": enum, "errorMessage": "string", "jobDetails": { "isDefinedInJob": enum, "isMonitoredByJob": enum, "lastJobId": "string", "lastJobRunTime": "string" }, "lastAutomatedDiscoveryTime": "string", "lastUpdated": "string", "objectCount": integer, "objectCountByEncryptionType": { "customerManaged": integer, "kmsManaged": integer, "s3Managed": integer, "unencrypted": integer, "unknown": integer }, "publicAccess": { "effectivePermission": enum, "permissionConfiguration": { "accountLevelPermissions": { "blockPublicAccess": { "blockPublicAcls": boolean, "blockPublicPolicy": boolean, "ignorePublicAcls": boolean, "restrictPublicBuckets": boolean } }, "bucketLevelPermissions": { "accessControlList": { "allowsPublicReadAccess": boolean, "allowsPublicWriteAccess": boolean }, "blockPublicAccess": { "blockPublicAcls": boolean, "blockPublicPolicy": boolean, "ignorePublicAcls": boolean, "restrictPublicBuckets": boolean }, "bucketPolicy": { "allowsPublicReadAccess": boolean, "allowsPublicWriteAccess": boolean } } } }, "region": "string", "replicationDetails": { "replicated": boolean, "replicatedExternally": boolean, "replicationAccounts": [ "string" ] }, "sensitivityScore": integer, "serverSideEncryption": { "kmsMasterKeyId": "string", "type": enum }, "sharedAccess": enum, "sizeInBytes": integer, "sizeInBytesCompressed": integer, "tags": [ { "key": "string", "value": "string" } ], "unclassifiableObjectCount": { "fileType": integer, "storageClass": integer, "total": integer }, "unclassifiableObjectSizeInBytes": { "fileType": integer, "storageClass": integer, "total": integer }, "versioning": boolean } ], "nextToken": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }
{ "message": "string" }

Properties

AccessControlList

Provides information about the permissions settings of the bucket-level access control list (ACL) for an S3 bucket.

PropertyTypeRequiredDescription
allowsPublicReadAccess

boolean

False

Specifies whether the ACL grants the general public with read access permissions for the bucket.

allowsPublicWriteAccess

boolean

False

Specifies whether the ACL grants the general public with write access permissions for the bucket.

AccessDeniedException

Provides information about an error that occurred due to insufficient access to a specified resource.

PropertyTypeRequiredDescription
message

string

False

The explanation of the error that occurred.

AccountLevelPermissions

Provides information about the account-level permissions settings that apply to an S3 bucket.

PropertyTypeRequiredDescription
blockPublicAccess

BlockPublicAccess

False

The block public access settings for the AWS account that owns the bucket.

BlockPublicAccess

Provides information about the block public access settings for an S3 bucket. These settings can apply to a bucket at the account or bucket level. For detailed information about each setting, see Blocking public access to your Amazon S3 storage in the Amazon Simple Storage Service User Guide.

PropertyTypeRequiredDescription
blockPublicAcls

boolean

False

Specifies whether Amazon S3 blocks public access control lists (ACLs) for the bucket and objects in the bucket.

blockPublicPolicy

boolean

False

Specifies whether Amazon S3 blocks public bucket policies for the bucket.

ignorePublicAcls

boolean

False

Specifies whether Amazon S3 ignores public ACLs for the bucket and objects in the bucket.

restrictPublicBuckets

boolean

False

Specifies whether Amazon S3 restricts public bucket policies for the bucket.

BucketCriteria

Specifies, as a map, one or more property-based conditions that filter the results of a query for information about S3 buckets.

PropertyTypeRequiredDescription

*

object

False

BucketCriteriaAdditionalProperties

Specifies the operator to use in a property-based condition that filters the results of a query for information about S3 buckets.

PropertyTypeRequiredDescription
eq

Array of type string

False

The value for the property matches (equals) the specified value. If you specify multiple values, Amazon Macie uses OR logic to join the values.

gt

integer

Format: int64

False

The value for the property is greater than the specified value.

gte

integer

Format: int64

False

The value for the property is greater than or equal to the specified value.

lt

integer

Format: int64

False

The value for the property is less than the specified value.

lte

integer

Format: int64

False

The value for the property is less than or equal to the specified value.

neq

Array of type string

False

The value for the property doesn't match (doesn't equal) the specified value. If you specify multiple values, Amazon Macie uses OR logic to join the values.

prefix

string

False

The name of the bucket begins with the specified value.

BucketLevelPermissions

Provides information about the bucket-level permissions settings for an S3 bucket.

PropertyTypeRequiredDescription
accessControlList

AccessControlList

False

The permissions settings of the access control list (ACL) for the bucket. This value is null if an ACL hasn't been defined for the bucket.

blockPublicAccess

BlockPublicAccess

False

The block public access settings for the bucket.

bucketPolicy

BucketPolicy

False

The permissions settings of the bucket policy for the bucket. This value is null if a bucket policy hasn't been defined for the bucket.

BucketMetadata

Provides statistical data and other information about an S3 bucket that Amazon Macie monitors and analyzes for your account. If an error occurs when Macie attempts to retrieve and process metadata from Amazon S3 for the bucket and the bucket's objects, the value for the versioning property is false and the value for most other properties is null. Key exceptions are accountId, bucketArn, bucketCreatedAt, bucketName, lastUpdated, and region. To identify the cause of the error, refer to the errorCode and errorMessage values.

PropertyTypeRequiredDescription
accountId

string

False

The unique identifier for the AWS account that owns the bucket.

allowsUnencryptedObjectUploads

string

Values: TRUE | FALSE | UNKNOWN

False

Specifies whether the bucket policy for the bucket requires server-side encryption of objects when objects are uploaded to the bucket. Possible values are:

  • FALSE - The bucket policy requires server-side encryption of new objects. PutObject requests must include a valid server-side encryption header.

  • TRUE - The bucket doesn't have a bucket policy or it has a bucket policy that doesn't require server-side encryption of new objects. If a bucket policy exists, it doesn't require PutObject requests to include a valid server-side encryption header.

  • UNKNOWN - Amazon Macie can't determine whether the bucket policy requires server-side encryption of new objects.

Valid server-side encryption headers are: x-amz-server-side-encryption with a value of AES256 or aws:kms, and x-amz-server-side-encryption-customer-algorithm with a value of AES256.

bucketArn

string

False

The Amazon Resource Name (ARN) of the bucket.

bucketCreatedAt

string

Format: date-time

False

The date and time, in UTC and extended ISO 8601 format, when the bucket was created, or changes such as edits to the bucket's policy were most recently made to the bucket.

bucketName

string

False

The name of the bucket.

classifiableObjectCount

integer

Format: int64

False

The total number of objects that Amazon Macie can analyze in the bucket. These objects use a supported storage class and have a file name extension for a supported file or storage format.

classifiableSizeInBytes

integer

Format: int64

False

The total storage size, in bytes, of the objects that Amazon Macie can analyze in the bucket. These objects use a supported storage class and have a file name extension for a supported file or storage format.

If versioning is enabled for the bucket, Macie calculates this value based on the size of the latest version of each applicable object in the bucket. This value doesn't reflect the storage size of all versions of each applicable object in the bucket.

errorCode

BucketMetadataErrorCode

False

Specifies the error code for an error that prevented Amazon Macie from retrieving and processing information about the bucket and the bucket's objects. If this value is ACCESS_DENIED, Macie doesn't have permission to retrieve the information. For example, the bucket has a restrictive bucket policy and Amazon S3 denied the request. If this value is null, Macie was able to retrieve and process the information.

errorMessage

string

False

A brief description of the error (errorCode) that prevented Amazon Macie from retrieving and processing information about the bucket and the bucket's objects. This value is null if Macie was able to retrieve and process the information.

jobDetails

JobDetails

False

Specifies whether any one-time or recurring classification jobs are configured to analyze data in the bucket, and, if so, the details of the job that ran most recently.

lastAutomatedDiscoveryTime

string

Format: date-time

False

The date and time, in UTC and extended ISO 8601 format, when Amazon Macie most recently analyzed data in the bucket while performing automated sensitive data discovery for your account. This value is null if automated sensitive data discovery is currently disabled for your account.

lastUpdated

string

Format: date-time

False

The date and time, in UTC and extended ISO 8601 format, when Amazon Macie most recently retrieved both bucket and object metadata from Amazon S3 for the bucket.

objectCount

integer

Format: int64

False

The total number of objects in the bucket.

objectCountByEncryptionType

ObjectCountByEncryptionType

False

The total number of objects that are in the bucket, grouped by server-side encryption type. This includes a grouping that reports the total number of objects that aren't encrypted or use client-side encryption.

publicAccess

BucketPublicAccess

False

Specifies whether the bucket is publicly accessible due to the combination of permissions settings that apply to the bucket, and provides information about those settings.

region

string

False

The AWS Region that hosts the bucket.

replicationDetails

ReplicationDetails

False

Specifies whether the bucket is configured to replicate one or more objects to buckets for other AWS accounts and, if so, which accounts.

sensitivityScore

integer

Format: int32

False

The sensitivity score for the bucket, ranging from -1 (classification error) to 100 (sensitive). This value is null if automated sensitive data discovery is currently disabled for your account.

serverSideEncryption

BucketServerSideEncryption

False

Specifies whether the bucket encrypts new objects by default and, if so, the type of server-side encryption that's used.

sharedAccess

string

Values: EXTERNAL | INTERNAL | NOT_SHARED | UNKNOWN

False

Specifies whether the bucket is shared with another AWS account. Possible values are:

  • EXTERNAL - The bucket is shared with an AWS account that isn't part of the same Amazon Macie organization.

  • INTERNAL - The bucket is shared with an AWS account that's part of the same Amazon Macie organization.

  • NOT_SHARED - The bucket isn't shared with other AWS accounts.

  • UNKNOWN - Amazon Macie wasn't able to evaluate the shared access settings for the bucket.

sizeInBytes

integer

Format: int64

False

The total storage size, in bytes, of the bucket.

If versioning is enabled for the bucket, Amazon Macie calculates this value based on the size of the latest version of each object in the bucket. This value doesn't reflect the storage size of all versions of each object in the bucket.

sizeInBytesCompressed

integer

Format: int64

False

The total storage size, in bytes, of the objects that are compressed (.gz, .gzip, .zip) files in the bucket.

If versioning is enabled for the bucket, Amazon Macie calculates this value based on the size of the latest version of each applicable object in the bucket. This value doesn't reflect the storage size of all versions of each applicable object in the bucket.

tags

Array of type KeyValuePair

False

An array that specifies the tags (keys and values) that are associated with the bucket.

unclassifiableObjectCount

ObjectLevelStatistics

False

The total number of objects that Amazon Macie can't analyze in the bucket. These objects don't use a supported storage class or don't have a file name extension for a supported file or storage format.

unclassifiableObjectSizeInBytes

ObjectLevelStatistics

False

The total storage size, in bytes, of the objects that Amazon Macie can't analyze in the bucket. These objects don't use a supported storage class or don't have a file name extension for a supported file or storage format.

versioning

boolean

False

Specifies whether versioning is enabled for the bucket.

BucketMetadataErrorCode

The error code for an error that prevented Amazon Macie from retrieving and processing metadata from Amazon S3 for an S3 bucket and the bucket's objects.

  • ACCESS_DENIED

BucketPermissionConfiguration

Provides information about the account-level and bucket-level permissions settings for an S3 bucket.

PropertyTypeRequiredDescription
accountLevelPermissions

AccountLevelPermissions

False

The account-level permissions settings that apply to the bucket.

bucketLevelPermissions

BucketLevelPermissions

False

The bucket-level permissions settings for the bucket.

BucketPolicy

Provides information about the permissions settings of the bucket policy for an S3 bucket.

PropertyTypeRequiredDescription
allowsPublicReadAccess

boolean

False

Specifies whether the bucket policy allows the general public to have read access to the bucket.

allowsPublicWriteAccess

boolean

False

Specifies whether the bucket policy allows the general public to have write access to the bucket.

BucketPublicAccess

Provides information about the permissions settings that determine whether an S3 bucket is publicly accessible.

PropertyTypeRequiredDescription
effectivePermission

string

Values: PUBLIC | NOT_PUBLIC | UNKNOWN

False

Specifies whether the bucket is publicly accessible due to the combination of permissions settings that apply to the bucket. Possible values are:

  • NOT_PUBLIC - The bucket isn't publicly accessible.

  • PUBLIC - The bucket is publicly accessible.

  • UNKNOWN - Amazon Macie can't determine whether the bucket is publicly accessible.

permissionConfiguration

BucketPermissionConfiguration

False

The account-level and bucket-level permissions settings for the bucket.

BucketServerSideEncryption

Provides information about the default server-side encryption settings for an S3 bucket. For detailed information about these settings, see Setting default server-side encryption behavior for Amazon S3 buckets in the Amazon Simple Storage Service User Guide.

PropertyTypeRequiredDescription
kmsMasterKeyId

string

False

The Amazon Resource Name (ARN) or unique identifier (key ID) for the AWS KMS key that's used by default to encrypt objects that are added to the bucket. This value is null if the bucket uses an Amazon S3 managed key to encrypt new objects or the bucket doesn't encrypt new objects by default.

type

string

Values: NONE | AES256 | aws:kms

False

The type of server-side encryption that's used by default when storing new objects in the bucket. Possible values are:

  • AES256 - New objects are encrypted with an Amazon S3 managed key. They use SSE-S3 encryption.

  • aws:kms - New objects are encrypted with an AWS KMS key (kmsMasterKeyId), either an AWS managed key or a customer managed key. They use SSE-KMS encryption.

  • NONE - New objects aren't encrypted by default. Default encryption is disabled for the bucket.

BucketSortCriteria

Specifies criteria for sorting the results of a query for information about S3 buckets.

PropertyTypeRequiredDescription
attributeName

string

False

The name of the bucket property to sort the results by. This value can be one of the following properties that Amazon Macie defines as bucket metadata: accountId, bucketName, classifiableObjectCount, classifiableSizeInBytes, objectCount, sensitivityScore, or sizeInBytes.

orderBy

string

Values: ASC | DESC

False

The sort order to apply to the results, based on the value specified by the attributeName property. Valid values are: ASC, sort the results in ascending order; and, DESC, sort the results in descending order.

ConflictException

Provides information about an error that occurred due to a versioning conflict for a specified resource.

PropertyTypeRequiredDescription
message

string

False

The explanation of the error that occurred.

DescribeBucketsRequest

Specifies criteria for filtering, sorting, and paginating the results of a query for statistical data and other information about S3 buckets.

PropertyTypeRequiredDescription
criteria

BucketCriteria

False

The criteria to use to filter the query results.

maxResults

integer

Format: int32

False

The maximum number of items to include in each page of the response. The default value is 50.

nextToken

string

False

The nextToken string that specifies which page of results to return in a paginated response.

sortCriteria

BucketSortCriteria

False

The criteria to use to sort the query results.

DescribeBucketsResponse

Provides the results of a query that retrieved statistical data and other information about one or more S3 buckets that Amazon Macie monitors and analyzes for your account.

PropertyTypeRequiredDescription
buckets

Array of type BucketMetadata

False

An array of objects, one for each bucket that matches the filter criteria specified in the request.

nextToken

string

False

The string to use in a subsequent request to get the next page of results in a paginated response. This value is null if there are no additional pages.

InternalServerException

Provides information about an error that occurred due to an unknown internal server error, exception, or failure.

PropertyTypeRequiredDescription
message

string

False

The explanation of the error that occurred.

JobDetails

Specifies whether any one-time or recurring classification jobs are configured to analyze data in an S3 bucket, and, if so, the details of the job that ran most recently.

PropertyTypeRequiredDescription
isDefinedInJob

string

Values: TRUE | FALSE | UNKNOWN

False

Specifies whether any one-time or recurring jobs are configured to analyze data in the bucket. Possible values are:

  • TRUE - The bucket is explicitly included in the bucket definition (S3BucketDefinitionForJob) for one or more jobs and at least one of those jobs has a status other than CANCELLED. Or the bucket matched the bucket criteria (S3BucketCriteriaForJob) for at least one job that previously ran.

  • FALSE - The bucket isn't explicitly included in the bucket definition (S3BucketDefinitionForJob) for any jobs, all the jobs that explicitly include the bucket in their bucket definitions have a status of CANCELLED, or the bucket didn't match the bucket criteria (S3BucketCriteriaForJob) for any jobs that previously ran.

  • UNKNOWN - An exception occurred when Amazon Macie attempted to retrieve job data for the bucket.

isMonitoredByJob

string

Values: TRUE | FALSE | UNKNOWN

False

Specifies whether any recurring jobs are configured to analyze data in the bucket. Possible values are:

  • TRUE - The bucket is explicitly included in the bucket definition (S3BucketDefinitionForJob) for one or more recurring jobs or the bucket matches the bucket criteria (S3BucketCriteriaForJob) for one or more recurring jobs. At least one of those jobs has a status other than CANCELLED.

  • FALSE - The bucket isn't explicitly included in the bucket definition (S3BucketDefinitionForJob) for any recurring jobs, the bucket doesn't match the bucket criteria (S3BucketCriteriaForJob) for any recurring jobs, or all the recurring jobs that are configured to analyze data in the bucket have a status of CANCELLED.

  • UNKNOWN - An exception occurred when Amazon Macie attempted to retrieve job data for the bucket.

lastJobId

string

False

The unique identifier for the job that ran most recently and is configured to analyze data in the bucket, either the latest run of a recurring job or the only run of a one-time job.

This value is typically null if the value for the isDefinedInJob property is FALSE or UNKNOWN.

lastJobRunTime

string

Format: date-time

False

The date and time, in UTC and extended ISO 8601 format, when the job (lastJobId) started. If the job is a recurring job, this value indicates when the most recent run started.

This value is typically null if the value for the isDefinedInJob property is FALSE or UNKNOWN.

KeyValuePair

Provides information about the tags that are associated with an S3 bucket or object. Each tag consists of a required tag key and an associated tag value.

PropertyTypeRequiredDescription
key

string

False

One part of a key-value pair that comprises a tag. A tag key is a general label that acts as a category for more specific tag values.

value

string

False

One part of a key-value pair that comprises a tag. A tag value acts as a descriptor for a tag key. A tag value can be an empty string.

ObjectCountByEncryptionType

Provides information about the number of objects that are in an S3 bucket and use certain types of server-side encryption, use client-side encryption, or aren't encrypted.

PropertyTypeRequiredDescription
customerManaged

integer

Format: int64

False

The total number of objects that are encrypted with a customer-provided key. The objects use customer-provided server-side encryption (SSE-C).

kmsManaged

integer

Format: int64

False

The total number of objects that are encrypted with an AWS KMS key, either an AWS managed key or a customer managed key. The objects use AWS KMS encryption (SSE-KMS).

s3Managed

integer

Format: int64

False

The total number of objects that are encrypted with an Amazon S3 managed key. The objects use Amazon S3 managed encryption (SSE-S3).

unencrypted

integer

Format: int64

False

The total number of objects that aren't encrypted or use client-side encryption.

unknown

integer

Format: int64

False

The total number of objects that Amazon Macie doesn't have current encryption metadata for. Macie can't provide current data about the encryption settings for these objects.

ObjectLevelStatistics

Provides information about the total storage size (in bytes) or number of objects that Amazon Macie can't analyze in one or more S3 buckets. In a BucketMetadata or MatchingBucket object, this data is for a specific bucket. In a GetBucketStatisticsResponse object, this data is aggregated for all the buckets in the query results. If versioning is enabled for a bucket, storage size values are based on the size of the latest version of each applicable object in the bucket.

PropertyTypeRequiredDescription
fileType

integer

Format: int64

False

The total storage size (in bytes) or number of objects that Amazon Macie can't analyze because the objects don't have a file name extension for a supported file or storage format.

storageClass

integer

Format: int64

False

The total storage size (in bytes) or number of objects that Amazon Macie can't analyze because the objects use an unsupported storage class.

total

integer

Format: int64

False

The total storage size (in bytes) or number of objects that Amazon Macie can't analyze because the objects use an unsupported storage class or don't have a file name extension for a supported file or storage format.

ReplicationDetails

Provides information about settings that define whether one or more objects in an S3 bucket are replicated to S3 buckets for other AWS accounts and, if so, which accounts.

PropertyTypeRequiredDescription
replicated

boolean

False

Specifies whether the bucket is configured to replicate one or more objects to any destination.

replicatedExternally

boolean

False

Specifies whether the bucket is configured to replicate one or more objects to an AWS account that isn't part of the same Amazon Macie organization.

replicationAccounts

Array of type string

False

An array of AWS account IDs, one for each AWS account that the bucket is configured to replicate one or more objects to.

ResourceNotFoundException

Provides information about an error that occurred because a specified resource wasn't found.

PropertyTypeRequiredDescription
message

string

False

The explanation of the error that occurred.

ServiceQuotaExceededException

Provides information about an error that occurred due to one or more service quotas for an account.

PropertyTypeRequiredDescription
message

string

False

The explanation of the error that occurred.

ThrottlingException

Provides information about an error that occurred because too many requests were sent during a certain amount of time.

PropertyTypeRequiredDescription
message

string

False

The explanation of the error that occurred.

ValidationException

Provides information about an error that occurred due to a syntax error in a request.

PropertyTypeRequiredDescription
message

string

False

The explanation of the error that occurred.

See also

For more information about using this API in one of the language-specific AWS SDKs and references, see the following:

DescribeBuckets