Menu
Amazon DevPay
Developer Guide (API Version 2007-12-01)

Using Amazon S3 Logs with DevPay

This section covers how Amazon DevPay products can produce reports for customers based on the Amazon Simple Storage Service server access logs.

Before You Begin

We assume you understand how Amazon S3 server access logs work and how Amazon S3 access control lists (ACLs) work. If you aren't already familiar with Amazon S3 logs or ACLs, we recommend you read about them in the Amazon Simple Storage Service Developer Guide first and then come back to this section.

Overview

You might want your Amazon S3 DevPay product to give your customers reports related to their use of Amazon S3. Your product can use the information in the Amazon S3 server access logs to produce those reports. Each record in a bucket's server access log contains details about an individual Amazon S3 request on that bucket, such as the request type, the resource the request worked with, and the date and time the request was processed.

When you enable logging for a bucket (the source bucket), you specify where you want the logs to be delivered (the target bucket), and Amazon S3 aggregates available log records into log files and delivers them to the target bucket (typically every few hours). The target bucket can be the same as the source bucket or a different bucket. However, the same Amazon S3 user must own both the source and target buckets.

Note

Because Amazon S3 requires the same user to own both the source and target buckets, you can't create a single bucket in Amazon S3 to hold all the log files for all your DevPay customers.

Who Pays for Log File Storage and Download?

Log files are essentially just objects in a customer's bucket, so the customer pays the same price you charge for storing objects and downloading objects with your DevPay product. Likewise, you pay AWS for the corresponding Amazon S3 costs that you would for storing objects for the customer. AWS does not charge to enable logging on a bucket, and there is no data transfer charge for log file delivery to the customer's target bucket.

Because log files are just like other objects in the bucket, any log file storage or access that your DevPay product performs is included in the usage statistics and dollar amount the customer sees on the Application Billing page. You should design your product to access the log files efficiently and delete the files as soon as they are no longer needed so that you minimize costs to the customer.

Access to Log Files

Because log files are essentially just like the other objects in the customer's bucket, your product accesses them on behalf of the customer the same way it accesses any other object in the bucket (using the customer credentials your product obtained from a call to the License Service). Objects in the customer's bucket (including log files) can be accessed only through your DevPay product (for more information, see Customer Access Stored Data).

Your DevPay product can use the Amazon S3 access control functionality to share any object in the customer's bucket with other users of your DevPay product. Likewise, the product can give other users of your product access to the log files. The important thing to remember is that all requests to access content in either the source or target bucket must come from your DevPay product. For example, your customer or another Amazon S3 user who has permission to access data in either bucket can't get access by using the Amazon S3 Firefox plugin. They must use your product.

Setting Up and Using Logs

This section lists the tasks your product must perform to enable and use logs, and to share logs with other Amazon S3 users. It is assumed that the target bucket already exists. For specific instructions on how your product uses the Amazon S3 API to execute these tasks, see the Amazon Simple Storage Service Developer Guide.

Basic Process for Enabling Logging and Using Logs

1

Your product sets up the target bucket.

This consists of setting the bucket's ACL to grant WRITE and READ_ACP permissions to the Amazon S3 log delivery group. This lets Amazon S3 deliver the logs to the target bucket.

2

Your product enables logging on the source bucket.

This consists of modifying the source bucket's BucketLoggingStatus resource.

3

Amazon S3 produces log files and puts them in the target bucket.

4

Your product gets the log files, produces the report for the customer, and deletes the log files.


Sharing Log Files

The following table describes what your product must do to share logs with another user of your product. For specific instructions on how your product uses the Amazon S3 API to execute these tasks, see the Amazon Simple Storage Service Developer Guide.

You might want another user of your product to have the ability to read and delete the log files in the target bucket. That user also needs to be able to list the objects in the bucket to know which log files are available to read and delete.

To give the other user these abilities, your DevPay product grants the following Amazon S3 permissions:

  • READ on the target bucket (which enables listing all the objects in the bucket)

  • WRITE on the target bucket (which enables deleting the log files and any other objects in the bucket)

  • READ on the BucketLoggingResource for the source bucket (which enables the user to read only the logs—any other objects in the target bucket can't be read)

The following XML snippet shows an example of the Grant elements your product adds to the target bucket's ACL to allow the log delivery group to write the log files.

To help you understand how sharing works for log files, this example also shows grants related to sharing log files with another user of your product with the AWS account login user@example.com. The two grants are the READ and WRITE permissions that let the user list the contents of the target bucket and delete objects from the bucket. The XML snippet that follows this one shows the additional grant required to let the user actually read the log files.

For complete information about using Amazon S3 access control policies, go to the Amazon Simple Storage Service Developer Guide.

Example XML for Setting Up the Target Bucket


<Grant>
   <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
      <URI>http://acs.amazonaws.com/groups/s3/LogDelivery</URI>
   </Grantee>
   <Permission>WRITE</Permission>
</Grant>
<Grant>
   <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
      <URI>http://acs.amazonaws.com/groups/s3/LogDelivery</URI>
   </Grantee>
   <Permission>READ_ACP</Permission>
</Grant>

<!-- The following grants let the user list and delete the objects in the bucket  -->
<Grant>
   <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
           xsi:type="AmazonCustomerByEmail">
      <EmailAddress>user@example.com</EmailAddress>
   </Grantee>
   <Permission>READ</Permission>
</Grant>
<Grant>
   <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
           xsi:type="AmazonCustomerByEmail">
      <EmailAddress>user@example.com</EmailAddress>
   </Grantee>
   <Permission>WRITE</Permission>
</Grant>

The following XML snippet shows an example of the BucketLoggingStatus element your product provides to enable logging on the source bucket. This example:

  • Uses the TargetBucket element to set the target bucket to CustomerA_LogBucket

  • Uses the TargetPrefix element to add the literal string CustomerA-access_log- as a prefix to the name of all the log files

This example also continues with the preceding example of showing how to share log files with the user@example.com user. The following XML includes a READ grant that lets the user actually read the log files.

For complete information about using the BucketLoggingStatus element, go to the Amazon Simple Storage Service Developer Guide.

Example XML for Enabling Logging on the Source Bucket


<?xml version="1.0" encoding="UTF-8"?>
<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01">
   <LoggingEnabled>
      <TargetBucket>CustomerA_LogBucket</TargetBucket>
      <TargetPrefix>CustomerA-access_log-/</TargetPrefix>
      
      <!-- The following grant lets the user read only the log files in the target bucket -->
      <TargetGrants>
         <Grant>
            <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:type="AmazonCustomerByEmail">
               <EmailAddress>user@example.com</EmailAddress>
            </Grantee>
            <Permission>READ</Permission>
         </Grant>
      </TargetGrants>
   </LoggingEnabled>
</BucketLoggingStatus>