Menu
Amazon EMR
Management Guide

Create an AWSCredentialsProvider for EMRFS

You can create a custom credentials provider which implements both the AWSCredentialsProvider and the Hadoop Configurable classes for use with EMRFS when it makes calls to Amazon S3. Creating a custom credentials provider in this way is useful when an Amazon EMR user may need to provide different credentials depending on which Amazon S3 bucket is being accessed.

You must specify the full class name of the provider by setting fs.s3.customAWSCredentialsProvider in the emrfs-site configuration classification. You set this property at cluster creation time using the AWS CLI. For example, the following code sets fs.s3.customAWSCredentialsProvider to MyAWSCredentialsProvider.

Use the following configuration file and save it locally or in Amazon S3:

Copy
[ { "Classification": "emrfs-site", "Properties": { "fs.s3.customAWSCredentialsProvider":"MyAWSCredentialsProvider" } } ]

Use the configuration you created with the following syntax:

Copy
aws emr create-cluster --release-label emr-4.1.0emr-4.2.0 --applications Name=Hive \ --instance-type m3.xlarge --instance-count 2 --configurations file://./myConfig.json

An example implementation follows:

Copy
public class MyAWSCredentialsProvider implements AWSCredentialsProvider, Configurable { private Configuration conf; private String accessKey; private String secretKey; private void init() { accessKey = conf.get("my.accessKey"); secretKey = conf.get("my.secretKey"); } @Override public AWSCredentials getCredentials() { return new BasicAWSCredentials(accessKey, secretKey); } @Override public void refresh() { } @Override public void setConf(Configuration configuration) { this.conf = configuration; init(); } @Override public Configuration getConf() { return this.conf; } }

An alternative implementation allows you to provide the bucket URI when creating an object. That follows this signature:

Copy
class MyCustomCredentialProvider implements AWSCredentialsProvider , Configurable { public MyCustomCredentialProvider (Uri uri , Configuration configuration) { } }