AWS SDK Version 2 for .NET
API Reference

AWS services or capabilities described in AWS Documentation may vary by region/location. Click Getting Started with Amazon AWS to see specific differences applicable to the China (Beijing) Region.

.NET Framework 4.5
 
Class that makes it easy to define Hadoop Streaming steps.

See also: Hadoop Streaming

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
IAmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials);
            
HadoopJarStepConfig config = new StreamingStep {
    Inputs = new List<string> { "s3://elasticmapreduce/samples/wordcount/input" },
    Output = "s3://my-bucket/output/",
    Mapper = "s3://elasticmapreduce/samples/wordcount/wordSplitter.py",
    Reducer = "aggregate"
}.ToHadoopJarStepConfig();
            
StepConfig wordCount = new StepConfig {
    Name = "Word Count",
    ActionOnFailure = "TERMINATE_JOB_FLOW",
    HadoopJarStep = config
};
            
RunJobFlowRequest request = new RunJobFlowRequest {
    Name = "Word Count",
    Steps = new List<StepConfig> { wordCount },
    LogUri = "s3://log-bucket/",
    Instances = new JobFlowInstancesConfig {
        Ec2KeyName = "keypair",
        HadoopVersion = "0.20",
        InstanceCount = 5,
        KeepJobFlowAliveWhenNoSteps = true,
        MasterInstanceType = "m1.small",
        SlaveInstanceType = "m1.small"
   }
};
            
RunJobFlowResponse response = emr.RunJobFlow(request);
             

Inheritance Hierarchy

System.Object
  Amazon.ElasticMapReduce.Model.StreamingStep

Namespace: Amazon.ElasticMapReduce.Model
Assembly: AWSSDK.dll
Version: (assembly version)

Syntax

C#
public class StreamingStep : Object

The StreamingStep type exposes the following members

Constructors

NameDescription
Public Method StreamingStep() Creates a new default StreamingStep.

Properties

NameTypeDescription
Public Property HadoopConfig System.Collections.Generic.Dictionary<System.String, System.String> Gets and sets the hadoopConfig.
Public Property Inputs System.Collections.Generic.List<System.String> Gets and sets list of step input paths.
Public Property Mapper System.String Gets and sets the mapper.
Public Property Output System.String Gets and sets the output path.
Public Property Reducer System.String Gets and sets the reducer.

Methods

NameDescription
Public Method AddHadoopConfig(string, string) Add a Hadoop config override (-D value)
Public Method AddInputs(String[]) Add more input paths to this step.
Public Method ToHadoopJarStepConfig() Creates the final HadoopJarStepConfig once you are done configuring the step. You can use this as you would any other HadoopJarStepConfig.

Version Information

.NET Framework:
Supported in: 4.5, 4.0, 3.5

.NET for Windows Store apps:
Supported in: Windows 8