AWS SDK Version 3 for .NET
API Reference

AWS services or capabilities described in AWS Documentation may vary by region/location. Click Getting Started with Amazon AWS to see specific differences applicable to the China (Beijing) Region.

Class that makes it easy to define Hadoop Streaming steps.

See also: Hadoop Streaming

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
IAmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials);
            
HadoopJarStepConfig config = new StreamingStep {
    Inputs = new List { "s3://elasticmapreduce/samples/wordcount/input" },
    Output = "s3://my-bucket/output/",
    Mapper = "s3://elasticmapreduce/samples/wordcount/wordSplitter.py",
    Reducer = "aggregate"
}.ToHadoopJarStepConfig();
            
StepConfig wordCount = new StepConfig {
    Name = "Word Count",
    ActionOnFailure = "TERMINATE_JOB_FLOW",
    HadoopJarStep = config
};
            
RunJobFlowRequest request = new RunJobFlowRequest {
    Name = "Word Count",
    Steps = new List { wordCount },
    LogUri = "s3://log-bucket/",
    Instances = new JobFlowInstancesConfig {
        Ec2KeyName = "keypair",
        HadoopVersion = "0.20",
        InstanceCount = 5,
        KeepJobFlowAliveWhenNoSteps = true,
        MasterInstanceType = "m1.small",
        SlaveInstanceType = "m1.small"
   }
};
            
RunJobFlowResponse response = emr.RunJobFlow(request);
             

Inheritance Hierarchy

System.Object
  Amazon.ElasticMapReduce.Model.StreamingStep

Namespace: Amazon.ElasticMapReduce.Model
Assembly: AWSSDK.ElasticMapReduce.dll
Version: 3.x.y.z

Syntax

C#
public class StreamingStep

The StreamingStep type exposes the following members

Constructors

NameDescription
Public Method StreamingStep()

Creates a new default StreamingStep.

Properties

NameTypeDescription
Public Property HadoopConfig System.Collections.Generic.Dictionary<System.String, System.String>

Gets and sets the hadoopConfig.

Public Property Inputs System.Collections.Generic.List<System.String>

Gets and sets list of step input paths.

Public Property Mapper System.String

Gets and sets the mapper.

Public Property Output System.String

Gets and sets the output path.

Public Property Reducer System.String

Gets and sets the reducer.

Methods

Note:

Asynchronous operations (methods ending with Async) in the table below are for .NET 4.5 or higher. For .NET 3.5 the SDK follows the standard naming convention of BeginMethodName and EndMethodName to indicate asynchronous operations - these method pairs are not shown in the table below.

NameDescription
Public Method AddHadoopConfig(string, string)

Add a Hadoop config override (-D value)

Public Method AddInputs(String[])

Add more input paths to this step.

Public Method ToHadoopJarStepConfig()

Creates the final HadoopJarStepConfig once you are done configuring the step. You can use this as you would any other HadoopJarStepConfig.

Version Information

.NET Standard:
Supported in: 1.3

.NET Framework:
Supported in: 4.5, 4.0, 3.5

Portable Class Library:
Supported in: Windows Store Apps
Supported in: Windows Phone 8.1
Supported in: Xamarin Android
Supported in: Xamarin iOS (Unified)
Supported in: Xamarin.Forms