AWS services or capabilities described in AWS Documentation may vary by region/location. Click Getting Started with Amazon AWS to see specific differences applicable to the China (Beijing) Region.
Class that makes it easy to define Hadoop Streaming steps.
See also: Hadoop Streaming
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); IAmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials); HadoopJarStepConfig config = new StreamingStep { Inputs = new List<string> { "s3://elasticmapreduce/samples/wordcount/input" }, Output = "s3://my-bucket/output/", Mapper = "s3://elasticmapreduce/samples/wordcount/wordSplitter.py", Reducer = "aggregate" }.ToHadoopJarStepConfig(); StepConfig wordCount = new StepConfig { Name = "Word Count", ActionOnFailure = "TERMINATE_JOB_FLOW", HadoopJarStep = config }; RunJobFlowRequest request = new RunJobFlowRequest { Name = "Word Count", Steps = new List<StepConfig> { wordCount }, LogUri = "s3://log-bucket/", Instances = new JobFlowInstancesConfig { Ec2KeyName = "keypair", HadoopVersion = "0.20", InstanceCount = 5, KeepJobFlowAliveWhenNoSteps = true, MasterInstanceType = "m1.small", SlaveInstanceType = "m1.small" } }; RunJobFlowResponse response = emr.RunJobFlow(request);
Namespace: Amazon.ElasticMapReduce.Model
Assembly: AWSSDK.ElasticMapReduce.dll
Version: 3.x.y.z
public class StreamingStep
The StreamingStep type exposes the following members
Name | Description | |
---|---|---|
StreamingStep() |
Creates a new default StreamingStep. |
Name | Type | Description | |
---|---|---|---|
HadoopConfig | System.Collections.Generic.Dictionary<System.String, System.String> |
Gets and sets the hadoopConfig. |
|
Inputs | System.Collections.Generic.List<System.String> |
Gets and sets list of step input paths. |
|
Mapper | System.String |
Gets and sets the mapper. |
|
Output | System.String |
Gets and sets the output path. |
|
Reducer | System.String |
Gets and sets the reducer. |
Name | Description | |
---|---|---|
AddHadoopConfig(string, string) |
Add a Hadoop config override (-D value) |
|
AddInputs(String[]) |
Add more input paths to this step. |
|
ToHadoopJarStepConfig() |
Creates the final HadoopJarStepConfig once you are done configuring the step. You can use this as you would any other HadoopJarStepConfig. |
.NET Core App:
Supported in: 3.1
.NET Standard:
Supported in: 2.0
.NET Framework:
Supported in: 4.5, 4.0, 3.5