Class that makes it easy to define Hadoop Streaming steps.
CopyC#
See also: Hadoop Streaming
CopyC#AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials); HadoopJarStepConfig config = new StreamingStep() .WithInputs("s3://elasticmapreduce/samples/wordcount/input") .WithOutput("s3://my-bucket/output/") .WithMapper("s3://elasticmapreduce/samples/wordcount/wordSplitter.py") .WithReducer("aggregate") .ToHadoopJarStepConfig(); StepConfig wordCount = new StepConfig() .WithName("Word Count") .WithActionOnFailure("TERMINATE_JOB_FLOW") .WithHadoopJarStep(config); RunJobFlowRequest request = new RunJobFlowRequest() .WithName("Word Count") .WithSteps(wordCount) .WithLogUri("s3://log-bucket/") .WithInstances(new JobFlowInstancesConfig() .WithEc2KeyName("keypairt") .WithHadoopVersion("0.20") .WithInstanceCount(5) .WithKeepJobFlowAliveWhenNoSteps(true) .WithMasterInstanceType("m1.small") .WithSlaveInstanceType("m1.small")); RunJobFlowResult result = emr.RunJobFlow(request).RunJobFlowResult;
Declaration Syntax| C# |
public class StreamingStep
Members| All Members | Constructors | Methods | Properties | ||
| Icon | Member | Description |
|---|---|---|
 | StreamingStep()()()() |
Creates a new default StreamingStep.
|
| Equals(Object) | (Inherited from Object.) |
| GetHashCode()()()() | Serves as a hash function for a particular type. (Inherited from Object.) |
| GetType()()()() | Gets the type of the current instance. (Inherited from Object.) |
 | HadoopConfig |
Gets and sets the hadoopConfig.
|
| Inputs |
Gets and sets list of step input paths.
|
| Mapper |
Gets and sets the mapper.
|
| Output |
Gets and sets the output path.
|
| Reducer |
Gets and sets the reducer.
|
| ToHadoopJarStepConfig()()()() |
Creates the final HadoopJarStepConfig once you are done configuring the step. You can use
this as you would any other HadoopJarStepConfig.
|
| ToString()()()() | Returns a string that represents the current object. (Inherited from Object.) |
| WithHadoopConfig(String, String) |
Add a Hadoop config override (-D value)
|
| WithInputs(array<String>[]()[][]) |
Add more input paths to this step.
|
| WithMapper(String) |
Set the mapper.
|
| WithOutput(String) |
Set the output path for this step.
|
| WithReducer(String) |
Set the reducer.
|
Inheritance Hierarchy| Object | |
 | StreamingStep |
