Class that makes it easy to define Hadoop Streaming steps.
See also: Hadoop Streaming
CopyC#
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials); HadoopJarStepConfig config = new StreamingStep { Inputs = new List<string> { "s3://elasticmapreduce/samples/wordcount/input" }, Output = "s3://my-bucket/output/", Mapper = "s3://elasticmapreduce/samples/wordcount/wordSplitter.py", Reducer = "aggregate" }.ToHadoopJarStepConfig(); StepConfig wordCount = new StepConfig { Name = "Word Count", ActionOnFailure = "TERMINATE_JOB_FLOW", HadoopJarStep = config }; RunJobFlowRequest request = new RunJobFlowRequest { Name = "Word Count", Steps = new List<StepConfig> { wordCount }, LogUri = "s3://log-bucket/", Instances = new JobFlowInstancesConfig { Ec2KeyName = "keypair", HadoopVersion = "0.20", InstanceCount = 5, KeepJobFlowAliveWhenNoSteps = true, MasterInstanceType = "m1.small", SlaveInstanceType = "m1.small" } }; RunJobFlowResult result = emr.RunJobFlow(request).RunJobFlowResult;
Declaration Syntax
C# |
public class StreamingStep
Members
All Members | Constructors | Methods | Properties | ||
Icon | Member | Description |
---|---|---|
StreamingStep()()()() |
Creates a new default StreamingStep.
| |
Equals(Object) | (Inherited from Object.) | |
GetHashCode()()()() | Serves as a hash function for a particular type. (Inherited from Object.) | |
GetType()()()() | Gets the type of the current instance. (Inherited from Object.) | |
HadoopConfig |
Gets and sets the hadoopConfig.
| |
Inputs |
Gets and sets list of step input paths.
| |
Mapper |
Gets and sets the mapper.
| |
Output |
Gets and sets the output path.
| |
Reducer |
Gets and sets the reducer.
| |
ToHadoopJarStepConfig()()()() |
Creates the final HadoopJarStepConfig once you are done configuring the step. You can use
this as you would any other HadoopJarStepConfig.
| |
ToString()()()() | Returns a string that represents the current object. (Inherited from Object.) | |
WithHadoopConfig(String, String) | Obsolete.
Add a Hadoop config override (-D value)
| |
WithInputs(array<String>[]()[][]) | Obsolete.
Add more input paths to this step.
| |
WithMapper(String) | Obsolete.
Set the mapper.
| |
WithOutput(String) | Obsolete.
Set the output path for this step.
| |
WithReducer(String) | Obsolete.
Set the reducer.
|
Inheritance Hierarchy
Object | |
StreamingStep |