EmrCreateCluster

class aws_cdk.aws_stepfunctions_tasks.EmrCreateCluster(scope, id, *, instances, name, additional_info=None, applications=None, auto_scaling_role=None, bootstrap_actions=None, cluster_role=None, configurations=None, custom_ami_id=None, ebs_root_volume_size=None, kerberos_attributes=None, log_uri=None, release_label=None, scale_down_behavior=None, security_configuration=None, service_role=None, tags=None, visible_to_all_users=None, comment=None, heartbeat=None, input_path=None, integration_pattern=None, output_path=None, result_path=None, timeout=None)

Bases: aws_cdk.aws_stepfunctions.TaskStateBase

(experimental) A Step Functions Task to create an EMR Cluster.

The ClusterConfiguration is defined as Parameters in the state machine definition.

OUTPUT: the ClusterId.

Stability

experimental

Parameters
  • scope (Construct) –

  • id (str) –

  • instances (InstancesConfigProperty) – (experimental) A specification of the number and type of Amazon EC2 instances.

  • name (str) – (experimental) The Name of the Cluster.

  • additional_info (Optional[str]) – (experimental) A JSON string for selecting additional features. Default: - None

  • applications (Optional[List[ApplicationConfigProperty]]) – (experimental) A case-insensitive list of applications for Amazon EMR to install and configure when launching the cluster. Default: - EMR selected default

  • auto_scaling_role (Optional[IRole]) – (experimental) An IAM role for automatic scaling policies. Default: - A role will be created.

  • bootstrap_actions (Optional[List[BootstrapActionConfigProperty]]) – (experimental) A list of bootstrap actions to run before Hadoop starts on the cluster nodes. Default: - None

  • cluster_role (Optional[IRole]) – (experimental) Also called instance profile and EC2 role. An IAM role for an EMR cluster. The EC2 instances of the cluster assume this role. This attribute has been renamed from jobFlowRole to clusterRole to align with other ERM/StepFunction integration parameters. Default: - - A Role will be created

  • configurations (Optional[List[ConfigurationProperty]]) – (experimental) The list of configurations supplied for the EMR cluster you are creating. Default: - None

  • custom_ami_id (Optional[str]) – (experimental) The ID of a custom Amazon EBS-backed Linux AMI. Default: - None

  • ebs_root_volume_size (Optional[Size]) – (experimental) The size of the EBS root device volume of the Linux AMI that is used for each EC2 instance. Default: - EMR selected default

  • kerberos_attributes (Optional[KerberosAttributesProperty]) – (experimental) Attributes for Kerberos configuration when Kerberos authentication is enabled using a security configuration. Default: - None

  • log_uri (Optional[str]) – (experimental) The location in Amazon S3 to write the log files of the job flow. Default: - None

  • release_label (Optional[str]) – (experimental) The Amazon EMR release label, which determines the version of open-source application packages installed on the cluster. Default: - EMR selected default

  • scale_down_behavior (Optional[EmrClusterScaleDownBehavior]) – (experimental) Specifies the way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized. Default: - EMR selected default

  • security_configuration (Optional[str]) – (experimental) The name of a security configuration to apply to the cluster. Default: - None

  • service_role (Optional[IRole]) – (experimental) The IAM role that will be assumed by the Amazon EMR service to access AWS resources on your behalf. Default: - A role will be created that Amazon EMR service can assume.

  • tags (Optional[Mapping[str, str]]) – (experimental) A list of tags to associate with a cluster and propagate to Amazon EC2 instances. Default: - None

  • visible_to_all_users (Optional[bool]) – (experimental) A value of true indicates that all IAM users in the AWS account can perform cluster actions if they have the proper IAM policy permissions. Default: true

  • comment (Optional[str]) – An optional description for this state. Default: - No comment

  • heartbeat (Optional[Duration]) – Timeout for the heartbeat. Default: - None

  • input_path (Optional[str]) – JSONPath expression to select part of the state to be the input to this state. May also be the special value JsonPath.DISCARD, which will cause the effective input to be the empty object {}. Default: - The entire task input (JSON path ‘$’)

  • integration_pattern (Optional[IntegrationPattern]) – AWS Step Functions integrates with services directly in the Amazon States Language. You can control these AWS services using service integration patterns Default: IntegrationPattern.REQUEST_RESPONSE

  • output_path (Optional[str]) – JSONPath expression to select select a portion of the state output to pass to the next state. May also be the special value JsonPath.DISCARD, which will cause the effective output to be the empty object {}. Default: - The entire JSON node determined by the state input, the task result, and resultPath is passed to the next state (JSON path ‘$’)

  • result_path (Optional[str]) – JSONPath expression to indicate where to inject the state’s output. May also be the special value JsonPath.DISCARD, which will cause the state’s input to become its output. Default: - Replaces the entire input with the result (JSON path ‘$’)

  • timeout (Optional[Duration]) – Timeout for the state machine. Default: - None

Stability

experimental

Methods

add_catch(handler, *, errors=None, result_path=None)

Add a recovery handler for this state.

When a particular error occurs, execution will continue at the error handler instead of failing the state machine execution.

Parameters
  • handler (IChainable) –

  • errors (Optional[List[str]]) – Errors to recover from by going to the given state. A list of error strings to retry, which can be either predefined errors (for example Errors.NoChoiceMatched) or a self-defined error. Default: All errors

  • result_path (Optional[str]) – JSONPath expression to indicate where to inject the error data. May also be the special value DISCARD, which will cause the error data to be discarded. Default: $

Return type

TaskStateBase

add_prefix(x)

Add a prefix to the stateId of this state.

Parameters

x (str) –

Return type

None

add_retry(*, backoff_rate=None, errors=None, interval=None, max_attempts=None)

Add retry configuration for this state.

This controls if and how the execution will be retried if a particular error occurs.

Parameters
  • backoff_rate (Union[int, float, None]) – Multiplication for how much longer the wait interval gets on every retry. Default: 2

  • errors (Optional[List[str]]) – Errors to retry. A list of error strings to retry, which can be either predefined errors (for example Errors.NoChoiceMatched) or a self-defined error. Default: All errors

  • interval (Optional[Duration]) – How many seconds to wait initially before retrying. Default: Duration.seconds(1)

  • max_attempts (Union[int, float, None]) – How many times to retry this particular error. May be 0 to disable retry for specific errors (in case you have a catch-all retry policy). Default: 3

Return type

TaskStateBase

bind_to_graph(graph)

Register this state as part of the given graph.

Don’t call this. It will be called automatically when you work with states normally.

Parameters

graph (StateGraph) –

Return type

None

metric(metric_name, *, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

Return the given named metric for this Task.

Parameters
  • metric_name (str) –

  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • sum over 5 minutes

Return type

Metric

metric_failed(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

Metric for the number of times this activity fails.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • sum over 5 minutes

Return type

Metric

metric_heartbeat_timed_out(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

Metric for the number of times the heartbeat times out for this activity.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • sum over 5 minutes

Return type

Metric

metric_run_time(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

The interval, in milliseconds, between the time the Task starts and the time it closes.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • average over 5 minutes

Return type

Metric

metric_schedule_time(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

The interval, in milliseconds, for which the activity stays in the schedule state.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • average over 5 minutes

Return type

Metric

metric_scheduled(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

Metric for the number of times this activity is scheduled.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • sum over 5 minutes

Return type

Metric

metric_started(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

Metric for the number of times this activity is started.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • sum over 5 minutes

Return type

Metric

metric_succeeded(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

Metric for the number of times this activity succeeds.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • sum over 5 minutes

Return type

Metric

metric_time(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

The interval, in milliseconds, between the time the activity is scheduled and the time it closes.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • average over 5 minutes

Return type

Metric

metric_timed_out(*, account=None, color=None, dimensions=None, label=None, period=None, region=None, statistic=None, unit=None)

Metric for the number of times this activity times out.

Parameters
  • account (Optional[str]) – Account which this metric comes from. Default: - Deployment account.

  • color (Optional[str]) – The hex color code, prefixed with ‘#’ (e.g. ‘#00ff00’), to use when this metric is rendered on a graph. The Color class has a set of standard colors that can be used here. Default: - Automatic color

  • dimensions (Optional[Mapping[str, Any]]) – Dimensions of the metric. Default: - No dimensions.

  • label (Optional[str]) – Label for this metric when added to a Graph in a Dashboard. Default: - No label

  • period (Optional[Duration]) – The period over which the specified statistic is applied. Default: Duration.minutes(5)

  • region (Optional[str]) – Region which this metric comes from. Default: - Deployment region.

  • statistic (Optional[str]) – What function to use for aggregating. Can be one of the following: - “Minimum” | “min” - “Maximum” | “max” - “Average” | “avg” - “Sum” | “sum” - “SampleCount | “n” - “pNN.NN” Default: Average

  • unit (Optional[Unit]) – Unit used to filter the metric stream. Only refer to datums emitted to the metric stream with the given unit and ignore all others. Only useful when datums are being emitted to the same metric stream under different units. The default is to use all matric datums in the stream, regardless of unit, which is recommended in nearly all cases. CloudWatch does not honor this property for graphs. Default: - All metric datums in the given metric stream

Default
  • sum over 5 minutes

Return type

Metric

next(next)

Continue normal execution with the given state.

Parameters

next (IChainable) –

Return type

Chain

to_state_json()

Return the Amazon States Language object for this state.

Return type

Mapping[Any, Any]

to_string()

Returns a string representation of this construct.

Return type

str

Attributes

auto_scaling_role

(experimental) The autoscaling role for the EMR Cluster.

Only available after task has been added to a state machine.

Stability

experimental

Return type

IRole

cluster_role

(experimental) The instance role for the EMR Cluster.

Only available after task has been added to a state machine.

Stability

experimental

Return type

IRole

end_states

Continuable states of this Chainable.

Return type

List[INextable]

id

Descriptive identifier for this chainable.

Return type

str

node

The construct tree node associated with this construct.

Return type

ConstructNode

service_role

(experimental) The service role for the EMR Cluster.

Only available after task has been added to a state machine.

Stability

experimental

Return type

IRole

start_state

First state of this Chainable.

Return type

State

state_id

Tokenized string that evaluates to the state’s ID.

Return type

str

Static Methods

classmethod filter_nextables(states)

Return only the states that allow chaining from an array of states.

Parameters

states (List[State]) –

Return type

List[INextable]

classmethod find_reachable_end_states(start, *, include_error_handlers=None)

Find the set of end states states reachable through transitions from the given start state.

Parameters
  • start (State) –

  • include_error_handlers (Optional[bool]) – Whether or not to follow error-handling transitions. Default: false

Return type

List[State]

classmethod find_reachable_states(start, *, include_error_handlers=None)

Find the set of states reachable through transitions from the given start state.

This does not retrieve states from within sub-graphs, such as states within a Parallel state’s branch.

Parameters
  • start (State) –

  • include_error_handlers (Optional[bool]) – Whether or not to follow error-handling transitions. Default: false

Return type

List[State]

classmethod is_construct(x)

Return whether the given object is a Construct.

Parameters

x (Any) –

Return type

bool

classmethod prefix_states(root, prefix)

Add a prefix to the stateId of all States found in a construct tree.

Parameters
  • root (IConstruct) –

  • prefix (str) –

Return type

None

ApplicationConfigProperty

class EmrCreateCluster.ApplicationConfigProperty(*, name, additional_info=None, args=None, version=None)

Bases: object

(experimental) Properties for the EMR Cluster Applications.

Applies to Amazon EMR releases 4.0 and later. A case-insensitive list of applications for Amazon EMR to install and configure when launching the cluster.

See the RunJobFlow API for complete documentation on input parameters

Parameters
  • name (str) – (experimental) The name of the application.

  • additional_info (Optional[Mapping[str, str]]) – (experimental) This option is for advanced users only. This is meta information about third-party applications that third-party vendors use for testing purposes. Default: No additionalInfo

  • args (Optional[List[str]]) – (experimental) Arguments for Amazon EMR to pass to the application. Default: No args

  • version (Optional[str]) – (experimental) The version of the application. Default: No version

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_Application.html

Stability

experimental

Attributes

additional_info

(experimental) This option is for advanced users only.

This is meta information about third-party applications that third-party vendors use for testing purposes.

Default

No additionalInfo

Stability

experimental

Return type

Optional[Mapping[str, str]]

args

(experimental) Arguments for Amazon EMR to pass to the application.

Default

No args

Stability

experimental

Return type

Optional[List[str]]

name

(experimental) The name of the application.

Stability

experimental

Return type

str

version

(experimental) The version of the application.

Default

No version

Stability

experimental

Return type

Optional[str]

AutoScalingPolicyProperty

class EmrCreateCluster.AutoScalingPolicyProperty(*, constraints, rules)

Bases: object

(experimental) An automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster.

Parameters
  • constraints (ScalingConstraintsProperty) – (experimental) The upper and lower EC2 instance limits for an automatic scaling policy. Automatic scaling activity will not cause an instance group to grow above or below these limits.

  • rules (List[ScalingRuleProperty]) – (experimental) The scale-in and scale-out rules that comprise the automatic scaling policy.

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_AutoScalingPolicy.html

Stability

experimental

Attributes

constraints

(experimental) The upper and lower EC2 instance limits for an automatic scaling policy.

Automatic scaling activity will not cause an instance group to grow above or below these limits.

Stability

experimental

Return type

ScalingConstraintsProperty

rules

(experimental) The scale-in and scale-out rules that comprise the automatic scaling policy.

Stability

experimental

Return type

List[ScalingRuleProperty]

BootstrapActionConfigProperty

class EmrCreateCluster.BootstrapActionConfigProperty(*, name, script_bootstrap_action)

Bases: object

(experimental) Configuration of a bootstrap action.

See the RunJobFlow API for complete documentation on input parameters

Parameters
  • name (str) – (experimental) The name of the bootstrap action.

  • script_bootstrap_action (ScriptBootstrapActionConfigProperty) – (experimental) The script run by the bootstrap action.

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_BootstrapActionConfig.html

Stability

experimental

Attributes

name

(experimental) The name of the bootstrap action.

Stability

experimental

Return type

str

script_bootstrap_action

(experimental) The script run by the bootstrap action.

Stability

experimental

Return type

ScriptBootstrapActionConfigProperty

CloudWatchAlarmComparisonOperator

class EmrCreateCluster.CloudWatchAlarmComparisonOperator(value)

Bases: enum.Enum

(experimental) CloudWatch Alarm Comparison Operators.

Stability

experimental

Attributes

GREATER_THAN

(experimental) GREATER_THAN.

Stability

experimental

GREATER_THAN_OR_EQUAL

(experimental) GREATER_THAN_OR_EQUAL.

Stability

experimental

LESS_THAN

(experimental) LESS_THAN.

Stability

experimental

LESS_THAN_OR_EQUAL

(experimental) LESS_THAN_OR_EQUAL.

Stability

experimental

CloudWatchAlarmDefinitionProperty

class EmrCreateCluster.CloudWatchAlarmDefinitionProperty(*, comparison_operator, metric_name, period, dimensions=None, evaluation_periods=None, namespace=None, statistic=None, threshold=None, unit=None)

Bases: object

(experimental) The definition of a CloudWatch metric alarm, which determines when an automatic scaling activity is triggered.

When the defined alarm conditions are satisfied, scaling activity begins.

Parameters
  • comparison_operator (CloudWatchAlarmComparisonOperator) – (experimental) Determines how the metric specified by MetricName is compared to the value specified by Threshold.

  • metric_name (str) – (experimental) The name of the CloudWatch metric that is watched to determine an alarm condition.

  • period (Duration) – (experimental) The period, in seconds, over which the statistic is applied. EMR CloudWatch metrics are emitted every five minutes (300 seconds), so if an EMR CloudWatch metric is specified, specify 300.

  • dimensions (Optional[List[MetricDimensionProperty]]) – (experimental) A CloudWatch metric dimension. Default: - No dimensions

  • evaluation_periods (Union[int, float, None]) – (experimental) The number of periods, in five-minute increments, during which the alarm condition must exist before the alarm triggers automatic scaling activity. Default: 1

  • namespace (Optional[str]) – (experimental) The namespace for the CloudWatch metric. Default: ‘AWS/ElasticMapReduce’

  • statistic (Optional[CloudWatchAlarmStatistic]) – (experimental) The statistic to apply to the metric associated with the alarm. Default: CloudWatchAlarmStatistic.AVERAGE

  • threshold (Union[int, float, None]) – (experimental) The value against which the specified statistic is compared. Default: - None

  • unit (Optional[CloudWatchAlarmUnit]) – (experimental) The unit of measure associated with the CloudWatch metric being watched. The value specified for Unit must correspond to the units specified in the CloudWatch metric. Default: CloudWatchAlarmUnit.NONE

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_CloudWatchAlarmDefinition.html

Stability

experimental

Attributes

comparison_operator

(experimental) Determines how the metric specified by MetricName is compared to the value specified by Threshold.

Stability

experimental

Return type

CloudWatchAlarmComparisonOperator

dimensions

(experimental) A CloudWatch metric dimension.

Default
  • No dimensions

Stability

experimental

Return type

Optional[List[MetricDimensionProperty]]

evaluation_periods

(experimental) The number of periods, in five-minute increments, during which the alarm condition must exist before the alarm triggers automatic scaling activity.

Default

1

Stability

experimental

Return type

Union[int, float, None]

metric_name

(experimental) The name of the CloudWatch metric that is watched to determine an alarm condition.

Stability

experimental

Return type

str

namespace

(experimental) The namespace for the CloudWatch metric.

Default

‘AWS/ElasticMapReduce’

Stability

experimental

Return type

Optional[str]

period

(experimental) The period, in seconds, over which the statistic is applied.

EMR CloudWatch metrics are emitted every five minutes (300 seconds), so if an EMR CloudWatch metric is specified, specify 300.

Stability

experimental

Return type

Duration

statistic

(experimental) The statistic to apply to the metric associated with the alarm.

Default

CloudWatchAlarmStatistic.AVERAGE

Stability

experimental

Return type

Optional[CloudWatchAlarmStatistic]

threshold

(experimental) The value against which the specified statistic is compared.

Default
  • None

Stability

experimental

Return type

Union[int, float, None]

unit

(experimental) The unit of measure associated with the CloudWatch metric being watched.

The value specified for Unit must correspond to the units specified in the CloudWatch metric.

Default

CloudWatchAlarmUnit.NONE

Stability

experimental

Return type

Optional[CloudWatchAlarmUnit]

CloudWatchAlarmStatistic

class EmrCreateCluster.CloudWatchAlarmStatistic(value)

Bases: enum.Enum

(experimental) CloudWatch Alarm Statistics.

Stability

experimental

Attributes

AVERAGE

(experimental) AVERAGE.

Stability

experimental

MAXIMUM

(experimental) MAXIMUM.

Stability

experimental

MINIMUM

(experimental) MINIMUM.

Stability

experimental

SAMPLE_COUNT

(experimental) SAMPLE_COUNT.

Stability

experimental

SUM

(experimental) SUM.

Stability

experimental

CloudWatchAlarmUnit

class EmrCreateCluster.CloudWatchAlarmUnit(value)

Bases: enum.Enum

(experimental) CloudWatch Alarm Units.

Stability

experimental

Attributes

BITS

(experimental) BITS.

Stability

experimental

BITS_PER_SECOND

(experimental) BITS_PER_SECOND.

Stability

experimental

BYTES

(experimental) BYTES.

Stability

experimental

BYTES_PER_SECOND

(experimental) BYTES_PER_SECOND.

Stability

experimental

COUNT

(experimental) COUNT.

Stability

experimental

COUNT_PER_SECOND

(experimental) COUNT_PER_SECOND.

Stability

experimental

GIGA_BITS

(experimental) GIGA_BITS.

Stability

experimental

GIGA_BITS_PER_SECOND

(experimental) GIGA_BITS_PER_SECOND.

Stability

experimental

GIGA_BYTES

(experimental) GIGA_BYTES.

Stability

experimental

GIGA_BYTES_PER_SECOND

(experimental) GIGA_BYTES_PER_SECOND.

Stability

experimental

KILO_BITS

(experimental) KILO_BITS.

Stability

experimental

KILO_BITS_PER_SECOND

(experimental) KILO_BITS_PER_SECOND.

Stability

experimental

KILO_BYTES

(experimental) KILO_BYTES.

Stability

experimental

KILO_BYTES_PER_SECOND

(experimental) KILO_BYTES_PER_SECOND.

Stability

experimental

MEGA_BITS

(experimental) MEGA_BITS.

Stability

experimental

MEGA_BITS_PER_SECOND

(experimental) MEGA_BITS_PER_SECOND.

Stability

experimental

MEGA_BYTES

(experimental) MEGA_BYTES.

Stability

experimental

MEGA_BYTES_PER_SECOND

(experimental) MEGA_BYTES_PER_SECOND.

Stability

experimental

MICRO_SECONDS

(experimental) MICRO_SECONDS.

Stability

experimental

MILLI_SECONDS

(experimental) MILLI_SECONDS.

Stability

experimental

NONE

(experimental) NONE.

Stability

experimental

PERCENT

(experimental) PERCENT.

Stability

experimental

SECONDS

(experimental) SECONDS.

Stability

experimental

TERA_BITS

(experimental) TERA_BITS.

Stability

experimental

TERA_BITS_PER_SECOND

(experimental) TERA_BITS_PER_SECOND.

Stability

experimental

TERA_BYTES

(experimental) TERA_BYTES.

Stability

experimental

TERA_BYTES_PER_SECOND

(experimental) TERA_BYTES_PER_SECOND.

Stability

experimental

ConfigurationProperty

class EmrCreateCluster.ConfigurationProperty(*, classification=None, configurations=None, properties=None)

Bases: object

(experimental) An optional configuration specification to be used when provisioning cluster instances, which can include configurations for applications and software bundled with Amazon EMR.

See the RunJobFlow API for complete documentation on input parameters

Parameters
  • classification (Optional[str]) – (experimental) The classification within a configuration. Default: No classification

  • configurations (Optional[List[ConfigurationProperty]]) – (experimental) A list of additional configurations to apply within a configuration object. Default: No configurations

  • properties (Optional[Mapping[str, str]]) – (experimental) A set of properties specified within a configuration classification. Default: No properties

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_Configuration.html

Stability

experimental

Attributes

classification

(experimental) The classification within a configuration.

Default

No classification

Stability

experimental

Return type

Optional[str]

configurations

(experimental) A list of additional configurations to apply within a configuration object.

Default

No configurations

Stability

experimental

Return type

Optional[List[ConfigurationProperty]]

properties

(experimental) A set of properties specified within a configuration classification.

Default

No properties

Stability

experimental

Return type

Optional[Mapping[str, str]]

EbsBlockDeviceConfigProperty

class EmrCreateCluster.EbsBlockDeviceConfigProperty(*, volume_specification, volumes_per_instance=None)

Bases: object

(experimental) Configuration of requested EBS block device associated with the instance group with count of volumes that will be associated to every instance.

Parameters
  • volume_specification (VolumeSpecificationProperty) – (experimental) EBS volume specifications such as volume type, IOPS, and size (GiB) that will be requested for the EBS volume attached to an EC2 instance in the cluster.

  • volumes_per_instance (Union[int, float, None]) – (experimental) Number of EBS volumes with a specific volume configuration that will be associated with every instance in the instance group. Default: EMR selected default

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_EbsBlockDeviceConfig.html

Stability

experimental

Attributes

volume_specification

(experimental) EBS volume specifications such as volume type, IOPS, and size (GiB) that will be requested for the EBS volume attached to an EC2 instance in the cluster.

Stability

experimental

Return type

VolumeSpecificationProperty

volumes_per_instance

(experimental) Number of EBS volumes with a specific volume configuration that will be associated with every instance in the instance group.

Default

EMR selected default

Stability

experimental

Return type

Union[int, float, None]

EbsBlockDeviceVolumeType

class EmrCreateCluster.EbsBlockDeviceVolumeType(value)

Bases: enum.Enum

(experimental) EBS Volume Types.

Stability

experimental

Attributes

GP2

(experimental) gp2 Volume Type.

Stability

experimental

IO1

(experimental) io1 Volume Type.

Stability

experimental

STANDARD

(experimental) Standard Volume Type.

Stability

experimental

EbsConfigurationProperty

class EmrCreateCluster.EbsConfigurationProperty(*, ebs_block_device_configs=None, ebs_optimized=None)

Bases: object

(experimental) The Amazon EBS configuration of a cluster instance.

Parameters
  • ebs_block_device_configs (Optional[List[EbsBlockDeviceConfigProperty]]) – (experimental) An array of Amazon EBS volume specifications attached to a cluster instance. Default: - None

  • ebs_optimized (Optional[bool]) – (experimental) Indicates whether an Amazon EBS volume is EBS-optimized. Default: - EMR selected default

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_EbsConfiguration.html

Stability

experimental

Attributes

ebs_block_device_configs

(experimental) An array of Amazon EBS volume specifications attached to a cluster instance.

Default
  • None

Stability

experimental

Return type

Optional[List[EbsBlockDeviceConfigProperty]]

ebs_optimized

(experimental) Indicates whether an Amazon EBS volume is EBS-optimized.

Default
  • EMR selected default

Stability

experimental

Return type

Optional[bool]

EmrClusterScaleDownBehavior

class EmrCreateCluster.EmrClusterScaleDownBehavior(value)

Bases: enum.Enum

(experimental) Valid valus for the Cluster ScaleDownBehavior.

Stability

experimental

Attributes

TERMINATE_AT_INSTANCE_HOUR

(experimental) Indicates that Amazon EMR terminates nodes at the instance-hour boundary, regardless of when the request to terminate the instance was submitted.

This option is only available with Amazon EMR 5.1.0 and later and is the default for clusters created using that version

Stability

experimental

TERMINATE_AT_TASK_COMPLETION

(experimental) Indicates that Amazon EMR blacklists and drains tasks from nodes before terminating the Amazon EC2 instances, regardless of the instance-hour boundary.

Stability

experimental

InstanceFleetConfigProperty

class EmrCreateCluster.InstanceFleetConfigProperty(*, instance_fleet_type, instance_type_configs=None, launch_specifications=None, name=None, target_on_demand_capacity=None, target_spot_capacity=None)

Bases: object

(experimental) The configuration that defines an instance fleet.

Parameters
  • instance_fleet_type (InstanceRoleType) – (experimental) The node type that the instance fleet hosts. Valid values are MASTER,CORE,and TASK.

  • instance_type_configs (Optional[List[InstanceTypeConfigProperty]]) – (experimental) The instance type configurations that define the EC2 instances in the instance fleet. Default: No instanceTpeConfigs

  • launch_specifications (Optional[InstanceFleetProvisioningSpecificationsProperty]) – (experimental) The launch specification for the instance fleet. Default: No launchSpecifications

  • name (Optional[str]) – (experimental) The friendly name of the instance fleet. Default: No name

  • target_on_demand_capacity (Union[int, float, None]) – (experimental) The target capacity of On-Demand units for the instance fleet, which determines how many On-Demand instances to provision. Default: No targetOnDemandCapacity

  • target_spot_capacity (Union[int, float, None]) – (experimental) The target capacity of Spot units for the instance fleet, which determines how many Spot instances to provision. Default: No targetSpotCapacity

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceFleetConfig.html

Stability

experimental

Attributes

instance_fleet_type

(experimental) The node type that the instance fleet hosts.

Valid values are MASTER,CORE,and TASK.

Stability

experimental

Return type

InstanceRoleType

instance_type_configs

(experimental) The instance type configurations that define the EC2 instances in the instance fleet.

Default

No instanceTpeConfigs

Stability

experimental

Return type

Optional[List[InstanceTypeConfigProperty]]

launch_specifications

(experimental) The launch specification for the instance fleet.

Default

No launchSpecifications

Stability

experimental

Return type

Optional[InstanceFleetProvisioningSpecificationsProperty]

name

(experimental) The friendly name of the instance fleet.

Default

No name

Stability

experimental

Return type

Optional[str]

target_on_demand_capacity

(experimental) The target capacity of On-Demand units for the instance fleet, which determines how many On-Demand instances to provision.

Default

No targetOnDemandCapacity

Stability

experimental

Return type

Union[int, float, None]

target_spot_capacity

(experimental) The target capacity of Spot units for the instance fleet, which determines how many Spot instances to provision.

Default

No targetSpotCapacity

Stability

experimental

Return type

Union[int, float, None]

InstanceFleetProvisioningSpecificationsProperty

class EmrCreateCluster.InstanceFleetProvisioningSpecificationsProperty(*, spot_specification)

Bases: object

(experimental) The launch specification for Spot instances in the fleet, which determines the defined duration and provisioning timeout behavior.

Parameters

spot_specification (SpotProvisioningSpecificationProperty) – (experimental) The launch specification for Spot instances in the fleet, which determines the defined duration and provisioning timeout behavior.

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceFleetProvisioningSpecifications.html

Stability

experimental

Attributes

spot_specification

(experimental) The launch specification for Spot instances in the fleet, which determines the defined duration and provisioning timeout behavior.

Stability

experimental

Return type

SpotProvisioningSpecificationProperty

InstanceGroupConfigProperty

class EmrCreateCluster.InstanceGroupConfigProperty(*, instance_count, instance_role, instance_type, auto_scaling_policy=None, bid_price=None, configurations=None, ebs_configuration=None, market=None, name=None)

Bases: object

(experimental) Configuration defining a new instance group.

Parameters
  • instance_count (Union[int, float]) – (experimental) Target number of instances for the instance group.

  • instance_role (InstanceRoleType) – (experimental) The role of the instance group in the cluster.

  • instance_type (str) – (experimental) The EC2 instance type for all instances in the instance group.

  • auto_scaling_policy (Optional[AutoScalingPolicyProperty]) – (experimental) An automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. Default: - None

  • bid_price (Optional[str]) – (experimental) The bid price for each EC2 Spot instance type as defined by InstanceType. Expressed in USD. Default: - None

  • configurations (Optional[List[ConfigurationProperty]]) – (experimental) The list of configurations supplied for an EMR cluster instance group. Default: - None

  • ebs_configuration (Optional[EbsConfigurationProperty]) – (experimental) EBS configurations that will be attached to each EC2 instance in the instance group. Default: - None

  • market (Optional[InstanceMarket]) – (experimental) Market type of the EC2 instances used to create a cluster node. Default: - EMR selected default

  • name (Optional[str]) – (experimental) Friendly name given to the instance group. Default: - None

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceGroupConfig.html

Stability

experimental

Attributes

auto_scaling_policy

(experimental) An automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster.

Default
  • None

Stability

experimental

Return type

Optional[AutoScalingPolicyProperty]

bid_price

(experimental) The bid price for each EC2 Spot instance type as defined by InstanceType.

Expressed in USD.

Default
  • None

Stability

experimental

Return type

Optional[str]

configurations

(experimental) The list of configurations supplied for an EMR cluster instance group.

Default
  • None

Stability

experimental

Return type

Optional[List[ConfigurationProperty]]

ebs_configuration

(experimental) EBS configurations that will be attached to each EC2 instance in the instance group.

Default
  • None

Stability

experimental

Return type

Optional[EbsConfigurationProperty]

instance_count

(experimental) Target number of instances for the instance group.

Stability

experimental

Return type

Union[int, float]

instance_role

(experimental) The role of the instance group in the cluster.

Stability

experimental

Return type

InstanceRoleType

instance_type

(experimental) The EC2 instance type for all instances in the instance group.

Stability

experimental

Return type

str

market

(experimental) Market type of the EC2 instances used to create a cluster node.

Default
  • EMR selected default

Stability

experimental

Return type

Optional[InstanceMarket]

name

(experimental) Friendly name given to the instance group.

Default
  • None

Stability

experimental

Return type

Optional[str]

InstanceMarket

class EmrCreateCluster.InstanceMarket(value)

Bases: enum.Enum

(experimental) EC2 Instance Market.

Stability

experimental

Attributes

ON_DEMAND

(experimental) On Demand Instance.

Stability

experimental

SPOT

(experimental) Spot Instance.

Stability

experimental

InstanceRoleType

class EmrCreateCluster.InstanceRoleType(value)

Bases: enum.Enum

(experimental) Instance Role Types.

Stability

experimental

Attributes

CORE

(experimental) Core Node.

Stability

experimental

MASTER

(experimental) Master Node.

Stability

experimental

TASK

(experimental) Task Node.

Stability

experimental

InstanceTypeConfigProperty

class EmrCreateCluster.InstanceTypeConfigProperty(*, instance_type, bid_price=None, bid_price_as_percentage_of_on_demand_price=None, configurations=None, ebs_configuration=None, weighted_capacity=None)

Bases: object

(experimental) An instance type configuration for each instance type in an instance fleet, which determines the EC2 instances Amazon EMR attempts to provision to fulfill On-Demand and Spot target capacities.

Parameters
  • instance_type (str) – (experimental) An EC2 instance type.

  • bid_price (Optional[str]) – (experimental) The bid price for each EC2 Spot instance type as defined by InstanceType. Expressed in USD. Default: - None

  • bid_price_as_percentage_of_on_demand_price (Union[int, float, None]) – (experimental) The bid price, as a percentage of On-Demand price. Default: - None

  • configurations (Optional[List[ConfigurationProperty]]) – (experimental) A configuration classification that applies when provisioning cluster instances, which can include configurations for applications and software that run on the cluster. Default: - None

  • ebs_configuration (Optional[EbsConfigurationProperty]) – (experimental) The configuration of Amazon Elastic Block Storage (EBS) attached to each instance as defined by InstanceType. Default: - None

  • weighted_capacity (Union[int, float, None]) – (experimental) The number of units that a provisioned instance of this type provides toward fulfilling the target capacities defined in the InstanceFleetConfig. Default: - None

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceTypeConfig.html

Stability

experimental

Attributes

bid_price

(experimental) The bid price for each EC2 Spot instance type as defined by InstanceType.

Expressed in USD.

Default
  • None

Stability

experimental

Return type

Optional[str]

bid_price_as_percentage_of_on_demand_price

(experimental) The bid price, as a percentage of On-Demand price.

Default
  • None

Stability

experimental

Return type

Union[int, float, None]

configurations

(experimental) A configuration classification that applies when provisioning cluster instances, which can include configurations for applications and software that run on the cluster.

Default
  • None

Stability

experimental

Return type

Optional[List[ConfigurationProperty]]

ebs_configuration

(experimental) The configuration of Amazon Elastic Block Storage (EBS) attached to each instance as defined by InstanceType.

Default
  • None

Stability

experimental

Return type

Optional[EbsConfigurationProperty]

instance_type

(experimental) An EC2 instance type.

Stability

experimental

Return type

str

weighted_capacity

(experimental) The number of units that a provisioned instance of this type provides toward fulfilling the target capacities defined in the InstanceFleetConfig.

Default
  • None

Stability

experimental

Return type

Union[int, float, None]

InstancesConfigProperty

class EmrCreateCluster.InstancesConfigProperty(*, additional_master_security_groups=None, additional_slave_security_groups=None, ec2_key_name=None, ec2_subnet_id=None, ec2_subnet_ids=None, emr_managed_master_security_group=None, emr_managed_slave_security_group=None, hadoop_version=None, instance_count=None, instance_fleets=None, instance_groups=None, master_instance_type=None, placement=None, service_access_security_group=None, slave_instance_type=None, termination_protected=None)

Bases: object

(experimental) A specification of the number and type of Amazon EC2 instances.

See the RunJobFlow API for complete documentation on input parameters

Parameters
  • additional_master_security_groups (Optional[List[str]]) – (experimental) A list of additional Amazon EC2 security group IDs for the master node. Default: - None

  • additional_slave_security_groups (Optional[List[str]]) – (experimental) A list of additional Amazon EC2 security group IDs for the core and task nodes. Default: - None

  • ec2_key_name (Optional[str]) – (experimental) The name of the EC2 key pair that can be used to ssh to the master node as the user called “hadoop.”. Default: - None

  • ec2_subnet_id (Optional[str]) – (experimental) Applies to clusters that use the uniform instance group configuration. To launch the cluster in Amazon Virtual Private Cloud (Amazon VPC), set this parameter to the identifier of the Amazon VPC subnet where you want the cluster to launch. Default: EMR selected default

  • ec2_subnet_ids (Optional[List[str]]) – (experimental) Applies to clusters that use the instance fleet configuration. When multiple EC2 subnet IDs are specified, Amazon EMR evaluates them and launches instances in the optimal subnet. Default: EMR selected default

  • emr_managed_master_security_group (Optional[str]) – (experimental) The identifier of the Amazon EC2 security group for the master node. Default: - None

  • emr_managed_slave_security_group (Optional[str]) – (experimental) The identifier of the Amazon EC2 security group for the core and task nodes. Default: - None

  • hadoop_version (Optional[str]) – (experimental) Applies only to Amazon EMR release versions earlier than 4.0. The Hadoop version for the cluster. Default: - 0.18 if the AmiVersion parameter is not set. If AmiVersion is set, the version of Hadoop for that AMI version is used.

  • instance_count (Union[int, float, None]) – (experimental) The number of EC2 instances in the cluster. Default: 0

  • instance_fleets (Optional[List[InstanceFleetConfigProperty]]) – (experimental) Describes the EC2 instances and instance configurations for clusters that use the instance fleet configuration. The instance fleet configuration is available only in Amazon EMR versions 4.8.0 and later, excluding 5.0.x versions. Default: - None

  • instance_groups (Optional[List[InstanceGroupConfigProperty]]) – (experimental) Configuration for the instance groups in a cluster. Default: - None

  • master_instance_type (Optional[str]) – (experimental) The EC2 instance type of the master node. Default: - None

  • placement (Optional[PlacementTypeProperty]) – (experimental) The Availability Zone in which the cluster runs. Default: - EMR selected default

  • service_access_security_group (Optional[str]) – (experimental) The identifier of the Amazon EC2 security group for the Amazon EMR service to access clusters in VPC private subnets. Default: - None

  • slave_instance_type (Optional[str]) – (experimental) The EC2 instance type of the core and task nodes. Default: - None

  • termination_protected (Optional[bool]) – (experimental) Specifies whether to lock the cluster to prevent the Amazon EC2 instances from being terminated by API call, user intervention, or in the event of a job-flow error. Default: false

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_JobFlowInstancesConfig.html

Stability

experimental

Attributes

additional_master_security_groups

(experimental) A list of additional Amazon EC2 security group IDs for the master node.

Default
  • None

Stability

experimental

Return type

Optional[List[str]]

additional_slave_security_groups

(experimental) A list of additional Amazon EC2 security group IDs for the core and task nodes.

Default
  • None

Stability

experimental

Return type

Optional[List[str]]

ec2_key_name

(experimental) The name of the EC2 key pair that can be used to ssh to the master node as the user called “hadoop.”.

Default
  • None

Stability

experimental

Return type

Optional[str]

ec2_subnet_id

(experimental) Applies to clusters that use the uniform instance group configuration.

To launch the cluster in Amazon Virtual Private Cloud (Amazon VPC), set this parameter to the identifier of the Amazon VPC subnet where you want the cluster to launch.

Default

EMR selected default

Stability

experimental

Return type

Optional[str]

ec2_subnet_ids

(experimental) Applies to clusters that use the instance fleet configuration.

When multiple EC2 subnet IDs are specified, Amazon EMR evaluates them and launches instances in the optimal subnet.

Default

EMR selected default

Stability

experimental

Return type

Optional[List[str]]

emr_managed_master_security_group

(experimental) The identifier of the Amazon EC2 security group for the master node.

Default
  • None

Stability

experimental

Return type

Optional[str]

emr_managed_slave_security_group

(experimental) The identifier of the Amazon EC2 security group for the core and task nodes.

Default
  • None

Stability

experimental

Return type

Optional[str]

hadoop_version

(experimental) Applies only to Amazon EMR release versions earlier than 4.0. The Hadoop version for the cluster.

Default
  • 0.18 if the AmiVersion parameter is not set. If AmiVersion is set, the version of Hadoop for that AMI version is used.

Stability

experimental

Return type

Optional[str]

instance_count

(experimental) The number of EC2 instances in the cluster.

Default

0

Stability

experimental

Return type

Union[int, float, None]

instance_fleets

(experimental) Describes the EC2 instances and instance configurations for clusters that use the instance fleet configuration.

The instance fleet configuration is available only in Amazon EMR versions 4.8.0 and later, excluding 5.0.x versions.

Default
  • None

Stability

experimental

Return type

Optional[List[InstanceFleetConfigProperty]]

instance_groups

(experimental) Configuration for the instance groups in a cluster.

Default
  • None

Stability

experimental

Return type

Optional[List[InstanceGroupConfigProperty]]

master_instance_type

(experimental) The EC2 instance type of the master node.

Default
  • None

Stability

experimental

Return type

Optional[str]

placement

(experimental) The Availability Zone in which the cluster runs.

Default
  • EMR selected default

Stability

experimental

Return type

Optional[PlacementTypeProperty]

service_access_security_group

(experimental) The identifier of the Amazon EC2 security group for the Amazon EMR service to access clusters in VPC private subnets.

Default
  • None

Stability

experimental

Return type

Optional[str]

slave_instance_type

(experimental) The EC2 instance type of the core and task nodes.

Default
  • None

Stability

experimental

Return type

Optional[str]

termination_protected

(experimental) Specifies whether to lock the cluster to prevent the Amazon EC2 instances from being terminated by API call, user intervention, or in the event of a job-flow error.

Default

false

Stability

experimental

Return type

Optional[bool]

KerberosAttributesProperty

class EmrCreateCluster.KerberosAttributesProperty(*, realm, ad_domain_join_password=None, ad_domain_join_user=None, cross_realm_trust_principal_password=None, kdc_admin_password=None)

Bases: object

(experimental) Attributes for Kerberos configuration when Kerberos authentication is enabled using a security configuration.

See the RunJobFlow API for complete documentation on input parameters

Parameters
  • realm (str) – (experimental) The name of the Kerberos realm to which all nodes in a cluster belong. For example, EC2.INTERNAL.

  • ad_domain_join_password (Optional[str]) – (experimental) The Active Directory password for ADDomainJoinUser. Default: No adDomainJoinPassword

  • ad_domain_join_user (Optional[str]) – (experimental) Required only when establishing a cross-realm trust with an Active Directory domain. A user with sufficient privileges to join resources to the domain. Default: No adDomainJoinUser

  • cross_realm_trust_principal_password (Optional[str]) – (experimental) Required only when establishing a cross-realm trust with a KDC in a different realm. The cross-realm principal password, which must be identical across realms. Default: No crossRealmTrustPrincipalPassword

  • kdc_admin_password (Optional[str]) – (experimental) The password used within the cluster for the kadmin service on the cluster-dedicated KDC, which maintains Kerberos principals, password policies, and keytabs for the cluster. Default: No kdcAdminPassword

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_KerberosAttributes.html

Stability

experimental

Attributes

ad_domain_join_password

(experimental) The Active Directory password for ADDomainJoinUser.

Default

No adDomainJoinPassword

Stability

experimental

Return type

Optional[str]

ad_domain_join_user

(experimental) Required only when establishing a cross-realm trust with an Active Directory domain.

A user with sufficient privileges to join resources to the domain.

Default

No adDomainJoinUser

Stability

experimental

Return type

Optional[str]

cross_realm_trust_principal_password

(experimental) Required only when establishing a cross-realm trust with a KDC in a different realm.

The cross-realm principal password, which must be identical across realms.

Default

No crossRealmTrustPrincipalPassword

Stability

experimental

Return type

Optional[str]

kdc_admin_password

(experimental) The password used within the cluster for the kadmin service on the cluster-dedicated KDC, which maintains Kerberos principals, password policies, and keytabs for the cluster.

Default

No kdcAdminPassword

Stability

experimental

Return type

Optional[str]

realm

(experimental) The name of the Kerberos realm to which all nodes in a cluster belong.

For example, EC2.INTERNAL.

Stability

experimental

Return type

str

MetricDimensionProperty

class EmrCreateCluster.MetricDimensionProperty(*, key, value)

Bases: object

(experimental) A CloudWatch dimension, which is specified using a Key (known as a Name in CloudWatch), Value pair.

By default, Amazon EMR uses one dimension whose Key is JobFlowID and Value is a variable representing the cluster ID, which is ${emr.clusterId}. This enables the rule to bootstrap when the cluster ID becomes available

Parameters
  • key (str) – (experimental) The dimension name.

  • value (str) – (experimental) The dimension value.

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_MetricDimension.html

Stability

experimental

Attributes

key

(experimental) The dimension name.

Stability

experimental

Return type

str

value

(experimental) The dimension value.

Stability

experimental

Return type

str

PlacementTypeProperty

class EmrCreateCluster.PlacementTypeProperty(*, availability_zone=None, availability_zones=None)

Bases: object

(experimental) The Amazon EC2 Availability Zone configuration of the cluster (job flow).

Parameters
  • availability_zone (Optional[str]) – (experimental) The Amazon EC2 Availability Zone for the cluster. AvailabilityZone is used for uniform instance groups, while AvailabilityZones (plural) is used for instance fleets. Default: - EMR selected default

  • availability_zones (Optional[List[str]]) – (experimental) When multiple Availability Zones are specified, Amazon EMR evaluates them and launches instances in the optimal Availability Zone. AvailabilityZones is used for instance fleets, while AvailabilityZone (singular) is used for uniform instance groups. Default: - EMR selected default

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_PlacementType.html

Stability

experimental

Attributes

availability_zone

(experimental) The Amazon EC2 Availability Zone for the cluster.

AvailabilityZone is used for uniform instance groups, while AvailabilityZones (plural) is used for instance fleets.

Default
  • EMR selected default

Stability

experimental

Return type

Optional[str]

availability_zones

(experimental) When multiple Availability Zones are specified, Amazon EMR evaluates them and launches instances in the optimal Availability Zone.

AvailabilityZones is used for instance fleets, while AvailabilityZone (singular) is used for uniform instance groups.

Default
  • EMR selected default

Stability

experimental

Return type

Optional[List[str]]

ScalingActionProperty

class EmrCreateCluster.ScalingActionProperty(*, simple_scaling_policy_configuration, market=None)

Bases: object

(experimental) The type of adjustment the automatic scaling activity makes when triggered, and the periodicity of the adjustment.

And an automatic scaling configuration, which describes how the policy adds or removes instances, the cooldown period, and the number of EC2 instances that will be added each time the CloudWatch metric alarm condition is satisfied.

Parameters
  • simple_scaling_policy_configuration (SimpleScalingPolicyConfigurationProperty) – (experimental) The type of adjustment the automatic scaling activity makes when triggered, and the periodicity of the adjustment.

  • market (Optional[InstanceMarket]) – (experimental) Not available for instance groups. Instance groups use the market type specified for the group. Default: - EMR selected default

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_ScalingAction.html

Stability

experimental

Attributes

market

(experimental) Not available for instance groups.

Instance groups use the market type specified for the group.

Default
  • EMR selected default

Stability

experimental

Return type

Optional[InstanceMarket]

simple_scaling_policy_configuration

(experimental) The type of adjustment the automatic scaling activity makes when triggered, and the periodicity of the adjustment.

Stability

experimental

Return type

SimpleScalingPolicyConfigurationProperty

ScalingAdjustmentType

class EmrCreateCluster.ScalingAdjustmentType(value)

Bases: enum.Enum

(experimental) AutoScaling Adjustment Type.

Stability

experimental

Attributes

CHANGE_IN_CAPACITY

(experimental) CHANGE_IN_CAPACITY.

Stability

experimental

EXACT_CAPACITY

(experimental) EXACT_CAPACITY.

Stability

experimental

PERCENT_CHANGE_IN_CAPACITY

(experimental) PERCENT_CHANGE_IN_CAPACITY.

Stability

experimental

ScalingConstraintsProperty

class EmrCreateCluster.ScalingConstraintsProperty(*, max_capacity, min_capacity)

Bases: object

(experimental) The upper and lower EC2 instance limits for an automatic scaling policy.

Automatic scaling activities triggered by automatic scaling rules will not cause an instance group to grow above or below these limits.

Parameters
  • max_capacity (Union[int, float]) – (experimental) The upper boundary of EC2 instances in an instance group beyond which scaling activities are not allowed to grow. Scale-out activities will not add instances beyond this boundary.

  • min_capacity (Union[int, float]) – (experimental) The lower boundary of EC2 instances in an instance group below which scaling activities are not allowed to shrink. Scale-in activities will not terminate instances below this boundary.

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_ScalingConstraints.html

Stability

experimental

Attributes

max_capacity

(experimental) The upper boundary of EC2 instances in an instance group beyond which scaling activities are not allowed to grow.

Scale-out activities will not add instances beyond this boundary.

Stability

experimental

Return type

Union[int, float]

min_capacity

(experimental) The lower boundary of EC2 instances in an instance group below which scaling activities are not allowed to shrink.

Scale-in activities will not terminate instances below this boundary.

Stability

experimental

Return type

Union[int, float]

ScalingRuleProperty

class EmrCreateCluster.ScalingRuleProperty(*, action, name, trigger, description=None)

Bases: object

(experimental) A scale-in or scale-out rule that defines scaling activity, including the CloudWatch metric alarm that triggers activity, how EC2 instances are added or removed, and the periodicity of adjustments.

Parameters
  • action (ScalingActionProperty) – (experimental) The conditions that trigger an automatic scaling activity.

  • name (str) – (experimental) The name used to identify an automatic scaling rule. Rule names must be unique within a scaling policy.

  • trigger (ScalingTriggerProperty) – (experimental) The CloudWatch alarm definition that determines when automatic scaling activity is triggered.

  • description (Optional[str]) – (experimental) A friendly, more verbose description of the automatic scaling rule. Default: - None

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_ScalingRule.html

Stability

experimental

Attributes

action

(experimental) The conditions that trigger an automatic scaling activity.

Stability

experimental

Return type

ScalingActionProperty

description

(experimental) A friendly, more verbose description of the automatic scaling rule.

Default
  • None

Stability

experimental

Return type

Optional[str]

name

(experimental) The name used to identify an automatic scaling rule.

Rule names must be unique within a scaling policy.

Stability

experimental

Return type

str

trigger

(experimental) The CloudWatch alarm definition that determines when automatic scaling activity is triggered.

Stability

experimental

Return type

ScalingTriggerProperty

ScalingTriggerProperty

class EmrCreateCluster.ScalingTriggerProperty(*, cloud_watch_alarm_definition)

Bases: object

(experimental) The conditions that trigger an automatic scaling activity and the definition of a CloudWatch metric alarm.

When the defined alarm conditions are met along with other trigger parameters, scaling activity begins.

Parameters

cloud_watch_alarm_definition (CloudWatchAlarmDefinitionProperty) – (experimental) The definition of a CloudWatch metric alarm. When the defined alarm conditions are met along with other trigger parameters, scaling activity begins.

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_ScalingTrigger.html

Stability

experimental

Attributes

cloud_watch_alarm_definition

(experimental) The definition of a CloudWatch metric alarm.

When the defined alarm conditions are met along with other trigger parameters, scaling activity begins.

Stability

experimental

Return type

CloudWatchAlarmDefinitionProperty

ScriptBootstrapActionConfigProperty

class EmrCreateCluster.ScriptBootstrapActionConfigProperty(*, path, args=None)

Bases: object

(experimental) Configuration of the script to run during a bootstrap action.

Parameters
  • path (str) – (experimental) Location of the script to run during a bootstrap action. Can be either a location in Amazon S3 or on a local file system.

  • args (Optional[List[str]]) – (experimental) A list of command line arguments to pass to the bootstrap action script. Default: No args

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_ScriptBootstrapActionConfig.html

Stability

experimental

Attributes

args

(experimental) A list of command line arguments to pass to the bootstrap action script.

Default

No args

Stability

experimental

Return type

Optional[List[str]]

path

(experimental) Location of the script to run during a bootstrap action.

Can be either a location in Amazon S3 or on a local file system.

Stability

experimental

Return type

str

SimpleScalingPolicyConfigurationProperty

class EmrCreateCluster.SimpleScalingPolicyConfigurationProperty(*, scaling_adjustment, adjustment_type=None, cool_down=None)

Bases: object

(experimental) An automatic scaling configuration, which describes how the policy adds or removes instances, the cooldown period, and the number of EC2 instances that will be added each time the CloudWatch metric alarm condition is satisfied.

Parameters
  • scaling_adjustment (Union[int, float]) – (experimental) The amount by which to scale in or scale out, based on the specified AdjustmentType. A positive value adds to the instance group’s EC2 instance count while a negative number removes instances. If AdjustmentType is set to EXACT_CAPACITY, the number should only be a positive integer.

  • adjustment_type (Optional[ScalingAdjustmentType]) – (experimental) The way in which EC2 instances are added (if ScalingAdjustment is a positive number) or terminated (if ScalingAdjustment is a negative number) each time the scaling activity is triggered. Default: - None

  • cool_down (Union[int, float, None]) – (experimental) The amount of time, in seconds, after a scaling activity completes before any further trigger-related scaling activities can start. Default: 0

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_SimpleScalingPolicyConfiguration.html

Stability

experimental

Attributes

adjustment_type

(experimental) The way in which EC2 instances are added (if ScalingAdjustment is a positive number) or terminated (if ScalingAdjustment is a negative number) each time the scaling activity is triggered.

Default
  • None

Stability

experimental

Return type

Optional[ScalingAdjustmentType]

cool_down

(experimental) The amount of time, in seconds, after a scaling activity completes before any further trigger-related scaling activities can start.

Default

0

Stability

experimental

Return type

Union[int, float, None]

scaling_adjustment

(experimental) The amount by which to scale in or scale out, based on the specified AdjustmentType.

A positive value adds to the instance group’s EC2 instance count while a negative number removes instances. If AdjustmentType is set to EXACT_CAPACITY, the number should only be a positive integer.

Stability

experimental

Return type

Union[int, float]

SpotProvisioningSpecificationProperty

class EmrCreateCluster.SpotProvisioningSpecificationProperty(*, timeout_action, timeout_duration_minutes, block_duration_minutes=None)

Bases: object

(experimental) The launch specification for Spot instances in the instance fleet, which determines the defined duration and provisioning timeout behavior.

Parameters
  • timeout_action (SpotTimeoutAction) – (experimental) The action to take when TargetSpotCapacity has not been fulfilled when the TimeoutDurationMinutes has expired.

  • timeout_duration_minutes (Union[int, float]) – (experimental) The spot provisioning timeout period in minutes.

  • block_duration_minutes (Union[int, float, None]) – (experimental) The defined duration for Spot instances (also known as Spot blocks) in minutes. Default: No blockDurationMinutes

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_SpotProvisioningSpecification.html

Stability

experimental

Attributes

block_duration_minutes

(experimental) The defined duration for Spot instances (also known as Spot blocks) in minutes.

Default

No blockDurationMinutes

Stability

experimental

Return type

Union[int, float, None]

timeout_action

(experimental) The action to take when TargetSpotCapacity has not been fulfilled when the TimeoutDurationMinutes has expired.

Stability

experimental

Return type

SpotTimeoutAction

timeout_duration_minutes

(experimental) The spot provisioning timeout period in minutes.

Stability

experimental

Return type

Union[int, float]

SpotTimeoutAction

class EmrCreateCluster.SpotTimeoutAction(value)

Bases: enum.Enum

(experimental) Spot Timeout Actions.

Stability

experimental

Attributes

SWITCH_TO_ON_DEMAND

(experimental) SWITCH_TO_ON_DEMAND.

Stability

experimental

TERMINATE_CLUSTER

(experimental) TERMINATE_CLUSTER.

Stability

experimental

VolumeSpecificationProperty

class EmrCreateCluster.VolumeSpecificationProperty(*, volume_size, volume_type, iops=None)

Bases: object

(experimental) EBS volume specifications such as volume type, IOPS, and size (GiB) that will be requested for the EBS volume attached to an EC2 instance in the cluster.

Parameters
  • volume_size (Size) – (experimental) The volume size. If the volume type is EBS-optimized, the minimum value is 10GiB. Maximum size is 1TiB

  • volume_type (EbsBlockDeviceVolumeType) – (experimental) The volume type. Volume types supported are gp2, io1, standard.

  • iops (Union[int, float, None]) – (experimental) The number of I/O operations per second (IOPS) that the volume supports. Default: - EMR selected default

See

https://docs.aws.amazon.com/emr/latest/APIReference/API_VolumeSpecification.html

Stability

experimental

Attributes

iops

(experimental) The number of I/O operations per second (IOPS) that the volume supports.

Default
  • EMR selected default

Stability

experimental

Return type

Union[int, float, None]

volume_size

(experimental) The volume size.

If the volume type is EBS-optimized, the minimum value is 10GiB. Maximum size is 1TiB

Stability

experimental

Return type

Size

volume_type

(experimental) The volume type.

Volume types supported are gp2, io1, standard.

Stability

experimental

Return type

EbsBlockDeviceVolumeType