Class: Aws::EMR::Client
- Inherits:
-
Seahorse::Client::Base
- Object
- Seahorse::Client::Base
- Aws::EMR::Client
- Includes:
- ClientStubs
- Defined in:
- gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb
Overview
An API client for EMR. To construct a client, you need to configure a :region
and :credentials
.
client = Aws::EMR::Client.new(
region: region_name,
credentials: credentials,
# ...
)
For details on configuring region and credentials see the developer guide.
See #initialize for a full list of supported configuration options.
Instance Attribute Summary
Attributes inherited from Seahorse::Client::Base
API Operations collapse
-
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
-
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
-
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster.
-
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource.
-
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster.
-
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
-
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#create_studio_session_mapping(params = {}) ⇒ Struct
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
-
#delete_studio(params = {}) ⇒ Struct
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#delete_studio_session_mapping(params = {}) ⇒ Struct
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
-
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed.
-
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
-
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
-
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
-
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your AWS account in the current Region.
-
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
-
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
-
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this AWS account.
-
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
-
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
-
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active EC2 instances and EC2 instances terminated in the last 30 days, up to a maximum of 2,000.
-
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions.
-
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names.
-
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you specify
stepIds
with the request of filter byStepStates
. -
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change. -
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
-
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID.
-
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group.
-
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster.
-
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your AWS account in the current Region.
-
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster.
-
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an EMR cluster.
-
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified EMR cluster.
-
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource.
-
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow).
-
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the EC2 instances in the cluster cannot be terminated by user intervention, an API call, or in the event of a job-flow error.
-
#set_visible_to_all_users(params = {}) ⇒ Struct
Sets the Cluster$VisibleToAllUsers value, which determines whether the cluster is visible to all IAM users of the AWS account associated with the cluster.
-
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
-
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
-
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down.
-
#update_studio_session_mapping(params = {}) ⇒ Struct
The Amazon EMR Studio APIs are in preview release for Amazon EMR and are subject to change.
Instance Method Summary collapse
-
#initialize(options) ⇒ Client
constructor
A new instance of Client.
-
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
Methods included from ClientStubs
#api_requests, #stub_data, #stub_responses
Methods inherited from Seahorse::Client::Base
add_plugin, api, clear_plugins, define, new, #operation_names, plugins, remove_plugin, set_api, set_plugins
Methods included from Seahorse::Client::HandlerBuilder
#handle, #handle_request, #handle_response
Constructor Details
#initialize(options) ⇒ Client
Returns a new instance of Client.
334 335 336 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 334 def initialize(*args) super end |
Instance Method Details
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
424 425 426 427 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 424 def add_instance_fleet(params = {}, = {}) req = build_request(:add_instance_fleet, params) req.send_request() end |
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
532 533 534 535 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 532 def add_instance_groups(params = {}, = {}) req = build_request(:add_instance_groups, params) req.send_request() end |
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster. A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using SSH to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. For more information on how to do this, see Add More than 256 Steps to a Cluster in the Amazon EMR Management Guide.
A step specifies the location of a JAR file stored either on the master node of the cluster or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step.
Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully.
You can only add steps to a cluster that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.
609 610 611 612 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 609 def add_job_flow_steps(params = {}, = {}) req = build_request(:add_job_flow_steps, params) req.send_request() end |
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
651 652 653 654 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 651 def (params = {}, = {}) req = build_request(:add_tags, params) req.send_request() end |
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster. Available only
in Amazon EMR versions 4.8.0 and later, excluding version 5.0.0. A
maximum of 256 steps are allowed in each CancelSteps request.
CancelSteps is idempotent but asynchronous; it does not guarantee that
a step will be canceled, even if the request is successfully
submitted. You can only cancel steps that are in a PENDING
state.
698 699 700 701 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 698 def cancel_steps(params = {}, = {}) req = build_request(:cancel_steps, params) req.send_request() end |
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
739 740 741 742 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 739 def create_security_configuration(params = {}, = {}) req = build_request(:create_security_configuration, params) req.send_request() end |
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
Creates a new Amazon EMR Studio.
838 839 840 841 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 838 def create_studio(params = {}, = {}) req = build_request(:create_studio, params) req.send_request() end |
#create_studio_session_mapping(params = {}) ⇒ Struct
Maps a user or group to the Amazon EMR Studio specified by StudioId
,
and applies a session policy to refine Studio permissions for that
user or group.
902 903 904 905 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 902 def create_studio_session_mapping(params = {}, = {}) req = build_request(:create_studio_session_mapping, params) req.send_request() end |
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
924 925 926 927 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 924 def delete_security_configuration(params = {}, = {}) req = build_request(:delete_security_configuration, params) req.send_request() end |
#delete_studio(params = {}) ⇒ Struct
Removes an Amazon EMR Studio from the Studio metadata store.
951 952 953 954 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 951 def delete_studio(params = {}, = {}) req = build_request(:delete_studio, params) req.send_request() end |
#delete_studio_session_mapping(params = {}) ⇒ Struct
Removes a user or group from an Amazon EMR Studio.
1007 1008 1009 1010 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1007 def delete_studio_session_mapping(params = {}, = {}) req = build_request(:delete_studio_session_mapping, params) req.send_request() end |
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
The following waiters are defined for this operation (see #wait_until for detailed usage):
- cluster_running
- cluster_terminated
1108 1109 1110 1111 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1108 def describe_cluster(params = {}, = {}) req = build_request(:describe_cluster, params) req.send_request() end |
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed. We recommend you use ListClusters, DescribeCluster, ListSteps, ListInstanceGroups and ListBootstrapActions instead.
DescribeJobFlows returns a list of job flows that match all of the supplied parameters. The parameters can include a list of job flow IDs, job flow states, and restrictions on job flow creation date and time.
Regardless of supplied parameters, only job flows created within the last two months are returned.
If no parameters are supplied, then job flows matching either of the following criteria are returned:
Job flows created and completed in the last two weeks
Job flows created within the last two months that are in one of the following states:
RUNNING
,WAITING
,SHUTTING_DOWN
,STARTING
Amazon EMR can return a maximum of 512 job flow descriptions.
1235 1236 1237 1238 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1235 def describe_job_flows(params = {}, = {}) req = build_request(:describe_job_flows, params) req.send_request() end |
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
1279 1280 1281 1282 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1279 def describe_notebook_execution(params = {}, = {}) req = build_request(:describe_notebook_execution, params) req.send_request() end |
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
1312 1313 1314 1315 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1312 def describe_security_configuration(params = {}, = {}) req = build_request(:describe_security_configuration, params) req.send_request() end |
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
The following waiters are defined for this operation (see #wait_until for detailed usage):
- step_complete
1366 1367 1368 1369 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1366 def describe_step(params = {}, = {}) req = build_request(:describe_step, params) req.send_request() end |
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
Returns details for the specified Amazon EMR Studio including ID, Name, VPC, Studio access URL, and so on.
1417 1418 1419 1420 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1417 def describe_studio(params = {}, = {}) req = build_request(:describe_studio, params) req.send_request() end |
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your AWS account in the current Region. For more information see Configure Block Public Access for Amazon EMR in the Amazon EMR Management Guide.
1449 1450 1451 1452 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1449 def get_block_public_access_configuration(params = {}, = {}) req = build_request(:get_block_public_access_configuration, params) req.send_request() end |
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
1482 1483 1484 1485 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1482 def get_managed_scaling_policy(params = {}, = {}) req = build_request(:get_managed_scaling_policy, params) req.send_request() end |
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
Fetches mapping details for the specified Amazon EMR Studio and identity (user or group).
1549 1550 1551 1552 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1549 def get_studio_session_mapping(params = {}, = {}) req = build_request(:get_studio_session_mapping, params) req.send_request() end |
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1591 1592 1593 1594 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1591 def list_bootstrap_actions(params = {}, = {}) req = build_request(:list_bootstrap_actions, params) req.send_request() end |
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this AWS account. Allows you to filter the list of clusters based on certain criteria; for example, filtering by cluster creation date and time or by status. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListClusters calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1653 1654 1655 1656 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1653 def list_clusters(params = {}, = {}) req = build_request(:list_clusters, params) req.send_request() end |
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1729 1730 1731 1732 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1729 def list_instance_fleets(params = {}, = {}) req = build_request(:list_instance_fleets, params) req.send_request() end |
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1827 1828 1829 1830 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1827 def list_instance_groups(params = {}, = {}) req = build_request(:list_instance_groups, params) req.send_request() end |
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active EC2 instances and EC2 instances terminated in the last 30 days, up to a maximum of 2,000. EC2 instances in any of the following states are considered active: AWAITING_FULFILLMENT, PROVISIONING, BOOTSTRAPPING, RUNNING.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1908 1909 1910 1911 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1908 def list_instances(params = {}, = {}) req = build_request(:list_instances, params) req.send_request() end |
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions. You can filter the list
based on multiple criteria such as status, time range, and editor id.
Returns a maximum of 50 notebook executions and a marker to track the
paging of a longer notebook execution list across multiple
ListNotebookExecution
calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1997 1998 1999 2000 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1997 def list_notebook_executions(params = {}, = {}) req = build_request(:list_notebook_executions, params) req.send_request() end |
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListSecurityConfigurations calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2035 2036 2037 2038 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2035 def list_security_configurations(params = {}, = {}) req = build_request(:list_security_configurations, params) req.send_request() end |
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you
specify stepIds
with the request of filter by StepStates
. You can
specify a maximum of ten stepIDs
.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2102 2103 2104 2105 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2102 def list_steps(params = {}, = {}) req = build_request(:list_steps, params) req.send_request() end |
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
Returns a list of all user or group session mappings for the EMR
Studio specified by StudioId
.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2156 2157 2158 2159 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2156 def list_studio_session_mappings(params = {}, = {}) req = build_request(:list_studio_session_mappings, params) req.send_request() end |
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
Returns a list of all Amazon EMR Studios associated with the AWS account. The list includes details such as ID, Studio Access URL, and creation time for each Studio.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2201 2202 2203 2204 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2201 def list_studios(params = {}, = {}) req = build_request(:list_studios, params) req.send_request() end |
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
2235 2236 2237 2238 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2235 def modify_cluster(params = {}, = {}) req = build_request(:modify_cluster, params) req.send_request() end |
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID. The call either succeeds or fails atomically.
2273 2274 2275 2276 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2273 def modify_instance_fleet(params = {}, = {}) req = build_request(:modify_instance_fleet, params) req.send_request() end |
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group. The input parameters include the new target instance count for the group and the instance group ID. The call will either succeed or fail atomically.
2327 2328 2329 2330 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2327 def modify_instance_groups(params = {}, = {}) req = build_request(:modify_instance_groups, params) req.send_request() end |
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. The automatic scaling policy defines how an instance group dynamically adds and terminates EC2 instances in response to the value of a CloudWatch metric.
2434 2435 2436 2437 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2434 def put_auto_scaling_policy(params = {}, = {}) req = build_request(:put_auto_scaling_policy, params) req.send_request() end |
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your AWS account in the current Region. For more information see Configure Block Public Access for Amazon EMR in the Amazon EMR Management Guide.
2489 2490 2491 2492 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2489 def put_block_public_access_configuration(params = {}, = {}) req = build_request(:put_block_public_access_configuration, params) req.send_request() end |
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster. The managed scaling policy defines the limits for resources, such as EC2 instances that can be added or terminated from a cluster. The policy only applies to the core and task nodes. The master node cannot be scaled after initial configuration.
2528 2529 2530 2531 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2528 def put_managed_scaling_policy(params = {}, = {}) req = build_request(:put_managed_scaling_policy, params) req.send_request() end |
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an EMR cluster.
2557 2558 2559 2560 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2557 def remove_auto_scaling_policy(params = {}, = {}) req = build_request(:remove_auto_scaling_policy, params) req.send_request() end |
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified EMR cluster.
2580 2581 2582 2583 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2580 def remove_managed_scaling_policy(params = {}, = {}) req = build_request(:remove_managed_scaling_policy, params) req.send_request() end |
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
The following example removes the stack tag with value Prod from a cluster:
2617 2618 2619 2620 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2617 def (params = {}, = {}) req = build_request(:remove_tags, params) req.send_request() end |
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow). The
cluster runs the steps specified. After the steps complete, the
cluster stops and the HDFS partition is lost. To prevent loss of data,
configure the last step of the job flow to store results in Amazon S3.
If the JobFlowInstancesConfig KeepJobFlowAliveWhenNoSteps
parameter
is set to TRUE
, the cluster transitions to the WAITING state rather
than shutting down after the steps have completed.
For additional protection, you can set the JobFlowInstancesConfig
TerminationProtected
parameter to TRUE
to lock the cluster and
prevent it from being terminated by API call, user intervention, or in
the event of a job flow error.
A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. For more information on how to do this, see Add More than 256 Steps to a Cluster in the Amazon EMR Management Guide.
For long running clusters, we recommend that you periodically store your results.
3131 3132 3133 3134 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3131 def run_job_flow(params = {}, = {}) req = build_request(:run_job_flow, params) req.send_request() end |
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the EC2
instances in the cluster cannot be terminated by user intervention, an
API call, or in the event of a job-flow error. The cluster still
terminates upon successful completion of the job flow. Calling
SetTerminationProtection
on a cluster is similar to calling the
Amazon EC2 DisableAPITermination
API on all EC2 instances in a
cluster.
SetTerminationProtection
is used to prevent accidental termination
of a cluster and to ensure that in the event of an error, the
instances persist so that you can recover any data stored in their
ephemeral instance storage.
To terminate a cluster that has been locked by setting
SetTerminationProtection
to true
, you must first unlock the job
flow by a subsequent call to SetTerminationProtection
in which you
set the value to false
.
For more information, seeManaging Cluster Termination in the Amazon EMR Management Guide.
3184 3185 3186 3187 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3184 def set_termination_protection(params = {}, = {}) req = build_request(:set_termination_protection, params) req.send_request() end |
#set_visible_to_all_users(params = {}) ⇒ Struct
Sets the Cluster$VisibleToAllUsers value, which determines whether the
cluster is visible to all IAM users of the AWS account associated with
the cluster. Only the IAM user who created the cluster or the AWS
account root user can call this action. The default value, true
,
indicates that all IAM users in the AWS account can perform cluster
actions if they have the proper IAM policy permissions. If set to
false
, only the IAM user that created the cluster can perform
actions. This action works on running clusters. You can override the
default true
setting when you create a cluster by using the
VisibleToAllUsers
parameter with RunJobFlow
.
3222 3223 3224 3225 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3222 def set_visible_to_all_users(params = {}, = {}) req = build_request(:set_visible_to_all_users, params) req.send_request() end |
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
3303 3304 3305 3306 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3303 def start_notebook_execution(params = {}, = {}) req = build_request(:start_notebook_execution, params) req.send_request() end |
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
3325 3326 3327 3328 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3325 def stop_notebook_execution(params = {}, = {}) req = build_request(:stop_notebook_execution, params) req.send_request() end |
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down. When a job flow is shut down, any step not yet completed is canceled and the EC2 instances on which the cluster is running are stopped. Any log files not already saved are uploaded to Amazon S3 if a LogUri was specified when the cluster was created.
The maximum number of clusters allowed is 10. The call to
TerminateJobFlows
is asynchronous. Depending on the configuration of
the cluster, it may take up to 1-5 minutes for the cluster to
completely terminate and release allocated resources, such as Amazon
EC2 instances.
3357 3358 3359 3360 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3357 def terminate_job_flows(params = {}, = {}) req = build_request(:terminate_job_flows, params) req.send_request() end |
#update_studio_session_mapping(params = {}) ⇒ Struct
Updates the session policy attached to the user or group for the specified Amazon EMR Studio.
3417 3418 3419 3420 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3417 def update_studio_session_mapping(params = {}, = {}) req = build_request(:update_studio_session_mapping, params) req.send_request() end |
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
Basic Usage
A waiter will call an API operation until:
- It is successful
- It enters a terminal state
- It makes the maximum number of attempts
In between attempts, the waiter will sleep.
# polls in a loop, sleeping between attempts
client.wait_until(waiter_name, params)
Configuration
You can configure the maximum number of polling attempts, and the delay (in seconds) between each polling attempt. You can pass configuration as the final arguments hash.
# poll for ~25 seconds
client.wait_until(waiter_name, params, {
max_attempts: 5,
delay: 5,
})
Callbacks
You can be notified before each polling attempt and before each
delay. If you throw :success
or :failure
from these callbacks,
it will terminate the waiter.
started_at = Time.now
client.wait_until(waiter_name, params, {
# disable max attempts
max_attempts: nil,
# poll for 1 hour, instead of a number of attempts
before_wait: -> (attempts, response) do
throw :failure if Time.now - started_at > 3600
end
})
Handling Errors
When a waiter is unsuccessful, it will raise an error. All of the failure errors extend from Waiters::Errors::WaiterFailed.
begin
client.wait_until(...)
rescue Aws::Waiters::Errors::WaiterFailed
# resource did not enter the desired state in time
end
Valid Waiters
The following table lists the valid waiter names, the operations they call,
and the default :delay
and :max_attempts
values.
waiter_name | params | :delay | :max_attempts |
---|---|---|---|
cluster_running | #describe_cluster | 30 | 60 |
cluster_terminated | #describe_cluster | 30 | 60 |
step_complete | #describe_step | 30 | 60 |
3529 3530 3531 3532 3533 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3529 def wait_until(waiter_name, params = {}, = {}) w = waiter(waiter_name, ) yield(w.waiter) if block_given? # deprecated w.wait(params) end |