Class: Aws::EMR::Client
- Inherits:
-
Seahorse::Client::Base
- Object
- Seahorse::Client::Base
- Aws::EMR::Client
- Includes:
- ClientStubs
- Defined in:
- gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb
Overview
An API client for EMR. To construct a client, you need to configure a :region
and :credentials
.
client = Aws::EMR::Client.new(
region: region_name,
credentials: credentials,
# ...
)
For details on configuring region and credentials see the developer guide.
See #initialize for a full list of supported configuration options.
Instance Attribute Summary
Attributes inherited from Seahorse::Client::Base
API Operations collapse
-
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
-
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
-
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster.
-
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource.
-
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster.
-
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
-
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
Creates a new Amazon EMR Studio.
-
#create_studio_session_mapping(params = {}) ⇒ Struct
Maps a user or group to the Amazon EMR Studio specified by
StudioId
, and applies a session policy to refine Studio permissions for that user or group. -
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
-
#delete_studio(params = {}) ⇒ Struct
Removes an Amazon EMR Studio from the Studio metadata store.
-
#delete_studio_session_mapping(params = {}) ⇒ Struct
Removes a user or group from an Amazon EMR Studio.
-
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
-
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed.
-
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
-
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
-
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
-
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
Returns details for the specified Amazon EMR Studio including ID, Name, VPC, Studio access URL, and so on.
-
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your AWS account in the current Region.
-
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
-
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
Fetches mapping details for the specified Amazon EMR Studio and identity (user or group).
-
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
-
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this AWS account.
-
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
-
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
-
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active EC2 instances and EC2 instances terminated in the last 30 days, up to a maximum of 2,000.
-
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions.
-
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names.
-
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you specify
stepIds
with the request of filter byStepStates
. -
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
Returns a list of all user or group session mappings for the Amazon EMR Studio specified by
StudioId
. -
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
Returns a list of all Amazon EMR Studios associated with the AWS account.
-
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
-
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID.
-
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group.
-
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster.
-
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your AWS account in the current Region.
-
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster.
-
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an EMR cluster.
-
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified EMR cluster.
-
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource.
-
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow).
-
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the EC2 instances in the cluster cannot be terminated by user intervention, an API call, or in the event of a job-flow error.
-
#set_visible_to_all_users(params = {}) ⇒ Struct
Sets the Cluster$VisibleToAllUsers value, which determines whether the cluster is visible to all IAM users of the AWS account associated with the cluster.
-
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
-
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
-
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down.
-
#update_studio(params = {}) ⇒ Struct
Updates an Amazon EMR Studio configuration, including attributes such as name, description, and subnets.
-
#update_studio_session_mapping(params = {}) ⇒ Struct
Updates the session policy attached to the user or group for the specified Amazon EMR Studio.
Instance Method Summary collapse
-
#initialize(options) ⇒ Client
constructor
A new instance of Client.
-
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
Methods included from ClientStubs
#api_requests, #stub_data, #stub_responses
Methods inherited from Seahorse::Client::Base
add_plugin, api, clear_plugins, define, new, #operation_names, plugins, remove_plugin, set_api, set_plugins
Methods included from Seahorse::Client::HandlerBuilder
#handle, #handle_request, #handle_response
Constructor Details
#initialize(options) ⇒ Client
Returns a new instance of Client.
334 335 336 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 334 def initialize(*args) super end |
Instance Method Details
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
424 425 426 427 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 424 def add_instance_fleet(params = {}, = {}) req = build_request(:add_instance_fleet, params) req.send_request() end |
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
532 533 534 535 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 532 def add_instance_groups(params = {}, = {}) req = build_request(:add_instance_groups, params) req.send_request() end |
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster. A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using SSH to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. For more information on how to do this, see Add More than 256 Steps to a Cluster in the Amazon EMR Management Guide.
A step specifies the location of a JAR file stored either on the master node of the cluster or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step.
Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully.
You can only add steps to a cluster that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.
609 610 611 612 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 609 def add_job_flow_steps(params = {}, = {}) req = build_request(:add_job_flow_steps, params) req.send_request() end |
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
651 652 653 654 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 651 def (params = {}, = {}) req = build_request(:add_tags, params) req.send_request() end |
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster. Available only
in Amazon EMR versions 4.8.0 and later, excluding version 5.0.0. A
maximum of 256 steps are allowed in each CancelSteps request.
CancelSteps is idempotent but asynchronous; it does not guarantee that
a step will be canceled, even if the request is successfully
submitted. You can only cancel steps that are in a PENDING
state.
698 699 700 701 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 698 def cancel_steps(params = {}, = {}) req = build_request(:cancel_steps, params) req.send_request() end |
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
739 740 741 742 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 739 def create_security_configuration(params = {}, = {}) req = build_request(:create_security_configuration, params) req.send_request() end |
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
Creates a new Amazon EMR Studio.
834 835 836 837 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 834 def create_studio(params = {}, = {}) req = build_request(:create_studio, params) req.send_request() end |
#create_studio_session_mapping(params = {}) ⇒ Struct
Maps a user or group to the Amazon EMR Studio specified by StudioId
,
and applies a session policy to refine Studio permissions for that
user or group.
893 894 895 896 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 893 def create_studio_session_mapping(params = {}, = {}) req = build_request(:create_studio_session_mapping, params) req.send_request() end |
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
915 916 917 918 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 915 def delete_security_configuration(params = {}, = {}) req = build_request(:delete_security_configuration, params) req.send_request() end |
#delete_studio(params = {}) ⇒ Struct
Removes an Amazon EMR Studio from the Studio metadata store.
937 938 939 940 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 937 def delete_studio(params = {}, = {}) req = build_request(:delete_studio, params) req.send_request() end |
#delete_studio_session_mapping(params = {}) ⇒ Struct
Removes a user or group from an Amazon EMR Studio.
988 989 990 991 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 988 def delete_studio_session_mapping(params = {}, = {}) req = build_request(:delete_studio_session_mapping, params) req.send_request() end |
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
The following waiters are defined for this operation (see #wait_until for detailed usage):
- cluster_running
- cluster_terminated
1089 1090 1091 1092 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1089 def describe_cluster(params = {}, = {}) req = build_request(:describe_cluster, params) req.send_request() end |
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed. We recommend you use ListClusters, DescribeCluster, ListSteps, ListInstanceGroups and ListBootstrapActions instead.
DescribeJobFlows returns a list of job flows that match all of the supplied parameters. The parameters can include a list of job flow IDs, job flow states, and restrictions on job flow creation date and time.
Regardless of supplied parameters, only job flows created within the last two months are returned.
If no parameters are supplied, then job flows matching either of the following criteria are returned:
Job flows created and completed in the last two weeks
Job flows created within the last two months that are in one of the following states:
RUNNING
,WAITING
,SHUTTING_DOWN
,STARTING
Amazon EMR can return a maximum of 512 job flow descriptions.
1216 1217 1218 1219 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1216 def describe_job_flows(params = {}, = {}) req = build_request(:describe_job_flows, params) req.send_request() end |
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
1260 1261 1262 1263 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1260 def describe_notebook_execution(params = {}, = {}) req = build_request(:describe_notebook_execution, params) req.send_request() end |
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
1293 1294 1295 1296 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1293 def describe_security_configuration(params = {}, = {}) req = build_request(:describe_security_configuration, params) req.send_request() end |
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
The following waiters are defined for this operation (see #wait_until for detailed usage):
- step_complete
1347 1348 1349 1350 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1347 def describe_step(params = {}, = {}) req = build_request(:describe_step, params) req.send_request() end |
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
Returns details for the specified Amazon EMR Studio including ID, Name, VPC, Studio access URL, and so on.
1393 1394 1395 1396 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1393 def describe_studio(params = {}, = {}) req = build_request(:describe_studio, params) req.send_request() end |
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your AWS account in the current Region. For more information see Configure Block Public Access for Amazon EMR in the Amazon EMR Management Guide.
1425 1426 1427 1428 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1425 def get_block_public_access_configuration(params = {}, = {}) req = build_request(:get_block_public_access_configuration, params) req.send_request() end |
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
1458 1459 1460 1461 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1458 def get_managed_scaling_policy(params = {}, = {}) req = build_request(:get_managed_scaling_policy, params) req.send_request() end |
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
Fetches mapping details for the specified Amazon EMR Studio and identity (user or group).
1520 1521 1522 1523 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1520 def get_studio_session_mapping(params = {}, = {}) req = build_request(:get_studio_session_mapping, params) req.send_request() end |
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1562 1563 1564 1565 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1562 def list_bootstrap_actions(params = {}, = {}) req = build_request(:list_bootstrap_actions, params) req.send_request() end |
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this AWS account. Allows you to filter the list of clusters based on certain criteria; for example, filtering by cluster creation date and time or by status. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListClusters calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1624 1625 1626 1627 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1624 def list_clusters(params = {}, = {}) req = build_request(:list_clusters, params) req.send_request() end |
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1700 1701 1702 1703 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1700 def list_instance_fleets(params = {}, = {}) req = build_request(:list_instance_fleets, params) req.send_request() end |
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1798 1799 1800 1801 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1798 def list_instance_groups(params = {}, = {}) req = build_request(:list_instance_groups, params) req.send_request() end |
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active EC2 instances and EC2 instances terminated in the last 30 days, up to a maximum of 2,000. EC2 instances in any of the following states are considered active: AWAITING_FULFILLMENT, PROVISIONING, BOOTSTRAPPING, RUNNING.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1879 1880 1881 1882 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1879 def list_instances(params = {}, = {}) req = build_request(:list_instances, params) req.send_request() end |
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions. You can filter the list
based on multiple criteria such as status, time range, and editor id.
Returns a maximum of 50 notebook executions and a marker to track the
paging of a longer notebook execution list across multiple
ListNotebookExecution
calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1968 1969 1970 1971 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1968 def list_notebook_executions(params = {}, = {}) req = build_request(:list_notebook_executions, params) req.send_request() end |
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListSecurityConfigurations calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2006 2007 2008 2009 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2006 def list_security_configurations(params = {}, = {}) req = build_request(:list_security_configurations, params) req.send_request() end |
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you
specify stepIds
with the request of filter by StepStates
. You can
specify a maximum of 10 stepIDs
.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2073 2074 2075 2076 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2073 def list_steps(params = {}, = {}) req = build_request(:list_steps, params) req.send_request() end |
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
Returns a list of all user or group session mappings for the Amazon
EMR Studio specified by StudioId
.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2122 2123 2124 2125 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2122 def list_studio_session_mappings(params = {}, = {}) req = build_request(:list_studio_session_mappings, params) req.send_request() end |
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
Returns a list of all Amazon EMR Studios associated with the AWS account. The list includes details such as ID, Studio Access URL, and creation time for each Studio.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2162 2163 2164 2165 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2162 def list_studios(params = {}, = {}) req = build_request(:list_studios, params) req.send_request() end |
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
2196 2197 2198 2199 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2196 def modify_cluster(params = {}, = {}) req = build_request(:modify_cluster, params) req.send_request() end |
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID. The call either succeeds or fails atomically.
2234 2235 2236 2237 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2234 def modify_instance_fleet(params = {}, = {}) req = build_request(:modify_instance_fleet, params) req.send_request() end |
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group. The input parameters include the new target instance count for the group and the instance group ID. The call will either succeed or fail atomically.
2288 2289 2290 2291 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2288 def modify_instance_groups(params = {}, = {}) req = build_request(:modify_instance_groups, params) req.send_request() end |
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. The automatic scaling policy defines how an instance group dynamically adds and terminates EC2 instances in response to the value of a CloudWatch metric.
2395 2396 2397 2398 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2395 def put_auto_scaling_policy(params = {}, = {}) req = build_request(:put_auto_scaling_policy, params) req.send_request() end |
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your AWS account in the current Region. For more information see Configure Block Public Access for Amazon EMR in the Amazon EMR Management Guide.
2450 2451 2452 2453 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2450 def put_block_public_access_configuration(params = {}, = {}) req = build_request(:put_block_public_access_configuration, params) req.send_request() end |
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster. The managed scaling policy defines the limits for resources, such as EC2 instances that can be added or terminated from a cluster. The policy only applies to the core and task nodes. The master node cannot be scaled after initial configuration.
2489 2490 2491 2492 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2489 def put_managed_scaling_policy(params = {}, = {}) req = build_request(:put_managed_scaling_policy, params) req.send_request() end |
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an EMR cluster.
2518 2519 2520 2521 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2518 def remove_auto_scaling_policy(params = {}, = {}) req = build_request(:remove_auto_scaling_policy, params) req.send_request() end |
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified EMR cluster.
2541 2542 2543 2544 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2541 def remove_managed_scaling_policy(params = {}, = {}) req = build_request(:remove_managed_scaling_policy, params) req.send_request() end |
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
The following example removes the stack tag with value Prod from a cluster:
2578 2579 2580 2581 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2578 def (params = {}, = {}) req = build_request(:remove_tags, params) req.send_request() end |
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow). The
cluster runs the steps specified. After the steps complete, the
cluster stops and the HDFS partition is lost. To prevent loss of data,
configure the last step of the job flow to store results in Amazon S3.
If the JobFlowInstancesConfig KeepJobFlowAliveWhenNoSteps
parameter
is set to TRUE
, the cluster transitions to the WAITING state rather
than shutting down after the steps have completed.
For additional protection, you can set the JobFlowInstancesConfig
TerminationProtected
parameter to TRUE
to lock the cluster and
prevent it from being terminated by API call, user intervention, or in
the event of a job flow error.
A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. For more information on how to do this, see Add More than 256 Steps to a Cluster in the Amazon EMR Management Guide.
For long running clusters, we recommend that you periodically store your results.
3092 3093 3094 3095 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3092 def run_job_flow(params = {}, = {}) req = build_request(:run_job_flow, params) req.send_request() end |
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the EC2
instances in the cluster cannot be terminated by user intervention, an
API call, or in the event of a job-flow error. The cluster still
terminates upon successful completion of the job flow. Calling
SetTerminationProtection
on a cluster is similar to calling the
Amazon EC2 DisableAPITermination
API on all EC2 instances in a
cluster.
SetTerminationProtection
is used to prevent accidental termination
of a cluster and to ensure that in the event of an error, the
instances persist so that you can recover any data stored in their
ephemeral instance storage.
To terminate a cluster that has been locked by setting
SetTerminationProtection
to true
, you must first unlock the job
flow by a subsequent call to SetTerminationProtection
in which you
set the value to false
.
For more information, seeManaging Cluster Termination in the Amazon EMR Management Guide.
3145 3146 3147 3148 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3145 def set_termination_protection(params = {}, = {}) req = build_request(:set_termination_protection, params) req.send_request() end |
#set_visible_to_all_users(params = {}) ⇒ Struct
Sets the Cluster$VisibleToAllUsers value, which determines whether the
cluster is visible to all IAM users of the AWS account associated with
the cluster. Only the IAM user who created the cluster or the AWS
account root user can call this action. The default value, true
,
indicates that all IAM users in the AWS account can perform cluster
actions if they have the proper IAM policy permissions. If set to
false
, only the IAM user that created the cluster can perform
actions. This action works on running clusters. You can override the
default true
setting when you create a cluster by using the
VisibleToAllUsers
parameter with RunJobFlow
.
3183 3184 3185 3186 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3183 def set_visible_to_all_users(params = {}, = {}) req = build_request(:set_visible_to_all_users, params) req.send_request() end |
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
3264 3265 3266 3267 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3264 def start_notebook_execution(params = {}, = {}) req = build_request(:start_notebook_execution, params) req.send_request() end |
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
3286 3287 3288 3289 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3286 def stop_notebook_execution(params = {}, = {}) req = build_request(:stop_notebook_execution, params) req.send_request() end |
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down. When a job flow is shut down, any step not yet completed is canceled and the EC2 instances on which the cluster is running are stopped. Any log files not already saved are uploaded to Amazon S3 if a LogUri was specified when the cluster was created.
The maximum number of clusters allowed is 10. The call to
TerminateJobFlows
is asynchronous. Depending on the configuration of
the cluster, it may take up to 1-5 minutes for the cluster to
completely terminate and release allocated resources, such as Amazon
EC2 instances.
3318 3319 3320 3321 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3318 def terminate_job_flows(params = {}, = {}) req = build_request(:terminate_job_flows, params) req.send_request() end |
#update_studio(params = {}) ⇒ Struct
Updates an Amazon EMR Studio configuration, including attributes such as name, description, and subnets.
3363 3364 3365 3366 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3363 def update_studio(params = {}, = {}) req = build_request(:update_studio, params) req.send_request() end |
#update_studio_session_mapping(params = {}) ⇒ Struct
Updates the session policy attached to the user or group for the specified Amazon EMR Studio.
3418 3419 3420 3421 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3418 def update_studio_session_mapping(params = {}, = {}) req = build_request(:update_studio_session_mapping, params) req.send_request() end |
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
Basic Usage
A waiter will call an API operation until:
- It is successful
- It enters a terminal state
- It makes the maximum number of attempts
In between attempts, the waiter will sleep.
# polls in a loop, sleeping between attempts
client.wait_until(waiter_name, params)
Configuration
You can configure the maximum number of polling attempts, and the delay (in seconds) between each polling attempt. You can pass configuration as the final arguments hash.
# poll for ~25 seconds
client.wait_until(waiter_name, params, {
max_attempts: 5,
delay: 5,
})
Callbacks
You can be notified before each polling attempt and before each
delay. If you throw :success
or :failure
from these callbacks,
it will terminate the waiter.
started_at = Time.now
client.wait_until(waiter_name, params, {
# disable max attempts
max_attempts: nil,
# poll for 1 hour, instead of a number of attempts
before_wait: -> (attempts, response) do
throw :failure if Time.now - started_at > 3600
end
})
Handling Errors
When a waiter is unsuccessful, it will raise an error. All of the failure errors extend from Waiters::Errors::WaiterFailed.
begin
client.wait_until(...)
rescue Aws::Waiters::Errors::WaiterFailed
# resource did not enter the desired state in time
end
Valid Waiters
The following table lists the valid waiter names, the operations they call,
and the default :delay
and :max_attempts
values.
waiter_name | params | :delay | :max_attempts |
---|---|---|---|
cluster_running | #describe_cluster | 30 | 60 |
cluster_terminated | #describe_cluster | 30 | 60 |
step_complete | #describe_step | 30 | 60 |
3530 3531 3532 3533 3534 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3530 def wait_until(waiter_name, params = {}, = {}) w = waiter(waiter_name, ) yield(w.waiter) if block_given? # deprecated w.wait(params) end |