Using Scaling Profiles
With Amazon EMR Serverless, you can choose how quickly your Apache Spark jobs add and release
executors. You set one configuration property, spark.emr-serverless.scalingProfile,
instead of tuning individual Spark dynamic allocation properties. The property takes one of
three profiles:
-
performance-optimized– Adds executors as quickly as possible, for the shortest run time. -
balanced– Adds executors more slowly and uses fewer of them. -
cost-optimized– Adds the fewest executors, for the lowest cost.
A scaling profile is a named set of spark.dynamicAllocation.* values.
EMR Serverless writes the values of the profile that you choose into the job's Spark
configuration when the job starts. Any spark.dynamicAllocation.* property that you
set yourself keeps your value. Starting with release emr-spark-8.2.0, Spark jobs
and sessions run with the balanced profile unless you choose a different one. For
details, refer to How EMR Serverless chooses the profile.
Key benefits
Scaling Profiles provide the following benefits.
-
One property instead of six – A profile sets up to six
spark.dynamicAllocation.*properties together, so you change scaling behavior for a job or an application by changing one value. -
Cost control for batch jobs – The
balancedandcost-optimizedprofiles wait longer between executor requests. They also aim for half or a third of the executors that the job's tasks could use. As a result, a job with short bursts of waiting tasks runs on fewer executors and accrues less vCPU and memory usage. -
Latency control for time-sensitive jobs – The
performance-optimizedprofile requests executors 1 second after tasks start to wait. Use it for jobs where run time matters more than cost. It keeps the scaling behavior of releases earlier thanemr-spark-8.2.0. -
Your settings take precedence – EMR Serverless writes a profile value only for a property that you didn't set, so existing tuning in your jobs and applications keeps working.
Available profiles
EMR Serverless provides three profiles. Each profile trades job run time against the
number of executors that the job uses. The following table lists what each property controls
and the value that each profile writes. performance-optimized writes the Apache
Spark defaults. A dash (–) means that the profile doesn't write the property, so the
property keeps its EMR Serverless default. For the defaults, refer to Spark job properties.
| Property | What it controls | performance-optimized |
balanced |
cost-optimized |
|---|---|---|---|---|
spark.dynamicAllocation.schedulerBacklogTimeout |
How long tasks wait before the job requests its first executors | 1s |
5s |
10s |
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout |
How often the job requests more executors while tasks still wait | 1s |
15s |
30s |
spark.dynamicAllocation.executorAllocationRatio |
The share of executors that the job aims for, out of the executors needed to run all running and waiting tasks at once | 1.0 |
0.5 |
0.33 |
spark.dynamicAllocation.executorIdleTimeout |
How long an executor stays idle before the job releases it | 60s |
60s |
30s |
spark.dynamicAllocation.cachedExecutorIdleTimeout |
How long an executor that holds cached data stays idle before the job releases it | – | 120s |
60s |
spark.dynamicAllocation.maxExecutors |
The most executors that the job can use | – | – | 100 |
How EMR Serverless chooses the profile
EMR Serverless resolves one profile for each job run or session when it starts. It uses the property value that you supply and the release label of the application.
-
You set the property – The job runs with the profile that you set, on any release label.
-
You don't set the property, with release
emr-spark-8.2.0or higher – The job runs with thebalancedprofile. -
You don't set the property, with a release earlier than
emr-spark-8.2.0– EMR Serverless writes no profile values. The job runs with the Apache Spark dynamic allocation defaults, which are the same values thatperformance-optimizedwrites. Existing jobs on these releases don't change behavior. -
You set
spark.dynamicAllocation.enabledtofalse– EMR Serverless applies no profile, because the job doesn't scale its executors.
A profile that you set on the application applies to every job run and session on that application. When you set the property in more than one place, the setting closest to the job run takes precedence:
-
--confin the job'ssparkSubmitParametersoverrides both of the other settings. -
The
spark-defaultsclassification in the job's or session'sconfigurationOverridesoverrides the application's setting. -
The
spark-defaultsclassification in the application'sruntimeConfigurationapplies only when the job run or session sets neither of the above.
For example, if the application sets balanced and the job's
configurationOverrides sets cost-optimized, the job runs with
cost-optimized. For the full precedence order of Spark properties, refer to
Using Spark configurations when you run EMR Serverless jobs.
After EMR Serverless resolves the profile, it applies the profile one property at a
time. For each property in the profile, EMR Serverless writes the profile value only if you
didn't set that property at any level. For example, if you choose
cost-optimized and also set
spark.dynamicAllocation.maxExecutors=500, the job runs with the
cost-optimized timeouts and ratio, and with 500 as the executor cap.
Setting a scaling profile
You can set a scaling profile for every job on an application, for one job run, or for
one session. In each case, you set spark.emr-serverless.scalingProfile to
performance-optimized, balanced, or cost-optimized.
EMR Serverless ignores letter case and leading or trailing spaces in the value.
Setting a profile for an application
The following example creates a Spark application on which every job run and session
uses the cost-optimized profile, unless the job sets a different profile. The
--runtime-configuration parameter carries the profile. The other parameters
are the same as for creating any other application.
aws emr-serverless create-application \ --type SPARK \ --namemy-application-name\ --release-label emr-spark-8.2.0 \ --runtime-configuration '[ { "classification": "spark-defaults", "properties": { "spark.emr-serverless.scalingProfile": "cost-optimized" } } ]' \ --regionregion
To change the profile of an existing application, update the application while it's in
the CREATED or STOPPED state. If the application is
STARTED, update-application rejects the change with a
ValidationException, so stop the application first. The
--runtime-configuration that you send replaces the application's whole
runtime configuration. Include every classification and property that the application
needs, not only the scaling profile. Job runs and sessions that start after the update use
the new profile.
aws emr-serverless update-application \ --application-idapplication-id\ --runtime-configuration '[ { "classification": "spark-defaults", "properties": { "spark.emr-serverless.scalingProfile": "performance-optimized" } } ]' \ --regionregion
Setting a profile for one job run
The following example runs one job with the performance-optimized profile
by passing the property with --conf in sparkSubmitParameters. This
value overrides any profile set on the application.
aws emr-serverless start-job-run \ --application-idapplication-id\ --execution-role-arnjob-role-arn\ --job-driver '{ "sparkSubmit": { "entryPoint": "s3://amzn-s3-demo-bucket/scripts/my-script.py", "sparkSubmitParameters": "--conf spark.emr-serverless.scalingProfile=performance-optimized" } }' \ --regionregion
You can also set the property in the spark-defaults classification of
applicationConfiguration in the job's
--configuration-overrides.
aws emr-serverless start-job-run \ --application-idapplication-id\ --execution-role-arnjob-role-arn\ --job-driver '{ "sparkSubmit": { "entryPoint": "s3://amzn-s3-demo-bucket/scripts/my-script.py" } }' \ --configuration-overrides '{ "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.emr-serverless.scalingProfile": "balanced" } } ] }' \ --regionregion
Setting a profile for a Spark Connect session
The following example starts a Spark Connect session with the
cost-optimized profile. Sessions take the property in
runtimeConfiguration inside --configuration-overrides. The
application must have interactive sessions turned on. To turn them on, pass
--interactive-configuration '{"sessionEnabled": true}' to
create-application or update-application.
aws emr-serverless start-session \ --application-idapplication-id\ --execution-role-arnexecution-role-arn\ --configuration-overrides '{ "runtimeConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.emr-serverless.scalingProfile": "cost-optimized" } } ] }' \ --regionregion
Setting a profile for a Livy or notebook session
With the Livy endpoint, pass the property in the conf map of the request
that creates the session. The application must have the Livy endpoint turned on
(--interactive-configuration '{"livyEndpointEnabled": true}'). The following
example is the body of a POST /sessions request.
{ "kind": "pyspark", "conf": { "emr-serverless.session.executionRoleArn": "execution-role-arn", "spark.emr-serverless.scalingProfile": "cost-optimized" } }
In a Jupyter notebook that connects through Livy, the %%configure magic
sends the same conf map when it creates the session.
%%configure -f { "conf": { "spark.emr-serverless.scalingProfile": "cost-optimized" } }
Checking the applied profile
You can check which profile a job ran with in the Spark UI. When EMR Serverless applies
a profile, it passes spark.emr-serverless.scalingProfile with the resolved
profile name to the Spark driver, together with the spark.dynamicAllocation.*
values that took effect. To see them, open the Spark UI for the job run or session and choose
the Environment tab. The Spark History Server shows the same values for
completed jobs.
The GetJobRun, GetSession, and GetApplication API
operations return only the configuration that you sent. A profile that comes from the release
default doesn't appear in their responses. For more information, refer to How EMR Serverless chooses the profile.
Considerations and limitations
-
Spark only – Scaling Profiles apply to Spark job runs and sessions. The property has no effect on Hive applications.
-
The default changes with
emr-spark-8.2.0– When you upgrade an application that doesn't set the property toemr-spark-8.2.0or higher, its job runs and sessions change from the earlier dynamic allocation defaults to thebalancedprofile. They can run longer on fewer executors. To keep the earlier behavior, set the profile toperformance-optimizedon the application or job. -
Profiles require dynamic allocation – A profile changes only
spark.dynamicAllocation.*properties. If you setspark.dynamicAllocation.enabledtofalse, EMR Serverless applies no profile, and the job runs with the executor count that you configure. -
cost-optimizedcaps the job at 100 executors – A job that needs more than 100 executors to meet its run-time target runs longer withcost-optimized. To raise the cap, setspark.dynamicAllocation.maxExecutorsyourself, or choose another profile. -
balancedandcost-optimizedrelease executors that hold cached data – Apache Spark keeps an executor that holds cached data until the job ends, unlessspark.dynamicAllocation.cachedExecutorIdleTimeoutis set. These two profiles set it to 120 and 60 seconds, so a job that reads cached data after a longer idle period recomputes that data. To keep cached executors longer, setspark.dynamicAllocation.cachedExecutorIdleTimeoutyourself. -
Application capacity limits still apply – A profile doesn't change the application's
maximumCapacity. When the application reaches its maximum capacity, the job can't add executors, regardless of the profile. -
The profile is resolved when the job starts – A change to the application's profile applies to job runs and sessions that start after the change.
Troubleshooting
The following are symptoms that you might see when you use Scaling Profiles, and what to do about each.
The request fails with a ValidationException about scalingProfile
This happens when the property value isn't one of the three profile names. The error
message lists the valid values, for example Invalid value 'fast' for scalingProfile;
valid values are performance-optimized, balanced, cost-optimized. Correct the value
in the application, job, or session configuration, and retry the request.
The Spark UI doesn't show spark.emr-serverless.scalingProfile
This happens when EMR Serverless applies no profile to the job. Check whether the job
sets spark.dynamicAllocation.enabled to false, or whether the
application runs a release earlier than emr-spark-8.2.0 without the property.
For more information, refer to How EMR Serverless chooses the profile.
A dynamic allocation value differs from the profile table
This happens when you set that spark.dynamicAllocation.* property yourself
at the application, job, or session level. Your value takes precedence over the profile
value. To use the profile value, remove your setting.
A job runs longer after an upgrade to emr-spark-8.2.0
This happens when the job doesn't set the property, so it moves to the
balanced default. That profile waits longer between executor requests and
aims for fewer executors. Set the profile to performance-optimized for that job
or application, as described in Setting a scaling profile.