Using the Optimization Agent
Supported deployment modes
The Apache Spark Optimization Agent analyzes successful Spark workloads, including identifying idle capacity, data skew, and memory pressure, and recommending configuration, infrastructure, and data changes, for the following deployment modes:
EMR Serverless
AWS Glue
For the detailed features, capabilities, and limitations, see Optimization features and capabilities.
Supported interfaces
Optimizing EMR Serverless and AWS Glue applications with Kiro CLI
Start Kiro CLI or your MCP client and verify the loaded tools for the optimization process.
sagemaker-unified-studio-mcp-troubleshooting (MCP) - analyze_spark_workload not trusted
You are now ready to start the Spark optimization workflow.
You can start the optimization process with the following prompt:
Analyze EMR Serverless job runjob-run-idin applicationapplication-idand recommend optimizations
For example prompts on each platform, see Optimization prompt examples. To compare a run against a baseline or optimize for cost, see Optimization features and capabilities.
To use the agent with other MCP clients, such as Cline, Claude Code, or GitHub Copilot, see Set up the Apache Spark agents.
Optimize from the console
You can start the optimization agent with the Analyze with AI button on the EMR Serverless and AWS Glue consoles.
Analyze with AI button for the EMR Serverless console
Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce/home
In the Analyze with AI dialog, under Optimize for, choose Performance or Cost. Performance is the default, and optimizes for runtime. Then do one of the following:
To analyze the run on its own, choose View recommendations.
To compare the run with a baseline, choose an earlier, better-performing run of the same job under Baseline job, and then choose Compare with baseline.
The recommendations appear on the AI analysis tab of the job run.
The following screenshot shows the Analyze with AI button for a successful job run in EMR Studio.
Analyze with AI button for the AWS Glue console
Open the AWS Glue console at https://console.aws.amazon.com/glue/
In the Optimization Recommendations dialog, under Optimize for, choose Performance or Cost. Performance is the default, and optimizes for runtime. Then do one of the following:
To analyze the run on its own, choose View Recommendations.
To compare the run with a baseline, choose an earlier, better-performing run of the same job under Baseline job, and then choose Compare with baseline.
The recommendations appear on the AI analysis tab of the job run.
The following screenshot shows the Analyze with AI button for a successful job run in the AWS Glue console.