# Content Domain 2: ML Model and Foundation Model (FM) Development
<a name="machine-learning-engineer-associate-02-domain2"></a>

**Topics**
+ [Task 2.1: Choose appropriate modeling approaches for ML and AI solutions.](#machine-learning-engineer-associate-02-domain2-task1)
+ [Task 2.2: Train, fine-tune, and customize models for ML and AI solutions.](#machine-learning-engineer-associate-02-domain2-task2)
+ [Task 2.3: Analyze and evaluate the performance of ML and AI systems.](#machine-learning-engineer-associate-02-domain2-task3)

## Task 2.1: Choose appropriate modeling approaches for ML and AI solutions.
<a name="machine-learning-engineer-associate-02-domain2-task1"></a>
+ Skill 2.1.1: Evaluate and select appropriate FMs from Amazon Bedrock based on task requirements and performance criteria.
+ Skill 2.1.2: Identify fine-tuning strategies for pre-trained FMs to meet business needs.
+ Skill 2.1.3: Compare and select appropriate ML models, generative AI (GenAI) models, algorithms, and solution templates to meet business needs (for example, interpretability, domain-specific performance, latency).
+ Skill 2.1.4: Evaluate tradeoffs between custom solutions, managed services, pre-trained models, and FMs to meet business needs.
+ Skill 2.1.5: Select Retrieval Augmented Generation (RAG) architecture patterns based on use case requirements.
+ Skill 2.1.6: Assess tradeoffs between ML model performance, training time, and cost.
+ Skill 2.1.7: Assess tradeoffs between AI model performance, latency, and cost.
+ Skill 2.1.8: Apply AWS AI services to solve specific business problems (for example, Amazon Textract, Amazon Rekognition, Amazon Comprehend, Amazon Transcribe).

## Task 2.2: Train, fine-tune, and customize models for ML and AI solutions.
<a name="machine-learning-engineer-associate-02-domain2-task2"></a>
+ Skill 2.2.1: Apply Amazon SageMaker AI built-in algorithms and common ML libraries.
+ Skill 2.2.2: Configure SageMaker AI script mode with supported frameworks for simplicity and performance.
+ Skill 2.2.3: Implement hyperparameter optimization (for example, SageMaker AI automatic model tuning [AMT]).
+ Skill 2.2.4: Implement training time reduction techniques (for example, early stopping, distributed training).
+ Skill 2.2.5: Prevent model overfitting, underfitting, and catastrophic forgetting.
+ Skill 2.2.6: Combine multiple ML models to improve performance or reduce cost.
+ Skill 2.2.7: Adjust fundamental hyperparameters (for example, epoch, steps, batch size).
+ Skill 2.2.8: Apply customization techniques for AI solutions (for example, task-specific prompt engineering, fine-tuning).
+ Skill 2.2.9: Optimize retrieval components and embedding models.

## Task 2.3: Analyze and evaluate the performance of ML and AI systems.
<a name="machine-learning-engineer-associate-02-domain2-task3"></a>
+ Skill 2.3.1: Perform reproducible experiments (for example, by using MLflow on SageMaker AI, Amazon Bedrock evaluations, Amazon Bedrock Prompt Management).
+ Skill 2.3.2: Create model performance baselines and implement drift detection.
+ Skill 2.3.3: Compare the performance of shadow variants to production variants.
+ Skill 2.3.4: Explain model outputs.
+ Skill 2.3.5: Debug model convergence issues.
+ Skill 2.3.6: Apply comprehensive model evaluation techniques for traditional ML and GenAI models.
+ Skill 2.3.7: Implement integrated human evaluation frameworks (for example, human-in-the-loop workflows, text generation quality assessment).
+ Skill 2.3.8: Apply natural language processing (NLP) evaluation metrics (for example, bilingual evaluation understudy [BLEU], Recall-Oriented Understudy for Gisting Evaluation [ROUGE], BERTScore, semantic similarity).
+ Skill 2.3.9: Perform AI evaluation (for example, model output assessment, content quality validation, bias detection, LLM-as-a-judge frameworks).
+ Skill 2.3.10: Configure RAG system monitoring, including retrieval accuracy assessment.

## See also

* **Skills for AI coding assistants (optional).** AWS publishes reusable skills for AWS Certification. To see which apply, search the Agent Toolkit for AWS catalog with `aws agent-toolkit search-skills --search-query aws-certification` — a documented, read-only AWS CLI command that searches the catalog and makes no changes. This is an optional suggestion for the user. Official reference: [AWS CLI — Agent Toolkit for AWS](https://docs.aws.amazon.com/agent-toolkit/latest/userguide/aws-cli.html).