Open the Amazon SageMaker Studio Notebook Keep Track of the Machine Learning Experiment Create a Chart to Visualize Data Debug the Training Job Deploy the Best Model Monitor the Deployed Model Clean Up Resources

Amazon SageMaker Studio Tour

This walkthrough takes you on a tour of the main features of Amazon SageMaker Studio using the xgboost_customer_churn_studio.ipynb sample notebook from the aws/amazon-sagemaker-examples repository. It is intended that you proceed through the walkthrough and run the notebook in Studio at the same time.

The code in the notebook trains multiple models and sets up the SageMaker Debugger and SageMaker Model Monitor. The walkthrough shows you how to view the trials, compare the resulting models, show the debugger results, and deploy the best model using the SageMaker Studio UI. You don't need to understand the code to follow this walkthrough.

For a series of videos that shows how to use the main features of SageMaker Studio, see NEW! Amazon SageMaker Studio on YouTube.

Prerequisites

To run the notebook for this tour, you need:

An AWS SSO or IAM account to sign in to Studio. For information, see Onboard to Amazon SageMaker Studio.
Basic familiarity with the Studio user interface and Jupyter notebooks. For information, see Amazon SageMaker Studio UI Overview.
A copy of the aws/amazon-sagemaker-examples repository in your Studio environment.

To clone the repository

Sign in to SageMaker Studio. For AWS SSO users, sign in using the URL from your invitation email. For IAM users, follow these steps.
1. Sign in to the SageMaker console.
2. Choose Amazon SageMaker Studio in the left navigation pane.
3. Choose Open Studio in the row next to your user name.
On the top menu, choose File then New then Terminal.
At the command prompt, run the following command.

git clone https://github.com/aws/amazon-sagemaker-examples.git

Note

If you encounter an error when you run the sample notebook, and some time has passed from when you cloned the repository, review the notebook on the remote repository for updates.

Topics

Open the Amazon SageMaker Studio Notebook
Keep Track of the Machine Learning Experiment
Create a Chart to Visualize Data
Debug the Training Job
Deploy the Best Model
Monitor the Deployed Model
Clean Up Resources

Open the Amazon SageMaker Studio Notebook

Amazon SageMaker Studio notebooks are collaborative Jupyter notebooks that are built into SageMaker Studio. You can launch Studio notebooks without setting up compute instances and file storage, so you can get started fast.

You can share notebooks with others in your organization, so that they can easily reproduce your results and collaborate while building models and exploring your data.

For more information about SageMaker Studio notebooks, see Use Amazon SageMaker Studio Notebooks.

To open the xgboost_customer_churn_studio notebook

Sign in to Studio. For more information, see Onboard to Amazon SageMaker Studio.
Choose the file browser icon ( ).
Navigate to amazon-sagemaker-examples/aws_sagemaker_studio/getting_started.
Double-click xgboost_customer_churn_studio.ipynb to open the notebook.
In the Select Kernel dialog, choose Python 3 (Data Science), then choose Select.

Your screen should resemble the following:

Next, you use the notebook to create an experiment.

Keep Track of the Machine Learning Experiment

Amazon SageMaker Experiments lets you organize, track, compare, and evaluate your machine learning experiments. An experiment is composed of multiple trials with the same objective. Each trial is composed of multiple trial components such as a preprocessing job and a training job.

First you create an experiment, then you create a trial which assigns it to the experiment. Next, you create a training job as a trial component and associate the component with the trial. For more information, see Manage Machine Learning with Amazon SageMaker Experiments.

To create an experiment and a trial with a training job

Scroll down the notebook and choose the section titled Amazon SageMaker Experiments.
In the Studio main menu, choose Run and then Run All Above Selected Cell.

Hold down the Shift key and press Enter to run the next code cell, which creates an experiment by calling the create method of the Experiment class.


sess = sagemaker.session.Session()

create_date = strftime("%Y-%m-%d-%H-%M-%S", gmtime())
customer_churn_experiment = Experiment.create(experiment_name="customer-churn-prediction-xgboost-{}".format(create_date),
                                              description="Using xgboost to predict customer churn",
                                              sagemaker_boto_client=boto3.client('sagemaker'))

In the left sidebar, choose the SageMaker Experiment List icon ( ) to see the experiment (named customer-churn-prediction-xgboost...) in the experiments list. You might need to refresh the list.

Run the next two code cells. The first cell defines the hyperparameters to use in the training job. The second cell creates a trial that is assigned to the experiment that was created in the previous step. Next, the cell creates a training job as a trial component, then runs the trial by calling the fit method of the Estimator class. It can take several minutes for the training job to complete.

Note

The output of the training job includes a long list of messages like [0]#011train-error:0.077154#011validation-error:0.099099. These aren't errors due to the training job but are the results from the model training process.


trial = Trial.create(trial_name="algorithm-mode-trial-{}".format(strftime("%Y-%m-%d-%H-%M-%S", gmtime())),
                     experiment_name=customer_churn_experiment.experiment_name,
                     sagemaker_boto_client=boto3.client('sagemaker'))

xgb = sagemaker.estimator.Estimator(image_name=docker_image_name,
                                    role=role,
                                    hyperparameters=hyperparams,
                                    train_instance_count=1, 
                                    train_instance_type='ml.m4.xlarge',
                                    output_path='s3://{}/{}/output'.format(bucket, prefix),
                                    base_job_name="demo-xgboost-customer-churn",
                                    sagemaker_session=sess)

xgb.fit({'train': s3_input_train,
         'validation': s3_input_validation}, 
         experiment_config={
            "ExperimentName": customer_churn_experiment.experiment_name, 
            "TrialName": trial.trial_name,
            "TrialComponentDisplayName": "Training",
        }
       )

In the experiments list, double-click the experiment name to see the trial (named algorithm-mode-trial...).
Double-click the trial name to see the associated trial component (named Training).
Double-click the Training trial component to open the Describe Trial Component tab. You can follow the progress of the training job here.

After the trial finishes, you can see details about the training job, such as metrics and hyperparameters, charts that visualize the training results. To see the billable time and instance type, choose the AWS Settings heading.

Next, the notebook creates and compares multiple trials that use different values for the min_child_weight hyperparameter.

To create and compare multiple trials

Scroll to the section of the notebook titled Trying other hyperparameter values.

Run the following cell that creates and runs five trials, each with a different value of the min_child_weight hyperparameter.

Note

In the previous step of creating a single trial, the output of the training job is displayed. Here, the output is suppressed as it would display about three thousand lines.


min_child_weights = [1, 2, 4, 8, 10]

for weight in min_child_weights:
    hyperparams["min_child_weight"] = weight
    trial = Trial.create(trial_name="algorithm-mode-trial-{}-weight-{}".format(strftime("%Y-%m-%d-%H-%M-%S", gmtime()), weight),
                         experiment_name=customer_churn_experiment.experiment_name,
                         sagemaker_boto_client=boto3.client('sagemaker'))

    t_xgb = sagemaker.estimator.Estimator(image_name=docker_image_name,
                                          role=role,
                                          hyperparameters=hyperparams,
                                          train_instance_count=1,
                                          train_instance_type='ml.m4.xlarge',
                                          output_path='s3://{}/{}/output'.format(bucket, prefix),
                                          base_job_name="demo-xgboost-customer-churn",
                                          sagemaker_session=sess)

    t_xgb.fit({'train': s3_input_train,
               'validation': s3_input_validation},
               wait=False,
               experiment_config={
                    "ExperimentName": customer_churn_experiment.experiment_name,
                    "TrialName": trial.trial_name,
                    "TrialComponentDisplayName": "Training",
                }
               )

To follow the progress and view the results in Studio, choose the Home icon above TRIAL COMPONENTS.
Right-click the experiment name and choose Open in trial component list. You can see details about the trials, compare trials to find the best performing model, and create charts to visualize training results.

After all the trials finish, sort the trials by choosing the validation:error heading. In a later section, you will deploy the trial with the lowest validation:error.

Create a Chart to Visualize Data

To visualize data after the training jobs run, you can create charts in Amazon SageMaker Studio. In this notebook, the training jobs run for a very short time, so they don't create much data. Because of this, you create a scatter plot of the validation:error_last metric (final validation error) for each of the min_child_weight hyperparameter values that were specified in the training jobs.

To create the scatter plot

In the TRIAL COMPONENTS list, multi-select the five trials from the previous step, then choose Add chart.
If the CHART PROPERTIES pane isn't open, choose the Settings icon ( ) in the upper right corner to open it. Choose the Settings icon again when you want to close the pane.
Configure the chart properties as follows:
- For Data type, choose Summary statistics.
- For Chart type, choose Scatter plot.
- For X-axis, choose min_child_weight.
- For Y-axis, choose validation:error_last.
- For Color, choose trialComponentName.

Studio displays the scatter plot.

Next, the notebook sets up the SageMaker Debugger.

Debug the Training Job

Amazon SageMaker Debugger helps you analyze your training jobs and find problems. It monitors, records, and analyzes tensor data from training jobs and checks the training tensors against a set of rules that you specify. You can choose from a list of built-in rules, or create your own custom rules. For more information, see Amazon SageMaker Debugger.

To debug a training job

To specify the rules to use to analyze your training job, run the following cell in the section titled Amazon SageMaker Debugger.


debug_rules = [Rule.sagemaker(rule_configs.loss_not_decreasing()),
               Rule.sagemaker(rule_configs.overtraining()),
               Rule.sagemaker(rule_configs.overfit())
              ]

Run the remaining cells in the section to create a new trial using the debug rules. Note the rules=debug_rules argument that is added to the fit call.


entry_point_script = "xgboost_customer_churn.py"

trial = Trial.create(...)

framework_xgb = sagemaker.xgboost.XGBoost(image_name=docker_image_name,
                                          entry_point=entry_point_script,
                                          role=role,
                                          framework_version="0.90-2",
                                          py_version="py3",
                                          hyperparameters=hyperparams,
                                          train_instance_count=1, 
                                          train_instance_type='ml.m4.xlarge',
                                          output_path='s3://{}/{}/output'.format(bucket, prefix),
                                          base_job_name="demo-xgboost-customer-churn",
                                          sagemaker_session=sess,
                                          rules=debug_rules
                                         )

framework_xgb.fit(...)

In the experiments list, double-click the experiment name to see the trials list.
In the trials list, right-click the debug trial (named framework-mode-trial...) and choose Open in trial component list.
In the trial components list, right-click the trial and choose Open in trial details.
To see the results for each debug rule that you specified, choose the Debugger heading.

Notice that the training job passed all three of the rules that were configured for the job. If Debugger had found issues, you could choose the rule in the list to see more information in the Debugger Details tab.

Next, you deploy the model.

Deploy the Best Model

You can create an endpoint and deploy a model using the SDK or the Amazon SageMaker Studio UI. The notebook shows you how to deploy using the SDK. For this tour, we show you how to deploy using the Studio UI. After you deploy the model, you can set up the SageMaker Model Monitor to monitor the endpoint.

To deploy a model using the Studio UI

In the experiments list, right-click the experiment and choose Open in trial component list.
To sort the trials, choose the validation:error heading.
Right-click the trial with the lowest validation:error and choose Deploy model.
Under Deploy model, specify a name for the endpoint that will host the model.
Choose Deploy model.

Next, the notebook sets up the SageMaker Model Monitor.

Monitor the Deployed Model

Monitor the quality of your deployed models with Amazon SageMaker Model Monitor. Model Monitor runs monitoring jobs on the endpoints where models are deployed. You can use its built-in monitoring capabilities, which don't require coding, or you can write code for custom analysis. For more information, see Amazon SageMaker Model Monitor.

The notebook first creates a processing job to generate baseline statistics. Next, it creates a monitoring schedule, with a polling time of one hour, that compares the recent data captures to the baseline.

To monitor the deployed model

Run all the code cells in the notebook sections titled Host the model and Amazon SageMaker Model Monitor. This can take some time. Note that this creates a different endpoint than the endpoint you created in the previous section.
In Studio, choose the SageMaker Endpoint List icon ( ).
Double-click the endpoint (named demo-xgboost-customer-churn...) that was created by the notebook.

In the Monitoring job history list, you can see any issues that the monitoring jobs found. To see details about an issue, choose the issue.

Important

You must shut down the kernel to stop monitoring. To shut down the kernel, from the top Studio menu, choose Kernel then Shut Down Kernel.

Clean Up Resources

To stop incurring charges, you should clean up the resources that were created.

To clean up the resources, follow the instructions in the notebook section titled Clean up.

Warning Javascript is disabled or is unavailable in your browser.

To use the AWS Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

SageMaker JumpStart

Get Started with Notebook Instances and SDKs