Amazon SageMaker
Developer Guide

Tuning a RCF Model

Automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many jobs that test a range of hyperparameters on your dataset. You choose the tunable hyperparameters, a range of values for each, and an objective metric. You choose the objective metric from the metrics that the algorithm computes. Automatic model tuning searches the hyperparameters chosen to find the combination of values that result in the model that optimizes the objective metric.

The Amazon SageMaker RCF algorithm is an unsupervised anomaly-detection algorithm that requires a labeled test dataset for hyperparameter optimization. It calculates anomaly scores for test datapoints and then labels the datapoints as anomalous if their scores are beyond three standard deviations from the mean score. This is known as the three-sigma limit heuristic. The F1 score is emitted based on the difference between calculated labels and actual labels. The hyperparameter tuning job finds the model that maximizes that score. The success of hyperparameter optimization depends on the applicability of the three-sigma limit heuristic to the test dataset.

For more information about model tuning, see Automatic Model Tuning.

Metrics Computed by the RCF Algorithm

The RCF algorithm computes the following metric during training. When tuning the model, choose this metric as the objective metric.

Metric Name Description Optimization Direction
test:f1

F1 score on the test dataset, based on the difference between calculated labels and actual labels.

Maximize

Tunable Hyperparameters

You can tune a RCF model with the following hyperparameters.

Parameter Name Parameter Type Recommended Ranges
num_samples_per_tree

IntegerParameterRanges

MinValue: 1, MaxValue:2048

num_trees

IntegerParameterRanges

MinValue: 50, MaxValue:1000