Anomaly detection

Anomaly detection in Amazon SageMaker Unified Studio identifies unexpected changes in your data by tracking statistics over time and flagging values that deviate from historical patterns. It works at both the dataset level (for example, sudden changes in row count) and the column level (for example, an unusual spike in null values for a specific column).

Anomaly detection operates on the statistics produced by data profile runs and data quality rule evaluations. As you accumulate profile and rule evaluation history, the system builds a baseline of expected behavior and highlights data points that fall outside that range.

Anomaly detection is available on the Data profile tab in the data catalog and on the Anomalies tab for data quality transform results in Visual ETL jobs.

For details about the underlying anomaly detection engine and algorithms, see Anomaly detection and dynamic rules in AWS Glue Data Quality in the AWS Glue documentation.

Viewing anomalies in trend charts

Anomalous data points are highlighted red on trend charts. When you hover over an anomalous point, a tooltip shows:

• The actual value

• The expected range

• An option to provide feedback on detected anomalies and classify the data point as Expected or Anomaly

When the system flags a data point as anomalous, you can provide feedback to improve future predictions. This feedback trains the anomaly detection model by telling it whether the flagged value was a true anomaly or a legitimate business change.

• Expected: Marks the flagged data point as a normal value. This is included in the model baseline, so similar values in the future are treated as normal.

• Anomaly: Marks the data point as a true anomaly. This is excluded from the model baseline, so similar values continue to be flagged as anomalous in future runs.

To mark a data point from the trend chart

1. On a trend chart, hover over the data point you want to provide feedback on.

2. In the tooltip, choose Expected or Anomaly.

3. A confirmation toast appears. The point is factored into or excluded from (based on your choice) future anomaly detection.

To mark a data point from the anomalies table

1. Click on Anomalies below the profile snapshot.

2. Locate the data point you want to reclassify.

3. In the classification column, choose between Expected and Anomaly.

The trend chart and anomalies table stay in sync - changes made in one are immediately reflected in the other.

Anomalies table

The anomalies table lists all detected anomalies with their details:

Column	Description
Metric	The metric that was flagged (along with the column name)
History	A quick visual of the statistic trend.
Actual value	The actual value recorded for this statistic in the latest run.
Expected range	The range the system predicted based on historical data.
Suggested rules	Recommended data quality checks based on the expected range.
Classification	Provide feedback to the anomaly detection model by marking the data point as Expected or Anomaly.

Permissions

Only project owners and members with edit permissions can mark data points as Expected or Anomaly. Members with read-only access can view trend charts and the anomalies table but cannot reclassify data points.