Request Syntax Request Parameters Response Syntax Response Elements Errors See Also

GetAccuracyMetrics

Provides metrics on the accuracy of the models that were trained by the CreatePredictor operation. Use metrics to see how well the model performed and to decide whether to use the predictor to generate a forecast. For more information, see Predictor Metrics.

This operation generates metrics for each backtest window that was evaluated. The number of backtest windows (NumberOfBacktestWindows) is specified using the EvaluationParameters object, which is optionally included in the CreatePredictor request. If NumberOfBacktestWindows isn't specified, the number defaults to one.

The parameters of the filling method determine which items contribute to the metrics. If you want all items to contribute, specify zero. If you want only those items that have complete data in the range being evaluated to contribute, specify nan. For more information, see FeaturizationMethod.

Note

Before you can get accuracy metrics, the Status of the predictor must be ACTIVE, signifying that training has completed. To get the status, use the DescribePredictor operation.

Request Syntax


{
   "PredictorArn": "string"
}

Request Parameters

The request accepts the following data in JSON format.

PredictorArn

The Amazon Resource Name (ARN) of the predictor to get metrics for.

Type: String

Length Constraints: Maximum length of 256.

Pattern: arn:([a-z\d-]+):forecast:.*:.*:.+

Required: Yes

Response Syntax


{
   "AutoMLOverrideStrategy": "string",
   "IsAutoPredictor": boolean,
   "OptimizationMetric": "string",
   "PredictorEvaluationResults": [ 
      { 
         "AlgorithmArn": "string",
         "TestWindows": [ 
            { 
               "EvaluationType": "string",
               "ItemCount": number,
               "Metrics": { 
                  "AverageWeightedQuantileLoss": number,
                  "ErrorMetrics": [ 
                     { 
                        "ForecastType": "string",
                        "MAPE": number,
                        "MASE": number,
                        "RMSE": number,
                        "WAPE": number
                     }
                  ],
                  "RMSE": number,
                  "WeightedQuantileLosses": [ 
                     { 
                        "LossValue": number,
                        "Quantile": number
                     }
                  ]
               },
               "TestWindowEnd": number,
               "TestWindowStart": number
            }
         ]
      }
   ]
}

Response Elements

If the action is successful, the service sends back an HTTP 200 response.

The following data is returned in JSON format by the service.

AutoMLOverrideStrategy

Note

The LatencyOptimized AutoML override strategy is only available in private beta. Contact AWS Support or your account manager to learn more about access privileges.

The AutoML strategy used to train the predictor. Unless LatencyOptimized is specified, the AutoML strategy optimizes predictor accuracy.

This parameter is only valid for predictors trained using AutoML.

Type: String

Valid Values: LatencyOptimized | AccuracyOptimized

IsAutoPredictor

Whether the predictor was created with CreateAutoPredictor.

Type: Boolean

OptimizationMetric

The accuracy metric used to optimize the predictor.

Type: String

Valid Values: WAPE | RMSE | AverageWeightedQuantileLoss | MASE | MAPE

PredictorEvaluationResults

An array of results from evaluating the predictor.

Type: Array of EvaluationResult objects

Errors

InvalidInputException

We can't process the request because it includes an invalid value or a value that exceeds the valid range.

HTTP Status Code: 400

ResourceInUseException

The specified resource is in use.

HTTP Status Code: 400

ResourceNotFoundException

We can't find a resource with that Amazon Resource Name (ARN). Check the ARN and try again.

HTTP Status Code: 400