Performance metrics

Several metrics are provided for evaluating how well the predictions match the target time series.
The metrics are correlation, MAE, WAPE and R₂, as described below.

All the metrics can be evaluated on either of: — - the time series as they are

• the change in the time series versus the previous target value
• the YoY change of the time series versus the target value of the same quarter in the
  previous year

Note that for the latter two transformations, also for predictions we are calculating the relative
change from the previous target (actual) value to the current prediction, not the change from the
previous prediction to the current prediction.

correlation()

correlation(target, prediction, growth_period=None, delta=False, start_date='1900-01-01', min_points=5)

Calculate the correlation between the two time series.

The output is a time series, where the value for a given date is the correlation for all the observations from
the start_date up to and including the given date.

• Parameters:
  • target – The target signal that was predicted, such as reported sales numbers.
  • prediction – The prediction time series.
  • growth_period – If specified, transforms both the target and the prediction time series into percentage change
    versus a previous target value before calculating the correlation.
    If growth_period=1, the change versus the previous target value is calculated.
    For quarterly data, if growth_period=4, the change versus the target value for the same quarter
    in the previous year is calculated.
  • delta – If set to True, subtracts the previous target value from both the target and the prediction before calculating the correlation. This option cannot be combined with growth_period.
  • start_date – Predictions on or after this date are used to calculate the correlation.
  • min_points – The minimum number of data points needed to calculate the correlation.

Example

Calculate the correlation between the actual year-over-year growth in sales versus the predicted
year-over-year growth in sales:

correlation(Sales_Actual, Sales_Prediction_Card_Spending, growth_period=4)

mae()

mae(target, prediction, growth_period=None, start_date='1900-01-01', min_points=1)

Calculate the Mean Absolute Error (MAE) of the predictions.

The output is a time series, where the value for a given date is the MAE for all the observations from
the start_date up to and including the given date.

• Parameters:
  • target – The target signal that was predicted, such as reported sales numbers.
  • prediction – The prediction time series.
  • growth_period – If specified, transforms both the target and the prediction time series into percentage change
    versus a previous target value before calculating the MAE.
    If growth_period=1, the change versus the previous target value is calculated.
    For quarterly data, if growth_period=4, the change versus the target value for the same quarter
    in the previous year is calculated.
  • start_date – Predictions on or after this date are used to calculate the correlation.
  • min_points – The minimum number of data points needed to calculate the correlation.

Example

Calculate the MAE of the sales predictions:

mae(Sales_Actual, Sales_Prediction_Card_Spending)

wape()

wape(target, prediction, growth_period=None, start_date='1900-01-01', min_points=1)

Calculate the Weighted Absolute Percentage Error (WAPE) of the predictions.
This is the average error divided by the average target value.

The output is a time series, where the value for a given date is the WAPE for all the
observations from the start_date up to and including the given date.

• Parameters:
  • target – The target signal that was predicted, such as reported sales numbers.
  • prediction – The prediction time series.
  • growth_period – If specified, transforms both the target and the prediction time series into
    percentage change versus a previous target value before calculating the WAPE. If the value
    is 1, the change versus the previous target value is calculated. For quarterly data, if the
    value is 4, the change versus the target value for the same quarter in the previous year
    is calculated.
  • start_date – Predictions on or after this date are used to calculate the correlation.
  • min_points – The minimum number of data points needed to calculate the correlation.

Examples

Calculate the WAPE of the sales predictions:

wape(Sales_Actual, Sales_Prediction_Card_Spending)

Calculate the WAPE of the sales estimates from 2018 onwards:

wape(Sales_Actual, Sales_Estimate_fiscal, start_date='2018-01-01')

r2_score()

r2_score(target, prediction, growth_period=None, delta=False, start_date='1900-01-01', min_points=3)

Calculate the R-squared (R₂) metric of the predictions.
This represents the proportion of the variance of the target that’s explained by the
predictions.

The output is a time series, where the value for a given date is the (R₂) score for
all the observations from the start_date up to and including the given date.

• Parameters:
  • target – The target signal that was predicted, such as reported sales numbers.
  • prediction – The prediction time series.
  • growth_period – If specified, transforms both the target and the prediction time series
    into percentage change versus a previous target value before calculating the R₂.
    If growth_period=1, the change versus the previous target value is calculated. For quarterly
    data, if growth_period=4, the change versus the target value for the same quarter in the
    previous year is calculated.
  • delta – If set to True, subtracts the previous target value from both the target and the
    prediction before calculating the R₂. This is the calculation shown in the
    Prediction Model UI.
    This option cannot be combined with growth_period.
  • start_date – Predictions on or after this date are used to calculate the correlation.
  • min_points – The minimum number of data points needed to calculate the correlation.

Examples

Calculate the R₂ of the sales predictions, in terms of their ability to predict the delta
versus the previous quarter:

r2_score(Sales_Actual, Sales_Prediction_Card_Spending, delta=True)

Calculate the R₂ of the sales predictions, in terms of their ability to predict the growth
rate of the sales numbers versus a year ago:

r2_score(Sales_Actual, Sales_Prediction_Card_Spending, growth_period=4)