Modelling KPIs

Modelling is the step of taking one or more KPI mappings and combining them to train an optimized model that makes accurate predictions of the KPI. Such models often combine data from multiple vendors in order to reduce prediction error.

The Models view in the KPI Analyzer allows you to easily browse existing models, run new models that combine data from multiple sources, deep-dive to analyze historical model performance.

Browsing models

In the Models view, the page header shows the company, KPI, frequency, and period being viewed (quarterly / semi-annual / annual). You can use the dropdowns to quickly switch companies or view a different KPI. The Abs/QoQ/YoY buttons let you toggle between absolute, QoQ and YoY predictions.

Models table

The models table shows all the models that are available for this KPI. Your own models, and those that have been shared to you, are at the top, followed by curated models from data vendors.

Select one or more models to view their predictions and other details in the charts on the page.

There are 3 categories of models shown in the table:

• Auto model: denoted by the Auto badge; this is automatically run when you enter the Models view, and considers all available inputs (i.e the standalone models you see in this table) in order to train an optimized model.
• Custom models: denoted by the Custom badge; these are models that you may choose to build with your choice of inputs (standalone models), model type, and model parameters.
• Standalone models: each KPI mapping (ref Mappings view) has its own standalone model, which uses only data from that KPI mapping to predict the KPI. These are then be combined into the Auto model and custom models.
  You'll find standalone models from each vendor subscription that you have, but if you create your own custom KPI mappings, those will also have standalone models.

👍

Non-subscribed vendor results are also shown

Similar to the Mappings view, model results from vendors that you are not subscribed to are also shown in the Models view, sorted to the bottom. Model predictions are not shown, but the backtested errors are available to view.

The following columns are shown in the table:

• Prediction: model prediction for the period being viewed. You can view predictions for the last / current / next period through the dropdown in the page header.
• Delta: delta vs consensus mean of the KPI. This is shown as a percentage for most KPIs, and in absolute terms for ratio KPIs (e.g. gross margin).
• Est. error: model estimated mean error, using point-in-time modelling based on where we are in the quarter/period. E.g. if we are 30 days from quarter-end, this calculates the mean error of historical predictions made using data available 30 days from quarter-end for each historical quarter.

📘

Consensus estimated error

The consensus row also has an estimated error, which is calculated on a like-for-like basis to models.

For example, if we are 30 days from quarter-end, we look at consensus as-of 30 days from quarter-end for each historical quarter, and calculate its mean error on that basis.

• Est. hit rate: directional accuracy of predictions vs consensus, using point-in-time modelling based on where we are in the quarter/period. Calculated as the % of the time when the model predicts a beat/miss, that the company also reports a beat/miss.
• Final error: mean error of historical predictions, when all data is available for each quarter/period.
• Final hit rate: directional accuracy of predictions vs consensus, when all data is available for each quarter/period. Calculated as the % of the time when the model predicts a beat/miss, that the company also reports a beat/miss.
• Model type: the type of model being used. "Ratio prediction" is the proprietary Exabel model used by default, though you can train your own custom models with other model types.

📘

Error & hit rate are calculated on the most recent 8 quarters

We calculated the estimated & final error / hit rate based on the most recent 8 quarters / periods.

In some cases, this may be based on fewer than 8 periods as there was insufficient training data to produce 8 backtests. However, these metrics will only be calculated if there are at least 4 periods.

Consensus estimated & final error are also calculated on the most recent 8 quarters, in order to provide a like-for-like comparison.

Model status

You may find status information about each Auto & custom model by hovering over the model and clicking on the information icon. This shows you the user who created the model, when it was created, latest run (models are run daily).

The point-in-time backtest, which is required to calculate the estimated error and hit rate metrics, is run only once when the model is first created. This takes longer to run (minutes), so when you create a new model, you may track its status here.

Model composition table

The model composition table shows how your Auto models and custom models combine data from your available KPI mappings when making predictions. This is not applicable to standalone models, which use data from only 1 KPI mapping.

📘

Weights are shown for the current quarter/period

Note that the model composition table always shows weights for the current quarter/period, even if you have chosen to view the last/next period.

Weights for historical quarters may be different from weights for the current quarter. That is because our models optimize for predictive power by choosing the inputs (KPI mappings) that have been most accurate at each point in time. This is always done in Auto models and by default in custom models, unless you fix the model weights (see Customizing models below).

Providing visibility on model weights for every quarter is on our near-term product roadmap.

Model weights are determined by the training & backtesting process. Generally, the more predictive power that a standalone model has (lower "Final error" in the models table), the higher the weight it will be assigned.

Under the hood, Exabel uses an algorithm to minimize the expected model variance (error), based on the full error covariance matrix of all inputs (KPI mappings). Hence, in some cases the model will choose to put higher weight on a combination of standalone models, that individually may have higher prediction errors, but in combination are expected to have a lower error.

Predictions / errors chart

The predictions chart shows the quarterly series of model predictions, overlaid with the actual (reported) KPI, and consensus mean. You can select multiple models in the Models table to view them in the same chart.

Following market convention, consensus for historical quarters is shown as-of immediately before the company reports results, and consensus for future quarters is the live consensus as of today.

📘

Model predictions are a mix of backtests & live predictions

When you first run a model, we aim to generate backtested predictions for the most recent 8 quarters. (Sometime, there are fewer than 8 if the KPI mappings have limited history.) These backtested predictions will appear in this chart, alongside the predictions for the current and next quarters.

Example: a model is first run in February 2024, after the company has already reported 4Q-2023 results. It generates backtests for the most recent 8 quarters, ie 1Q-2022 - 4Q-2023, and starts making predictions for the current (1Q-2024) and next (2Q-2024) quarters. Fast-forward 3 months, and the company has now reported 1Q-2024 results. The predictions chart would show 9 historical quarters of predictions (8 from the backtest & 1 live prediction), as well as predictions for 2Q-2024 (now the current quarter) and 3Q-2024 (now the next quarter).

The date range may be changed through the Settings button, and there is a download button to get an Excel file with the chart data.

In the same widget, you can switch to the predictions errors chart by clicking on the bar-chart icon in the top-right corner. This shows the quarterly errors relative to the actual KPI value for each historical quarter.

Positive numbers indicate where consensus/models predicted above what was reported, and vice versa.

This can help to more easily identify quarters where the model performed strongly / poorly, visualize the relative errors between models and consensus, and analyze if model performance has been improving / worsening over time.

Revisions chart

The revisions chart overlays consensus and model revisions in the same chart. Model revisions are only tracked once a model has been run, so you may need to wait a few days before a revision history line starts to build.

Intra-quarter chart

The intra-quarter chart shows how each model "builds-up" through a quarter to its prediction. Auto models & custom models are shown as a point at the quarter-end, while the standalone models that they use are shown as lines that build up to the prediction based on the intra-quarter data.

Probability distribution chart

The probability distribution chart plots an estimated distribution of predicted outcomes. This helps to visualize the relative confidence of the model vs consensus, as well as where the model is predicting a significantly non-consensus outcome.

For each selected model and consensus, a normal distribution is plotted centred on the model prediction / consensus mean, with a width determined by its estimated error.

Point-in-time analysis chart

---

🚧

Under construction

Customizing models

Model actions

Re-run, share, delete