Improved DSL for temporal change calculations
We have improved the signal.change() and signal.relative_change() DSL functions for temporal change (eg QoQ, YoY) to better handle irregular quarterly calendars.
The new logic uses smarter defaults for comparing each data point with the correct historical point. Previously, quarterly time series with irregular quarter-end dates would have an unintuitive result when calculating QoQ / YoY change, as the functions would forward fill the time series to find the comparable data point. These changes now mean that the functions return the intuitively expected results by default, but still give users options to override.
The described changes do not apply to the
relative_change(signal, ...)syntax, which is kept unchanged for backwards compatibility of existing signals. See also the accompanying deprecation notice for this legacy syntax.
Example: year-on-year (YoY) change
In the example below, we look at Kroger, which reports fiscal quarters with an irregular frequency (try it out). The 1st column shows quarterly sales, the 2nd column shows YoY absolute change with the new behaviour, and the 3rd column shows the old unintuitive behaviour.
YoY change on an irregular quarterly time series - new, default behaviour in 2nd column; vs old, unintuitive result in 3rd column
Consider the 21 May 2022 data point. for a YoY calculation, the comparable historical data point should be on 21 May 2021, but there is no data point here (the closest is 22 May 2021). The old logic forward filled the underlying time series, such that the 30 Jan 2021 was used as the comparable data point, giving an unintuitive result (44600 - 30737 = 13863). The new logic looks for the nearest data point, even if it is later, in this case by 1 day on 22 May 2021, giving the intuitively expected result (44600 - 41298 = 3302).
Example: daily change
Yet, in the financial and investment domain, it would be incorrect to always default to such 'nearest' matching. In the example below (try it out), we have a daily time series of share prices, with missing values on weekends.
Day-on-day change on daily share price time series - default behaviour in 2nd column; and undesired behaviour from always using 'nearest' matching
Consider the data point on Monday 29 August 2022 - the comparable historical point should be 1 day earlier, on Sunday 28 August 2022, where no data point exists. The correct intuitive behaviour (2nd column) in this context would be to compare it to the last data point on Friday 26 August 2022 (158.9 - 161.4 = -2.5). Nearest matching (3rd column) would give an unintuitive result, as the nearest data point to 28 August 2022 is the original data point (158.9 - 158.9 = 0).
Summary
To handle such cases generally, we have introduced method and limit parameters to signal.change() and signal.relative_change(), and selected opinionated defaults that produce intuitive results for the financial / investment domain:
- For temporal changes >= 3 months (QoQ / YoY / etc), we default to nearest matching and allow a limit of up to 31 days. This limit controls how far we may look before/after to find a historical data point. 31 days is sufficient to handle extreme cases of companies with 12/12/12/16-week quarters, like Kroger.
- For temporal changes >= 1 and <3 months (MoM), we default to nearest matching and allow a limit of up to 10 days.
- For temporal changes <1 month (WoW / DoD), we default to forward filling and allow a limit of up to 10 days. The default limit of 10 days is made with long market holidays in mind, which lead to longer gaps in daily market data.