Multi-layer forecasting to predict business outcomes during times of market volatility

Predicting business outcomes often happens in the form of forecasts. These carry the implicit assumption “what happened in the past reflects the future”. This is very often the case. However, external factors are present in the market and obviously have huge impacts on business outcomes. Taking these factors into account is critical for making reasonable projections of business outcomes, and in the process, can identify the market sensitivity to these external factors. Often it is just as important to know “how sensitive is the market to factor X?” as it is “when do we expect the market to recover?” Taking external factors into account requires multi-layer forecasting, since the external factors themselves need to be projected into the future in order to support forecasts of a metric of interest. What’s the process for making predictions with multi-layer forecasting? Let’s discuss a few guiding principles to get us started.

Principle 1: Stack models to achieve layered forecasts

Before we consider which models to use, or how to stick them together, let’s talk about the first principle of multi-layer forecasting. The goal of layering forecasts is to integrate more information to better inform a prediction. Not only do we wish to forecast the metric of interest (in today’s example, the effect of COVID case load on business outcomes), but we also want to predict the underlying drivers that influence this metric.

Below is a diagram showing three unique values and a time threshold, which could be today or the end of a known or observed period.

There are many types of values that can work together in a stacked forecast, but here is one set of labels that would make sense for a COVID to sales scenario.

Principle 2: All models should make sense

The “smell test” or “common sense test” should apply to all models. The primary metric should be predicted within reasonable limits: in our scenario, the unemployment rate shouldn’t be 30%, but it also shouldn’t be 0.01%. It’s important to note: be prepared for outcomes that are dire, or in some cases, rosier than expected. Of course, simple plots often do enough to show whether something makes sense, but be careful of plots that don’t show enough scale to be useful. Sometimes, a log scale is needed to fit the data and trends onto one graph.

Data types are also an important consideration, since they drive the model. We are in the era of machine learning, which has few assumptions, is often incredibly accurate with only moderate amounts of work, and has become easier than ever to use. Machine learning models have just a few drawbacks:

1. ML models are not readily interpretable. Few give us the ability to see the effects of an individual term on an outcome.
2. ML models don’t readily offer us confidence intervals, which are an important way to check the feasibility of our models.
3. ML models often exchange low error for high variance. In our case, we can accept some error, but want the forecasts from our model to have a low volatility, also known as variance.
4. Few ML models offer constraints on the outcome. The most important of these is the Poisson constraint, i.e., the restriction that the output can never be negative. It doesn’t make sense, for instance, to have negative sales, a negative unemployment rate, or negative COVID cases. These are all strictly positive, and any model which can deviate below zero, likely will at some future point.

Principle 3: Use common sense to anticipate driving factors

Creating a meaningful forecast can be overwhelming, so start simple. Often just a few of the most critical factors are enough to get reasonable projections. Why not use hundreds? In this rare case, it may be wiser to reduce the number of driving factors to a much smaller range, say anywhere from 1 to 5 “support” models. Why? Uncertainty stacks, and since we also have to forecast each of the driving factors, the model can quickly become overly sensitive to the model fits of the driving factor forecasts. We are seeking simplicity and even more so, stability. We don’t want drastic sensitivities to tiny deviations in a future forecast.

Step-by-step instructions for multi-layer forecasting

1. Choose metric of interest and collect proposed driving factors. Optionally, you can assess correlations between metric of interest and driving factors, reduce to those which are highly (but not above say 80%) correlated to the metric of interest, and try to select driving factors which are not highly correlated to each other. Use common sense to select those that you might want to later explain the relationship to the metric of interest. For instance, you could state “as unemployment decreases by 1%, and with COVID cases held constant, sales increase by $4 million dollars a quarter in the Southwest region.”

2. Create forecasts for each of the driving factors. These should extend to the range desired for forecasting the metric of interest.

3. Create a forecast of the metric of interest which uses the forecasts of driving factors as “exogenous” or “endogenous” factors as appropriate. Specification of this model depends on knowledge of the mathematics or statistics of the chosen forecasting method. Note that most forecasting methods do not accept external variables.

To view an example walk through, read the full article here: https://www.logic2020.com/insight/multi-layer-forecasting