RDP 2017-01: Gauging the Uncertainty of the Economic Outlook Using Historical Forecasting Errors: The Federal Reserve's Approach 3. Methods for Gauging Uncertainty

How might central banks go about estimating the uncertainty associated with the outlook?^[5] The approach employed by the FOMC and several other central banks is to look to past prediction errors as a rough guide to the magnitude of forecast errors that may occur in the future.^[6] For example, if most actual outcomes over history fell within a band of a certain width around the predicted outcomes, then a forecaster might expect future outcomes to cluster around his or her current projection to a similar degree. Such an error-based approach has two attractive features. First, the relationship of the uncertainty estimates to historical experience is clear. Second, the approach focuses on the actual historical performance of forecasters under true ‘field conditions’ and does not rely on after-the-fact analytic calculations, using various assumptions, of what their accuracy might have been.^[7]

The approach of the FOMC is somewhat unusual in that historical estimates are compared with qualitative assessments of how uncertainty in the forecast period may differ from usual. Most FOMC participants have judged the economic outlook to be more uncertain than normal in well over half of SEPs published since late 2007.^[8] A majority has also assessed the risks to some aspect of the economic outlook to be skewed to the upside or downside in more than half of the SEPs released to date, and in many other releases a substantial minority has reported the risks as asymmetric.

These qualitative comparisons address two potential drawbacks with the error-based approach. First, the error-based approach assumes that the past is a good guide to the future. Although this assumption in one form or another underlies all statistical analyses, there is always a risk that structural changes to the economy may have altered its inherent predictability. Indeed, there is evidence of substantial changes in predictability over the past 30 years, which we discuss below. These signs of instability suggest a need to be alert to evidence of structural change and other factors that may alter the predictability of economic outcomes for better or worse. Given that structural changes are very difficult to quantify in real time, qualitative assessments can provide a practical method of recognizing these risks.

Second, estimates based on past predictive accuracy may not accurately reflect policymakers' perceptions of the uncertainty attending the current economic outlook. Under the FOMC's approach, participants report their assessments of uncertainty conditional on current economic conditions. Thus, perceptions of the magnitude of uncertainty and the risks to the outlook may change from period to period in response to specific events.^[9] And while analysis by Knüppel and Schultefrankenfeld (2012) calls into question the retrospective accuracy of the judgmental assessments of asymmetric risks provided by the Bank of England, such assessments are nonetheless valuable in understanding the basis for monetary policy decisions.

Model simulations provide another way to gauge the uncertainty of the economic outlook. Given an econometric model of the economy, one can repeatedly simulate it while subjecting the model to stochastic shocks of the sort experienced in the past. This approach is employed by Norges Bank to construct the fan charts reported in their quarterly Monetary Report, using NEMO, a New Keynesian DSGE model of the Norwegian economy. Similarly, the staff of the Federal Reserve Board regularly use the FRB/US model to generate fan charts for the staff Tealbook forecast.^[10] Using this methodology, central bank staff can approximate the entire probability distribution of possible outcomes for the economy, potentially controlling for the effects of systematic changes in monetary policy over time, the effective lower bound on nominal interest rates, and other factors. Moreover, staff economists can generate these distributions as far into the future as desired and in as much detail as the structure of the model allows. Furthermore, the model-based approach permits analysis of the sources of uncertainty and can help explain why uncertainty might change over time.

However, the model-based approach also has its limitations. First, the estimates are specific to the model used in the analysis. If the forecaster and his or her audience are worried that the model in question is not an accurate depiction of the economy (as is always the case to some degree), they may not find its uncertainty estimates credible. Second, the model-based approach also relies on the past being a good guide to the future, in the sense that the distribution of possible outcomes is constructed by drawing from the model's set of historical shocks. Third, this methodology abstracts from both the difficulties and advantages of real-time forecasting: It tends to understate uncertainty by exploiting after-the-fact information to design and estimate the model, and it tends to overstate uncertainty by ignoring extra-model information available to forecasters at the time. Finally, implementing the model-based approach requires a specific characterization of monetary policy, such as the standard Taylor rule, and it may be difficult for policymakers to reach consensus about what policy rule (if any) would be appropriate to use in such an exercise.^[11] Partly for these reasons, Wallis (1989, pp 55–56) questions whether the model-based approach really is of practical use. These concerns notwithstanding, in at least some cases model-based estimates of uncertainty are reasonably close to those generated using historical errors.^[12]

A third approach to gauging uncertainty is to have forecasters provide their own judgmental estimates of the confidence intervals associated with their projections. Such an approach does not mean that forecasters generate probability estimates with no basis in empirical fact; rather, the judgmental approach simply requires the forecaster, after reviewing the available evidence, to write down his or her best guess about the distribution of risks. Some central banks combine judgment with other analyses to construct subjective fan charts that illustrate the uncertainty surrounding their outlooks. For example, such subjective fan charts have been a prominent feature of the Bank of England's Inflation Report since the mid-1990s.

Judgmental estimates might not be easy for the FOMC to implement, particularly if it were to try to emulate other central banks that release a single unified economic forecast together with a fan chart characterization of the risks to the outlook. Given the large size of the Committee and its geographical dispersion, achieving consensus on the modal outlook alone would be difficult enough, as was demonstrated in 2012 when the Committee tested the feasibility of producing a consensus forecast and concluded that the experiment (at least for the time being) was not worth pursuing further considering the practical difficulties.^[13] Trying to achieve consensus on risk assessments as well would only have made the task harder. And while the FOMC needs to come to a decision on the stance of policy, it is not clear that asking it to agree on detailed features of the forecast is a valuable use of its time.

Alternatively, the FOMC could average the explicit subjective probability assessments of individual policymakers, similar to the approaches used by the Bank of the Japan (until 2015), the Survey of Professional Forecasts, and the Primary Dealers Survey.^[14] The relative merits of this approach compared to what the FOMC now does are unclear. Psychological studies find that subjective estimates of uncertainty are regularly too low, often by large margins, because people have a systematic bias towards overconfidence.^[15] Contrary to what might be suspected, this bias is not easily overcome; overconfidence is found among experts and among survey subjects who have been thoroughly warned about it. This same phenomenon suggests that the public may well have unrealistic expectations for the accuracy of forecasts in the absence of concrete evidence to the contrary – which, as was noted earlier, is a reason for central banks to provide information on historical forecasting accuracy.