RDP 2017-01: Gauging the Uncertainty of the Economic Outlook Using Historical Forecasting Errors: The Federal Reserve's Approach 4. Collecting Historical Forecast Data

To provide a benchmark against which to assess the uncertainty associated with the projections provided by individual Committee participants, one obvious place to turn is the FOMC's own forecasting record – and indeed, we exploit this information in our analysis. For several reasons, however, the approach taken in the SEP also takes account of the projection errors of other forecasters as well. First, although the Committee has provided projections of real activity and inflation for almost forty years, the horizon of these forecasts was, for quite a while, considerably shorter than it is now – at most one and a half years ahead as compared with roughly four years under the current procedures. Second, the specific measure of inflation projected by FOMC participants has changed over time, making it problematic to relate participants' past prediction errors to its current forecasts. Finally, consideration of other forecasts reduces the likelihood of placing undue weight on a potentially unrepresentative record. For these reasons, supplementing the Committee's record with that of other forecasters is likely to yield more reliable estimates of forecast uncertainty.

In addition to exploiting multiple sources of forecast information, the approach used in the Summary of Economic Projections also controls for differences in the release date of projections. At the time of this writing, the FOMC schedule involves publishing economic projections following the March, June, September, and December FOMC meetings. Accordingly, the historical data used in our analysis is selected to have publication dates that match this quarterly schedule as closely as possible.

Under the FOMC's current procedures, each quarter the Committee releases projections of real GDP growth, the civilian unemployment rate, total personal consumption expenditures (PCE) chain-weighted price inflation, and core PCE chain-weighted price inflation (that is, excluding food and energy). Each participant also reports his or her personal assessment of the level of the federal funds rate at the end of each projection year that would be consistent with the Committee's mandate. The measures projected by forecasters in the past do not correspond exactly to these definitions. Inflation forecasts are available from a variety of forecasters over a long historical period only on a CPI basis; similarly, data are available for historical projections of the 3-month Treasury bill rate but not the federal funds rate. Fortunately, analysis presented below suggests that forecast errors are about the same whether inflation is measured using the CPI or the PCE price index, or short-term interest rates are measured using the T-bill rate or the federal funds rate.

A final issue in data collection concerns the appropriate historical period for evaluating forecasting accuracy. In deciding how far back in time to go, there are tradeoffs. On the one hand, collecting more data by extending the sample further back in time should yield more accurate estimates of forecast accuracy if the forecasting environment has been stable over time. Specifically, it would reduce the sensitivity of the results to whether extreme rare events happen to fall within the sample. On the other hand, if the environment has in fact changed materially because of structural changes to the economy or improvements in forecasting techniques, then keeping the sample period relatively short should yield estimates that more accurately reflect current uncertainty. Furthermore, given the FOMC's qualitative comparison to a quantitative benchmark, it is useful for that measure to be salient and interpretable, to which other information and judgements can be usefully compared. In balancing these considerations, in this paper we follow current FOMC procedures and employ a moving fixed-length 20-year sample window to compute root mean squared forecast errors and other statistics, unless otherwise noted. We also conform to the FOMC's practice of rolling the window forward after a new full calendar year of data becomes available; hence, the Summary of Economic Projections released in June 2016 reported average errors for historical predictions of what conditions would be in the years 1996 to 2015.^[16]

Footnote

Obviously, different conclusions are possible about the appropriate sample period and other methodical choices in using historical errors to gauge future uncertainty. For example, while the European Central Bank also derives its uncertainty estimates using historical forecasting errors that extend back to the mid-1990s, it effectively shortens the sample size by excluding ‘outlier’ errors whose absolute magnitudes are greater than two standard deviations. In contrast, information from a much longer sample period is used to construct the model-based confidence intervals regularly reported in the Tealbook, which are based on stochastic simulations of the FRB/US model that randomly draw from the equation residuals observed from the late 1960s through the present. In this case, however, some of the drawbacks of using a long sample period are diminished because the structure of the model controls for some important structural changes that have occurred over time, such as changes in the conduct of monetary policy. [16]