RDP 2017-01: Gauging the Uncertainty of the Economic Outlook Using Historical Forecasting Errors: The Federal Reserve's Approach 5. Data Sources

For the reasons just discussed, the FOMC computes historical forecast errors based on projections made by a variety of forecasters. The first source for these errors is the FOMC itself, using the mid-point of the central tendency ranges reported in past releases of the Monetary Policy Report and (starting in late 2007) its replacement, the Summary of Economic Projections.^[17] The second source is the Federal Reserve Board staff, who prepare a forecast prior to each FOMC meeting; these projections were reported in a document called the Greenbook until 2010, when a change in the color of the (restructured) report's cover led it to be renamed the Tealbook. For brevity, we will refer to both as Tealbook forecasts in this paper.^[18] The third and fourth sources are the Congressional Budget Office (CBO) and the Administration, both of which regularly publish forecasts as part of the federal budget process. Finally, the historical forecast database draws from two private data sources – the monthly Blue Chip consensus forecasts and the mean responses to the quarterly Survey of Professional Forecasters (SPF). Both private surveys include many business forecasters; the SPF also includes forecasters from universities and other nonprofit institutions.

Differences between these six forecasters create some technical and conceptual issues for the analysis of historical forecasting accuracy. Table 1A shows differences in timing and frequency of publication, horizon, and reporting basis. We discuss these below, then address several other issues important to the analysis of past predictive accuracy and future uncertainty, such as how to define ‘truth’ in assessing forecasting performance, the mean versus modal nature of projections, and the implications of conditionality.

5.1 Data Coverage

The FOMC currently releases a summary of participants' forecasts late each quarter, immediately following its March, June, September, and December meetings. However, as shown in the second column of Table 1A, the various forecasts in the historical dataset necessarily deviate from this late-quarter release schedule somewhat. For example, the CBO and the Administration only publish forecasts twice a year, as did the FOMC prior to late 2007; in addition, the SPF is released in the middle month of each quarter, rather than the last month. Generally, each historical forecast is assigned to a specific quarter based on when that forecast is usually produced.^[19] In some cases, the assigned quarter differs from the actual release date. Because of long publication lags, the Administration forecasts released in late January and late May are assumed to have been completed late in the preceding quarter. Also, those FOMC forecasts that were released in July (generally as part of the mid-year Monetary Policy Report) are assigned to the second quarter because participants submitted their individual projections either in late June or the very beginning of July. Finally, because the Blue Chip survey reports extended-horizon forecasts only in early March and early October, the third-quarter Blue Chip forecasts are the projections for the current year and the coming year reported in the September release, extended with the longer-run projections published in the October survey.

With respect to coverage of variables, all forecasters in our sample except the FOMC have published projections of real GDP/GNP growth, the unemployment rate, CPI inflation, and the 3-month Treasury bill rate since at least the early 1980s. In contrast, the FOMC has never published forecasts of the T-bill rate, and only began publishing forecasts of the federal funds rate in January 2012 – too late to be of use for the analysis in this paper. As for inflation, the definition used by FOMC participants has changed several times since forecasts began to be published in 1979. For the first ten years, inflation was measured using the GNP/GDP deflator; in 1989 this series was replaced with the CPI, which in turn was replaced with the chain-weighted PCE price index in 2000 and the core chain-weighted PCE price index in 2005. Since late 2007, FOMC participants have released projections of both total and core PCE inflation. Because these different price measures have varying degrees of predictability – in part reflecting differences in their sensitivity to volatile food and energy prices – the Committee's own historical inflation forecasts are not used to estimate the uncertainty of the outlook.

5.2 Variations in Horizon and Reporting Basis

The horizon of the projections in the historical error dataset varies across forecaster and time of year. At one extreme are the FOMC's projections, which prior to late 2007 extended only over the current year in the case of the Q1 projection and the following year in the case of the Q2 projection. At the other extreme are the projections published by the CBO, the Administration, and the March and October editions of the Blue Chip, which extend many years into the future.

In addition, the six primary data sources report forecasts in different ways, depending on the variable and horizon. In some cases, the published unemployment rate and T-bill rate projections are for the Q4 level, in other cases for the annual average. Similarly, in some cases the real GDP growth and CPI inflation projections are expressed as Q4-over-Q4 percent changes, while in other cases they are reported as calendar-year-over-calendar-year percent changes. Details are provided in Table 1A. These differences in reporting basis are potentially important because annual average projections tend to be more accurate than forecasts of the fourth-quarter average, especially for current-year and coming-year projections; to a somewhat lesser extent, the same appears to be true for year-over-year projections relative to Q4-over-Q4 forecasts.^[20] For this reason, projections on a Q4 basis or Q4-over-Q4 basis are used wherever possible to correspond to the manner in which the FOMC reports its forecasts.^[21] In addition, current and next-year forecasts of the unemployment rate and the T-bill rate are excluded from the calculation of average RMSEs when reported on an annual-average basis, as are GDP growth and CPI inflation when reported on a calendar year-over-year basis. However, differences in recording basis are ignored in the calculation of average RMSEs for longer horizon projections, both because of the sparsity of forecasts on the desired reporting basis and because a higher overall level of uncertainty reduces the importance of the comparability issue.^[22]

5.3 Defining ‘Truth’

To compute forecast errors one needs a measure of ‘truth.’ One simple approach is to use the most recently published estimates. For the unemployment rate, CPI inflation, and the Treasury bill rate, this approach is satisfactory because their reported value in a given quarter or year changes little if at all as new vintages of published historical data are released. In the case of real GDP growth, however, this definition of truth has the drawback of incorporating the effects of definitional changes, the use of new source material, and other measurement innovations that were introduced well after the forecast was generated. Because forecasters presumably did not anticipate these innovations, they effectively were forecasting a somewhat different series in the past than the historical GDP series now reported in the national accounts. Forecasters predicted fixed weight GDP prior to 1995, GDP ex-software investment prior to 1999, and GDP ex-investment in intangibles before 2014, in contrast to the currently-published measure that uses chain weighting and includes investment in software and intangibles. To avoid treating the effects of these measurement innovations as prediction errors, ‘true’ real GDP growth is measured using the latest published historical data, adjusted for the estimated effect of the switch to chain-weighting and the inclusion of investment in software and intangibles.^[23]

5.4 Mean Versus Modal Forecasts

Another issue of potential relevance to our forecast comparisons is whether they represent mean predictions as opposed to median or modal forecasts. As documented by Bauer and Rudebusch (2016), this issue can be important for short-horizon interest rate forecasts because the distribution of possible outcomes becomes highly skewed when interest rates approach zero. Until recently, many forecasters saw the most likely (modal) outcome was for interest rates to remain near zero for the next year or two. Because there was a small chance of interest rates declining slightly, but a sizeable chance of large increases, the implicit mean of the distribution was greater than the mode. As we discuss below, this has implications for how confidence intervals about the interest rate outlook should be constructed.

The projections now produced by FOMC participants are explicitly modal forecasts in that they represent participants' projections of the most likely outcome under their individual assessments of appropriate monetary policy, with the distribution of risks about the published projections viewed at times as materially skewed. However, we do not know whether participants' projections in the past had this modal characteristic. In contrast, the CBO's forecasts, past and present, are explicitly mean projections. In the case of the Tealbook projections, the Federal Reserve Board staff typically views them as modal forecasts. As for our other sources, we have no reason to believe that they are not mean projections, although we cannot rule out the possibility that some of these forecasters may have had some objective other than minimizing the root mean squared error of their predictions.

5.5 Policy Conditionality

A final issue of comparability concerns the conditionality of forecasts. Currently, FOMC participants condition their individual projections on their own assessments of appropriate monetary policy, defined as the future policy most likely to foster trajectories for output and inflation consistent with each participant's interpretation of the Committee's statutory goals. Although the definition of ‘appropriate monetary policy’ was less explicit in the past, Committee participants presumably had a similar idea in mind when making their forecasts historically. Whether the other forecasters in our sample (aside from the Tealbook) generated their projections on a similar basis is unknown, but we think it reasonable to assume that most sought to maximize the accuracy of their predictions and so conditioned their forecasts on their assessment of the most likely outcome for monetary policy.

This issue also matters for the Tealbook because the Federal Reserve Board staff, to avoid inserting itself into the FOMC's internal policy debate, has eschewed guessing what monetary policy actions would be most consistent with the Committee's objectives. Instead, the staff has traditionally conditioned the outlook on a ‘neutral’ assumption for policy. In the past, this approach sometimes took the form of an unchanged path for the federal funds rate, although it was more common to instead condition on paths that modestly rose or fell over time in a manner that signaled the staff's assessment that macroeconomic stability would eventually require some adjustment in policy. More recently, the Tealbook path for the federal funds rate has been set using a simple policy rule, with a specification that has changed over time. In principle, these procedures could have impaired the accuracy of the Tealbook forecasts because they were not intended to reflect the staff's best guess for the future course of monetary policy. But as we will show in the next section, this does not appear to have been the case – a result consistent with the findings of Faust and Wright (2009).

Fiscal policy represents another area where conditioning assumptions could have implications for using historical forecast errors to gauge current uncertainty. The projections reported in the Monetary Policy Report, the Tealbook, the Blue Chip, and the Survey of Professional Forecasters presumably all incorporate assessments of the most likely outcome for federal taxes and government outlays. This assumption is often not valid for the forecasts produced by the CBO and the Administration because the former conditions its baseline forecast on unchanged policy and the latter conditions its baseline projection on the Administration's proposed fiscal initiatives. As was the case with the Tealbook's approach to monetary policy, the practical import of this type of ‘neutral’ conditionality for this study may be small. For example, such conditionality would not have a large effect on longer-run predictions of aggregate real activity and inflation if forecasters project monetary policy to respond endogenously to stabilize the overall macroeconomy; by the same logic, however, they could matter for interest rate forecasts.

Footnotes

Until recently, the Monetary Policy Report and the Summary of Economic Projections reported only two summary statistics of participants' individual forecasts – the range across all projections (between sixteen and nineteen, depending on the number of vacancies at the time on the Federal Reserve Board) and a trimmed range intended to express the central tendency of the Committee's views. For each year of the projection, the central tendency is the range for each series after excluding the three highest and three lowest projections. Beginning in September 2015, the SEP began reporting medians of participants' projections as well, and we use these medians in place of the mid-point of the central tendency in our analysis. [17]

The statistics reported for the Tealbook in this paper are based on the full 20-year sample. Individual Tealbooks, which contain detailed information on the outlook, become publicly available after approximately five years. [18]

In contrast to the approach employed by Reifschneider and Tulip (2007), we do not interpolate to estimate projections for missing publication quarters in the case of the CBO and the Administration. In addition, because of the FOMC's semi-annual forecasting schedule prior to October 2007, FOMC forecasts are used in the analysis of predictive accuracy for forecasts made in the first and second quarters of the year only. [19]

These differences in relative accuracy occur for two reasons. First, averaging across quarters eliminates some quarter-to-quarter noise. Second, the annual average is effectively closer in time to the forecast than the fourth-quarter average because the mid-point of the former precedes the mid-point of the latter by more than four months. This shorter effective horizon is especially important for current-year projections of the unemployment rate and the Treasury bill rate because the forecaster will already have good estimates of some of the quarterly data that enter the annual average. Similar considerations apply to out-year projections of real GDP growth and CPI inflation made on a calendar-year-over-calendar-year basis. [20]

Strictly speaking, no historical forecasts of short-term interest rates conform with the basis employed by the FOMC, the target level of the federal funds rate (or mid-point of the target range) most likely to be appropriate on the last day of the year. But except for the current-year projections released at the September and December meetings, the practical difference between interest rate forecasts made on this basis and projections for average conditions in the fourth quarter are probably small. [21]

An alternative approach might have been to use annual or year-over-year projections to back out implied forecasts on the desired Q4 average or Q4-over-Q4 basis, using a methodology such as that discussed by Knüppel and Vladu (2016). Whether the potential gain in accuracy from adopting such an approach would offset the resulting loss in simplicity and transparency is not obvious, however. [22]

See Federal Reserve Board (2014) for details. An alternative to adjusting the current vintage of published data, and one that we employed in our earlier 2007 study, would be to define truth using data published relatively soon after the release of forecast – an approach that would increase the likelihood that the definition of the published series is the same or similar to that used when the variable was projected. One drawback with this quasi-real-time approach is that the full set of source data used to construct estimates of real GDP does not become available for several years, implying that the quasi-real-time series used to define truth often do not fully incorporate all the source data that will eventually be used to construct the national accounts, even if the definition of real GDP remains otherwise unchanged. As discussed in Federal Reserve Board (2014), this drawback is a serious one in that revisions to realtime data are substantial and much larger than the methodological adjustments used in this paper. [23]