RDP 2023-06: Firms' Price-setting Behaviour: Insights from Earnings Calls 4. Index Validation

4.1 Expected information content

First and foremost, earnings calls are backward looking: they cover financial results for the previous six months and are delivered with a two-month lag. Taking this as our starting point, it is important to consider why the aggregate sentiment indices we develop would contain any information that is relevant for assessing current economic conditions. That is, it is important to ask why aggregate indices derived from earnings calls would contain any information that is already available in real-time indicators of current economic conditions taken from surveys of businesses or from business liaison programs, such as those run by central banks. If insights from earnings calls about current economic conditions are contemporaneous or help predict insights from survey respondents or liaison contacts from a similar set of businesses, then earnings calls must influence these respondents in some way. In this section, we argue there are three reasons this might occur.

• First, and most obvious, is that executives do, at times, reveal new facts about the current operating environment or the outlook. For example, executives of commercial banks often provide updates and forecasts for net interest margins over the year ahead. Forward-looking information is also elicited as part of the Q&A sessions.
• Second, executives often reveal new facts about decision-making processes, prompting a reassessment of the outlook. For instance, an executive could reveal that some divisional managers have been requested to reduce costs on an ongoing basis, thereby explaining persistent declines in expenses.
• Third, the clear and concise synthesis of existing information provided by executives during the earnings presentation and Q&A session could cause the outlook of others – such as business survey respondents or business liaison contacts – to be updated or revised through a process of fact-free learning (Aragones et al 2005).

The first two reasons require little explanation. New facts are presented, integrated into an existing knowledge base and the assessment of current and future economic conditions is modified. Within a Bayesian framework, these modifications are done mechanically according to Bayes' rule.

The third requires some explanation. We argue that executives can also change the assessment of current conditions as well as the outlook without communicating new facts, but by pointing out new regularities in an existing set of information and explaining how these are used to make decisions. For example, consider a survey respondent or business liaison contact seeking to understand why its firm increased prices by appealing to the macroeconomic determinants of the decision, which come from readily available information. The respondent favours a clear and concise explanation because they need to communicate this information to others. We argue the complexity of delivering a clear and concise explanation – that is, choosing a small number of explanatory factors out of a large set of possible candidates to achieve a given level of explainability – means that simple regularities based on existing facts are often overlooked (Aragones et al 2005). Once these regularities are clearly explained by their firm executives in the earnings calls, the logic can seem evident, and causes an update of the outlook due to a reassessment about how decisions will be made in the future.

This process of fact-free learning from earnings calls has parallels with recent explanations for why central bank communication can change the market's assessment of current economic conditions – the so-called ‘communication channel’ of monetary policy. Bauer and Swanson (2020) argue that a differential assessment of publicly available information between the central bank and the private sector can cause the private sector to revise its outlook, which the authors term the ‘response to news’ channel. For instance, the central bank could be putting a different weight on certain variables at different times to explain why particular decisions were made. Regardless of the precise mechanism, both explanations highlight why an expert evaluation of publicly available data can change an assessment of current economic conditions.

The three reasons outlined above give rise to some simple testable hypotheses, which if confirmed, would suggest there is contemporaneous or forward-looking information in the earnings call transcripts that is useful for understanding firms' price-setting behaviour:

• H1: the indices should have a strong contemporaneous or leading relationship with regular surveys of firms designed specifically to measure business conditions, including with respect to our concepts of interest (e.g. input costs, demand and final prices).
• H2: the indices should have a strong contemporaneous or leading relationship with similar real-time information obtained from the Reserve Bank of Australia's extensive business liaison program.

Confirming these hypotheses only provides suggestive evidence of the existence of real-time information in the earnings calls that is relevant for assessing current economic conditions. It could also be the case that earnings calls, regular surveys of firms and the information extracted from business liaisons are all similarly backward looking. To address this, we also examine whether our indices have a strong contemporaneous (or predictive) relationship with related official statistical measures in their reference quarter.^[8]

4.2 Results

Before testing the hypotheses above, we first describe the usefulness of our aggregate indices as a narrative tool for explaining aggregate fluctuations in economic conditions. Figure 5 shows that the aggregate indices from both methodologies can identify significant turning points in the economy. For instance, all three indices (input costs, final prices and demand) fell sharply during the global financial crisis of 2008 and at the onset of the pandemic and have increased sharply since the middle of 2021. Sentiment regarding demand, prices and input costs has moderated lately, with the demand index identified using the zero-shot text classifier leading the declines in the price indices. This same dynamic also occurred following the global financial crisis. Disaggregated input cost indices from the zero-shot classifier are also consistent with recent developments, with sizable increases in references to general supply shortages, hiring difficulties and labour costs (Figure 6). In addition to this, the dictionary-based subindices show large increases in concerns related to rents (Figure A1).

To perform a preliminary test of our hypotheses above about the timeliness of the information derived from earnings calls, Figure 7 compares sentiment indices obtained from the earnings calls to those constructed from two other sources.

1. The Reserve Bank of Australia's business liaison program – this is a formal program of economic intelligence gathering established over 20 years ago, through which Reserve Bank of Australia staff meet frequently with firms from a pool of around 900 active contacts (Dwyer, McLoughlin and Walker 2022). Details of these discussions are systematically recorded in confidential ‘diary notes’. We use the text of these notes to construct indices for input costs, demand, final prices and labour costs using a similar approach as that applied to firms' earnings calls.
2. A monthly survey of around 400 firms from the National Australia Bank (NAB) – this is a survey designed to produce statistical indices related to business conditions. We compare our text-based indices to the NAB survey-based indices for purchase costs, forward orders, selling prices and labour costs.

As shown in Figure 7, all three series – from earnings calls, business liaison and business surveys – appear tightly associated, hinting that there is real-time information in the earnings calls that is relevant for assessing current economic conditions.^[9]

To test our hypotheses more formally, Table 2 shows the results of statistical tests for predictive or contemporaneous relationships. Our sentiment indices for input costs, demand, labour costs and prices are compared to comparable indices derived from business liaison and business surveys. In doing so, we do not account for publication lags in the release of the business survey indices (which are typically released with a three-week lag). In addition to this, the sentiment indices for input costs, demand, labour costs and final prices are compared to official statistics for growth in producer prices (PPI), domestic final demand (DFD), compensation of employees (COE) and consumer prices (CPI), respectively. Again, in these comparisons we pretend data on growth in the PPI, DFD, COE and the CPI are available immediately after the end of the quarter. That is, we assume there is no lag in the release of these statistics, while in reality there is a sizable publication lag.

The key finding from this exercise, before going into the details below, is that there is robust evidence information derived from earnings calls is contemporaneous with real-time updates received from the Reserve Bank of Australia's business contacts, regular business conditions surveys and with related statistical measures. This is consistent with the hypothesis that information from the earnings calls is relevant for assessing current economic conditions.

More specifically, Table 2 shows the results of bivariate Granger causality tests between the indices derived from earnings calls and their counterparts from either business conditions surveys, the Reserve Bank of Australia's business liaison or from related statistical releases.^[10] To do this we run 12 bivariate VARs of the form:

where y_t is the measure of interest. For example, in the first regression, y_t includes the input cost sentiment index from earnings calls and the related measure from a survey of firms' purchase costs, with the results shown in the first row of Table 2. In the second regression, y_t includes the input cost sentiment index from earnings calls and the related measure from the business liaison program, with the result shown in the second row of Table 2, and so on and so forth. The procedure we use to establish Granger causality follows Toda and Yamamoto (1995), using information criterion to determine the appropriate maximum lag length for the variables. Importantly, by including lagged values of both series, this approach deals with serial correlation in the macroeconomic situation, which might explain how backward-looking indicators may still have predictive power for future variables. The results in Table 2 restricts the analysis to the March and September quarters only, which is when most firms hold their earnings calls. Using quarter-end dates puts the earnings calls at a slight informational disadvantage, as the main reporting season occurs over the month of February and August, one month before the end of the quarter.

In the comparison with business surveys there is evidence that past information from the sentiment indices can help to predict future values. That is, the sentiment indices from earnings calls ‘Granger cause’ their conceptual counterparts from the business conditions survey (except for the series for selling prices). In the comparison with business liaison, the sentiment indices for input costs and demand are useful for predicting information from liaison; however, past information in liaison about labour costs and final prices appears informative for future information extracted from earnings calls. In the comparison against official statistical measures, the sentiment indices for input costs and final prices can help predict official statistics for growth in the PPI and CPI in the reference period. Finally, the last column of Table 2 shows peak correlations between each of the series and the period in which these occur. Peak correlations all occur in the same period, except for the input cost index, with the peak correlation occurring when liaison information leads information from earnings calls by one period, and the final price index, with the peak correlation occurring when earnings calls leads inflation by one period. Table 3 repeats the results but at a quarterly frequency, which disadvantages the earnings calls further still, given far fewer firms hold earnings calls in the June and December quarters. Regardless, the results from this comparison are broadly similar.

Footnotes

It is important to note here that we are only trying to establish if information taken from earnings calls are relevant for assessing current economic conditions. We are not examining if there is any information about the statistical measures over and above that contained in other indicators of current economic conditions. That is, we are not concerned with examining if the information extracted from earnings calls is useful for making marginal improvements to nowcasts of these variables. [8]

These correlations are still strong, albeit a little weaker, if we exclude the period since 2021. [9]

In comparing earnings calls to business liaison, we also ran Granger causality tests using a matched sample of 180 firms appearing in both the earnings call sample and the Reserve Bank of Australia's business liaison program. To record a match, firms had to appear only once in both samples – we did not require that firms also appear in the same time period. The results are like those reported for the full sample, although correlations are weaker (see Appendix B). [10]