Research Discussion Paper – RDP 2019-08 Non-technical summary for ‘The Well-meaning Economist’
September 2019
Most economic research seeks to reveal truths about economic relationships. This paper is different because it reflects on how we reveal those truths, focusing on a research strategy that is common in the data analysis stage. The paper introduces alternative strategies and shows that they can produce very different conclusions. Importantly, those conclusions have properties that often make them more relevant for policymakers. The sequence of arguments is targeted at researchers but a simplified hypothetical example, given here, can convey the key ideas.
Suppose we discover today that in 1990 a random subset of Australian schoolchildren were given a badly misprinted version of the standard mathematical textbook. Its answers were wrong and the explanations were nonsense. Suppose also that today we can survey these and the unaffected schoolchildren (all now adults) about their incomes. Besides objecting to the injustice of the misprint, an economic researcher might see this as a unique opportunity to assess the value of effective educational materials for career outcomes.
Conducting the survey, our hypothetical researcher records that half of the affected students now have annual incomes of $40k and half have $100k. For the unaffected students, half have annual incomes of $60k and half have $80k. For reasons that I leave to the paper, the standard strategy in this simplified situation would be to summarise the salaries of each group with their so-called ‘arithmetic mean’, which is a basic type of average. Since both groups have arithmetic mean incomes of $70k, the headline conclusion for the policymaker is that the misprint was unimportant. Even in complex research situations, summarising outcomes with numbers akin to these arithmetic means is a standard strategy.
But what if we choose a different summary measure, like a ‘geometric mean’, or any other mean in the ‘quasilinear’ family? Leaving an explanation of these concepts aside, the point is that often the conclusions will change. For instance, in the textbook misprint case, the geometric mean income for the affected students is $63k and for the unaffected students is $69k. Hence the headline conclusion for the policymaker is that the misprint was detrimental. The reason for the change is that the geometric mean penalises inequality, which is higher among the affected students. The penalty is an attractive feature here, because in western democracies it is evident from tax and social security systems that policymakers view income inequality as undesirable. The question is only what amount of penalty is appropriate.
In addition to this inequality aversion (or ‘risk aversion’), the paper discusses three other criteria that researchers can use to choose between means. Two of them will not be discussed here because they are technical. The third relates to the ease of working with different types of means, because researchers might reject them on account of feeling more comfortable using tools that are built around arithmetic ones. The paper demonstrates that even in complex research situations we can adapt existing tools to work with different types of means. In many cases, working with the different types can simplify the research process, saving time and improving transparency.
We would do well to choose our means well.