RDP 2021-02: Star Wars at Central Banks 2. Researcher Bias is about Undisclosed Exaggeration

Our use of the term researcher bias is based on an influential paper in the medical sciences, by Ioannidis (2005), who defines it as ‘the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced’. It is not to be confused with ‘chance variability that causes some findings to be false by chance even though the study design, data, analysis, and presentation are perfect’ (p 0697).

We use a modified definition, mainly to clarify the most important aspects of what we understand Ioannidis to mean. The main change is to elaborate on ‘should’, to reflect our view that however much a procedure distorts findings, disclosing it exonerates the researcher from bias. In other words, it is non-disclosure, intentional or not, that makes the distortion of results most problematic and worthy of investigation. Hence our definition focuses on undisclosed procedures.

Other research on this topic, including Simonsohn et al (2014) and Brodeur et al (2016), uses different language. For example, Brodeur et al (2016) use the term inflation and define it as a residual from a technical decomposition (the z-curve method). Brodeur et al (2020) use the more loaded term p-hacking. The intentions behind these other terms and definitions, as we understand them, are the same as ours.