RDP 2021-02: Star Wars at Central Banks Appendix B: Replicating Simonsohn et al (2014) and Brodeur et al (2016)

In our reproduction of Brodeur et al (2016), we found only minor problems:

1. The paper presents many kernel densities for distributions of test statistics, which it characterises as having a ‘two-humped’ camel shape. The kernel densities do not have any boundary correction, even though the test statistics have a bounded domain. The first hump in the camel shape is in many cases just an artefact of this error. This problem has no effect on the paper's key quantitative conclusions. Moreover, the raw distributions are pictured, providing full transparency.
2. Whenever a paper showed only point estimates and standard errors, the authors infer z-scores assuming a null hypothesis of zero and a two-sided alternative hypothesis. But 73 of these cases have non-zero null hypotheses and/or one-sided alternative hypotheses, meaning the inferred z-scores are incorrect. The effects of this problem are trivial because the tests in question make up less than 0.2 per cent of the sample. Note that whenever we present results from the top journals sample, we drop these incorrect scores.
3. The results presented in Table 2 of Brodeur et al (2016) differ noticeably from the output of their code (hence why the results presented in our Table 2, some of which are meant to line up with Table 2 of Brodeur et al, look different). We attribute this discrepancy to transcription error, because elsewhere in their paper, any discrepancies between their code output and published tables are trivial. The errors in Table 2 of Brodeur et al are not systematically positive or negative, and do not affect the main conclusions.
4. The authors claim that ‘[n]onmonotonic patterns in the distribution of test statistics, like in the case of a two-humped shape, cannot be explained by selection alone’ (p 11). This statement is incorrect, as we illustrate in Figure 2 of this paper. Since the two-humped shape features prominently in their paper, we feel it worthy of discussion here. In other parts of their paper, most notably Section IV, the authors seem to recognise that a two-humped shape can be explained by selection. So we discuss the issue here only as a matter of clarification.

We successfully replicated the results of Simonsohn et al (2014) using a web-based application that the authors make available (version 4.06) and their two datasets (obtained via personal communication). The only differences were immaterial and we think attributable to slight improvements to the web application (see Simonsohn, Simmons and Nelson (2015) for details). The authors provided us with their data soon after our request.