Low Power and Exaggeration

What are some risks of implementing low statistical analyses?

Date

September 25, 2024

Objective

After this session, you should be able to run a simulation, vary its parameters and have a basic understanding of the statistical power, its drivers and what its lack thereof entails.

Summary

In this session, we focus on statistical power and the impact of lack thereof.

First, we study some of its drivers by expanding our simulation. We vary parameters values (sample size, effect size, proportion of treated) and look at their impact on statistical power. Then, we present the related concept of exaggeration and start discussing how it may interact with causal identification strategies.

Session Outline

  1. Summary from last week
  2. R coding on your own: varying parameters
  3. A primer on exaggeration
  4. Introduction to causal exaggeration
  5. Exercise: implementing a complex simulation

Materials

Open slides

Exercise

Before class next week (October 2nd), please send me via email a html document, generated with Quarto, implementing the analysis required and answering the questions listed in this document.

Please implement the whole simulation from scratch. You can copy/paste what we have done before and use this as a starting point.

Specific resources for this lecture

Publication Bias in Economics

  • Doucouliagos and Stanley (2013) 60% of research areas in economics feature substantial publication bias (strongest when dominant theory stronger because difficult to defend results that go against it)
  • Brodeur et al. (2016) document publication bias in top econ journals (+ show that it comes more from the author’s side)
  • Vivalt (2019) studies the extent of p-hacking in impact evaluations (but decrease over time for RCTs)
  • Andrews and Kasy (2019): provides a publication bias correction based on the probability of publication conditional on result + method to identify this probability
  • Brodeur et al. (2020) compare to what extent different causal identification strategies suffer from publication bias. IV (and DiD) suffer more than RCT and RDD
  • Chopra et al (2023) using an experiment with researchers as subject, show that “studies with a null result are perceived to be less publishable, of lower quality and of lower importance”
  • Brodeur et al (2023): issues of marginal significance (publication bias) come more from authors’ behavior than from the peer review process
  • Table 2 in Christensen and Miguel (2018) summarizes this literature

Evidence of low statistical power (and exaggeration)

In Economics

  • Ioannidis et al (2017) use meta-analyses to compute the statistical power of the studies “contained” in these meta-analyses
  • Ferraro and Shukla (2020) use the same techniques as Ioannidis et al (2017) to show that there are power issues in environmental economics
  • Ferraro and Shuklla (2022) same in agricultural economics
  • DellaVigna and Linos (2022): shows that academic papers studying nudges find effects that are much larger than in large nudge experiment ran by nudges companies. Explain this with low power
  • Black et al (2022): show the importance of taking power into account and show how to implement power calculations
  • Young (2022) documents a lack of power of IVs in economics (among other things)
  • In a non-directly related context, Roth (2023) underlines that a lack of power of pre-trend tests in event-study designs can lead to bias on the main estimate

In Political Science

  • Arel-Bundock et al (2022) documents a lack of power in political sciences (median power 10% and only 1 in 10 tests have 80% power to detect the consencessus effects reported in the literature)
  • Lal et al. (2024) documents a lack of power of IVs in political science (among other things)
  • Stommes et al (2023) shows in RD in political sciences are under-powered to detect anything but large effects and lead to exaggeration

Mechanisms behind exaggeration