The credibility revolution in economics has made causal inference methods ubiquitous. Simultaneously, an increasing amount of evidence highlights that the literature strongly favors statistically significant results. I show that these two phenomena interact in a way that can substantially worsen the reliability of published estimates: while causal identification strategies alleviate bias caused by confounders, they reduce statistical power and can create another type of bias—exaggeration— when combined with selection on significance. This is consequential as estimates are routinely turned into decision-making parameters for policy makers conducting cost-benefit analyses. I characterize this confounding-exaggeration trade-off theoretically and using realistic Monte Carlo simulations replicating prevailing identification strategies and document its prevalence in the literature. I then discuss potential avenues to address this issue.
Accurate Estimation of Small Effects: Illustration Through Air Pollution and Health
The last version of the working paper can be downloaded here.
This paper identifies tangible design parameters that might lead to inaccurate estimates of relatively small effects. Low statistical power not only makes relatively small effect difficult to detect but resulting published estimates also exaggerate true effect sizes. We first document the existence of this issue in the epidemiology and economics literature of interest. Then, we identify its drivers using real data simulations that replicate most prevailing inference methods. Finally, we argue relevance to many other literatures and propose a principled workflow to evaluate and avoid exaggeration when conducting a non-experimental study.
Work in progress
The Environment in the Public Sphere: Structural vs Individual Discourses
