The credibility revolution in economics has made causal inference methods ubiquitous. Simultaneously, an increasing amount of evidence highlights that the literature strongly favors statistically significant results. I show that these two phenomena interact in a way that can substantially worsen the reliability of published estimates: while causal identification strategies alleviate bias caused by confounders, they reduce statistical power and can create another type of bias—exaggeration— when combined with selection on significance. This is consequential as estimates are routinely turned into decision-making parameters for policy makers conducting cost-benefit analyses. I characterize this confounding-exaggeration trade-off theoretically and using realistic Monte Carlo simulations replicating prevailing identification strategies and document its prevalence in the literature. I then discuss potential avenues to address this issue.
This paper identifies tangible design parameters that might lead to inaccurate estimates of relatively small effects. Low statistical power not only makes relatively small effects difficult to detect but resulting published estimates also exaggerate true effect sizes. Through the case of the literature on the short-term health effects of air pollution, we document the prevalence of this issue and identify its drivers using real data simulations replicating most prevailing identification strategies used in economics. While the analysis builds on a specific literature, it draws out insights that expand beyond this setting. Finally, we discuss approaches to evaluate and avoid exaggeration when conducting a non-experimental study.
