The credibility revolution has made causal inference methods ubiquitous in economics, yet it coexists with selection on significance. I show that these two phenomena interact in ways that reduce the reliability of published estimates: while causal identification strategies alleviate bias from confounders, they reduce statistical power and can generate another type of bias–exaggeration–when combined with selection on significance. I characterize this confounding-exaggeration trade-off theoretically and via realistic Monte Carlo simulations, and document its prevalence in the literature. I then propose practical solutions, including a tool to identify the variation actually driving identification.
This paper identifies tangible design parameters that might lead to inaccurate estimates of relatively small effects. Low statistical power not only makes relatively small effects difficult to detect but resulting published estimates also exaggerate true effect sizes. Through the case of the literature on the short-term health effects of air pollution, we document the prevalence of this issue and identify its drivers using real data simulations replicating most prevailing identification strategies used in economics. While the analysis builds on a specific literature, it draws out insights that expand beyond this setting. Finally, we discuss approaches to evaluate and avoid exaggeration when conducting a non-experimental study.
Work in progress
Climate Change Narratives: Environmental News in French Newscasts
