The credibility revolution has made causal inference methods ubiquitous in economics. Yet there is widespread evidence of selection on statistical significance and associated biases in the literature. I show that these two phenomena interact to reduce the reliability of published estimates: while causal identification strategies alleviate bias from confounders, by restricting the variation used for identification they reduce statistical power and can exacerbate another bias–exaggeration. I characterize this confounding-exaggeration trade-off theoretically and through realistic Monte Carlo simulations, and explore its prevalence in the literature. In realistic settings, exaggeration can exceed the confounding bias these methods aim to eliminate. Finally, I propose practical solutions to navigate this trade-off, including a versatile tool to identify the variation and observations actually driving identification in applied causal studies.
This paper identifies design parameters that can lead to inaccurate estimates of small effects. Low statistical power not only makes such effects difficult to detect but resulting significant estimates also necessarily exaggerate true effect sizes on average. Through the literature on short-term health effects of air pollution, I explore this issue and its policy implications. Exaggeration can be substantial and power low even with large sample sizes. Real-data simulations highlight key additional drivers: the number of exogenous shocks, instrument strength, and outcome count. I propose a workflow to evaluate and mitigate exaggeration risk in non-experimental studies.
Selected work in progress
Climate Change Narratives: Environmental News in French Newscasts
