Causal Exaggeration and Calibration

Causal identification strategies are essential to make causal claims. They can however also come at a cost and create bias (exaggeration). We discuss this trade-off in this session.

Objective

After this session, you should understand the importance of the variation used to identify the effect of interest and be able to calibrate your simulation.

Summary

This session focuses on causal exaggeration¹. Causal identification strategies only use part of the variation, the exogeneous part. This reduces precision and statistical power and can create exaggeration, leading significant and published estimates to exaggerate the true effect, even when the estimator is unbiased in the traditional sense. The variation used for identification is the key driver of the resulting trade-off between confounding and exaggeration.

To illustrate this, I built fake-data simulations that I calibarated based on actual studies. This lead us to discuss why and how calibrating simulations.

Session Outline

1. Homework: Treatment Effect Heterogeneity
2. Summary from last week
3. Causal exaggeration
4. Calibrating simulations

Materials

Open slides

Exercise

Before Friday October 11, 7pm, please submit here a html document, generated with Quarto, implementing the analysis described in this document.

Specific resources for this lecture

• This class is based on one of my working papers. You can find more details there.

Comparison IV and OLS

• Young (2022) replicate 30 papers from the economics literature (AEA journals). Find that:
  • 75% of the 2SLS 95% CI contain the corresponding OLS point estimates (67.3% of main results)
  • IV estimates often larger (in absolute terms) or opposite sign than OLS: “greater than 0.5 times the absolute value of the OLS point estimate in .73 of headline regressions”
  • 2SLS estimates usually do not provide meaningful information regarding the extent to which OLS are biased
• Lal et al. (2024) replicate 67 papers from the political science literature. Find that:
  • For 97% of designs studied 2SLS > OLS (34% at least 5 times larger)

Footnotes

1. This session builds on a working paper I am working on.