Lecture 4 - Causal Exaggeration

class: right, middle, inverse, title-slide

.title[
# Lecture 4 - Causal Exaggeration
]
.subtitle[
## <br> Topics in Econometrics
]
.author[
### Vincent Bagilet
]
.date[
### 2024-10-02
]

---

class: right, middle, inverse

# Exercise

## Treatment effect heterogeneity

---
class: titled, middle

# Simulating treatment effect heterogeneity

- How did you model heterogeneity?

- Did your model represent the true DGP? What happens if it does not?

- Questions about calibration?

- Bonus Question: how to proceed to keep track of all analyses (with varying DGP)?

---
class: titled, middle

# For next week

- Calibration exercise

- Any comments on the exercise?

- Is there anything you would like me to discuss before the end of next class?

---
class: right, middle, inverse

# Summary from last week

---
class: titled, middle

# Summary from last week

- Simulations: varying parameters and complexifying

- Fire alarm break
  
- Low statistical power leads to exaggerated significant estimates

---
class: right, middle, inverse

# Causal Exaggeration

---
# The Confounding-Exaggeration Trade-Off

.pull-left[

- Challenge of empirical economics: identifying causal effects

- Only use part of the variation: the exogenous part

- Reduces precision and statistical power

- `$\mathbb{E}[\hat{\beta}] = \beta$` but `$\mathbb{E}[\hat{\beta} | \text{ Significant}] \neq \beta$`
]
--
.pull-right[
<img src="data:image/png;base64,#images/intuition_trade_off.png" width="600" style="display: block; margin: auto;" />
]

---
# Unconfounded but Inflated

- Problem: `$\exists$` **publication bias**

- Can explain part of the replication crisis in economics

- In economics, nearly 80% of estimates are exaggerated by a factor of 2 (!!)

- Exaggeration could be on par with OVB

- Trade-off confounders/exaggeration: **avoiding a bias creates another type of bias**

- Shared mechanism (causal strategies limit variation) but specific channels

- Thinking of causal stragtegies as control approaches ties all channels together

---
class: titled, middle

# A Potential Example of Exaggeration

---

# Intuition for controls and exaggeration
  
  
- Let `$y_{i} = \alpha + \beta x_{i} + \delta w_{i} + u_{i}$`, thus

`$$\sigma_{ovb}^2 =
          \dfrac{\sigma^{2}_{u_{ovb}}}{n \ \sigma_{x}^{2}} = 
				  \dfrac{\sigma^{2}_{y^{\perp x}}}{n \ \sigma_{x}^{2}}
				  \qquad \text{and} \qquad
				  \sigma_{ctrl}^2 = 
          \dfrac{\sigma^{2}_{u_{ctrl}}}{n \ \sigma_{x^{\perp w}}^{2}} = 
				  \dfrac{\sigma^{2}_{y^{\perp x, w}}}{n \ \sigma_{x^{\perp w}}^{2}}$$`
		
						
--

- Controlling `$\nearrow$` variance (and exaggeration) if the fraction of the variance unexplained by `$w$` is smaller for `$x$` than for `$y^{\perp x}$`

- `$\Leftrightarrow$` if it absorbs more of the variation in `$x$` than in the residual part of `$y$` ( `$y^{\perp x}$`)

---

# RDD

- **Identification approach**: only uses  observations inside the bandwidth

-  Economics of education: impact of additional lessons on test scores

- `$Final_{i} = \beta_{0}  + \beta_{1} T_i + \eta Qual_{i} +  \delta W_i^{3} + u_{i}$`

- `$Qual_i = \mu + \gamma W_i^{3} + \epsilon_i$`

---

---

# IV

- **Identification approach**: only uses the part of the variation in `$x$` explained by the instrument

- Political economy: impact of turnout on vote share

- `$Turnout_{i} = \pi_{0} + \pi_{1} Rain_{i} + \gamma w_{i} + e_{i}$`

- `$Share_{i} = \beta_{0} + \beta_{1} Turnout_{i} + \delta w_{i} + u_{i}$`

---

---

# Exogenous shocks

- **Identification approach**: only uses exogenous variation from shocks

- Impact of air pollution (plant closure) on birthweight

- `$bw_{z,t} = \beta_{0} + \beta_{1} T_{z, t} + \zeta_z + \tau_t + u_{z,t}$`

---

---

# Fixed Effects

- **Identification approach**: only uses the remainin variation after partialling out fixed effects

- Impact of high temperature on worker productivity

---

---

---
class: titled, middle

# Gauging Risks of OVB and Exaggeration

- In essence impossible to measure

- For OVB: **sensitivity analyses** (e.g. Cinelli and Hazlett, 2020)

- Power calculations: not only useful to know if we will be able to detect an effect

- **Prospective simulations**: simulate the DGP

- Helps think about effect sizes, relation between the variables, etc

- **Retrospective calculations**: retrodesign(beta, se)

---
class: titled, middle

# Driver of the Trade-off

.pull-left[
- Exaggeration `$\Rightarrow$` variance matters even **after** a significant estimate has been obtained

- Barely significant result: sign of low power and potential exaggeration

- Evaluate the variation used for identification

- Visualize where the variation comes from
]
.pull-right[

<img src="data:image/png;base64,#images/starwars.png" width="90%" style="display: block; margin: auto;" />
]

---
class: titled, middle

# Structural Approaches

- Without publication bias this issue disappears

- Abandoning the 5% significance threshold

- Interpretation of CI’s width to embrace uncertainty

- Replication of studies with similar designs

---
class: titled, middle

# Summary

- Lack of power and significance filter in economics

- Recipe for exaggeration

- What are the drivers?

- This paper: **causal methods can exacerbate exaggeration**

- Addressing the issue:

- Power calculations
  
  - Highlighting where the variation comes from

---
class: right, middle, inverse

# Calibrating simulations

---
class: titled, middle

# Why calibrating?

- Make simulations more realistic

- But simulations will never be truly realistic

- But can still allow to run some sort of **robustness check**

- Again,

---
layout:true

# Fake data simulations

---

### Distributions of the variables

- Emulate the distribution of variables in existing data sets

---
class: titled, middle

### Relationships between variables

- Read the literature

- Get a sense of **typical effect sizes and of relationships** between variables

- Make assumptions on those relationships. Acknowledge them.

- Complexify later if needed. You choose when you stop

- Varying parameters values might change the distribution of "variables" (*eg* the error term)

---
layout: false
class: titled, middle

# Real data simulations
### General approach

- Start from an existing data set

- Not yours. At least not the subset you are interested in

- Try to pick a subset where there is not already a treatment effect

- Define a treatment allocation mechanism

- Add an artificial treatment effect to the outcome variable in our initial data set ( `$y_i = Y_i(0)$` )

`$$Y_i(1) = Y_i(0) + \beta_i T_i$$`

---
# Real data simulations
### Complexifying

- Only one artificial aspect: the treatment

- We can play on only 2 components:

.pull-left[
- **Who is treated?** *Treatment allocation*
  
  - Everyone
  
  - Only a subset of the population
]
.pull-right[
- **How?** *Treatment effect*

- Homogenous
  
  - Heterogenous but random
  
  - Some specific correlation structure
]
  
---
class: right, middle, inverse

# Summary

---
class: titled, middle

# Punchlines

- Credibility revolution + convincing identification methods *BUT* replication failures

- When power is low, significant estimates **are always** far from the true effect

- The bias/variance trade-off can in fact be a bias/bias trade-off

- **Variance matters**, even once a significant estimate has been obtained

- Calibrate your simulation based on existing information

???

- Even if the distribution of our causal estimator is centered around the true effect, the distribution we actually draw estimates from may not be.

---
class: right, middle, inverse

# Thank you!