Exercise - Simulating treatment effect heterogeneity

The impact of extra lessons on students’ grades, a statistical power analysis.

Instructions

Before class next week (October 2nd), please send me via email a .html document (not a .qmd document), generated with Quarto, implementing the analysis required and answering the questions listed in the present document.

Please implement the whole simulation from scratch. You can copy/paste what we have done before and use this as a starting point.

Make sure to include the following lines as an option to your document (at the very top of your .qmd document, between the two series of ---. Don’t forget to remove the pre-existing format: html line, if there is one). It will produce a self-contained html, ie, a nicely rendered html that stands alone:

format:
  html:
    embed-resources: true

So far, the power analysis we implemented for our economics of education experiment assumed that we accurately represented the true DGP. However, it is very likely that the actual DGP would be different from the one we modeled. For instance, effects are probably heterogenous across individuals.

Note

We are often interested in estimating a version of an ATE (Average Treatment Effect). The wording itself (“average”) implies that we indeed expect effects to be heterogenous across individuals.

In this exercise, you will implement the same power analysis as the one we implemented before but with some sort of heterogeneity in treatment effects.

Question 1

What other reasons than heterogeneity in treatment effect could cause the DGP to be different from the one we simulated so far?

Although there are other reasons for the DGP to be different, we will focus on heterogeneity here. To add heterogeneity to the analysis, we need to modify .

There are of course many ways to model heterogeneity in treatment effect. Pick one, modify what is needed and display your code below.

# Copy/paste and modify the code we wrote together
# (only the necessary bits)

Question 2

Justify your choices in maximum 5 lines (there is no correct answer, you just need to justify your choice a bit).

As discussed in class, the true effect size matters a lot for the resulting statistical power. Quickly explore the economics of education literature (for instance Kraft (2020)) to get a sense of the typical magnitude of treatment effects in this literature.

Question 3

What value would you choose for the true effect? Very briefly explain your choice.

Run the power simulation, varying the sample size.

# Copy/paste the rest of the code necessary to run the simulation

Question 4

Which sample size would you choose? Explain and return one graph (explaining why you chose this type of graph).

Which condition is necessary for your answer to hold?

The true DGP needs to be similar to what you simulated.

Now imagine you would like to modify the DGP again. Modifying the code and running the analysis again would prevent you from keeping track of the previous analysis and would not be very reproducible. Copying and pasting the code before modifying it would not be reproducible either.

Bonus Question

How would you proceed so that you would keep track of your previous analyses, with a different DGP? Answer in just a couple of lines.