Post estimation analysis
Let’s focus on the example of described for the RDD simulations, studying the impact of a school teaching program on student final scores. Assume that using real world data, we find an estimate of the treatment effect to be equal to 0.22 with standard deviation of 0.2. We want to compute what could be the power and exaggeration. These values depend on the true effect size. We therefore compute them for a range of credible true effect sizes. Another way to look at the problem is to wonder whether, if the true effect was smaller, the design of this study would be “good enough” to detect an effect and by how much we can expect the estimate to be inflated.
Replicating the analysis with the same design would yield an estimate with a similar standard error as the original study. We therefore postulate different true effect sizes and feed them to retrodesign:
se <- 0.2
retro <- retrodesign::retro_design(as.list(seq(0.1, 0.7, by = 0.01)), se) %>%
unnest(cols = c(effect_size, power, type_s, type_m)) %>%
select(-type_s) %>%
mutate(power = power * 100) %>%
rename(
"Statistical Power (%)" = power,
"Exaggeration Ratio" = type_m
) %>%
pivot_longer(
cols = -c(effect_size),
names_to = "statistic",
values_to = "value"
)We can then build simple graphs representing what would be the power and exaggeration ratio for a range of hypothetical true effect sizes:
Show code used to generate the graph
retro %>%
ggplot(aes(x = effect_size, y = value)) +
geom_line(size = 1.1) +
facet_wrap( ~ statistic, scales = "free") +
# geom_vline(aes(xintercept = 0.4)) +
geom_vline(aes(xintercept = 0.3)) +
scale_y_continuous(breaks = scales::pretty_breaks(n = 5)) +
labs(
title = "Evolution of power and exaggeration with true effect size",
x ="Hypothetical True Effect Size",
y = "",
caption = paste("Standard error =", se)
)