Core Idea
Definition
Quasi-experimental designs estimate causal effects from nonrandomized data by exploiting patterns such as timing changes, thresholds, instruments, or matched comparisons that create more credible counterfactuals.
In Plain English
When you cannot run a true experiment, look for situations where the world created something close enough to one.
Framework Structure
Components
Flow
Define causal question -> Find natural comparison structure -> State identification assumption -> Estimate and challenge the effect
How to Apply
- 1.Clarify the causal effect you want to estimate
- 2.Choose a design that fits the available data and context, such as DiD, RDD, IV, or matching
- 3.State the key identifying assumption explicitly
- 4.Check whether the data and context make that assumption plausible
- 5.Interpret the estimated effect with appropriate humility about design limits
When to Use
- •Policy, product, and business questions where RCTs are infeasible
- •Retrospective causal analysis
- •Observational settings with useful natural variation
- •High-stakes decisions where plain correlation is too weak
- •Estimating intervention effects after the fact
When NOT to Use
- •When the identifying assumption is obviously implausible
- •When a causal claim is being forced from data that support only description
- •When stakeholders want RCT-level certainty from very weak natural experiments
- •When the design choice is driven by convenience rather than fit
Example
Problem
A company wants to know whether a policy change improved retention, but it was rolled out to everyone at once.
Application
- 1.Look for natural comparison structure such as staggered rollout, thresholds, or comparable untreated groups
- 2.Choose an appropriate quasi-experimental design
- 3.State the assumptions required for that design to approximate the counterfactual
- 4.Estimate the effect and stress-test whether the result is robust
Conclusion
The company may gain useful causal insight without randomization, but only if the identification logic is credible.
Takeaway
Quasi-experiments are powerful because they respect the need for causal structure even when perfect experiments are unavailable.
Common Mistakes
- •Using a design whose assumptions are not defended
- •Treating a quasi-experiment as automatically causal without testing robustness
- •Ignoring that different designs estimate different kinds of effects
- •Overlooking selection or timing distortions
- •Confusing technical sophistication with credible identification
How to Practice
assumption first reading
When reviewing a quasi-experiment, identify the key identification assumption before looking at the headline result.
alternative story check
Ask what noncausal story could still explain the pattern if the design assumptions fail.
design fit review
Compare whether another quasi-experimental approach would better match the structure of the problem.
Related Cognitive Biases
correlation causation confusion
Quasi-experiments are an attempt to do better than naive before-after comparison.
overconfidence
Complex methods can create more certainty than the design really deserves.
selection bias
Many quasi-experimental methods exist precisely to reduce distortions from nonrandom assignment.
Related Frameworks
Related Skills
Variants & Extensions
Typical Failure Modes
- •Weak identification
- •Hidden selection bias
- •Overclaimed certainty
Further Reading
- Mostly Harmless Econometrics by Joshua D. Angrist and Jörn-Steffen Pischke
- Mastering 'Metrics by Joshua D. Angrist and Jörn-Steffen Pischke
- Causal Inference: What If by Miguel Hernán and James Robins