Core Idea
Definition
A randomized controlled trial infers causal effects by comparing outcomes across groups whose treatment assignment was determined by randomization.
In Plain English
If similar people or units are randomly split into groups, differences in outcome are more credibly attributed to the intervention.
Framework Structure
Components
Flow
Randomly assign units -> Apply treatment to one group -> Measure outcomes -> Compare treated and control groups
How to Apply
- 1.Define the intervention and the outcome clearly
- 2.Randomly assign units to treatment and control conditions
- 3.Keep other conditions as comparable as possible
- 4.Measure outcomes consistently across groups
- 5.Interpret the difference in outcomes as evidence about the causal effect, while checking implementation quality
When to Use
- •Testing interventions where random assignment is feasible and ethical
- •Product experiments and A/B tests
- •Medical, educational, or policy evaluation
- •High-stakes causal questions where observational evidence is weak
- •Separating real effects from confounded correlations
When NOT to Use
- •When randomization is unethical or infeasible
- •When compliance, spillovers, or attrition undermine the design badly
- •When the trial environment becomes too artificial to generalize
- •When the question is exploratory and not yet ready for formal intervention testing
Example
Problem
A product team wants to know whether a new onboarding sequence increases activation.
Application
- 1.Randomly assign new users to the old or new onboarding flow
- 2.Ensure assignment is implemented reliably
- 3.Measure activation and downstream quality metrics consistently
- 4.Compare outcomes between groups to estimate the intervention's causal effect
Conclusion
Because assignment was randomized, the team can interpret differences more credibly as caused by the onboarding change.
Takeaway
RCTs are strongest when the design, execution, and interpretation are all handled with discipline.
Common Mistakes
- •Assuming randomization guarantees perfect execution
- •Ignoring attrition or noncompliance that reintroduce bias
- •Treating one successful trial as universally generalizable
- •Measuring only the easiest outcome while missing more important ones
- •Overlooking interference between treatment and control groups
How to Practice
assignment audit
Whenever reviewing an experiment, ask how treatment assignment actually happened and whether it was preserved.
outcome completeness check
Look for attrition, missingness, or compliance issues before trusting the comparison.
external validity note
After seeing the result, ask where else the finding should or should not be expected to hold.
Related Cognitive Biases
selection bias
Randomization is designed to reduce distortions caused by who receives the intervention.
confirmation bias
Even with strong design, people may still interpret ambiguous results in self-serving ways.
survivorship bias
Attrition or missing outcomes can quietly bias the comparison if ignored.
Related Frameworks
Related Skills
Variants & Extensions
Typical Failure Modes
- •Noncompliance
- •Attrition bias
- •Weak generalization
Further Reading
- Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang, and Ya Xu
- The Book of Why by Judea Pearl and Dana Mackenzie
- Causal Inference in Statistics by Judea Pearl, Madelyn Glymour, and Nicholas P. Jewell