Core Idea
Definition
Calibration practice measures whether your stated confidence levels align with how often you are actually right.
In Plain English
If you say you are 80 percent sure, then roughly 8 out of 10 similar judgments should be correct.
Framework Structure
Components
Flow
Make forecast -> State confidence explicitly -> Observe outcome -> Compare confidence with reality -> Adjust future judgment
How to Apply
- 1.Make predictions in explicit probabilistic terms instead of vague language
- 2.Track the prediction and its confidence level
- 3.Review outcomes over enough cases to see patterns
- 4.Notice whether you are systematically overconfident or underconfident
- 5.Adjust your future probability estimates based on the feedback
When to Use
- •Forecasting and planning
- •Managers making repeated judgment calls
- •Analysts, investors, operators, and product teams
- •Any domain where probabilities are stated or implied repeatedly
- •Improving metacognitive accuracy over time
When NOT to Use
- •When outcomes are too vague to score honestly
- •When the decision is purely one-off and no learning loop is possible
- •When confidence is being tracked performatively rather than used for improvement
- •When feedback is so delayed that the signal is almost unusable
Example
Problem
A manager wants to improve how accurately they estimate whether projects will ship on time.
Application
- 1.For each project checkpoint, assign a probability of on-time delivery
- 2.Track the forecast and the eventual result
- 3.Review whether projects labeled 70 percent likely actually ship on time around 70 percent of the time
- 4.Adjust future estimates if the manager is consistently too optimistic or too pessimistic
Conclusion
The manager gradually replaces vague confidence with a more reality-matched forecasting habit.
Takeaway
Calibration improves not by feeling smart in the moment, but by letting outcomes teach your confidence levels.
Common Mistakes
- •Using only yes or no language rather than confidence ranges
- •Recording predictions but never reviewing them
- •Treating calibration as ego management rather than accuracy training
- •Scoring too few cases and drawing strong conclusions
- •Ignoring ambiguity in what counts as a correct outcome
How to Practice
prediction log
Keep a running log of predictions, confidence levels, and outcomes.
range language
Replace words like maybe or likely with approximate probability ranges.
quarterly calibration review
Review a batch of past predictions to see where your confidence ran too high or too low.
Related Cognitive Biases
overconfidence
People systematically express more certainty than their accuracy deserves.
hindsight bias
After the outcome, people misremember how obvious it seemed beforehand.
self serving bias
People may grade their own predictions in ways that protect ego instead of improving accuracy.
Related Frameworks
Related Skills
Variants & Extensions
Typical Failure Modes
- •No outcome tracking
- •Too little data
- •Ego-protective review
Further Reading
- Superforecasting by Philip E. Tetlock and Dan Gardner
- Thinking in Bets by Annie Duke
- Noise by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein