Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.
Humans are bad at estimating. Really bad.
We are cursed with optimism and blind to points of failure. We plan assuming nothing will go wrong, we believe that the problem is clearly understood and the solution is well scoped.
There won’t be any interruptions or distractions, no blockers or delays. Things will flow gently down the stream, along the yellow brick road, like daisies on a gentle breeze.
Then reality hits. Like a brick wall to the face in a nuclear winter. The 6-week dream turns to 10-12-16-24!
This is the planning fallacy: We suffer from an optimism bias and always underestimate how long a task (or project) will take. This occurs even when we are asked to consider as much detail as possible about the work.
But this time I’ll be realistic
Study after study has shown that we are eternal optimists when estimating our own work. So much so that when asked for predictions based on:
- realistic “best guess” scenarios; vs
- hoped-for “best case” scenarios
we give the same answers!
Thankfully there is a way to temper that pesky optimism.
Take the Outside View
Forget about trying to break down a project into tasks, hours, days and then adding up the times. Instead, identify some roughly similar projects that you’ve previously completed and use their durations. That’s it.
Previous projects use real historical data. They are unbiased by optimism and take in the cold hard truth of delays and unknown unknowns.
“But past projects can’t know about the specifics of this project! What about this particular complexity? We need to consider it!”
But that’s precisely the point. Taking the “inside view” and trying to consider the granular specifics of a project will never account for the unexpected. Sadly, this is true even when we are asked to consider the worst case scenario.
Phone a friend
The alternative to thinking of your own past projects is to ask an expert. An outsider who isn’t aware of the details of a project but has plenty of experience to draw from will be more realistic (pessimistic!) and likely more accurate.
In the wild
We see the outside view applied to some common software development workflows:
Story point estimation. Teams assign a level of effort (not time!) relative to past tasks of similar effort. A good pointing process will periodically review past estimations and recalibrate as needed.
Even better, Kanban advocates for measuring Cycle Time: the time it takes for tasks to move from In Progress to Done. What’s important to note is that nowhere do you ask anyone to estimate anything. You simply take the historical data which allows you to forecast a rough backlog completion.
Step outside
A bias of optimism can often be a good thing. But when planning it is our downfall. Don’t sweat the small stuff, take the outside view; then baulk at your seemingly outrageous yet accurate estimates!
