--- title: 'Take the Outside View: A solution to the planning fallacy' teaser: "Optimism is our kryptonite but history is our saviour. \n" tags: planning,estimation,product management author: Jared Turner published_on: 2023-06-30 --- > Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. Humans are bad at estimating. Really bad. We are cursed with optimism and blind to points of failure. We plan assuming nothing will go wrong, we believe that the problem is clearly understood and the solution is well scoped. There won't be any interruptions or distractions, no blockers or delays. Things will flow gently down the stream, along the yellow brick road, like daisies on a gentle breeze. Then reality hits. Like a brick wall to the face in a nuclear winter. The 6-week dream turns to 10-12-16-24! This is the [planning][planning] [fallacy][fallacy]: We suffer from an optimism bias and always underestimate how long a task (or project) will take. This occurs even when we are asked to consider _as much detail as possible_ about the work. ## But this time I'll be realistic [Study][study1] [after][study2] [study][study3] has shown that we are eternal optimists when estimating our own work. So much so that when asked for predictions based on: - realistic "best guess" scenarios; vs - hoped-for "best case" scenarios we give [the same answers]! Thankfully there is a way to temper that pesky optimism. ## Take the Outside View Forget about trying to break down a project into tasks, hours, days and then adding up the times. Instead, identify some roughly similar projects that you've previously completed and use their durations. That's it. Previous projects use real historical data. They are unbiased by optimism and take in the cold hard truth of delays and unknown unknowns. > "But past projects can't know about the specifics of _this_ project! What about this particular complexity? We need to consider it!" But that's precisely the point. Taking the "inside view" and trying to consider the granular specifics of a project will never account for the unexpected. Sadly, this is true even when we are asked to consider the worst case scenario. ## Phone a friend The alternative to thinking of your own past projects is to ask an expert. An outsider who isn't aware of the details of a project but has plenty of experience to draw from will be more realistic (pessimistic!) and likely more accurate. ## In the wild We see the outside view applied to some common software development workflows: [Story point estimation]. Teams assign a level of effort (not time!) relative to _past tasks of similar effort_. A good pointing process will periodically review past estimations and recalibrate as needed. Even better, Kanban advocates for measuring Cycle Time: the time it takes for tasks to move from In Progress to Done. What's important to note is that nowhere do you ask anyone to estimate _anything_. You simply take the historical data which allows you to forecast a _rough_ backlog completion. ## Step outside A bias of optimism can often be a good thing. But when planning it is our downfall. Don't sweat the small stuff, take the outside view; then baulk at your seemingly outrageous yet accurate estimates! [planning]: https://en.wikipedia.org/wiki/Planning_fallacy [fallacy]: https://www.lesswrong.com/posts/CPm5LTwHrvBJCa9h5/planning-fallacy [study1]: https://web.archive.org/web/20130908065829/http://www.dtic.mil/dtic/tr/fulltext/u2/a047747.pdf [study2]: https://www.researchgate.net/publication/233153892_It's_About_Time_Optimistic_Predictions_in_Work_and_Love [study3]: https://www.researchgate.net/publication/232604065_Inside_the_Planning_Fallacy_The_Causes_and_Consequences_of_Optimistic_Time_Predictions [the same answers]: https://www.researchgate.net/publication/12309770_People_focus_on_optimistic_scenarios_and_disregard_pessimistic_scenarios_when_predicting_task_completion_times [Story point estimation]: https://www.atlassian.com/agile/project-management/estimation