It might have happened to you too. You think that a certain project will take about five days to get done. Maybe three if everything goes smoothly, a week if something comes up. Yet there you are, halfway through the second week, still working on the project. By this time, it feels as if you've definitely bitten off more than you can chew. Then by the end of the second week, you eventually get it done...

 

It's not that you procrastinated or didn't put out your best effort. It simply took more time than you had thought.

Like it or not, people simply suck at planning. We want to plan, and we're maybe a little obsessed with it too (perhaps in an attempt to control our destiny?), yet most of the time our plans are turned upside down. Sometimes it's because of outside factors, but other times - perhaps most times - it's because of our own inability to estimate how long it will take to get something done.

The planning fallacy

An article published in 1994 in The Journal of Personality and Social Psychology by Roger Buehler and his colleagues shows how epically we fail at planning. After five studies were conducted on nearly 500 undergrads, the researchers found out that not only did the subjects underestimate their own completion times for a number of academic and non-academic tasks, but they also didn't take into account their experiences of how a similar process happened in the past. In other words, they focused more on the plan and the future scenario than on what past experiences taught them. However, when they became observers, the opposite was true: they overestimated others' completion times and considered indicative past experiences more.

One study by the same group of researchers, had 37 psychology students estimate the time it would take them to finish their theses. The average estimate was around 34 days. They also estimated that if everything went as well as it possibly could, the theses would take around 27 days, and that if everything went as poorly as it possibly could, it would take them around 49 days to complete their theses. It shouldn't surprise you by now that only 30% of students actually completed their theses within the time frame they predicted. The actual average completion time was 55.5 days.

And that's not all.

A survey published in 1997 found that Canadian taxpayers mailed in their tax forms a week later than they had predicted, although they were fully aware of their past record of mailing in forms. They had just thought they would get it done quicker next time. This is what the planning fallacy is all about: insisting that our current predictions are realistic, even though we have been overly optimistic in the past.

We're optimistic about current predictions - our ego requires it

Roger Buehler and his colleagues suggest that an explanation for the planning fallacy might be the self-serving attributional bias in how we interpret our past experiences. In other words, we tend to attribute success to ourselves, while we attribute failures to external factors. We do this to protect ourselves and our self-esteem (think student who blames his bad marks on the teacher's poor teaching skills) and we do it in various situations - at work, at school, in sports, in our relationships and even in our consumer choices. We fool ourselves into thinking that what happened last time was a one-off that wasn't our fault; this time we'll surely get everything done quicker.

We don't know enough to plan

The truth is, it's not surprising that we often think 'this time will be different'. We want it to be so, and those of us who are more optimistic actually believe it will be so. But when starting any kind of project, even though aspects of it might be familiar, others most probably aren't. So when we're planning before getting started on something, we know less about what we have to do than we'll ever know again. This is a bad time to make decisions about when we'll finish the project.

And when we make plans, working becomes about meeting those plans, not about great performance. We focus more on the big bad deadline staring us in the face than on how we're actually doing what we do. It becomes a game of "I have to get this done!!"

How can we work better?

Here there's something everyone can learn from agile software development. Agile developers don't work for months to knock down huge projects. They work in iterations. They deliver to the customer quickly, breaking down big projects into smaller steps. And planning is often about "what can I deliver tomorrow that is valuable?" Not that agile software development doesn't have a big picture in mind - it does. But it doesn't attempt to tackle the whole big picture at once.

True agile developers don't get stuck on rules either. They adapt to change as it comes. Because rules that have no reason are just obstacles.

Finally, the worst thing about making plans is that we get hung up on them. We become unresponsive to change and we blindly follow them, because we think that abandoning the initial plan means we're somehow failing. When a plan makes you rigid to change, then it's failing to be good plan.

We'd love to hear about your experience with plans and estimates. How realistic or optimistic are you when it comes down to estimating when you'll finish a project? Let us know in the comments below.