Everyone is Bad at Predicting the Future

July 2
4 minute read

As individuals, we tend to have expectations about our knowledge of the future. Predicting the future is important, we think.

But if I can get a bit philosophical about this….

I’ll put it simply: I know that every single person reading this article is bad at predicting the future, including me. I can guarantee it. And there’s a really easy way that I can prove this. If you were good at predicting the future—even just a tiny bit—you would be independently wealthy. Instead, you have to work for a living because you haven’t been able to leverage your knowledge of the future into piles and piles of cash.

For example, you probably can’t tell me the exact price at 1 p.m. tomorrow of Apple stock.  You probably can’t tell me the wind speed at the top of the nearest hill at 1 p.m. tomorrow.  Work-related examples are also easy to see. If you are commuting to work tomorrow, you can’t tell me the license plate number of the last car you see driving in the opposite direction before you park.  You can’t tell me who is going to be the first person you talk to at your lunch break tomorrow.  These may seem like little examples, but they point towards an important concept related to our work environments and predictability… complexity.

Of course there are always exceptions to the rule. But in all the years I’ve been teaching the Certified ScrumMaster class, I’ve only had one person say “I’m just here for curiosity. I sold a business. I’ve got millions.”

Estimations are Wrong

But despite the fact that we’re terrible at predicting the future, we still rely on estimations. One of the principles in the Agile Manifesto is that the primary measure of progress is working software. In Scrum, we don’t care about our long-term plans. What matters is what value are we delivering right now. This is a responsibility of the Product Owner: to help a Scrum team choose what is the most valuable thing to do right now.

In 2003, I took a class from Ken Schwaber, one of the founders of Scrum. He recounted that he was talking to the CIO of a large IT organization.  This CIO told him, “you know I run these projects that are 12 to 18 months long and, at the end, I don’t get what I need. It’s really frustrating.”

Ken thought about it for a moment. Ken responded, “you know, with Scrum, I can deliver what you don’t need in a month.”

It all comes back to this idea of complexity. When we talk about capacity planning—even for a single Sprint or estimation for beyond a single Sprint—we just have to acknowledge that we’re bad at predicting the future. Instead we need to create the kind of environment where people don’t get punished for poor estimates.

Be Realistic About Predicting the Future

It’s okay to estimate things and it’s okay to try to predict the future. Just be realistic. You’re not going to do it well. No one will and that’s okay. And we just have to be really honest about it.

This honesty can be difficult because we’re also biased. You’ve probably heard the concept of cognitive biases. One of the most well-known is called confirmation bias. Confirmation bias is the simple idea that when you believe something for whatever reason, you will tend to only consider evidence that supports your belief. You will ignore any evidence that disputes that belief, or claim that it’s not really good evidence.

So, if you already believe that you’re good at predicting the future—if you believe that you’re good at project planning, for example—you’re wrong. But you’re ignoring the evidence that you’re wrong. There is a well-known study done every two years by the Standish Group called the Chaos report. The gist of it is that out of many thousands of IT projects they study, less than half are fully successful, regardless of the method used (agile or otherwise).

You might have gotten lucky once or twice and turned that into a cognitive bias where you’re ignoring or making excuses for all the times you weren’t good at making longer term plans.

And because Scrum dispenses with long-term estimations, as long as you keep delivering a potentially-releasable product with each Sprint, you never have to predict the future. Or break your promises to your customer.

You can stop being wrong and start working right.

Now, if you insist on trying to predict the future, read my article 9 Agile Estimation Techniques for some best practices.

This article was adapted from a transcript of a recent Certified ScrumMaster class that Mishkin taught in Toronto.

Berteig Consulting

Empower Your Entire Organization with BERTEIG Consulting in Agile, Scrum, Kanban, SAFe and LEAN.

Bruce Power
Capital One
Equitable Life of Canada