Estimating innovation.
Why accurate estimates are so difficult for innovation-based projects and how to reduce risk through iterative approaches.
tl;dr. Estimation accuracy relates directly to how much you know about a project. Scrum methodology helps by accepting that you know least at the beginning and iteratively improving forecasts.
Things you know, don’t know, don’t know you don’t know.
As a kid I remember evenings with our kitchen table covered by architects drawings and a printing calculator spewing long lengths of paper inked with long lists of long numbers. I didn’t understand what was going on, just that dad was “doing estimates” and that I should keep well away.
My parents ran a building company that would do everything from replacing a broken window to refitting public libraries, and as far as I could tell this meant a lot of time creating estimates.
Fast forward 30 years and I’ve spent my fair share of time estimating digital projects, sometimes successfully, other times less so. Now I’ve had the opportunity to compare the estimation of digital projects with those of a more physical construction, I’d like to share with you what I’ve learned.
Why is it so difficult to create an accurate estimate for innovative digital projects?
Every project’s total costs can be split into 3 categories.
1. Things you know.
This is your happy place. The more work you’ve completed, the more experienced you are. With this experience comes the knowledge to accurately predict how long it will take to do again. Even better, you might end up doing it more quickly than last time, as you’re on your way to becoming an expert.
2. Things you don’t know.
This category is considerably less fun. Sometimes you’ll have to estimate a project that has aspects you’re not familiar with. You have some adjacent experience though, so you’ll make an educated guess.
3. Things you don’t know you don’t know.
Here be dragons. If your work contains any type of innovation, you’re going to have lots of line items in this category. It’s the things you don’t know you don’t know that make innovation high risk, high reward.
The reason your estimates are not accurate.
So, how did my dad manage to create an estimate to refit an entire public library with an accuracy that ensured profitability, and a bid low enough to win the job, using only a printing calculator, architects drawings, and some product brochures?
With lots and lots of experience, and well defined objectives. Those architects drawings were accompanied by priced material specifications supplied by a quantity surveyor, for everything from flooring to ceiling tiles, and my dad’s 20 years of experience allowed for accurate predictions of the labour costs required to complete the job. As a result his huge, 500 page estimate was informed mostly by data from the first category, things he knew.
By comparison we do not have it easy. When you consider how much of our estimating is for things we don’t know then you start to see why so few estimates are accurate. And whenever you’re estimating things you don’t know, you introduce things you don’t know you don’t know.
How to reduce risk in your innovation based projects by avoiding being bitten in the ass by your estimates.
At the risk of sounding like a twelve step program, to overcome the problem we must first admit we have a problem. There is a direct correlation between the accuracy of your estimate and the amount of information you have about the thing to be done.
This information combines a definition of the thing to be done, and experience of having done this thing before. Once you’ve augmented your experience with advice from others, the only way to increase the accuracy of your estimate is to improve the definition.
And here lies the very essence of our problem — we create project estimates at the start of a project, the exact time when we know least about it.
Let’s return to our three categories, and consider the end of a successful project, when we know all of the things we needed to know to create an accurate estimate. It would be lovely to have this foresight but I’ve misplaced my flux capacitor so you’ll have to accept that at the start of the project, when you create your estimate, that there are things you don’t know and things you don’t know you don’t know.
As these unknowns are resolved, our estimate’s accuracy increases. If only there was a way to take advantage of this.
Enter Scrum.
If like most people you’ve heard of Scrum and all that “Agile nonsense” but haven’t actually tried it, then you’ll love this book. It’s a great read by an ex-fighter pilot with compelling anecdotes about millions wasted on large scale software projects that will trigger your schadenfreude and leave you wincing.
Accepting that we know the least about a project at the start is at the heart of Scrum. Estimating is an important part of the process but it’s done in a group to benefit from your full team’s experience. Then the same team proceeds to make the things they estimated, with weekly reviews of how much got done this week (the team’s velocity), what obstacles slowed them down, and what they want to do next week.
Dividing the remaining estimated work by the team’s velocity forecasts how much time will be required to complete the project with the accuracy of this prediction increasing each week.
This all sounds very wonderful in theory but I’ve experienced problems in reality that prevented projects from successfully adopting this approach. The biggest challenge is selling the approach to clients, especially to the marketers responsible for much of the digital projects made by creative vendors like advertising agencies.
Here, the issue is that historically, when a project discovers that its estimate wasn’t accurate and the budget has run out, the vendor tends to absorb the extra costs, with the client paying a fixed fee. To add insult to injury, the vendor then receives heavy criticism for late delivery as thanks for their generosity, with the client justified in their chagrin due to the delays being discovered and reported to them late in the project.
Scrum allows the the vendor to know earlier if the original estimates are correct. This knowledge allows the vendor and client to course correct by either modifying the launch date, or modifying the feature set. Again, this can be hard to sell, because clients have traditionally been told that the price, timings, and definition of a project can be reliably determined at the start of a project, only to be let down all too often.
I asked Yuri Takhteyev, Rangle.io’s CTO, how they use scrum with clients like this that are tied to the traditional fixed scope, fixed budget, fixed timescale model. “We don’t” was his reply. “We only work on a time and materials basis”. This is smart, and now that they have critical mass (they are three years old and have become the biggest JavaScript shop in the world) clients either sign up for Scrum and see more of their budget spent on developer hours, or go elsewhere for a more traditional approach. This has the added bonus for Rangle.io that they don’t need to worry as much about running out of budget as their traditional counterparts, because their clients are buying sprints — a finite number of hours of a finite number of people, instead of an estimated scope of work.
In Yuri’s talk at this year’s Web Unleashed conference he pointed out that the biggest reason that Scrum fails when people try it for the first time is because they don’t adopt all its aspects. For example, they adopt iterative development using sprints, but aren’t prepared to be flexible on the feature set or the launch date, or they follow the correct steps then don’t bother to carry out retrospectives — the review meetings critical to removing obstacles and increasing a team’s velocity. He recommends that you do it by the book, and yes, it’s the same book that I’m recommending here.
I’m happy to report that I see increasing numbers of projects where a brand manager has read the book, and is prepared to be realistic about how their new thing is going to be made. And without exception, these projects have over delivered on expectations without burning out anyone on the team. I acknowledge it’s not an easy mind shift, but if as a client you can be comfortable with a variable budget and launch date, or a variable feature set, you’ll see more of your money spent on making your thing instead of on meetings to discuss making your thing.
It’s worth noting that Scrum isn’t for everyone. Some people are simply allergic to it, and if you try to force it upon them, you are in for a world of pain. The book features anecdotes about a construction project where the author acting as client insists to the construction company remodelling his house that they use Scrum to manage the project, and everything goes well. I know exactly how badly this request would have been received by my dad — very, very badly — but that’s because his estimating and project management was firmly based in the world of things he knew, and he knew how to profitably deliver those projects on time.
For the rest of us, especially those of us planning complex software projects, accept your lack of knowledge at the start of your project and read the book.