One in every six IT projects suffers a cost overrun of 200% or more. That is the startling conclusion of new research by Oxford University and McKinsey; see the summary story here.
The study reviewed 1471 IT projects costing more than USD170M. The mean average cost overrun was under 30%. However, this statistic hides the truth about the deviation, with a significant proportion of IT projects running well out of control. The research also found that the probability of an IT project running out of control is 20 times greater than the risk predicted by standard risk management models.
In the article, Bent Flyvbjerg, BT professor and the founding chair of major programme management at Oxford University, highlighted five basic controls to avoid such massive cost overruns. These controls are simple and obvious, but are still worth reiterating. For example, one recurring problem was that business cases for IT projects are often works of fantasy, or as Flyvbjerg put it:
“Costs are underestimated, schedules are underestimated, and benefits are overestimated. If you have all these biases in the business case you’re going to make the wrong decisions – that’s simple. If you get misinformation in the business case instead of information you’re going to make the wrong decisions.”
However, I cannot agree with all the conclusions as presented. Flyvbjerg and the marketing of this report keep using the phrase ‘black swan’ to explain why so many projects overrun so far outside the forecasted bounds. In other words, the project failures are blamed on events of very low probability and high impact. These events have devastating consequences but cannot be accurately forecast. Forgive me for cynicism, but I suspect the real reason for throwing ‘black swans’ into the mix is because it makes for sexy headlines, beloved of both consulting firms and academics wanting more research funds. The advice given by Flyvbjerg on how to counter risk makes a mockery of the idea that black swans cause so many IT projects to spiral out of control. For example, nobody would think a biased business case that overstates benefits and understates cost is a black swan. The risk of bias in forecasting is known to even the most naive manager (whether they do anything about it is a different question).
Let me analyse these findings about IT project risk another way, to clarify why the failing projects cannot all be due to black swans. If you cross the road and get killed by a meteorite falling from the sky, that is a black swan. If you cross the road and get killed by a car, I may agree you were unlucky. Perhaps it was pitch black, the road looked empty, the driver did not have his lights on and you were wearing an iPod so did not hear the car’s engine. If you cross the road, get killed by a car, but you already knew that one in six people who try to cross that road get killed, then you are an idiot. With odds of 1 in 6, the chance of a massive IT project overrun is no different to the odds when playing Russian Roulette. If somebody you knew put a bullet in their brain when playing Russian Roulette, I doubt you would excuse it as ‘black swan unpredictability’. This research only confirms what most of us know anecdotally: that many IT projects overrun badly and that huge numbers of excuses are made for this, but as the years go by, still many IT projects continue to overrun by shocking amounts. It is not good enough to play the imbecile who always believes the excuses, and then keeps being surprised by each new failure. If we want to manage risk, we start with transparent and honest assessment of the data. A lot of IT projects fail for reasons which we should be perfectly able to predict, such as bias in initial estimates. The only way to manage risk successfully is to admit that and do something about it.
Mike Irizarry, CTO of U.S. Cellular out of Chicago gave a great keynote this year at the B/OSS Live conference. He gave a few pointers on transformation. They had just completed a big billing conversion. . . .
1. Be Clear on Scope of Project. You need to understand — based on your company, plans, business strategy and industry trends — how you can deliver services that your current systems can’t support and you must do it quickly. Time to market used to be a year. Now it’s a month, 3 months, or six months. It’s hard to meet that.
2. Lower customizations
Our rule was 85% of the vendor offering must be out of the box. We want to do configuration, not customization.
3. Business Case
We spent 6 months putting together our business case. A long time. We did time and motion studies and did focus groups with customers. It’s very important.
4. Reserve Contingency
You don’t know what you don’t know. Reserve contingency for time and money. Do a risk profile on the operation. It will save you from having to go back to the board and ask for more funds.
5. Best in Class vs. Best of Suite
Best in Suite is best for us. We wanted something integrated number one. We were willing to settle for components of the suite that would not be best in class. There is a significant cost impact of choosing best in class.
6. Build some wiggle room in your time line.
You don’t want to interrupt the customer experience.
7. Process is harder than Technology
People said there was no problem changing processes, but when the new systems came in, people were kicking and screaming. You need support from the executive level to say: we are going to adopt this. Otherwise you will be stuck in requirements hell.
8. Ask the Vendors to Stand up the System before you sign the Contract
You would never buy a car without driving it. But a billing system is big dollars. Make the vendor prove that it works and it works in an integrated way. Don’t avoid that. Take a couple of vendors all the way through the final stage. It’s going to upset the vendors, I know, but it allows you to achieve ultimate leverage and it’s in the best interest of the vendor too as to what’s really important.
9. Don’t think of the maturity of the Overall Suite You Buy
Look at the different components of the suite and understand the maturity of the different things inside. That will help you identify risk and give you a contingency plan to deal with that. It also means you can truly see where you want to be on your roadmap.
For a detailed critique of this article, see Zafft, Robert, “A White Elephant is not a
Black Swan: Why you can do more about IT-project risk than you think (A reply
to Flyvbjerg and Budzier), Journal of Risk Governance and Control: Financial
Markets and Institutions, Vol. 2, Issue 3 (2012) (Also available on SSRN: http://ssrn.com/abstract=2119800).