IT Project Risk: Black Swans Turn White

One in every six IT projects suffers a cost overrun of 200% or more. That is the startling conclusion of new research by Oxford University and McKinsey; see the summary story here.

The study reviewed 1471 IT projects costing more than USD170M. The mean average cost overrun was under 30%. However, this statistic hides the truth about the deviation, with a significant proportion of IT projects running well out of control. The research also found that the probability of an IT project running out of control is 20 times greater than the risk predicted by standard risk management models.

In the article, Bent Flyvbjerg, BT professor and the founding chair of major programme management at Oxford University, highlighted five basic controls to avoid such massive cost overruns. These controls are simple and obvious, but are still worth reiterating. For example, one recurring problem was that business cases for IT projects are often works of fantasy, or as Flyvbjerg put it:

“Costs are underestimated, schedules are underestimated, and benefits are overestimated. If you have all these biases in the business case you’re going to make the wrong decisions – that’s simple. If you get misinformation in the business case instead of information you’re going to make the wrong decisions.”

However, I cannot agree with all the conclusions as presented. Flyvbjerg and the marketing of this report keep using the phrase ‘black swan’ to explain why so many projects overrun so far outside the forecasted bounds. In other words, the project failures are blamed on events of very low probability and high impact. These events have devastating consequences but cannot be accurately forecast. Forgive me for cynicism, but I suspect the real reason for throwing ‘black swans’ into the mix is because it makes for sexy headlines, beloved of both consulting firms and academics wanting more research funds. The advice given by Flyvbjerg on how to counter risk makes a mockery of the idea that black swans cause so many IT projects to spiral out of control. For example, nobody would think a biased business case that overstates benefits and understates cost is a black swan. The risk of bias in forecasting is known to even the most naive manager (whether they do anything about it is a different question).

Let me analyse these findings about IT project risk another way, to clarify why the failing projects cannot all be due to black swans. If you cross the road and get killed by a meteorite falling from the sky, that is a black swan. If you cross the road and get killed by a car, I may agree you were unlucky. Perhaps it was pitch black, the road looked empty, the driver did not have his lights on and you were wearing an iPod so did not hear the car’s engine. If you cross the road, get killed by a car, but you already knew that one in six people who try to cross that road get killed, then you are an idiot. With odds of 1 in 6, the chance of a massive IT project overrun is no different to the odds when playing Russian Roulette. If somebody you knew put a bullet in their brain when playing Russian Roulette, I doubt you would excuse it as ‘black swan unpredictability’. This research only confirms what most of us know anecdotally: that many IT projects overrun badly and that huge numbers of excuses are made for this, but as the years go by, still many IT projects continue to overrun by shocking amounts. It is not good enough to play the imbecile who always believes the excuses, and then keeps being surprised by each new failure. If we want to manage risk, we start with transparent and honest assessment of the data. A lot of IT projects fail for reasons which we should be perfectly able to predict, such as bias in initial estimates. The only way to manage risk successfully is to admit that and do something about it.