The Curious Case of the Vanishing Targets

I have a very serious point to make today. But in order to make it, I will start with a comic daydream…

Usain Bolt struck his lightning pose for the TV cameras. He smiled, and it felt like the world smiled back. A hush descended around the stadium. The Jamaican shook each long limb in turn. He was a sprinter, built for speed like no man had ever been. He was relaxed. He was ready.

On your marks.

Bolt sauntered forward. He was casual, but deliberate too, carefully placing each foot into position, soles pressed firmly against the starting blocks. Bolt then knelt upright, and made the sign of the cross. Now was the time. He looked up at the sky, and blew a kiss to the heavens. And then he faced down, as if in prayer.

Set.

His mind suddenly raced ahead, imagining himself at the finish line. This was the 2016 Rio Olympics, and Bolt was going to win his third consecutive gold for 100 metres. And in doing so, he would run faster than ever before, faster than anyone had ever run. Bolt was sure of it.

The starting gun fired.

Bolt exploded from the blocks. His legs unwound, consuming the ground before him. And it was over, finishing so soon there was no time for thought, nor words. There was only speed, and sound. The crowd roared louder than a hurricane, overawed by Bolt’s power, blown away by what they had witnessed.

Bolt raised his hands, praising the crowd as they praised him. Somebody handed him a Jamaican flag, and he held it aloft, flying it behind him as he toured the track. Then, halfway round, he stopped, to watch the replay on the stadium’s big screen. Bolt waited to see the official time. He knew he had broken his record. The only question was whether he had done something previously unimaginable. Was Bolt the first man to run 100 metres in under 9 seconds?

Bolt waited. He raised an eyebrow. He waited. He smiled, but less broadly. He waited. Why was it taking so long to show the official time for his race?

“Is everything alright, Mr. Bolt? We need you to move on, the high jumpers use this part of the track for their run-up.”

“Where’s my time?” quizzed Bolt. “I want to know my time for the 100 metres.”

“Oh.” The diminutive Olympic official tugged the front of his blazer, and looked nervously around him. “Didn’t anyone tell you? We’re not bothering to measure the times in this Olympics. There’s no need.”

“What do you mean, there’s no need?” asked Bolt, perplexed.

“You had the other guys racing alongside. What’s the point of measuring the time you ran? You definitely came first!”

“I want to see if I did better or worse than before. I might have gone under 9 seconds in that final.”

“Yes, but targets like that promote the wrong sort of competition. You won’t become a faster runner by trying to beat a target like that.”

All trace of a smile was erased from Bolt’s face. His eyes widened. He stepped closer to the official, leaning over him. He uttered one word: “what?”

“The Olympic Committee has brought in a much better way to improve athletic performances. Instead of using targets that get objectively measured, we now have some sports scientists, coaches and doctors who will review the video tape of your performance. They’ll write you a report, explaining how you could improve your running process further.”

“They’re going to write some papers that will improve my running process? Nobody timed my run?”

“That’s right.”

Bolt bared his teeth again, but he was not grinning. He raised his hands again, but not to the sky. They were wrapped around the neck of the Olympic official, throttling him.

Okay, so that was a silly story, and so far I have not mentioned telecoms once. But sometimes I get confronted by supposedly serious people acting in such ridiculous ways that it is difficult to describe the enormity of their silliness. So imagine the following. You want to be the Usain Bolt of revenue assurance. You want your telco’s bills to be more accurate than they have ever been before. You want to be a real champion, delivering bills which are more accurate than anyone’s bills have ever been. You want this, because you really strive to always improve your performance. Now answer one question:

Is it better to set accuracy targets and objectively measure your performance against those targets? Or not?

I think you know which conclusion I am expecting you to draw. There is a reason why we record the performance of athletes using numbers, why we measure companies in terms of profits, and why even governments set themselves targets (even if they always miss them). So you will be as bemused as I am, that a bunch of ‘experts’ in the United Kingdom believe the exact opposite to you and me. The UK has a long tradition of believing its telecoms billing is more accurate than everyone else’s. They believed this because people wrote pieces of paper saying they met accuracy targets which were tougher than anyone else’s. But now, to become even better, they have decided the way forward is to not have any targets at all.

Common sense. Logic. Past experience. Real data. None of these apply, when regulators make decisions like these. Here is the rationale, point by point, per Britain’s regulator:

The removal of the target-based requirements and the retention of the existing process-based requirements should ensure that the approval and audit processes, and ongoing reporting by CPs to ABs, were focussed on CPs identifying and analysing all billing errors. Instead of having targets which envisaged an ‘acceptable’ error rate, the remaining provisions, while recognising that errors may occur, would aim to achieve ongoing improvement in CPs’ systems and processes to ensure that, where errors occur, corrective measures are put in place that address the risk of repetition. This would more closely align the Direction with the provisions of GC11.1 which requires CPs to ensure that all bills represent and do not exceed the true extent of any such service actually provided, rather than setting targets for billing accuracy.

Let us break this down. To begin with, CPs are comms providers i.e. the telcos, whilst ABs are approval bodies i.e. a kind of ‘specialist’ auditor that checks bill accuracy. Some of them believe, and the regulator agrees with them, that the previously mandated level of accuracy somehow discouraged people from measuring and responding to all the billing errors that occurred. The target somehow prevented them from doing these utterly sensible things, which they are legally and morally obliged to do anyway, even though the target was for the total billing error. Words fail me already. What have they been attempting to measure so far, if not the sum total of all billing errors? Why would the absence of any target make somebody work harder to detect and resolve errors?

GC11.1 is clause 1 of General Condition 11, a requirement to be satisfied by every UK telco. It says, in short, that telcos should not overcharge customers. Duh. And yet, as even the regulator admits, mistakes happen. Previously, this reality was dealt with by stating how much error would be tolerated, before a telco was punished. This was the accuracy ‘target’, so to speak, though it should properly be called a tolerance, as it was supposedly mandatory to stay within the target. The theory was that telcos who break the limits deserve punishment. This does not preclude punishing telcos for behaving badly even if they do not break the limits. Governments want people to drive safely, and whilst it is very possible to drive dangerously at low speeds, it also makes sense for law enforcement authorities to automatically punish people who exceed speed limits. But the UK regulator has somehow reversed that kind of thinking. They now believe that having a limit somehow encourages telcos to behave badly, because they will drive up to that limit. Already we are in fanciful territory, as if telcos routinely discover very very very small overcharging errors and then decide they should do nothing about correcting them. But the regulator now thinks that tolerances encourage telcos to believe they will not get punished if they stay within the tolerance, though nothing currently stops the regulator from enforcing GC11.1 more strictly, if it wanted to. However, the regulator’s new approach is to do away with any tolerance, and to leave it completely vague when anybody will be punished for any bad behaviour.

CPs point out that they use process-based requirements for their own internal audits and for ensuring billing accuracy, so compliance costs could be reduced. This could also encourage voluntary compliance with the Direction by CPs with annual relevant revenues under £40 million not covered by its scope.

I was an auditor, once. I mean, I was a real auditor, once. In the real audit world, fees have to be earned, budgets managed, profits made, and there were a fair few other audit firms competing for the same work. In that real world, there is a simple but important rule of thumb: process-based audits are cheap, substantive auditing is expensive. Auditing a process is cheap because it is subjective. You contemplate whether anything might go wrong, and if you can think of nothing that might go wrong, then you conclude that nothing will go wrong. Substantive auditing is expensive because you have the hard work of actually checking real data, to see if it really is right or wrong. As there might be a lot of complicated data to check, this can get expensive, even on a sample basis. But whilst substantive auditing is hard and expensive, you must do some, or else your audit might be a total crock.

Contrast that real world insight to the paragraph from the regulator. It implies that telcos, and with the full knowledge of some kinds of auditors, have already been favouring the cheaper kind of audit work, which involves more of the subjective stuff, and less of the actual checking of data. The regulator notes that the less real checking done by telcos, the cheaper their audits will be. This is correct. And if audits get cheaper, the regulator hopes telcos will start volunteering to do audits even if they are not mandatory. This is pure speculation, and hardly a desirable goal anyway. Why encourage the voluntary proliferation of weak auditing practice, instead of focusing stringent audits on business practices that really need auditing? And how is any of this a valid argument for ending a requirement that was designed to protect customers from too much overcharging?

The arrangements should be more adaptable and future-proof as they would be based solely on processes rather than targets which might need to be changed as usage and services changed.

Remember, this regulation is supposedly about protecting real customers of real comms providers. This clause implies customers will be better protected in future, if no objective targets are set. Why? Because setting objective targets is difficult. But if it is difficult to set objective targets, how much harder would it be to decide the appropriate punishment for a telco that demonstrably overcharges its customers, when there is not even a guideline target to compare their performance to? And what, if anything, is objectively measured by a process-based audit?

Although we would be removing elements of the current requirements, we believe the remaining requirements of the Direction would be adequate to protect consumers by ensuring that CPs processes were focussed on ensuring the accuracy of bills. Indeed, for the reasons explained above, we consider that the focus on the process-based requirements should result in a closer alignment with the objectives of GC 11.1 and should therefore be more effective at protecting consumers.

In other words, though they are making the regulations easier, they claim this does not really make them any easier. In fact, they somehow claim it makes the regulations tougher, and hence more effective at protecting customers. Telcos will no longer need to do things they previously had to do. By telcos not doing as much, somehow customers will be protected more than before.

There is a simple counter-argument to the regulator’s position. Hardly any countries mandate targets for bill accuracy. The regulator continues to be challenged by telcos who complain British bill accuracy regulations are much tougher than those found anywhere else. For many years, the British regulator insisted it was necessary to set targets, even though they knew other countries did not. Either they were wrong to set targets before, or they are wrong to do away with targets now. Which one is it? What else changed, that might justify this u-turn? How is it that the UK regulator argues it was right, when they said it was necessary to adopt the toughest billing accuracy tolerances that the world has ever seen, and the UK regulator argues it is still right, when they say there is no need to measure performance against any kind of tolerance at all?

In a recent podcast with Mike Willett, I said the UK’s bill accuracy scheme was prone to ‘shenanigans’. Whilst it can be long and boring to explain what is involved in these shenanigans, I believe this latest regulatory twist provides ample evidence of the fundamental problems with the UK’s billing accuracy regime. The historic source of these problems is straightforward: the regime began with unrealistic and unworkable targets, endorsed by people who simply never accumulated enough data to realize how badly they had underestimated the true extent of billing error. From that point on, nobody was allowed to lose face. The auditors could not lose face, for being less knowledgeable and experienced than they pretended. The telcos could not lose face, because any telco that did would unfairly suffer relative to every other telco. And the regulator could not lose face, because they would have to admit they implemented a regulation that delivered collective delusion instead of consumer protection.

Now UK customers are reaping the rewards of all that face-saving. Instead of protecting customers with realistic targets, backed by tough data-driven audits and a genuine desire to penalize the worst offender, the UK’s regulator has ensured nobody lost their job. To do this, they needed to deliver a through-the-looking-glass explanation for why the only thing better than having an insanely narrow accuracy target is to not have any target at all!

After his disappointment in Rio, Usain Bolt called a press conference, and said he would run just one more race, with the intention of setting a new 100 metres world record that would never be broken.

“Wow Usain!” shouted one of the press corps. “What’s your target for this unbreakable new record? Is it 0.01 seconds?”

“What are you smoking?” replied Bolt. “Nobody can run 100 metres in 0.01 seconds. That target would be madness.”

Another member of the press corps shouted out. “If you’re not going to run it in less than 0.01 seconds, why bother having a target at all?”

Bolt put his head in his hands. Mo Farah, his friend, sat alongside. Farah put his hand on Bolt’s shoulder, consoling his old pal. Bolt slowly turned to face Farah. “Don’t be down,” said Farah, “they just don’t understand what people like us are trying to accomplish.”

“Maybe they don’t even care,” mused Bolt.