Ofcom, the UK comms regulator, recently published the second document in a series that provides the most comprehensive case study of how a telco can allow hundreds of thousands of customers to be incorrectly charged without taking timely corrective action. Between the two documents, I doubt we will ever see a more exhaustive dissection of the real-life subject matter of revenue assurance being released into the public domain. When read together, the two publications present a litany of failures by both the telco responsible and the business paid to audit their metering and billing accuracy. They also provide a deep insight into how complicated the process of charging customers can be, and hence why superficial tests and auditing will never be sufficient to identify all the kinds of errors that can actually occur. In this instance, it took 15 years for the complaints of customers to eventually prompt a fundamental correction in the way flawed systems operated. However, it is highly unlikely that more than a few people will learn anything from the experience.
Although the two reports represent the pinnacle of an unusually large amount of work that went into analyzing and documenting how a phone bill can contain errors, neither the regulator, auditors nor the telco want this to be turned into a teachable moment that will help others to avoid mistakes. The UK regulator has long maintained a pretense of having a ‘zero tolerance’ stance on overcharging, but neither they nor most of the industry wants to admit how much work would need to be performed to deliver it in practice. The two documents describe the conclusions reached by Ofcom after it was found that systematic errors caused mobile operator O2, the UK division of Telefónica, to overcharge at least a quarter of a million customers between 2003 and 2019, and then led them to inadequately explain the faults to Ofcom when the regulator began its own investigation of the scale of mistakes that were made. Those Ofcom documents are:
- The investigation report published by Ofcom in early 2021 which was used to justify imposing a GBP10.5mn (USD14.5mn) fine on O2 for failing to bill correctly; Commsrisk already covered this report here.
- Last week’s report, which provides much more detail, and which explains the basis of an additional fine of GBP150,000 (USD200,000) because O2 failed to provide accurate answers to Ofcom’s questions during their investigation of the billing errors.
Before delving into this second report, it is worth stepping back and contemplating how unfortunate it is that Ofcom chose to spread relevant information about the same underlying mistakes across two unhelpfully-titled reports published almost a year apart. Ofcom can reasonably argue it is not their responsibility to teach telcos how to bill customers. Ofcom’s lack of competence in this domain is evidenced by the slow rate of progress for their investigation after customers have already endured so many years of failure. However, a great deal of work has ultimately gone into understanding what went wrong in O2, and there is no advantage to consumers if every other telco is forced to learn from their own mistakes just to spare the trivial additional effort involved in circulating information about what happened in O2. If nothing else, the time taken to complete Ofcom’s investigation of whether they were misinformed gives us a meaningful indication of how hard it can be to determine the full truth about billing failures. Per Ofcom’s own words: “Case opened 13 December 2019 Case closed 01 December 2021”.
Ofcom sets the scene at the beginning of their second report.
…Ofcom found that Telefónica UK Limited (trading as ‘O2’) had contravened its regulatory obligations in relation to accurate billing. O2 did so by failing to render or make available accurate final bills to be issued to customers after their cancellation of services (i.e. termination bills) and by overcharging a significant number of customers terminating their ‘Pay Monthly’ services with O2, between at least 26 May 2011 and 15 March 2019.
This document explains Ofcom’s separate and additional finding that O2 has also contravened its information requirements as part of our above-mentioned investigation. It also sets out why we consider this to be a serious breach of the information requirements and why we have decided to impose a penalty of £150,000 on O2.
The entire report is written from the perspective of a regulator that is totally dependent on the telco they are investigating to provide the information that the regulator needs to determine if customers were incorrectly charged. Neither report includes any suggestion that Ofcom might want independent sources of data about the accuracy of bills. This is worth highlighting for two reasons. First, there are countries where comms regulators obtain independent data about the accuracy of phone bills. Regulators hence have a choice in how they approach the challenge of protecting customers from overbilling. Second, the report repeatedly mentions BABT, the German-owned business tasked with approving the accuracy of O2’s metering and billing per obligations stipulated by Ofcom. At no point is it ever suggested that BABT might supply any independent insight into mistakes made by O2, even though they were paid to audit O2 throughout the whole time that customers were overcharged. This begs a serious question about the purpose of ‘approving’ systems to deliver accurate bills when the people tasked with making this approval decision also lack any mechanism to compare what their clients have said to independently-sourced data.
In June 2019, Ofcom received a notification under the Metering and Billing Direction (the ‘Direction’) from the Approval Body for O2’s metering and billing system, TUV SUD BABT (‘BABT’). That notification (the ‘BABT Notification’) informed Ofcom that O2 had experienced an Extraordinary Performance Failure (‘EPF’) in relation to its metering and billing system, whereby 93,259 customers had been overcharged a total of £959,706.19 between at least 1 January 2012 to 7 March 2019.
Let me apologize to international readers on behalf of the UK’s regulatory bureaucracy for the way they inject unhelpful jargon into topics that do not require it. What this paragraph says is that BABT, the organization responsible for approving the accuracy of the systems used by O2 to produce bills, told Ofcom in June 2019 about very serious errors that occurred between January 2012 and March 2019. These errors are so serious that BABT should not have given approval for those systems between January 2012 and March 2019, which is why the word ‘extraordinary’ occurs in the bureaucratic term ‘Extraordinary Performance Failure’. None of the people involved in this miserable regulatory process seem to appreciate it is deeply wrong to pretend that an error that consistently occurred for seven years in a row is ‘extraordinary’. Something that occurs every month for seven consecutive years is, in fact, ordinary. It was ordinary for O2 to overcharge many thousands of customers.
The reason to label this error as ‘extraordinary’ is to give the impression that neither BABT nor O2 could have been expected to prevent or identify these errors any sooner. And yet, the full history shows this error began much earlier than January 2012, and should also have been identified many years ago. This notification just reflected the first time that O2 told BABT they thought mistakes had been made. At no point does the report address why BABT had not identified the issue at an earlier stage. On the contrary, BABT is treated as if their role is just a mindless go-between, passing messages between O2 and Ofcom whilst having no opinion on whether they are true or false.
On 12 February 2021, following our completion of the Billing Investigation, we found that O2 had contravened its billing requirements by failing to render or make available accurate final bills to customers after their cancellation of services (i.e. termination bills) and by overcharging a significant number of customers terminating their ‘Pay Monthly’ services with O2, between at least 26 May 2011 and 15 March 2019 (the ‘Relevant Period’).
Note the use of the words ‘between at least 26 May 2011 and 15 March 2019′. We are going to come back to this choice of words because it is pertinent to understanding Ofcom’s own role in playing down the extent of mistakes that were made.
As explained in Section 2, BABT notified Ofcom on 12 June 2019 that O2 had experienced a Category 1 EPF between at least 1 January 2012 and 7 March 2019. The BABT Notification gave details of (among other things) O2’s explanation of its billing error…
Note again how the role of BABT is reduced to that of a messenger. They passed on O2’s explanation of what went wrong but were not expected to provide any explanation of their own, even though they were responsible for approving O2’s systems.
…In particular, it contained O2’s own description of the error, namely “[i]n a very specific scenario, the termination bill will take a duplicate payment as part of the final Direct Debit” and this had resulted in duplicate payments totaling £959,706.19 and affecting 93,259 accounts.
In light of that information, we served on 1 August 2019 our first demand (notice) for specified information from O2 to better understand various matters relating to its billing error (and to obtain associated evidence), such as:
- confirmations and clarifications about the accuracy and completeness of the
information given by O2 relating to the billing error in the BABT Notification;- information that O2 had made available on its termination bills to customers affected
by the billing error;- information and documents relating to its duration (with start and end dates) for
affected services;- documents relating to any corrective action taken by O2;
- explanations and documents detailing the governance systems in place during the
Relevant Period to escalate and address billing system errors; and- details of O2’s relevant turnover.
All of these questions are sensible and understandable, but note again how there is an expectation that O2 must furnish the information because Ofcom does not expect that BABT would have any relevant information that could be used to answer these questions or show if O2’s answers were wrong. O2 is even asked to satisfy the basic requirement of detailing the governance systems used to escalate and address billing system errors because there is no expectation that BABT would offer an opinion on how they work or why they failed, even though BABT was supposedly reviewing the output of those governance systems.
In the end, we had to prepare and serve a total of seven separate demands (notices) for specified information from O2 during the Billing Investigation to be satisfied that we had received sufficient reliable evidence from O2 relating to the billing error.
Billing can be complex. If we consider the time Ofcom spent drafting notices and reviewing the answers received we get an important insight into how much work can actually be involved in dissecting the causes of an error. Multiply this by all the kinds of errors that are possible and we soon get an appreciation of why it is tempting to take a superficial approach to assuring accuracy.
Upon reviewing O2’s First Response, we identified several discrepancies in the information provided to us by O2 on 5 September 2019, including descriptions of its billing error. We therefore highlighted those discrepancies to O2 on 17 September 2019, and requested to have a meeting with O2, together with [redacted] (who runs O2’s billing system), to discuss the billing error in more detail.
The likeliest explanation for discrepancies in O2’s answers is that nobody in that business had a firm understanding of what was going wrong. Here we see the first indication in the report that O2 depends on another party for the accuracy of its billing, which is why the business running O2’s billing system also needed to attend the first meeting with Ofcom. It baffles me that the name of this business was redacted in a document written by a body responsible for protecting consumers. That business also has a responsibility to end users of O2’s network.
On 27 September 2019, prior to the meeting, O2 sent a note drafted by [redacted] which set out details of three scenarios that led to the billing error.
So we almost immediately find the investigation of O2’s errors switches to becoming an interrogation of the business that manages O2’s billing system. It would hence be pertinent to evaluate whether O2 exercised sufficient oversight of this business.
During the Billing Investigation, we identified several concerns about the accuracy and completeness of some information provided by O2 in response to the First and Second Notices.
Having completed the Billing Investigation, we deal below with our findings of O2’s information contraventions. Those contraventions arise from several different specific information requirements, which we have grouped into four different broad heads of discussion below, namely O2’s information contraventions relating to:
- the nature and scope of the billing error;
- the duration of the billing error;
- O2’s awareness of the billing error; and
- establishing harm or potential harm.
That the investigation has to discuss O2’s awareness of the errors means there must have been a fundamental breakdown in the process that saw their billing systems routinely approved by BABT.
Prior to opening the Billing Investigation, Ofcom’s understanding of the billing error was based on the information given in the BABT Notification. In particular, we noted that the BABT Notification contained O2’s own description of the billing error:
“In a very specific scenario, the termination bill will take a duplicate payment as part of the final Direct Debit. This happens if the customer disconnects on a Saturday / Sunday and has an outstanding periodic bill. The termination invoice should take into account payments made against the periodic bill when calculating the final payment. However, in this scenario payment for the termination invoice includes the value of the periodic bill meaning that we take payment for that bill twice. This issue is now fixed and we won’t see any new instances.” (emphasis added)
We understood from that description that the billing error occurred if the customer (i) had an outstanding periodic bill; and (ii) disconnected (terminated) their account on a Saturday/Sunday.
O2 used some weaselly language in their description of the error. I particularly dislike the use of the phrase ‘a very specific scenario’, which Ofcom also chose to highlight. Anyone who understands the complexity of modern technology and the challenges involved in auditing or securing that technology knows that expecting systems to uniformly comply to a few general principles will get you nowhere. Technology like billing systems must deal with many thousands of ‘very specific scenarios’ and are expected to handle all of them correctly. If O2’s team had not deluded themselves about the specificity of this error then they might have identified and addressed other errors sooner, whilst giving more reliable answers to Ofcom.
Ofcom explicitly asked O2 to confirm the completeness and accuracy of the statement made in BABT’s notification. This is important because O2 said the statement was complete and accurate when it was not, establishing a legal basis for Ofcom to fine O2 for providing misinformation. But also note that this means Ofcom has no legal grounds for relying upon any information presented by BABT on behalf of O2 or any of the other telcos whose metering and billing systems are approved by BABT. Whilst the audit itself is mandatory, there is no outcome from the audit which is legally robust, begging the question of whether it would ever be possible to terminate the approval of any telco for failing to comply with the mandatory billing accuracy obligations. Referring to themselves as TUK, an abbreviation for Telefónica UK, O2 implied the BABT notification was reliable, although they used weasel words here too.
In its First Response, O2 responded to that Question 1(a) of the First Notice as follows:
Question 1(a) of the First Response
TUK has no reason to believe that the information provided on pages 1 and 2 the [sic] BABT notification is inaccurate or incomplete at the time of submission by TUK to BABT. We continue to be of this view.
It is a shame that O2 did not spend half as much effort on checking billing systems as they applied to lawyering their way out of responsibility for years of failure. Phrases like ‘no reason to believe’ and ‘at the time of submission’ hardly inspire confidence that O2 knows how their own systems work. Nevertheless, O2 elaborated on the aforementioned error:
A customer’s periodic bill is produced on a Saturday, e.g. 1st of the month.
- Payment for that periodic bill should be taken, two weeks later e.g. 14th of the month.
- However, in the interim the customer contacts O2 to terminate their account on a Sunday, e.g. 8th of the month.
- The Termination Bill should include the value of the outstanding periodic bill as a brought forward amount. Payment for the Termination Bill should be taken two weeks later e.g. on 22nd of the month, and should recognise any payment which has been received for the outstanding periodic bill and be reduced accordingly.
- As concerns the Billing Error, the Termination Bills of affected customers did not reflect payments received for any such outstanding periodic bills and therefore such customers were charged the same outstanding amount again. This error would have happened regardless of a customer’s debt position as this was a technical fault that did not recognise that payment had been received for the outstanding periodic bill.
However, Ofcom noticed that O2’s description of the error was inconsistent with the description of the systems given by the business paid to run them.
…we noted that the description of the billing error in the BABT Notification appeared to differ from an explanation of the billing error provided by [redacted] in one of the annexes provided as part of the First Response. Further, other annexes attached to the First Response also contained references to a separate scenario not described in the BABT Notification.
When challenged with the information presented by their supplier, O2 admitted that four different scenarios would lead to the same kind of billing error. Other misinformation given by O2 to Ofcom highlights that management in O2 were often wrong about how their systems and processes work in practice. For example, Ofcom noticed inconsistencies in the way O2 described the sequence of events when a customer terminates their contract. Once again, O2 could only offer a correct answer after checking with their supplier.
The timeframe within the customer journeys has been reviewed and has been confirmed by [redacted] as incorrect. The “7 days” should state “14 days”.
O2 also failed to give consistent answers for when the errors began and ended. They variously told Ofcom that the errors occurred from 7 March 2001 to 7 March 2019, from May 2003 to 14 March 2019, and from 5 December 2003 to 15 March 2019. However, Ofcom repeatedly referred to errors happening “between at least 26 May 2011 and 15 March 2019” in any press releases likely to be read by journalists. Whilst Ofcom is keen to penalize O2 for supplying bad information, they wish to avoid discussion of why they only fined O2 for errors that occurred since 26 May 2011. The reason for that has nothing to do with how billing systems work, or how they are audited, but relates to a point of law, as buried deep in the detail of Ofcom’s first report.
…due to the way in which the statutory scheme applies in relation to historical contraventions which pre-date the introduction of sections 96A to 96C of the Act on 26 May 2011, Ofcom is only able in O2’s case to consider the period of its contravention from 26 May 2011 for the purpose of giving a confirmation decision under section 96C of the Act.
Put simply, the law changed in 2011, but Ofcom sought to dupe the press and public by not referring to any errors before that date in the press releases that they knew would be used as the basis for all news articles about O2’s fine. Ofcom did not want to draw attention to the fact that billing errors could remain undetected for so very long, or to the fact they would also be powerless to punish other telcos if similarly historic errors now come to light.
It is already clear that BABT knows nothing about when errors occur, but Ofcom’s investigation showed they do not even know when errors have been resolved. The BABT notification which started the Ofcom investigation said this incident was ‘fully closed’ on 7 March 2019, but subsequent information supplied by O2 said the date of resolution was eight days later. Only one small portion of Ofcom’s report implies any criticism of BABT’s role in auditing these systems. Ofcom refers to a section of the BABT notification and then explicitly states that this section was completed by BABT, before reproducing the misinformation in that part of the notification. Ofcom’s approach is especially interesting because they did not merely copy the words of that notification, but reproduced it as an image, suggesting they want to avoid any hint of having altered the work done by BABT. That image is copied below.
BABT is so apathetic about verifying the performance of billing systems that they do not even check if an error has been resolved when the client they are auditing tells them it has. Ofcom included the confusion about dates as further grounds for fining O2, even though their second report slyly admits that “due to the way in which the statutory scheme applies in relation to our enforcement of historical contraventions, this inconsistency did not affect our final decision in the Billing Investigation”. It is remarkable how much effort Ofcom put into investigating the date on which errors began when this has no impact on the fine they can levy.
Ofcom’s investigation of when the error was identified and resolved casts some light on how much O2 relied on the business running their systems and how little they checked what that business was doing. The sequence of events that led to the error finally being acknowledged and resolved began when an unnamed individual with the title of Financial Analyst reported the error on 5 February 2019 in an email to O2. The email said the problem had only recently been noticed but may be “affecting a large number of customers who disconnect their account not long after their Periodic (sic) bill”. Given the date of the email, O2 must have promptly contacted BABT to tell them about it. However, the two organizations still managed to put the wrong date on the paperwork sent to Ofcom, which said O2 had identified the problem on 4 February. Ofcom highlights this discrepancy as further justification for fining O2 without commenting on any responsibility that BABT has for checking the facts included in the official notification they wrote.
A further reason to fine O2 stemmed from the operator not volunteering a complete analysis of how many customers were affected until Ofcom questioned some initial figures that only related to a subset of affected customers. As Ofcom pointed out in their report:
Had we not followed up on O2’s failure to provide the requested figures for the “total amount of customers affected”, we would not have had an accurate or complete understanding of the number of customers who had paid overcharges as a result of the billing error, or the amount that they had paid… the Billing Investigation would only have taken into account:
- c.80% of the total number of customers who paid overcharges; and
- c.49% of the total amount of overcharges paid by customers.
O2 claimed to have taken timely action to reimburse all affected customers but later told Ofcom they were waiting for reports from the business managing their billing system in order to identify what was owed to some affected customers. Confusion was created because some customers had already received refunds after complaining to O2’s Customer Services staff. This links to a problem identified by Ofcom in their first report, where the governance process used by O2 was supposed to incorporate insights into billing errors derived from customer complaints but somehow none of the complaints about billing errors for terminating customers were ever relayed to O2’s billing team or to BABT. One of O2’s responses included an email from the business running their billing system which said:
…such occurrences are seem [sic] to be corrected by CSA’s themselves by raising refunds, probably when customer complain [sic]
To get a better understanding of what O2 had already done to refund customers, Ofcom then asked for the following.
For customers who paid overcharges in 2018 and were subsequently refunded, please analyse a sample of 20 accounts to identify what triggered the refund to be made.
At long last, here is the first evidence of someone doing something that looks like actual audit work, though only for a meager sample of 20 from the quarter-million customers who were overcharged. It also shows how low the expectations really are, not just because of the failings of O2 and their supplier, but also because auditing is hard. Ofcom wants to see an analysis of only 20 accounts but does not even seek to pick the 20 accounts for themselves, as would be necessary to prevent bias in the selection of a terribly small sample. And the reason Ofcom is asking for this tiny sample is because BABT’s work does not even involve this meager amount of auditing for real customer accounts.
Ofcom treated the revelation that some customers had already been refunded as yet another example of O2 failing to provide complete information at the outset, though the more pertinent observation would be that incorrect claims have been made about the UK’s approval process for metering and billing systems. It has been said this approval process includes a review of customer complaints about bills to stop precisely this kind of situation where customers complain about inaccurate bills to one part of the business but these complaints are not brought to the attention of anyone with responsibility for billing. The information presented in Ofcom’s report shows that O2 was heavily reliant on customer complaints to correct errors in the bills of many customers who terminated their services. However, this knowledge was never conveyed to the billing function, and nor was this systematic shortfall identified when BABT reviewed the billing governance processes of their client.
One oddity in Ofcom’s determination is that they considered O2’s violation to have been exacerbated because the error was not ‘self-reported’ by O2 directly to Ofcom. O2 instead relied upon BABT to pass on the necessary information. This is a peculiar gripe. What purpose is satisfied by BABT’s audit if the regulator that mandates this audit does not even expect to receive information as a consequence of the audit? Contrary to the terminology used, there is nothing ‘extraordinary’ about BABT or their fellow approval bodies notifying Ofcom about Extraordinary Performance Failures. They were routinely used as an excuse for failure when I was working for UK telcos – though not for the telcos I worked for – and insiders tell me that BABT and their peers keep relying on them to excuse clients who make mistakes but whose metering and billing systems continue to receive their approval. It is as if Ofcom has reduced BABT to the role of messenger boy and then decided they cannot even be trusted to deliver messages reliably.
Ofcom makes a valid point when observing:
Ofcom acknowledges that O2 uses a third party, [redacted], to manage its billing system and that it was at times reliant on [redacted] to provide it with the necessary information needed to fully and accurately respond to some of the questions asked in our information requests. However, irrespective of whether O2 uses a third party to manage part of its business, it remains O2’s responsibility to carefully scrutinise the information it intends to provide in response to our requests and to ensure its accuracy and completeness before it is submitted to Ofcom.
As valid as this is, Ofcom should reflect on what this means for BABT and other approval bodies. Is anybody, anywhere, placing any reliance on what they do? If nobody places any reliance on the work performed by an auditor then there is no justification for imposing that audit. The audit fails to serve any actual purpose. So whilst O2 should be held responsible for the performance of all their suppliers, it is notable that Ofcom avoids discussion of the performance of BABT, another business which supplies services to both O2 and Ofcom. BABT seemingly find themselves in the luxurious position of being paid by O2 because of a regulatory obligation imposed by Ofcom without anyone being able to hold BABT accountable for any of their failings.
Indeed, we expect that all information within the scope of a statutory information request should be properly interrogated, cross-checked and reviewed through appropriate governance channels (including by the appropriate director/head of department) and responses to statutory information requests should only be sent to Ofcom when they are complete and accurate.
This is a reasonable expectation. So why does it not also apply to the information supplied to Ofcom by BABT? Ofcom wrote the law that says British telcos need to use billing auditors like BABT in order to protect the interests of phone users. If Ofcom has no faith in the information given by BABT then there is no rationale for their audit work.
Annex 3 and Annex 4 of Ofcom’s report provide a very readable explanation of the circumstances that led O2 customers to be overcharged when they terminated their accounts. These annexes should be required reading for anyone trained to do telecoms billing or revenue assurance, though they are too long to reproduce here. I recommend you download the report from here and use these closing sections whenever you need to give an ill-informed audience (c-level executives, other comms regulators, inexperienced or arrogant auditors) an authoritative illustration of why billing errors will remain undetected in the absence of thorough analysis and testing.
Combining the key points from Ofcom’s second report with information from their first report would help to address the naïvety that still surrounds the difficulty of assuring revenues and ensuring billing accuracy. But as you can tell from the length of this article, that is work that I will have to leave to another day, or to someone else. In that respect, I am guilty of the same fault manifest by almost everyone who takes an interest in billing accuracy but comes to appreciate that it involves much more hard and thankless work than they would like to take on.
