A recent survey concludes that almost three million Brits have been incorrectly charged by their mobile service provider, costing them GBP63.5mn (USD84mn) in total. Sadly, the survey was run by uSwitch, a British business that makes money by encouraging customers to change suppliers, causing them to lose objectivity. They were recently responsible for the dodgy assertion that British users are ‘wasting’ 3.4GB of mobile data each month, a claim which did not stand up to scrutiny. The worrying headline about billing errors per uSwitch’s press release has been repeated uncritically by all sorts of copy-paste journalists, but deserves proper examination before we trust it.
uSwitch.com surveyed a sample of 2,005 UK adults from the 23rd to 26th of March 2018. Results have been weighted to reflect a nationally representative criteria.
Respondents were asked ‘Have you ever noticed any unexpected costs on your mobile phone bill?’. The net response for ‘Yes’ was 38%. These 38% were then asked ‘What was the reason for these unexpected costs?’ 17% answered ‘It was a billing error.’ This equates to 5.34% of all respondents. There are 51,767,000 UK adults. 5.34% of 51,767,000 is 2,762,628.
Have they ever noticed a surprising cost on a bill? Mobile phones have been widely popular for over 20 years. Are we expecting the respondents to have an accurate memory of two decades’ of bills? And what is the value of a survey that aggregates errors from last month with those made in the 1990’s?
This paragraph also includes evidence that the numbers have been bungled by uSwitch. 17 percent of 38 percent is definitely not equal to 5.34 percent. 0.17*0.38 = 0.0646. Even if there were rounding errors and uSwitch are talking about 16 percent of 37 percent that would still equate to 5.9 percent of all respondents. Perhaps the worst failing of uSwitch is that they commission lots of opinion polls but refuse to share the raw data, making it impossible to check if they blundered with their calculations.
Nevertheless, if 5.34 percent of respondents have suffered a billing error then that would be just 107 people. Or about 6 people per year over a 20-year period. This is being used to generate a headline that total errors were worth GBP63.5mn. Clearly this sample is not representative of anything but uSwitch’s desire to generate cheap publicity by engineering headlines that lazy journalists will copy.
Respondents who had noticed unexpected costs on their mobile phone bills were then asked ‘On average, how much did these costs add to your bill?’ The average additional charge of those who had noticed a billing error in the past was £23. £23 multiplied by the 2,762,628 who had billing errors due to their mobile network equals £63,540,444.
The survey method is wonky irrespective of the quality of the data given by respondents. Respondents answered a question asking the average surprise addition to their bill. Only 17 percent of surprises were described as errors. If 83 percent of surprises are not errors then you cannot multiply the average value of surprises by the number of people who had an error and claim that is the total value of errors. The result of that calculation is just meaningless garbage.
But even if we give uSwitch the benefit of our doubt, and assume the average of all surprises is the same as the average for billing errors, then the calculation is still flawed. The same person could have suffered two or more distinct errors, especially if they have owned a mobile phone for 20 years. So the total error is not the number of people multiplied by the average error per person. The total error is the number of errors multiplied by the average error. Failing to ask how many errors each person has suffered means the survey has not collected the data necessary to calculate the total value of all errors.
Does GBP23 sound like a plausible ‘average’ error? Data from Ofcom, the UK regulator, says the average monthly household expenditure on mobile voice and data has fallen from the low 50’s to the mid 40’s, so uSwitch is claiming the ‘average’ error is worth half of the customer’s average monthly expenditure. Mistakes of that size would certainly be noticeable. But it further illustrates the flaw in relying on people to report an average error. Errors are going to exhibit a wide distribution by their very nature. I have seen a complaint from a customer who (wrongly) believed he had been overcharged by a single penny. On the other hand, the largest ever billing error was greater than all human wealth put together.
Respondents were asked ‘Are you the main bill payer on your mobile contract?’ 15% responded that they do not have a mobile phone contract, leaving a sample of 85% who do.
Hold on… Ofcom’s statistics say 94 percent of adult Brits have a mobile phone. How does this compare to the uSwitch 15/85 split? Are they saying 15 percent use prepaid phones? Where are the people who do not have phones in this survey? However you interpret these assertions, to get the total error you would need to multiply the average error by the number of postpaid bills, not by the number of adults in the country, which is the figure that uSwitch used. Not every adult receives a postpaid mobile bill.
Ru Bhikha, mobiles expert at uSwitch.com, comments… “A message to the networks – the fact that so many consumers cannot ‘be bothered’ to check their bill suggests it is seen by many as laborious task. We urge providers to make sure key information is easy to access, with the facts – the key information a customer wants to see – surfaced prominently so it is easy to ensure all is well.”
If anybody should be demanding clearer numbers it is the so-called journalists who have been uncritically repeating uSwitch’s flawed statistics.