Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.
H. James Harrington
What follows is a true story about information and misinformation. It concerns the statistics people are told about the frequency of SIM swap fraud in the UK, the difficulty in uncovering how the statistics are compiled, and the reality that shows the numbers are unreliable. The revelations in this story are drawn from Freedom of Information (FoI) requests submitted by my colleague, Rob Chapman, Chief Operating Officer of the Risk & Assurance Group (RAG). The story has three acts:
- Statistics reported by journalists and repeated by professionals
- Noticing anomalies in the statistics and investigating them
- The truth about the unreliable way the statistics were compiled
What You Are Told
Reports of Sim-swap fraud have gone up by 400% in five years
That quote is the lede of an April 2020 article published by Which?, the magazine of the Consumers’ Association, a nonprofit organization that describes itself as the largest consumer organization in the UK. This association is held in high esteem; over half a million people subscribe to their magazine. Their quote is based on figures supplied by Action Fraud, the function that captures and maintains records of frauds for Britain’s police. Action Fraud works closely with the National Fraud Intelligence Bureau (NFIB), a division of the City of London Police tasked with understanding and responding to fraud across the UK. This particular Action Fraud statistic about SIM swaps, and other statistics they provided alongside, were reiterated by many mainstream UK news providers during 2020. Here are just a few examples.
- “Fraud cases involving mobile SIM cards have rocketed by 400% in five years, watchdogs say” in the Daily Mirror
- “Figures from Action Fraud, the national fraud reporting centre, show the number of people falling victim to this type of scam has increased substantially since 2015 and that it has resulted in losses of more than £10m to UK consumers” in The Observer
- “The number of Sim swaps reported in 2018 was 1,832 per cent higher than in 2016… Action Fraud, the national service for reporting fraud and cybercrime, says that although the number of cases then fell to 875 in 2019, the average lost (sic) rose from £938 in 2018 to £3,172 last year” in The Sunday Times
The same statistics can also be found in publications for professionals.
- “…sim-swap fraud… have (sic) rocketed by 400% over the past five years” in Fintech Futures
- “According to Which?, SIM swap frauds have rocketed over 400% since 2015” in the LexisNexis blog
- “Since 2015, this type fraud (sic) has increased by a staggering 400%” in Information Security Buzz News
Before we go any further, note how statistics are repeated so lazily that words like ‘rocketed’ are often repeated too. But not all these claims are the same. Which? observed that the number of reports of fraud has risen. Many other commentators elided the number of reports with the number of crimes. That also shows a kind of laziness with the facts.
Unraveling the Truth
RAG held a one-day meeting in central London during December 2018, with the agenda focusing on UK issues for an audience of risk professionals from UK telcos and relevant specialist suppliers. The NFIB is based relatively close to the venue for that meeting, and police officers who represent the NFIB asked to participate. Detective Inspector Alexander Eristavi talked primarily about the underreporting of frauds by telcos, and sought to encourage the audience to use an updated online system to report more of the frauds suffered by telcos and their customers. Eristavi argued that increased reporting would be of mutual benefit. The audience was polite but some in the room had doubts about the benefits of telling the police about more crimes when there were few instances of the police responding to crime reports they already received.
Before we go any further, let us be clear that the police invested in improvements to the systems used to report frauds during 2018. They then actively encouraged an increase in the number of frauds reported. Other businesses may also report telco frauds; banks report SIM swaps because they are a gateway to stealing from online accounts. We can only speculate about how much success the police might have enjoyed when asking businesses to be more diligent about reporting crime; perhaps nobody left that December 2018 RAG meeting with the intention to spend more time informing the police about frauds. But it seems peculiar to me that when the number of reported crimes went up, the police seemingly wanted journalists to infer it was explained by a rise in crime, without taking credit for encouraging an increase in the proportion of crimes reported.
Despite the concerns that SIM swaps were being underreported to the police, Rob Chapman and I hit upon an idea to turn the information already given by Action Fraud to journalists into information sent more frequently to pertinent anti-fraud professionals. The plans seemed obvious to us; if these fraud managers and analysts are expected to submit reports to the police, why not motivate them by showing them a monthly tracker of the national tally? Then we would all have a measure of whether SIM swaps were rising or falling generally, and each telco could sense how well they were doing relative to peers. This approach seemed intuitively better than telco fraud managers occasionally learning a statistic by reading a blog written by a salesman who read a newspaper article written by a journalist who read a press release written by Action Fraud. We reasoned that if the NFIB could make time to attend a meeting with fraud managers because they wanted more data coming in, then their counterparts at Action Fraud could recycle the data collected to provide a monthly tracker of all SIM swaps reported in the UK each month.
Now I realize how naïve we were.
Rob spent the first few months of 2019 unsuccessfully asking the police for the statistics, even though we were only asking for the same information that they had willingly given to journalists. He was forced to submit a Freedom of Information request just to get visibility of the data the police had already provided to journalists, so we could see the detail instead of relying on a journalist’s interpretation of the underlying information. The police failed to satisfy that FoI request within the legal time limits, which means the police broke the law, in a not-so-technical manner of speaking.
This left me bemused at the time. Why would the police be so reluctant to share this data with telcos, after a senior officer made the effort to ask telcos for help with supplying the very same data? But then I looked at the detail they provided in response to Rob’s FoI request. And then I realized why the police might be reluctant to share what they know with people who have a specific interest in this field.
The data was poor.
The results from the March 2019 FoI request stated that 666 SIM swaps had been reported over a 39-month period, giving an average of just 17 per month. This is data for a country with over 94 million active mobile subscriptions. Such a low baseline for reporting means a rise or fall in the numbers might just reflect a change of behavior by those submitting reports. Even the police admitted this is true. 61 was the highest number of SIM swap reports in a single month, and this was blamed on a ‘bulk upload’ of reports by a single organization. If just one organization can suddenly decide to report a number of crimes that is three times higher than the national average then it would be foolish to perform a trend analysis using this data. So Rob and I shelved our plans to recycle the police data into a monthly tracker for SIM swaps.
There was a new burst of press headlines about SIM swaps at the end of 2019 and during the early months of 2020. These were based on new figures from Action Fraud that indicated a sharp rise in the number of SIM swaps reported since 2015. However, we could see there was something very wrong with the information being shown to the public. The new batch of figures from Action Fraud covered an overlapping period with their previous figures, but the two sets of statistics were inconsistent with each other.
That kind of inconsistency should not be possible. On both occasions Action Fraud was telling the press they were sharing the number of SIM swaps reported within each calendar month. The date of a report is the date it was received by the police; it can never change, and the police is consistent about this. But if you say there were 13 reports of SIM swap crimes submitted during January 2017, then later say there were 18 reports of SIM swap crimes submitted during January 2017, you cannot be using a consistent method to count the number of reports of SIM swap crimes submitted during January 2017. This begs the question of why Action Fraud cannot reliably count the number of SIM swap crimes that are reported to them.
The inconsistency was not trivial. In their first set of figures, the police said 252 SIM swaps were reported during 2018. Their second set of figures told the press there were 3,111 SIM swaps reported during 2018.
So Rob submitted a second FoI request.
What Little We Really Know
Thanks to Rob’s persistence, I now possess two reports from the NFIB, explaining why the police finds it so difficult to tell how many SIM swap crimes occur in the UK… although they seemingly gave the UK press the impression that the trend was obvious. The front pages of both reports state they cannot be copied or distributed “without prior authorisation”. What nonsense! The reports were handed over to satisfy a Freedom of Information request – a legal obligation – and they explain numbers that the police already chose to put into the public domain. So if you decide to download them from this website (links are given below) then you know not to be surprised about what they say on their covers.
The investigation began by observing that police numbers given to the media at one time were not consistent with police numbers given to the media on a later occasion. Rob’s second FoI request prompted the police to release a third set of numbers that were also inconsistent with the figures provided to the media in late 2019 and early 2020, although they were much closer to those numbers than the figures covered by Rob’s March 2019 FoI request. Rob enquired about the numbers the police had recently shared with journalists, but it was not possible for the police to precisely reproduce or explain those numbers. The best they could do was to produce some similar numbers, whilst explaining all the reasons why their numbers may vary each time they produce a statistic.
The police uses simple database queries to count any report that contains matching words
Though the police want people to report SIM swaps, they do not use a consistent code to archive and extract them from their database. As a consequence, they have to enter strings of text into a search query and see how many matches they find. The prime reason for the difference in numbers reported per each of Rob’s FoI requests is that different strings of text were used on each occasion. The police now say they used two database searches to produce the statistics covered by the March 2019 FoI request, whilst four database searches were used for more recent statistics included in 2020 newspaper articles.
Matching words in documents is an unsatisfactory way of measuring a specific type of crime. That much is evident from the police believing they need to seach for four different strings of characters to produce a single statistic about one type of crime. Whilst it is reasonable to assume that every report which mentions the words ‘SIM swap’ will refer to an actual or attempted SIM swap, it is far less obvious that the word ‘porting’ might not also be found in crime reports that have nothing to do with SIM swaps. The data supplied through the second FoI shows that the words ‘SIM swap’ are now only found in a minority of the reports the police counted for their second batch of statistics. If the statistics were compiled by only searching for the phrase ‘SIM swap’ then they would show the average monthly number of reported SIM swaps has gone up from 5.5 in 2015 to 16.7 in 2020.
The problem with counting search hits is worse than you might think possible
Computers behave consistently, so if you perform the same searches, and then add more searches, then you should have a higher number of hits as a result of the additional searches. However, the second batch of statistics created by the police were sometimes lower than the monthly statistics covered by Rob’s first FoI request in March 2019.
Though they know which search strings were used for their first batch of statistics, and which were used for their second batch of statistics, the police concluded “it is not possible to ascertain the reason for the discrepancy in the figures”. They speculate that some of the discrepancies in the number of responses may be due to changes in how words are typed, such as searching for “Simswap” instead of “SIM swap”. This might also explain the (much smaller) differences between the numbers in the second FoI response and those reported by journalists in late 2019 and early 2020.
The five-year ‘trend’ versus the one-year ‘trend’
Most journalists followed the lead of Which? by comparing the total number of reported SIM swaps in 2019 with the number reported for 2015, claiming that this reflects a long-term trend. One look at the graph Which? included in their article shows why we should question their analysis.
As you can see, and as The Sunday Times pointed out, the number of reported SIM swaps was 1,832 percent higher in 2018 than 2016, or 2,060 percent higher than 2015. That also means the number of reported SIM swaps fell by more than two-thirds between 2018 and 2019. One incident was to blame for more than half of all SIM swaps reported during those five years, ruining any neat theory about why SIM swaps may go up or down. That incident occured in 2018. Which brings me to…
The unexplained spike that was easily explained
Which? was happy to provide their five-year analysis of trends without offering any explanation for the huge spike in 2018. This is lazy reporting, but they attempted to shift blame by stating: “City of London Police says there is no intelligence to suggest why there was a spike in 2018.” That is odd, because the City of London Police, and specifically the National Fraud Intelligence Bureau which sits within the City of London Police, knows perfectly well what caused the spike. I doubt this could be called ‘intelligence’ if the police did not know why 2,005 SIM swaps were reported in June 2018, compared to an average of 100 SIM swaps reported during the other 11 months of that year!
In their FoI report, the police state that this spike in reported SIM swap frauds was due to mistakes made when TSB, a popular high street bank across the UK, migrated customer accounts to a new IT system. Many customers were locked out of their online bank accounts, and fraudsters exploited the situation by using a wide range of tricks to fool people into sharing their credentials. This was national headline news. It beggars belief that a specialist from the police, tasked with giving information about fraud to the press, could have briefed a Which? journalist about fraud intelligence without either of them knowing this massive spike in reported SIM swap fraud occurred at the same time as the TSB incident. It does not say much for the curiosity or investigative powers of a journalist who neither pressed the police for an explanation, nor looked for possible answers by reading her own magazine.
The rise in the value of losses that was nothing of the sort
The Sunday Times also observed that whilst the number of reported SIM swaps fell dramatically from 2018 to 2019, the average value of those SIM swaps rose equally dramatically. This is a byproduct of bad data. The 2018 figures were dominated by the TSB reports, but none of those reports had any monetary values attached to them. They were hence all treated as having nil value. If you exclude the TSB reports from the data set then it is clear the average reported value did not rise in 2019. If the press wanted to do a proper trend analysis over several years then they would find that the average value of reported SIM swaps has gone down over time.
Double counting of SIM swaps
One thing Which? got right was to refer to a rise in the number of reports about crime. They did not claim to be stating the number of crimes that took place. This is because there is not a one-to-one relationship between the number of reports received by the police and the number of crimes being reported. Sometimes the same crime is reported twice. Unfortunately, the police do not know how often this occurs.
Per the police’s FoI response, two reports for a single crime can occur when…
…both crime and information reports are included in any dataset, such as when the victim changes and the bank takes on the loss from the original victim. To exclude the information reports would provide a less accurate data set as, amongst other reasons, reports where the attempted SIM swap was identified but didn’t occur could be omitted.
The police gave a specific example of potential double-counting in their FoI response. There were two reports in 2017 that referred to a SIM swap which cost GBP450,000 (USD615,000) but the police cannot tell if both reports are about the same crime. The reason to highlight this example is obvious: a crime of this scale is so unusual that it is unlikely there were two distinct crimes which cost the same amount of money at the same point in time. However, the systems and methods used by the police do not permit them to know for sure. Hence both reports were added to the total of losses. So when the Guardian states that over GBP10mn was lost to SIM swaps during the last five years, they are not warning readers that almost half a million pounds might need to be substracted from that total because of the double-counting of a single crime. Just this one instance of double-counting would have added GBP78 to the average reported loss over the entire five-year period. We have no way of telling how many other crimes were double-counted when the police produced the totals repeated by the media.
Actual SIM swaps versus attempted SIM swaps
You may have just noticed another problem with drawing inferences about a rise in SIM swaps. The police does not distinguish between the report of a successful SIM swap and the report of a SIM swap that was attempted but failed. The distinction between actual and attempted SIM swaps was only mentioned obliquely during their second FoI response. Perhaps it does not occur to the police to warn journalists and the public about the possibility of confusion. Observing the difference between actual and failed SIM swaps would be essential to assessing how well customers are protected in practice, but no mention is made of this distinction in any of the news reports surrounding the police statistics. The failure to distinguish actual SIM swaps from failed SIM swaps discredits every story which assumes a rise in the number of reports shows there has been a rise in the number of victims of crime.
A Rise in Crime or a Rise in Bad Statistics?
What we now know is that the UK police cannot distinguish a rise in actual SIM swap crime from a rise in the number of SIM swap reports in their system. A second report for the same crime is not a second crime. The number of reports might go up because there are more crimes, or because people are making more effort to report crime. A report might tell you a SIM swap occurred, or failed. None of this nuance can be found in the way the mainstream UK press has reported on this topic, and you are even less likely to find it amongst the ‘expert’ commentators working for specialist firms or the trade press.
Fraudsters love misinformation. Confusion and doubt are their friends. When anti-fraud professionals spread or rely upon misinformation then it demonstrates they have a flawed understanding of fraud and the impact of the work being done to reduce fraud. It should give us pause that most Brits, and most UK risk professionals working in the communications sector, sincerely believe there has been a significant rise in SIM swap fraud, as encouraged by BBC fearmongering and other sensationalist media coverage.
SIM swap fraud is serious, it causes harm to people, and should be tackled. But we will not be effective at reducing SIM swap fraud if we mistakenly believe that real improvements have failed. That is why it is wrong to simplistically welcome every increase in fraud statistics as ‘proving’ the need for better fraud management. Flashy but flawed statistics also undermine confidence in the quality of work that has already been done. Perhaps SIM swap crime has gone up – but you cannot safely reach that conclusion from the shoddy data supplied to journalists by Action Fraud. Please keep that in mind next time you see experts repeating statistics from official sources.
Click for the comparative SIM swap numbers from their response to the original 2019 FoI request: data summary.