The Communications Fraud Control Association (CFCA) is a US organization that surveys the opinions of telecoms fraud managers every two years, with the result that its headline loss figures are reported widely (and often inaccurately). However, it is time to question how much faith is put in their findings. Like many other people, I listened intently as CFCA President Jason Lane-Sellers spoke at the recent Subex User Conference (pictured), the first time the 2019 survey results were presented outside of a CFCA meeting in the USA. Upon receiving the slide pack, and giving myself time to review the figures in detail, I came to realize that many of the numbers were flawed. Tomorrow I will go through the detail of the report that Jason presented, analyzing the conclusions and explaining where they are unlikely to be reliable, and where they can be proven to be wrong. Today I will preface that analysis by dealing with the most fundamental questions when assessing the quality of any sample: is it large enough, and is it representative?
How Many Telcos Responded to the CFCA Survey?
This figure was not explicitly stated in the report that Jason shared publicly. Luckily, it is possible to derive the number of survey responses from information given within report. Simple maths shows just 61 telcos answered one of the easiest questions in the survey. Whilst it is possible that some telcos may have skipped some survey questions, it seems unlikely that many will have skipped this particular question, which came near the beginning of the questionnaire.
Survey respondents were asked where their fraud team was situated within the telco organization chart; unlike the rest of the report, answers to this question were given to two decimal places.
Finance: 36.07%, Security: 27.87%, Operations: 13.11%, Risk Management: 11.48%, IT: 6.56%, Customer Care: 1.64%, Other: 19.67%
Every result is a whole number multiple of the smallest result, which was 1.64 percent. For example, 36.07 is almost exactly 12 times 1.64, whilst 6.56 is equal to 4 times 1.64. If 1.64 percent = 1 telco, then 61 * 1.64 = 100 percent. Hence the ratio between each figure supports the inference that there were 61 answers given to this question.
Whilst it is possible to obtain survey answers in the same ratios if the number of surveyed telcos were a multiple of 61 (i.e. 122, 183, 244 etc) that would be a very unlikely outcome. However, it is possible to confirm that the number of responses must be less than 122 by considering the answers given to a different survey question. When asked which associations the respondents belonged to, 26 percent of responses were said to come from CFCA members. 61 * 0.26 = 16, which would be only slightly fewer than the total number of telcos listed as members of the CFCA. It is fair to assume that most CFCA telcos would respond to this survey, but not enough telcos belong to the CFCA for it to be possible for them to generate a quarter of responses if a total of 122 telcos were surveyed. The CFCA’s association is simply not large enough to make those numbers work. Hence we can discount the possibility that 122 telcos were surveyed, and can safely conclude that 61 telcos responded.
To be generous, it is possible that some questions were skipped by some telcos, and perhaps more telcos answered other questions. But only 61 answered a simple question about where their Fraud Department is located in their company’s organization chart. If somebody does not know, or will not share where their fraud team sits, then why would we trust their other answers?
It is also fair to observe that 61 is a significant increase in the number of respondents to the CFCA survey. Similar analysis of the 2017 survey shows that some of those questions were answered by just 46 telcos.
Did the CFCA Survey a Representative Sample of Global Telcos?
Put simply, even the CFCA does not pretend that its ‘global’ survey is actually representative of what is happening across the world. This can be seen by simply looking at the answers given when telcos were asked about the region in which they were based.
Western Europe: 29.1%, North America: 26.6%, Asia: 10.1%, Eastern Europe and Russia: 10.1%, Africa: 8.9%, Central and South America: 6.3%, Middle East: 5.1%, South Pacific: 3.8%
For the first time, more of the CFCA survey responses came from Western Europe (29.1 percent, up from 15.2 percent in 2017) than North America (26.6 percent, down from 34.2 percent in 2017). In other words, over 55 percent of responses came from telcos based in regions that are home to just 13 percent of the world’s population.
These figures also suggest that the rise in the number of survey respondents from 46 in 2017 to 61 in 2019 can mostly be attributed to an increase in the number of responses from Western Europe. Consider that the number of North American responses appears to be static over the same period: 34.2 percent of a total of 46 telcos and 26.6 percent of a total of 61 telcos both roughly equate to 16 telcos from North America for both 2017 and 2019. In contrast, 15.2 percent of a sample of 46 telcos is equal to just 7 Western European telcos in 2017, whilst 29.1 percent of 61 telcos is roughly 18 telcos from Western Europe in 2019. 11 additional responses from Western Europe would account for most of the increase of 15 telcos from 2017 to 2019. This also shows there was little growth in the number of respondents from other regions, resulting in an increased bias towards Western countries than was manifest in 2017.
It might be argued that the bias towards Western Europe could be exaggerated because some big international telecoms groups are headquartered in Western Europe, but they serve customers in other regions too. However, this argument hardly works for a sample where just 10.1 percent of sampled telcos were based in Asia. The continent of Asia is home to well over half of the world’s population. The mega-telcos of China and India are headquartered in their home countries; it is difficult to see how you can claim to have a reliable survey of global telecoms fraud without incorporating players as large as China Mobile and Reliance Jio. Mergers and bankruptcy have reduced the number of telcos in some Asian countries, but a representative survey would need to obtain a fair sample of the 48 countries in Asia, compared to the 25 countries of Western Europe, and the 23 countries that belong to North America if you include the Caribbean and everywhere in Central America from Mexico to Panama.
Per any sensible measure, the telcos responding to the CFCA survey are heavily skewed towards telcos serving white Western countries and away from those serving Asians and Africans in developing economies. This is also an example of racial bias being embedded in the way telco risk professionals work.
The sample of telcos surveyed by the CFCA is too small to generate reliable estimates of fraud on a worldwide scale. Furthermore, the data is obviously biased towards rich white countries. Not even the CFCA dares to claim their sample is representative, but they do not have to: many have fallen into the habit of behaving as if it is.
With a sample this small, and this skewed, it is daft to talk about fraud ‘rising’ or ‘falling’. Anyone applying a degree of professional skepticism should know any variance from 2017 to 2019 is likely to be a statistical artefact, a consequence of the way the sample was taken rather than representing any real change in the world. Furthermore, there is little reason to surmise that the fraud endured by these telcos is equivalent to the fraud endured worldwide… unless you want to endorse the unjustified belief that white Westerners know more than everyone else.
But perhaps I am missing the point. Perhaps the people quoting these numbers do not care if they are true or not, because the real goal is to grab attention. I can hardly argue against their objective, because the sheer number of times the CFCA survey is quoted confirms our profession is addicted to bogus statistics. But if I can poke such straightforward holes in this report, do we really expect serious and powerful people to give it much credence? Fraudsters are liars, and anyone considering the risk of fraud will be mindful of the chances of being deceived. We should not be surprised at the failure of any business case for increased investment in fraud management if it depends on evidence as unreliable as these survey findings.
Tomorrow I will review the survey findings in detail, examining which are useful and which can be proven to be wrong.