Professionals working in cybersecurity, fraud management and revenue assurance have long been fascinated with the results of surveys that claim to provide a global estimate of the cost of risk management and its failures. As somebody who studied the science of statistics when I was younger, I find it equally fascinating to observe the ratio between:
- the large number of intelligent people who trust the figures produced by these surveys, and
- the tiny the number of professionals who actually contributed to the sample.
Industry polls like the CFCA Fraud Loss Survey struggle to obtain samples larger than 50. To put this into perspective, the FCC’s new Robocall Mitigation Database currently lists 6,340 comms providers that must comply with FCC rules governing voice calls. Those FCC rules only concern voice calls, and only if they terminate in the USA! So it is reasonable to question how much a telco in Vietnam or Tonga or Eswatini should be comparing the amount of fraud they endure with the mean average of a bunch of US telcos that represents less than a single percent of all the telcos listed in the US regulator’s anti-fraud database, never mind all the other comms providers that also exist around the planet.
These is a solution to the problem of unreliable survey results: obtain much larger samples. Last year’s big survey from the Risk & Assurance Group (RAG) received responses from 175 risk professionals working for comms providers. This was a huge step forward for the comms industry, especially as the 2020 survey was the first time RAG had asked comms providers to report how much leakage they suffer. But 175 is still nowhere near enough for a robust sample. That is why I am asking for your help today.
They say it is insane to keep doing the same things whilst expecting to get different results. The RAG leakage survey is different to others. The following principles explain why RAG’s survey has a chance of obtaining a genuinely robust measure of the scale of industry leakage.
- Ask few questions of many people, not many questions of few people. A long questionnaire depresses the number of people willing to answer. Some questionnaires take hours to complete. RAG’s survey asks just 16 questions which can be answered in 7 minutes.
- Be inclusive and respect diversity. If all the questions are written by people living in Wyoming, Washington and West Virginia then they will be much more relevant to comms providers in the USA than those operating in Uruguay, Uganda and the United Arab Emirates. The panel of 11 experts who selected the questions for the RAG survey were drawn from every continent.
- The standard deviation is as important as the average. Surveys with small samples refuse to share information about the variance of results because they want to hide the sample size. However, you can only usefully compare results if you understand the range and distribution of outcomes as well as the arithmetic mean of those outcomes. That is why RAG shares all the raw data collected.
- Work with global partners who share your interests. RAG is proud to have international assurance, analytics and security vendor Subex as sponsor of this year’s survey. I would rather have big businesses actively sharing our questionnaire with hundreds of their telco customers than have many ‘partnerships’ with organizations eager to repeat headline results but doing nothing to increase the survey’s sample size.
Updated 16 October 2021: The survey has now closed! All of the survey data will be collated, analyzed and then published on this page of the RAG website at the beginning of November.
