Review of the TMF Fraud Survey

I should not really be reviewing the report of the TM Forum’s fraud survey, which is currently being circulated for evaluation by TMF members. However, I have never once heard of any TMF member actually affecting the contents of documents like these (including the several times I have tried myself) so I will give you my preliminary review anyway. In short, the results are interesting enough to merit a quick read, but not so interesting they are likely to change the way you manage fraud. TMF members may want to download the report. The rest of you can afford to wait for whatever information they subsequently choose to make public.

The survey received responses from 51 telcos. That number could be worse, and the spread of telcos was pretty good, but this is hardly enough to justify reaching any reliable conclusions about fraud management worldwide. Per the title the survey was seemingly conducted in 2015, though it has taken a while to share the results. The TMF should probably consider whether their internal bureaucracy is preventing them from releasing survey findings on a timely basis. Evidently a fair number of the respondents were not TMF members; when asked about which fraud organizations they followed or were a member of, only 53 percent listed the TMF. (My head hurts when I try to imagine how almost half of the responses came from telcos who neither follow nor are members of the organization which ran this survey.) To be fair, only the GSMA Fraud Forum scored higher, being followed by 65 percent of respondents.

You can probably guess the answers given to questions about the way fraud responsibilities are organized, the scope of work, the extent of coverage, and so forth. Fraud management has been around for a long while, so there was a predictable mix of telcos who felt they did everything whilst others knew they needed to do more. However, a larger sample might have yielded more interesting results; part of the difficulty in extrapolating from data like this is that the least capable and engaged telcos are also least likely to contribute.

The survey questions were designed to push the need for broad acceptance of a standardized fraud maturity model. I am not sure that is wise. The problem with maturity models, as beautifully illustrated by what has happened to the TMF’s RA maturity model, is that they are easily degraded by including too much tactical detail at the expense of a coherent strategic vision. Tactical improvements excite consultants and vendors, because they want to help telcos execute specific focused projects with clearly stated deliverables, and so allow the telco to do something tomorrow which it did not do yesterday. But this level of detail is also too precise for a genuine maturity model, leading to unwarranted generalizations that fail to fit the circumstances of many real businesses. It is easier to generalize if you stick to strategy, and the RA maturity model was most successful when describing a strategic evolution path that stretched years ahead of telcos (and is still years ahead of where most telcos are). But that requires a common vision of where we want to go, and such a vision is neither easy to agree nor a convenient mechanism to boost sales. Unless the TMF fraud team has a distinctive strategic vision they can articulate, they will add little to the industry by pushing a list of tactical improvements that they like. Furthermore, competent fraud managers will know what tactical improvements they have already prioritized for their business, and so are unlikely to support the adoption of an external maturity model unless it happens to coincide with their own interests. For those reasons I was not surprised that the survey showed low levels of interest in maturity models, whilst I was surprised that the report writers argued so strongly for them.

I disagreed with the report writers when it came to interpreting the losses reported by telcos. In this survey, most telcos said they suffered less than 1 percent of leakage due to fraud. The report writers stated…

…this figure is at odds with other industry benchmarks, which consistently show the value as higher than this.

I do not know which benchmarks they are referring to, but that sounds wrong to me. The CFCA global fraud survey is arguably the most persuasive fraud survey run on a regular basis, and it has consistently shown that most telcos think they are losing less than 1 percent of revenues to fraud. However, it is true that the CFCA survey estimates of loss were higher when telcos are asked to estimate the losses that other telcos suffer. Clearly there is some potential for bias in the results that a telco reports, but the TMF survey was poorly designed because it failed to follow the CFCA’s approach of asking a pair of questions that would have helped to assess the potential impact of bias. Instead, I think the writers are showing their own bias, by comparing how telcos report their own leakage with typical survey estimates of leakage in general.

And please, if people really want to write authoritative research reports, will they do what every first year undergraduate is expected to do, and cite their sources? The low leakage estimate obviously upset the report writers, who later commented:

The majority of respondents state they are losing 1% or less of their revenues due to Fraud (sic) – that is below the industry benchmark range of 1.5% to 2%.

But no source is given for this ‘industry benchmark range’. This report is supposed to present objective research from an industry association; it is not meant to be a marketing white paper from a vendor. This ‘range’ deserves to be treated like bullshit if the authors do not refer to where it has been published. After all, the writers of this report are effectively warning readers that their own survey findings may be bullshit! Saying the survey does not match the results of some unnamed other survey leaves me wondering if I am wasting my time by reading this report.

I found the most interesting findings related to training. A minority of telcos said they received fraud management training from external providers during the past year. However, when asked what would boost performance in the short term, the top answer was increasing the training activities of the fraud team. The challenge is clear: how will education providers help telcos to close their training gap?

Perhaps this survey will improve after a few more iterations. Building up a track record will increase the number of telcos that take part. The interpretation of the data will improve if questions are refined to minimize bias and if comparative numbers are presented from previous surveys. For now, the TMF team needs to ask themselves what they hope to achieve by running their own survey, especially if they are going to undermine its credibility by hinting that other surveys deliver more reliable results.

TMF members have until 21st June to give their evaluation feedback. They can download the report from here.

Eric Priezkalns
Eric Priezkalns
Eric is a recognized expert on communications risk and assurance. He was Director of Risk Management for Qatar Telecom and has worked with Cable & Wireless, T‑Mobile, Sky, Worldcom and others.

Eric was lead author of Revenue Assurance: Expert Opinions for Communications Providers, published by CRC Press. He was a founding member of Qatar's National Committee for Internet Safety and the first leader of the TM Forum's Enterprise Risk Management team. Eric currently sits on the committee of the Risk & Assurance Group, and is an editorial advisor to Black Swan. He is a qualified chartered accountant, with degrees in information systems, and in mathematics and philosophy.

Commsrisk is edited by Eric. Look here for more about Eric's history as editor.