Regular readers of Commsrisk will be aware of the many criticisms of STIR/SHAKEN, the centerpiece of the US strategy for reducing unwanted robocalls, not least because foreign telcos and regulators are being pressured to adopt the expensive anti-spoofing technology. However, evidence is emerging of a previously unexamined problem within the US strategy for nuisance robocall reduction. With STIR/SHAKEN proving to be a disappointment, greater emphasis has been placed on using analytics to intelligently distinguish between good and bad traffic. Anyone with a robust understanding of data analysis will already apprehend the danger: relying on algorithms to categorize billions of phone calls will lead to some false positives.
A further risk is that data, and power, is concentrated. The market for call labeling analytics is dominated by just three firms: Hiya, First Orion and Transaction Network Services (TNS). The Federal Communications Commission (FCC) has helped to spur revenue growth for these firms by encouraging telcos to make greater use of analytics to tackle spam. What is still unknown is the degree of error manifest in practice, and the extent to which errors will be tolerated by regulators who choose to remain coy about mistakes, despite their inevitability. A recent example of the concerns comes in the form of a letter to the FCC from Raymond Pasquale, CEO of Unified Office, a managed service provider for small and medium-sized businesses.
We are a relatively small VoIP provider based in Nashua, New Hampshire with customers around the country spanning from large auto dealerships to over 1,200 restaurants. Recently, over the past few months, we began to receive complaints from our customers regarding a change in the presentation of Caller ID, particularly the CNAM (or customer name) portion. The complaints ranged from calls being labeled “spam risk” to “city, state” and other misleading labels.
The mislabeling of the calls has occurred despite Unified Office satisfying the anti-spam obligations that have been heaped upon businesses like theirs.
We’ve been Stir/Shaken compliant for several months and sign all of our calls.
Though the calls are not being blocked, the mislabeling of the calls is also damaging.
The removal of CNAM and the mislabeling of our calls has served to create harm both to our customers’ businesses and to our reputation as their provider. The worst of this is the resulting lack of faith developing in the public phone system’s seeming inability to label calls correctly as well as calls not completing because called parties are not answering their phones because the customer name is missing or the call is mislabeled.
Despite all the hoopla about the FCC’s crackdown on unwanted calls, one of Pasquale’s criticisms is that his business is now subject to determinations that are made by people, processes and systems that remain obscure.
…this recent phenomenon has been somewhat of a secret. We received no pertinent information from any of our upstream partners that would have indicated that this practice was occurring and what the possible remedy or remedies might be. As a matter of fact, we only discovered the possible reasons for this after doing a fair amount of digging via repeated Google searches uncovering companies like Hiya, TNS and First Orion that we understand are responsible for call analytics.
Note that this example features an established American business that has not been told about changes in the way their calls are handled or about the specific reasons their customer’s calls are being labeled as spam. If they find it difficult to obtain answers to their questions, imagine the difficulty that a foreign telco would face if the same analytics are applied to the calls they make to the USA.
We have agreements with both AT&T and Verizon and approached our account teams who had no idea why this was occurring, speculating that it might be related to robocall mitigation efforts. Registering our customer phone numbers to First Orion fixed some of the mislabeling but not all. It all seems to be a big mystery and there is no clear path forward that we are able to find along with our partners as to how to prevent this problem going forward.
Pasquale also highlights the burden being placed on his business because of the mistakes made by others.
…clearing spam markings on telephone numbers that we sign and we know are not illegal robocalls has been painful at best.
Pasquale ends his letter with emotive language that suggests the seriousness of the impact that call mislabeling can have on a legitimate business.
Our customers are distraught and we are trying our best to reassure them that this will be remedied.
Several industry insiders have separately warned me that the quality of spam labeling in the USA is unsatisfactory. It is difficult to say more because the big three of US call labeling analytics, Hiya, First Orion, and TNS, may be inclined to sue if their reputations are damaged by any defamatory statements that cannot be justified using hard facts. But the problem here is that only those three businesses seem to have the data needed to evaluate how well their algorithms are performing. They must update their algorithms based on the feedback they receive from people like Pasquale, who gained some relief by notifying First Orion of specific numbers being used to originate legitimate calls, and which were previously being wrongly associated with spam. If they are engaged in the business of improving their algorithms to make them more accurate then they must also have some awareness of the historic inaccuracy of those same algorithms. However, it appears that nobody in the USA is regulating the error rates for these analytics firms. This then leads to further concerns about who is taking responsibility for accurate call categorization when the same firms sell their services to telcos in other countries, some of which will have regulators that are more inclined to permit analytics-based blocking than the FCC currently is.
Many commentators on the US strategy for reducing robocalls talk as if it is a coherent whole, supported by consensus, and consistent over time. Either they do not know or they do not care to admit the reality. I am an outsider, but the more I speak to insiders, the more I discover a history of serious disagreements and changing tactics. At one stage some held the naïve belief that STIR/SHAKEN would feed into automated analysis that would be so accurate that bad traffic would be blocked automatically. This was fanciful. There is currently only a positive correlation between A-grade STIR/SHAKEN attestations and good traffic; calls which have a B- or C-grade attestation are much more likely to be spam than calls which have no STIR/SHAKEN signature at all. This is because the lower grades of attestation represent signatures which were applied some time after the call originated.
That STIR/SHAKEN is only truly effective when applied end-to-end also explains why the US has been pressuring other countries to adopt STIR/SHAKEN. The STIR/SHAKEN model that was adopted could only deliver significant benefits for US consumers if foreign telcos followed the lead of the US too, despite the absence of any cross-border governance model, and the absurdity of a small cabal of US professionals seeking to unilaterally decide a major change in the governance of all voice calls globally. It was as if they did not expect to encounter any resistance to an expensive method which was unlikely to deliver much benefit until the whole world had implemented it in practice, even though they must have known all the reasons why technical limitations meant it could only be partially implemented in the USA, never mind anywhere else. For reasons that are difficult to comprehend, the US built a strategy around a technology that could only deliver benefits if adopted universally, and so would have very limited impact in the short-term. Their face-saving maneuver involves placing greater reliance on analytics, despite STIR/SHAKEN not providing the level of input that was previously hoped for. But automated analysis of phone calls will never be 100 percent accurate.
Consider the analytics that have been applied to distinguishing between good traffic and bad traffic in a related domain: electronic mail. Google’s Gmail service has visibility of approximately one-quarter of all emails sent, but their filters still make mistakes. Their analysis is being performed in an environment that is much more conducive to their goal. Emails need not be delivered immediately; time can be spent reviewing an email before deciding how to label it. The entire content of the email is known before it is delivered; algorithms can review all of the text and use it to make a decision about whether it is spam. In contrast, decisions about whether to categorize a voice call as spam need to be made in milliseconds, before the recipient’s phone starts ringing. These decisions can only be based on what is known about the origin of the call, before anybody has said anything.
Despite all their advantages, Google still fails to correctly label all emails. A debate about error rates is not academic. The error rates for labeling will affect people in real life. The effect is greater for voice calls; if you refuse to take a call that was labeled as spam you will not have the opportunity to look for it in your junk folder later. Spam calls are a problem, but people still rely on voice calls when the need to communicate is most urgent. We cannot afford to dismiss the real consequences if legitimate calls are obstructed by flawed algorithms. These factors apply to a domain where no telecoms business on the planet comes anywhere near to seeing one-quarter of all phone calls in the way that Google sees a quarter of all emails. Perhaps Hiya, First Orion and TNS would like a three-way split of the global market for voice call labeling, just like they have currently split the US market. That would presumably help them to improve the effectiveness of their algorithms, but only by raising genuine concerns about a lack of competition and the need for oversight.
It is too early to say if the US anti-robocall strategy has taken another wrong turn by placing too much reliance on the technology and the businesses that label voice calls. But it is time to ask harder questions about who is scrutinizing the error rates attained in practice and ensuring that the power held by the analytics businesses cannot be abused.
