With the substantial public and private investment in combatting the problem comes a need to assess our progress and re-evaluate what we believe we know about illegal calling. Industry sources publish regular reports on call trends, but using proprietary and opaque methodologies. For example, call labelling must happen with limited context, and is therefore difficult or impossible to evaluate accuracy. Academic research uses transparent methods and evaluations, but comprise “one-shot” efforts describing a single moment in time. Collectively, the technical community and the public at large need a reassessment of the state of the network.
These observations sound like they were drawn from Commsrisk, but they come from “Characterizing Robocalls with Multiple Vantage Points”, an academic paper published on October 22. The paper was authored by a crack team of researchers at North Carolina State University (NCSU) whose work has been highlighted on Commsrisk before: Sathvik Prasad, Aleksandr Nahapetyan and Bradley Reaves. Prasad and Reaves also conducted the award-winning study in 2020 that used a massive honeypot to demonstrate the prevalence of unsolicited robocall campaigns in the USA. Per their own words, their new research highlights “the most promising directions for future efforts to characterize and stop telephone spam”. And the single most promising method would be to make increased use of phone honeypots!
The paper opens with the good news that multiple sources of data indicate a fall in the number of illegal calls received by Americans.
Longitudinal data indicates a secular decline in robocall volumes, though the magnitude of the decline may range from 25%-50%… We cannot attribute the decline to any single factor.
Fans of STIR/SHAKEN love saying it is not a silver bullet. The NCSU researchers demonstrate it is even less of a silver bullet than the fans care to admit.
We present the first analysis of the adoption of STIR/SHAKEN call authentication protocol by robocalls. Notwithstanding the decline mentioned previously, we show that remaining robocallers have succeeded in in (sic) assimilating to the new regime. We further discuss reasons that explain why STIR/SHAKEN has failed to dramatically reduce robocall volumes and propose recommendations to address them.
The US Federal Trade Commission (FTC) has tried to reassure the American public by publicizing action taken to tackle voice scams. The public may not be impressed if they saw an objective analysis of how promptly the FTC has acted.
Using traceback data, we find that about 99% of campaigns subject to enforcement actions through the Project Point of No Entry (PPoNE) investigation ceased to operate after the enforcement actions… We estimate that PPoNE investigated campaigns associated with 5.5% of all robocalls observed in this study, though on average the investigation lagged the first observed call by 387 days.
It is refreshing to read an academic analysis because academics respect the proper meaning of words instead of just repeating propaganda. This paper focuses on the facts as they are, not the promises made by increasingly unreliable ‘experts’ who are nothing other than shills for vendors.
STIR/SHAKEN… does not actually authenticate the source of the call, nor does it offer mutual authentication. It also does not provide mechanisms that are compatible with legacy infrastructure, so calls that originate, terminate, or transit non-VoIP networks irrecoverably lose this information. In addition to fundamental limits on what it guarantees or how compatible it is with the network, the short deployment timeframe led to a host of failures and incompatibilities so severe that many legitimate calls could not be authenticated.
India uses mandatory honeypots to measure the scale of unsolicited communications received by phone users. The NCSU researchers who wrote this paper are also advocates for collecting impartial data through the deployment of honeypots. If we really wanted to tackle phone scams, we would all be talking about the need for honeypots that will impartially tell us what scammers are doing. However, the authorities in Western countries like the USA have convinced themselves that tracking consumer complaints is a sufficiently accurate proxy for monitoring the bad traffic that ordinary people receive. That was always a reckless policy.
There are no grounds for assuming one statistic (the number of consumer complaints) is correlated to another statistic (the number of illegal calls that people receive) if nobody has ever measured the second statistic. Refusing to implement honeypots is also a stupid way to save money. Even a very large honeypot costs a tiny amount to run compared to the amounts spent on methods like STIR/SHAKEN, or the amounts that have to be spent on determining if a customer is complaining about communications that really were illegal or not. This new research from NCSU should provide the final word on whether countries like India were right to systematically use honeypots to monitor unsolicited communications. Spoiler alert: they were.
…we confirm previous studies of low overlap in honeypot and consumer complaint data. We also show that robocalling is such a large problem that two large honeypots can have high call volumes but low overlap.
If honeypots had been used systematically they would have already contradicted many of the unsubstantiated claims made about STIR/SHAKEN. One of the sources of data for the NCSU study was a large honeypot, the Robocall Observatory, that began operation before STIR/SHAKEN became mandatory in the USA.
STIR/SHAKEN was enabled on the observatory on 1st Nov 2021. Since then, Robocall Observatory has received 3,234,362 calls. Throughout the duration of Robocall Observatory’s operation, about 30.40% (983,519) of calls were signed with an A-level attestation, followed by 21.59% (698,278) with B-level attestation, 9.52% (307,986) with C-level attestation, and 38.48% (1,244,579) were unsigned calls.
Step back and reflect on these statistics for a while. Per this honeypot, phone numbers that were never used to solicit calls from any source received over 30 percent of their calls from the highest grade of sources ‘authenticated’ using STIR/SHAKEN. Over 60 percent of calls were signed with one of the three grades of STIR/SHAKEN signature. This tells us two things:
- Americans receive most of their unsolicited calls from numbers that are supposed to be safe because their use complies with STIR/SHAKEN rules
- STIR/SHAKEN has had zero impact on a large minority of unsolicited calls received by Americans
A second honeypot called RRAPTOR began operation at the beginning of 2024. It found even more unsolicited calls had received the highest grade of STIR/SHAKEN signature.
STIR/SHAKEN was enabled throughout the duration of RRAPTOR’s operation. About 44.3% (420,639) of unsolicited calls received by RRAPTOR were signed with an A-level attestation, followed by 24.5% (232,344) with C-level attestation, 12.8% (121,026) with B-level attestation. About 18.4% (174,766) calls were unsigned calls.
US STIR/SHAKEN provider TransNexus has reported a steady increase in the proportion of calls that are received with STIR/SHAKEN signatures; the difference between the figures generated by RRAPTOR and by the Robocall Observatory appears to reflect that overall trend. The researchers note that about 80 percent of calls received by the Robocall Observatory during May 2024 were signed; this was the final month of honeypot data included in the study. Analysis of the Robocall Observatory data also confirmed the trend towards higher grades of STIR/SHAKEN signatures over time, whilst the proportion of unsigned calls is declining.
Readers outside of the USA should be wary of the risk that different cultures are expected to assimilate American norms. Whether legal or not, it is considered normal for Americans to receive many unsolicited calls. These honeypots record trends for unsolicited calls, not for the kinds of calls that people consciously chose to receive, such as those from a family member or a business that they actively shared their phone number with. High volumes of authenticated but unsolicited calls indicates different expectations for acceptable corporate behavior in the USA compared to some other countries. The business case for STIR/SHAKEN depends on the belief that phone users want to receive large numbers of calls that are legitimate, and hence authenticated, even though they were not solicited. If people receive very few unsolicited calls then it follows that there will be even fewer illegal unsolicited calls.
Of the calls received by the Robocall Observatory that played a message which could be analyzed, 2.5 percent were in Spanish and 1.5 percent were in Mandarin Chinese. The respective proportions for RRAPTOR were 1.4 percent in Spanish and 0.9 percent were in Mandarin. This confirms past observations that a multilingual approach is needed to protect consumers from scams. It also begs further questions about the real motives of Americans who target their lobbying for STIR/SHAKEN at the governments of English-speaking countries like the UK and Australia whilst pretending US sanctions on China’s telecoms businesses are no obstacle to cooperation on international scam traffic. Their targets seem much better suited to boosting telesales by US-headquartered multinational corporations than to tackling the sources of illegal comms traffic.
In contrast, the relatively cheap technology of honeypots could have been used to identify and shut down many scammers and illegal spammers. As the NCSU researchers observe, many unsolicited robocalls include a spoken phone number which recipients are asked to call back.
In 2024, RRAPTOR captured 2,325 unique callback numbers from 2,519 campaigns…
Callback numbers tend to be live for around three months.
We see that the average lifetime of a callback number was 88 days, with a standard deviation of 219 days.
Instead of obsessing about the spoofing of CLIs, these callback numbers should have been the focus of immediate action whenever the content of a call appears to be illegal. We know that calls to callback phone numbers will be connected. It should be a priority to block outbound calls to callback numbers and to chase down the businesses that allowed those numbers to be used for criminal activity. Western societies have become stuck on the concept that the public receives scam calls despite a long history of scams, including wangiri, that were consciously designed to provoke the victim into dialing a number given to them by the scammer. Preventing the outbound calls made by victims to scammers will yield more immediate results than pouring effort into determining the origin of calls that scammers make. It should also be the responsibility of the authorities to stop numbers being allocated to parties which repeatedly allow them to be used for crime.
The NCSU researchers also highlighted a significant weakness in consumer protection strategies that focus on displaying warnings to recipients of phone calls.
We uncovered 3,784 campaigns with at least one call containing more than one call attempt within a 15 second window. We further filtered these campaigns to determine if more than 90% of calls within the campaign had more than one call request message (SIP INVITE) within 15 seconds. A total of 362 campaigns were using voicemail injection techniques to deliver robocalls. Such calls directly reach the user’s voicemail and bypasses the standard call presentation to the user, where labels like “Scam Likely” and “Spam Risk” are displayed when the phone is ringing.
Another reason to focus on callback numbers is that it is incredibly unlikely that we will ever compile a comprehensive list of numbers to be blocked or flagged when used as the CLI of incoming calls.
A blocklist curated from FTC DNC [Do Not Call] feed is 21.1% effective when applied to block calls made to Robocall Observatory, and is 12.1% effective when applied to block calls made to RRAPTOR. This is an upper bound because it assumes that the FTC DNC observed the call before the call was made to either of the honeypots. However, we found FTC DNC lags honeypots and often reported a caller ID after they were seen in honeypots. A blocklist curated from RRAPTOR when applied to block calls made to Robocall Observatory is only 16.3% effective. Similarly, a blocklist curated from Robocall Observatory when applied to block calls made to RRAPTOR is 9.8% effective.
…cross-source blocking of calls based on the caller ID is only marginally effective.
Waiting for consumers to complain about scams introduces an undesirable delay compared to the speed at which honeypots would identify the same scams.
…in most of the cases honeypots observe the reported caller IDs even before they are reported by consumers. Therefore, honeypots serve as early-visibility vantage points to track emerging robocall campaigns.
It is already difficult to tell if the recipients of two calls listened to the same recorded message, so consider what will happen as generative AI is used to construct interactive scambots that choose different words depending on what the intended victim says. Perhaps the most important takeaway from this research is that useful collaboration will not occur without a lot more thought and effort going into the conceptual tools that are an essential precursor to collaboration. We do not currently have the taxonomies needed to understand if two separate parties are talking about the same scam, but impartial academics could help us address that gap.
Outside of academic literature, the construct of a “call campaign” is undefined… By starting with a rigourous definition of campaigns, we were able to effectively compare campaigns across multiple data sources and modalities.
This paper made a compelling argument for the use of cheap and effective honeypots to identify, measure and shut down phone scams. Sadly, the cheapness and effectiveness of anti-scam honeypots is an obstacle to deploying them in the upside-down world of network crime reduction. Nobody would make a lot of money by operating them. Maintaining an objective source of ongoing information about unsolicited communications would be a threat to methods of tackling scams that are less effective but more lucrative.
If we really sought effective collaboration between businesses, regulators and law enforcement, we would start by basing our collective efforts on a reliable measure of which collaborative activities have the greatest impact on scams and other illegal traffic. Let us see which nominal supporters of collaboration want to follow the lead of India and the NCSU by implementing phone honeypots, and who prefers to ignore their work.
“Characterizing Robocalls with Multiple Vantage Points” by Sathvik Prasad, Aleksandr Nahapetyan and Bradley Reaves is freely available here.
