Google announced last week the addition of new AI-powered scam detection features for Android phones. They claim that the new tech will provide protection against voice and messaging scams in real time, during the course of an exchange with a scammer. The question for consumers and regulators is whether the new Android filters will be more or less effective than Google’s existing, and often frustrating, spam filtering for email. Wrongly diverting an email to a junk folder is annoying, but the consequences of wrongly interrupting a voice call or wrongly flagging text messages could be a lot worse.
The core of Google’s approach is an on-device AI which analyzes conversation patterns in real-time. The critical difference, and a potential point of failure, is that the technology is not simply scanning for keywords. It is supposed to look for contextual clues during the course of the victim’s interaction with a scammer. For instance, red flags should be raised when there are requests for sensitive information like one-time passwords (OTPs) and personal identification numbers (PINs) or demands for immediate transfers of money. The tech will look for patterns during the exchange of multiple text messages or in the midst of a live voice conversation.
Privacy and Limitations
Google emphasizes that this scam detection technology is executed entirely on the device, with no data sent to any of Google’s servers. This addresses the immediate privacy concerns, but it also presents a limitation. A centralized system, whilst potentially more invasive, could leverage a much larger dataset for training and adaptation. On-device AI is only as good as the data it has been trained on, and that data is inherently limited.
The initial roll-out of scam detection for messages will only occur in English, and only in the USA, UK and Canada. The exclusive focus on English is unfortunate because of the high frequency of scams that target victims who live in these countries but whose first language is not English. Presumably Google also calculated they could treat French-speaking Canadians as a lower priority without negatively impacting their reputation amongst that community.
Scam detection for calls has already been subjected to beta testing with some English-speaking users of Pixel 9 handsets in the USA. Google now intends to expand coverage to all US customers of Pixel 9 and above. Phones will play an audio warning and will vibrate when the AI suspects the phone user is conversing with a scammer. This will be alarming for users who have never experienced it before, so false positives could greatly damage the credibility of voice scam detection from the outset.
Much will depend on how much rigor is applied to testing not only during the initial launch phase, but also during the roll-out to additional languages and countries. In particular, it is important not to be seduced by performance metrics which are too narrow. Reporting a very high percentage success rate for a very limited range of scams conducted in English will not help us understand if scammers are compensating by devising new scams or shifting their focus to other languages. There will not be a consistent increase in trust for phone communications unless AI-based monitoring software protects all members of culturally diverse populations.
Optimism Tempered by Realism
Google’s development of on-phone scam detection capabilities is a welcome step. AI undoubtedly has potential as a technology to protect consumers from crime. However, I left the recent Mobile World Congress in the Barcelona with the impression that the comms industry is going to place too much reliance on AI to fight scams. It is easier to see how tech businesses can make money from AI than from other tools to fight crime, but that does not mean we can afford to concentrate investment in AI at the expense of other methods of detecting and preventing scam activity. For example, there is an unhelpful and uneconomic balance being created by loading the majority of anti-scam controls at the very endpoint for communications, the recipient’s phone, whilst the comms industry fails to impose consistent know-your-customer checks on businesses that instigate bulk communications. Google also deserves criticism for continuing to profit from the advertising of SMS blasters on YouTube.
We know that scammers are adept at changing their tactics. While an AI might catch patterns of scam activity that are already familiar, criminals respond to effective controls by evolving the methods they use. It is important not to become too dependent on techniques like filtering as an alternative to other staples of consumer protection, such as deterring crime with proper enforcement of laws and harsh penalties for those criminals who are caught. Criminals will experiment with changing their phrasing to evade the detection capabilities of AI. Similar issues have arisen in the context of email spam filters; they may miss sophisticated phishing attempts whilst wrongly flagging legitimate emails as spam.
I am a business customer of Google’s Gmail service and I also use email services provided by other firms. This allows me to observe some limitations of technology in actual practice. There has been a lot of experience accumulated with email filters but they are still not perfect. Google sees around one-quarter of all emails, giving them an enormous amount of data to learn from. Their filters still make mistakes. Accurately filtering other forms of communication is a much more extreme challenge than filtering emails. This leads me to be circumspect about how well these new AI-powered filters for voice and text messaging will be.
Email providers have developed ways to counter abuse such as DomainKeys Identified Mail (DKIM) for which there is no good analog in voice or text messaging. The words in an email are not split up between multiple messages, as they are when two phone users are exchanging texts in quick succession. All of the content of an email is available to Google prior to them delivering the message. The processing power they can apply to analyzing the contents of emails is not limited to that available on a single handset. There are no time constraints when analyzing emails before they are delivered, as there are when trying to analyze live voice conversations. Emails do not include non-verbal communications like the tone of a speaker’s voice. But Google still makes many mistakes with how it categorizes emails.
Harmful emails still get delivered to my inbox by Google. Innocent messages still get directed to the junk folder, even when they came from another one of my own email accounts. If Google cannot reliably tell that a guy called Eric Priezkalns is not impersonating a guy called Eric Priezkalns when forwarding an innocuous email from one account to another then their AI is bound to make mistakes when somebody is in the middle of a conversation with their bank or receives messages from their doctor’s surgery.
Academic research showed that email filters exhibited political bias during the 2020 US elections. To describe a filter as ‘biased’ is another way of saying it makes arbitrary decisions that users would not agree with. There will be pushback if filters like these are found to be ‘biased’ against legitimate businesses, or genuine messages that really did come from the police or government, or subsets of the community whose choice of words might sound like those of a scammer when interpreted by a machine.
The effectiveness of Google’s anti-scam tech will depend heavily on continuous learning, adaptation, and its ability to keep pace with the evolution of scams. The history of email spam filtering suggests this will not be straightforward. More importantly, Google’s monitoring tech is neutralized as soon as a scammer persuades a victim to switch to a communications channel that Google is not able to monitor. We must pursue a broad strategy that attacks scammers on all fronts instead of placing too much emphasis on automated filters.
Google’s blog about their new Android anti-scam capabilities can be found here.
