15.8 C
New York
Monday, April 15, 2024

Why It's So Hard to Count Twitter Bots

Is the Twitter account @ElonMusk a bot? One of the best algorithms for detecting fake accounts thinks it might be, which shows how challenging it is to quantify the proportion of fake accounts across the social network.

Counting Twitter bots has become a point of contention in Elon Musk’s ongoing $44 billion acquisition of Twitter. Last Friday, the billionaire tweeted that he was putting his purchase “temporarily on hold” until the company provided details to back up its claim (as stated in its latest SEC filing) that fewer than 5 percent of “monetizable daily active users” on Twitter are spam or fake. Musk also outlined a plan to count bots himself that involved sampling 100 @Twitter followers to see how many were bots and said the approach suggests over 20 percent of accounts are fake.

But accurately quantifying the percentage of bots on Twitter is a lot more difficult, according to experts.

Finding them isn’t hard if you know where to look. Certain accounts, including Musk’s, seem to attract plenty of them. “If you simply mention Elon Musk on Twitter, you immediately get engaged with a ton of crypto bots,” says Chris Bail, a professor of sociology at Duke University who studies social media.

Twitter is not the only social network to struggle with fake accounts. Facebook removes billions of bogus accounts every year. But it is hard to know for certain that an account on Twitter is a bot, since legitimate users may have few followers, rarely tweet, or have strange usernames. It is even more difficult to gauge the number of bots that operate across the platform as a whole.

To test Musk’s proposed methodology, IV.ai, an AI company, looked at 100 accounts that follow Musk’s car manufacturing company Tesla on Twitter.

An algorithmic examination of the accounts on Tuesday found that more than 20 accounts out of 100 have a high likelihood of being bots. A manual examination of the same 100 concluded that more than half may be bots. And an analysis of the topics discussed by those accounts did not find evidence that any of the suspected accounts were promotional. But many of those accounts also disappeared shortly after, suggesting that Twitter catches bots fairly quickly. Vince Lynch, CEO of IV.ai, says identifying dubious accounts is also inherently subjective and involves a degree of uncertainty. 

“It’s a very hard problem,” says Filippo Menczer, a professor at Indiana University who led the development of the Botometer algorithm, which gave Musk’s account a relatively high bot score. Menczer says that looking at 100 accounts will not be representative of Twitter’s daily active users, and different samples will produce wildly different results. “I want to hope that that was a joke,” Menczer says of the methodology.

Automated accounts have become more sophisticated and complex in recent years. Many fake accounts are partly operated by humans, as well as machines, or just amplify messages written by real people (what Menczer calls “cyborg accounts”). Other accounts use tricks designed to evade human and algorithmic detection, such as rapidly liking and unliking tweets or posting and deleting tweets. And of course there are plenty of automated or semi-automated accounts, such as those run by many companies, that aren’t actually harmful.

Most PopularBusinessThe End of Airbnb in New York

Amanda Hoover

BusinessThis Is the True Scale of New York’s Airbnb Apocalypse

Amanda Hoover

CultureStarfield Will Be the Meme Game for Decades to Come

Will Bedingfield

GearThe 15 Best Electric Bikes for Every Kind of Ride

Adrienne So

The Botometer algorithm uses machine learning to assess a wide range of public data tied to an account—not just the content of tweets, but when messages are sent, who follows an account, and so on—to determine the likelihood of it being a bot. Although the algorithm is state of the art, Menczer says, “a lot of accounts now fall to the range where the algorithm is basically not very sure.”

Menczer and others say that spotting bots is a game of cat and mouse. But they add that it may become significantly more challenging in the future as spammers use algorithms that are better able to generate convincing text and hold coherent conversations.

Twitter itself is better equipped to spot bots using machine learning because it has access to a lot more data about each account. This includes a user’s full history of activity, as well as the different IP addresses and devices they use. But Delip Rao, a machine learning expert who worked on spam detection at Twitter from 2011 to 2013, says the company may not be able to reveal how this works because doing so could disclose personal data or information that could be used to manipulate the platform’s recommendation system.

This week, Musk also got into a spat with Parag Agrawal, Twitter’s CEO, over how easily the company could disclose its methodology for finding bots. On Monday, Agrawal posted a thread explaining how complex the challenge still is. He noted that the private data Twitter holds may change calculations around the number of bots on the service. “FirstnameBunchOfNumbers with no profile pic and odd tweets might seem like a bot or spam to you, but behind the scenes we often see multiple indicators that it’s a real person,” he wrote in the thread. Agrawal also said that Twitter could not disclose details of these assessments.

If Twitter is unable, or unwilling, to reveal its methodology and Musk says he won’t proceed without details, the deal may remain in limbo. Of course, Musk could be using the issue as leverage to negotiate the price down.

For now, Musk seems dissatisfied with Twitter’s efforts to explain why finding bots is not as easy as he thinks. He responded to Agrawal’s long thread on Monday with a simple message that seemed far more fitting for a bot than a prospective buyer of Twitter: a single, smiling poop emoji.

Update 5/9/2022 12:00 ET: This piece has been updated to not imply that IV.ai singlehandedly identified bot-like activity among accounts amplifying misinformation about US voter fraud.

Related Articles

Latest Articles