Study reveals AI-driven fake accounts spreading misinformation on X

German researchers have identified nearly 8,000 bot-operated accounts on X, previously Twitter, using AI-generated profile pictures to amplify political messages and cryptocurrency scams.

The study, conducted by Ruhr University Bochum, GESIS Leibniz Institute, and CISPA Helmholtz Center, indicates that these synthetic profiles are part of a coordinated effort to manipulate online discourse.

According to the research, the accounts were created in bulk, mostly in 2023, with specific goals of spreading disruptive content and misinformation.

“A significant portion of accounts was created in bulk shortly before our data collection, which is a common pattern for accounts created for message amplification, disinformation campaigns, or similar disruptive activity,” the researchers stated.

Advances in generative AI technology have blurred the distinction between real and fake profiles, aiding this manipulation.

The study highlighted specific characteristics of fake accounts, including lower follower numbers, averaging around 393.35 compared to the 5,086.38 average of real accounts.

They also showed minimal interaction with other users, often posting messages without engaging.

Additionally, the study uncovered a cluster of accounts with exactly 106 followers, further indicating organised, inauthentic behavior.

These fake profiles focus on divisive subjects, including the war in Ukraine, U.S. elections, and COVID-19 discussions, mirroring findings by the Center for Countering Digital Hate (CCDH).

The CCDH report noted that 87 of Elon Musk's posts regarding U.S. elections were rated as misleading by fact-checkers, collectively drawing 2 billion views.

Beyond political discourse, the bot accounts also promote cryptocurrency scams and explicit content.

The research team plans to refine their detection methods to identify accounts using newer AI models and to expand their analysis to other social platforms in a bid to combat “coordinated inauthentic behavior.”