Hoax: Fake Russian “troll” error message

This case study was originally published in OpenAI’s October 2024⁠(opens in a new window) report.

On June 18, a post on X appeared to expose a Russian troll account whose credits for using GPT‑4o had expired. The post quickly went viral. Our investigation showed that this post was a hoax which could not have come from our models. However, earlier posts made by the same X account were generated using our models, apparently in an attempt to bait controversy. This activity likely originated in the United States. After further investigation, we banned this OpenAI account; public reporting shows that the X account has also been suspended.

This hoax consisted of an OpenAI account and an account on X. Initially, the OpenAI account used our models to generate short, adversarial comments in reply to other people’s posts on X. These comments were then posted by the X account. This activity occurred in mid-June. On June 18, the X account posted a comment that appeared to be a JSON error message from a Russian-speaking user who was trying to generate content supportive of President Trump, but who had run out of credits. This message was not generated by our models, but appears to have been manually created. It included a snippet of apparent JSON code which was not valid JSON, and which mis-referenced our model’s name.

The hoax post quickly went viral, with tweets about it generating thousands of retweets, posts on LinkedIn and Reddit, multiple media queries, and even a debunk of the incident.

Concerning the content that was actually generated using our models, the actor behind this account generated counter-arguments to other X users on a wide variety of topics. These ranged from fantasy gaming through motorcycles to arguments about whether the world is flat. In most cases, the model was primarily instructed to be argumentative, and it is this, more than any particular ideology, which was the common theme. Open-source reporting shows that one of the X account’s posts used an ableist term to denigrate the mental capacity of the person it was arguing with. This term was not generated by our models, but appears to have been added in by some other process before the comment was posted.

This was an unusual situation, and the reverse of the other cases discussed in this report. Rather than our models being used in an attempt to deceive people, likely non-AI activity was used to deceive people about the use of our models. The original tweet, as shown by one screenshot, achieved five reposts, 14 quotes, and three likes. Tweets about the tweet achieved at least a thousand times more spread—and were then further amplified on other social media platforms, and led to mainstream media inquiries. This would place the hoax at the upper end of Category 3 on the Breakout Scale, close to breaking out to Category 4 if mainstream media had amplified it. This is an object lesson in how quickly social media can amplify an appealing hoax, but it also shows how the mystique once possessed by Russian influence operations has been replaced by a far more skeptical view of their capabilities. One reason the hoax attracted an audience appears to have been that it appealed to a belief that Russian trolls are not only human, but sometimes laughably inept.