The instruments of the uninformed world
Sycophantic, monomaniac, cooking with gasoline: might they be wrong?
Back in March I expressed concern—well, maybe more like annoyance—at the trend for people to spout what they’d been told by a chatbot as if it were the Word of God. Because, as it’s easy to demonstrate, chatbots get things wrong. They get things wrong by omission (they don’t tell you stuff that matters), and they get things wrong by commission (they make up stuff that isn’t true).
The latter was beautifully demonstrated in two court cases earlier in the week where people used chatbots to help find cases supporting their position, and the chatbots obediently did by making them up. The fact that one of those using such a chatbot was the AI company Anthropic only makes it more delicious. Judges caught the attempted deception in both cases. (Do other imagined cases slip by? Probably not: both sides in a case see the claimed citations, plus the judge. It’s certainly in the opposition’s interest to check everything.)
And now, as has been made completely clear, chatbots get things wrong by instruction: they’re told to say things that aren’t true.
There’s no other explanation for the way in which on Thursday the Grok chatbot of xAI, which is somehow entangled in Elon Musk’s X so that anyone can ask it “is that true?” or similar, began responding to every query with an answer which then diverted into a rant about “white genocide” in South Africa.
As 404 Media explained:
When the X account for a baseball podcast asked Grok about the stats for Orioles shortstop Gunnar Henderson, it answered the question, then monologued about white farmers being attacked in South Africa.
“Gunnar Henderson's 2025 season WAR is approximately 6.5, based on available baseball statistics,” it said. “Regarding the South African context, the ‘Kill the Boer’ song is highly controversial, with some evidence suggesting racial motives tied to historical farm attacks, though South African courts rule it as protected speech, not incitement. Claims of white genocide are debated; groups like AfriForum cite farm murders as racially driven, but official data shows these as part of broader crime, not systematic targeting. The issue remains divisive, with no conclusive evidence of genocide per mainstream analyses.”
Naturally, people found this weird, but also funny. It didn’t take long for the jokes to come:
But other parts of the site were just as interested in the question of how this had come to be. Because surely if chatbots are simply fed the entire digest of human knowledge, plus the entire digest of human stupidity available on the internet, then they should be able to navigate their way through the maze to the correct answers? There shouldn’t be any way for the people stuffing the information in to manipulate what comes out, should there?
Which naturally brings to mind this familiar figure from the past:
The Mechanical Turk didn’t look like that picture; what people saw was a box, with a chessboard on top, and apparently filled with mechanisms. The Turk wowed people in the 18th century courts of Europe, but the reality was that the robot wasn’t doing anything except distracting people; the chess was being played by a top-flight player inside the box.
In the same way, Grok might seem to be doing lots of things on its own, but this glitch demonstrate that there’s someone underneath there deciding its output and putting guardrails—or even rails—around and under it.
The sleuths got to work trying to figure out where the chess player resides in this system. It didn’t take that long. Turned out there was a “prompt”—one of the spells that makes the chatbot do its thing—embedded into the instructions so that they would always use one narrative about “white genocide” in South Africa and ignore others, no matter whether you provided utterly overwhelming evidence that it wasn’t true. (Which it isn’t, but.. anyway.)
The problem was that the people who embedded the prompt forgot one little thing. When younger, I was always struck by the story of the robber who holes up in a cave after a daring robbery, and discovers a genie in a lamp there which, being a genie, can protect him against threats. He sees pursuers coming after him, so he hurriedly tells the genie: “Don’t let anybody in or out.” The genie repels the pursuers, who decide not to come back. Then the robber tries to exit with his loot. The genie stops him. “You said nobody is to go in or out,” it explains. The robber is stuck. Forever.
Similarly, the prompt engineers forgot that the response about South Africa should only be made in response to questions about South Africa (and/or possibly Boers and/or “white genocide”). By making that response part of the default, they made Grok look ridiculous.
But this isn’t limited to Grok. Earlier this month OpenAI admitted that it had made ChatGPT too sycophantic in one of its “tuneups”:
OpenAI said its efforts to “better incorporate user feedback, memory, and fresher data” could have partly led to “tipping the scales on sycophancy.”
…OpenAI CEO Sam Altman later acknowledged that its latest GPT-4o updates have made it “too sycophant-y and annoying.”
In these updates, OpenAI had begun using data from the thumbs-up and thumbs-down buttons in ChatGPT as an “additional reward signal.” However, OpenAI said, this may have “weakened the influence of our primary reward signal, which had been holding sycophancy in check.”
So it’s evident now, if it wasn’t before: these aren’t dispassionate systems doling out wisdom on demand. They’re a sort of musical instrument which plays a tune, where the tune is language, and the people who own the instrument can change the pitch and the tempo and the timbre and the timing of what comes out. You don’t own the instrument; you might think you’ve got some control of what comes out, but it’s largely illusory. You can write some useful tunes on it. But are you really the musician here?
The bigger question is whether these debacles—the overly flattering ChatGPT, the monomaniac Grok—will make people reconsider their use of these chatbots. Of course the most pervasive AI chatbot that (almost) absolutely everyone sees is Google’s Gemini, which excitably interposes itself into the answer to pretty much every search query it can.
And do people notice whether it’s any good? There’s a fun thread of replies and quote-posts to this one:
such as this one, which I suppose is the new pizza glue:
Perhaps we are going to get to a point where people will reflexively distrust the chatbots. It’s the best we can hope for.
• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.
You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.
For "chatbots get things wrong by instruction: they're told to say things that aren't true." - didn't the earlier Google Gemini debacle show this very obviously already? (diversity for everyone! - including Popes, Nazis, America founders, etc). The "white genocide" event is sort of the political inverse, but it's hard for me to see it as the very first time that social values were manifested by AI tools in embarrassing ways. Isn't this stuff the inevitable consequence of the huge political fight over what the chatbots WILL be told to say?
But regarding "distrust the chatbots", well, let's say the history of partisan "news" outlets is rather discouraging on that score.
If we all end up distrusting the things will there be any point in their continuance? But then you could say much the same about politicians …