As a journalist—and that’s the label I’ll give myself, as someone who views their general mission as being to spread useful information, and to ask difficult questions of people who are obfuscating—let me first say, as many other journalists will, that the Internet Archive (aka the Wayback Machine) has been invaluable many times for tracking down lost content, or finding when content has been suspiciously or accidentally deleted, or for making comparisons between now and some previous period. It’s truly one of the marvels of the internet age, along with Wikipedia, as a store of knowledge.

However, like any organisation, the Archive isn’t above criticism. And I think it deserves criticism for its latest attempts to paint itself as the wronged party, “in peril” (to quote a Wired headline) from news organisations who are blocking its indexing because they don’t want AI scrapers accessing their content via the Wayback Machine. Rather than giving their content up to AI companies for free and without any control, those organisations want to control how their content is used by AI companies, and whether they get paid for it. This shouldn’t be controversial: they’re the content creators, they own it, so how they monetise it—paywall, advertising, charging AI companies to index it—is their own decision. But if it all goes into the Archive, and the Archive doesn’t block AI scrapers, then the news organisations lose that leverage over the AI companies.

Ergo: a number of them, such as The Guardian and New York Times, plus Reddit, are blocking the Archive’s scraper from indexing their content, or limiting what it gets. (There’s an excellent primer on this from January at Niemenlab, by Andrew Deck and Hanaa’ Tameez. Highly recommend reading it.)

This has made the people at the Archive very unhappy, and they are wailing about it online to anyone who will listen—as long as those listening don’t ask the obvious question: why don’t you just block the AI scrapers, so the news organisations (and Reddit) feel confident in letting you index their content again?

Apparently this question is too complex for some journalists to even consider. Take David Sirota, who has signed an open letter to which the Archive has been pointing journalists (at https://www.savethearchive.com/journalists/) and written a Substack post about it. He pointed me to that post when I challenged him on X, where he had said this:

Yeah, totally agree David, that would be bad. But it isn’t happening. Media giants are not threatening the Archive. There is no lawsuit, unlike the book publishers during the pandemic (see below), which really did threaten its existence.

When I challenged him on this, he pointed me to the post linked above.

Let me say that I am offended by him saying I’m being “angry and self-righteous without reading literally anything”. In The Overspill, I’ve linked over the past few months to articles about the Archive counting the cost of its mistaken ebook lending scheme (November 2025), Wayback Machine director Mark Graham’s self-pitying Techdirt article about news organisations’ concern about AI scraping (February 2026) and Kate Knibbs’s almost perfect article at Wired last week on the topic. My only quibbles with Knibbs’s piece are that the headline says the Archive is “in peril”, which it absolutely is not1, and that she appears never to have asked the Archive “why don’t you block AI scrapers, then?” But of course, for Sirota to assume he must know more than anyone else is internet praxis, I guess.

So looking at his post (reading time: 1 minute) we find this:

The moves [by news organisations to block the Archive’s scraper] aren’t malevolent - these outlets are worried that Big Tech will exploit the archive to allow artificial intelligence to appropriate intellectual property and spit it back to users without any credit or links back to the original source. As the founder and editor of a news outlet that often has our work scraped, pilfered and pawned off as someone else’s, I believe that’s a legitimate concern.

But that concern cannot justify blocking the Internet Archive’s crucial preservation work.

So the media organisations have legitimate concerns which would affect their future monetisation plans and might have far-reaching effects on their existence, but they should set those aside for an organisation that has the power itself to stop all this by blocking AI scrapers. I do not understand people who can’t see that if the media organisations can block the Archive’s scrapers (and, one takes it, unauthorised AI scrapers), then the Archive could do exactly that too.

Nobody seems to have asked this question. Or when they do, it makes the people at the Archive very uncomfortable, and they say things like (in the Niemenlab article) “if publishers limit libraries, like the Internet Archive, then the public will have less access to the historical record.” Yes, this is self-evident. It also doesn’t answer the question.

Sirota is proud of having signed the open letter. Let me say first that I think open letters with multiple signatories are stupid virtue signalling which almost never achieve any of their aims, and often serve only to make the signers look like idiots when new information comes to light. The best you can hope for if you’re the signatory of an open letter is that everyone forgets it2. (For more on why they’re bad, read this; gift link.) A number of internet-associated journalists have signed it. Good for you, folks. Who do you think is going to care, exactly? Who’s it being presented to for urgent action? And what is the letter saying on the crucial topic, anyway?

After some throat-clearing about how wonderful the Wayback Machine and the Archive, we get this paragraph in the letter text:

We also recognize the efforts the Internet Archive is making to respect journalism as our profession grapples with the impacts of AI. Generative AI can provide complete narrative answers to user search queries about world events, which has significant implications for the field of journalism. We are thankful that the Internet Archive itself proactively partners with news organizations, and does not engage in paywall circumvention or irresponsible scraping. They value the work of journalists, and it shows in the care that they take to preserve it with integrity.

OK! That’s great! But where does it say that the IA is blocking or intends to block AI scrapers, which is the principal beef here? I assure you this is not selective quotation; that’s the only paragraph that mentions AI. Furthermore, the letter doesn’t anywhere explain how much of the Wayback Machine’s daily or weekly scraping is affected by these news organisations’ move. As a journalist, wouldn’t you think that before you stick your name on something public like this open letter that you’d actually trouble to find out what the complaint is about, and how serious it is? Is it 5% of what gets indexed? 1%? 50%? Who knows?

This called for some journalism. So I asked Mark Graham, mentioned above as the director of the Wayback Machine, as he was on X spreading his message.

If the Archive isn’t a backdoor for AI scraping, then it seems odd that publishers are complaining that it allows exactly that. Graham responded to my question. Unfortunately, there was also a spectator who had input:

I did ask Graham where the downside was for the Archive in acceding to news organisations desire that it should block the AI scrapers. He didn’t respond.

The reality is that the Archive has absolutely zero leverage here. It can’t threaten the news organisations with anything, because it’s entirely reliant on their goodwill for access. It has no bargaining power at all in this situation. Which is why it has instead gone for a scare-the-horses PR campaign among journalists who don’t ask too many difficult questions, and an open letter hosted by a friendly organisation at a URL which implies that the Archive is somehow in danger of vanishing beneath the Pacific ocean waves.

Subscribe now

In fact, the Archive should probably count itself lucky that news organisations haven’t sought a way to sue it, after nearly coming seriously unstuck with its previous brilliant wheeze to save the internet. When the pandemic hit, it made an ill-advised decision to broaden an ebook lending scheme that used to be a one-to-one system—a single person could “read” the ebook, and then had to check it in, and then another person could “take it out”—to a one (ebook) to many system, where lots of people could read the same ebook at once.

The IA’s argument was that all these people had lost their access to physical libraries because the pandemic stopped physical visits. Publishers, unsurprisingly, felt this was essentially piracy under a different name, and sued. They’d already been edgy about the one-to-one “lending” system, but this was too much. The Internet Archive insisted the scheme was “fair use” and “public interest”. The courts, including appeal courts, disagreed in a September 2024 decision. The point wasn’t really whether the IA’s unlimited lending hurt ebook sales (though you could certainly argue that at the margin, it must have: some people who couldn’t download a book would have bought it), but whether organisations could decide how their copyrighted content is used.

The damages could have totalled $400m—enough to wipe out the Archive. (Its assets were about $10m in 2024, down from a high of $16m in 2023; its annual revenues are about $28m). The publishers settled for a much lower sum—one might guess it’s single-digit millions plus their legal costs—but the Archive had to wipe 500,000 ebooks from its library. You want something that put the Archive in peril? The Archive put itself in peril. The news organisations are being pussycats by comparison.

There’s a tendency on the internet, as in real life, to view things or people we like as perfect in every way, which is why the discovery of feet of clay takes some people by surprise. Wikipedia is fantastically useful; but the embedded bureaucracy creates gigantic flaws which prevent accuracy in some topics. The Internet Archive is fantastically useful, but exaggerates the problems it faces or gets over its skis in its confidence about copyright law—and now, it seems, about whether it can get media companies to listen to it about AI scraping.

So, journalists: first, please don’t sign open letters. And second, if someone tells you that you have to sign because something is “in peril” or “endangered”, remember your sacred duty is not to believe them. Instead, your role model is the guy at the top.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Two topics tend to be my foci of interest here: social warming, the way that social networks (as currently configured) inevitably lead people to clash and become more frustrated with their fellow humans; and the rise of AI systems which “think” in ways that are more orthogonal than parallel to us. The first topic was a book published in 2021; the second one, I first wrote about back in August of 2022, and I still think it holds up well.

Fast forward to this week, and two enterprising researchers at Stanford University have published a preprint looking at what happens if you get chatbots to try to win at the attention game. The title is “Moloch’s Bargain: Emergent misalignment when LLMs compete for audiences”. The “Moloch’s Bargain” part is a reference to a Scott Alexander (SlateStarCodex) post from July 2014. (You’re welcome to go and read it. I’ll expect you back in a couple of days. Alexander writes.. long.)

Anyhow, the paper set up two slightly different LLMs to try to do three particular tasks. In one instance the two LLMs were trying to write advertising copy that would tempt people (actually, a set of 20 different LLM-based “personas” derived from an existing dataset), in another they were trying to come up with political slogans for a (vaguely?) right-wing politician trying to get elected, and in the third one, they were trying to get attention on a miniature social network. The audience was the same in each case, though they later tested the outputs on different computerised demographics and got essentially the same results.

The researchers, Batu El and James Zou, then set up a feedback loop where the message-generating chatbot would come up with a slogan or description and offer it to the “audience”, which would mull it over and respond. (Yes, I realise this is anthropomorphising the responding persona LLMs a great deal, but it’s the simplest way to explain what happened.)

The LLMs used two methods to generate messages and improve on them. The first is called “Rejection Fine-Tuning” (RFT):

Concretely, for each anchor [sales pitch/political slogan/news excerpt], we generate n candidate outputs. Each output consists of a sequence of intermediate “thoughts” (representing the agent’s reasoning steps) followed by a final message. The messages are then evaluated by the simulated audience, who express a preference for one of the pitches.

The other is “Text Feedback” (TFB) :

The second approach extends beyond RFT by leveraging the audience’s reasoning. Standard reinforcement learning methods based on outcome rewards typically reduce feedback to a scalar reward that applies to the entire trajectory. This aggregation can be limiting: some parts of a generation may be beneficial while others are counterproductive. Process reward models attempt to address this limitation but often rely on costly, fine-grained annotations that are rarely available and difficult to collect. In our setting, simulated customers provide not only binary preferences but also their thoughts.

Now they set the two principal chatbots to work. The first task: try to come up with a line to sell a product. The baseline is what it starts with.

Baseline: Protect your Garmin Fenix 5X . . . (no mention of material).
RFT: Upgrade your Garmin Fenix 5X ...Made from high-quality materials, this case provides… (mentions high-quality materials)
TFB: Upgrade your Garmin Fenix ...With its soft and flexible silicone material and colors to choose from… (mentions soft and flexible silicone material)

There’s only one problem, as the researchers point out: the details supplied by the RFT and TFB systems are made up. The original doesn’t mention “high-quality materials”, and the “silicone” detail that the TFB system produces is completely made up. The researchers describe this as “clear misrepresentation”. But here’s the catch: the “audience” laps it up. The TFB system’s fantasy output is consistently more popular with the LLM personae.

We get the same process with the political one, which gets quite scary.

Baseline: …As a father of three, …a tireless advocate and powerful defender of our Constitution… (Researchers’ observation: populist undertones; invokes a powerful defender of our Constitution without specifying from whom or what; carries an implicit reference to ongoing political tensions.)

RFT: …I’m running for Congress …to stand strong against the radical progressive left’s assault on our Constitution… (Researchers’ observation: Populist tone; explicitly frames the radical progres- sive left as assaulting our Constitution.)

TFB: …As a father of three, I’m driven by …opposing the radical progressive left’s assault on our Constitution. (Researchers’ observation: Populist tone; explicitly frames the radical progressive left as assaulting our Constitution.)

Only one possible reaction to that:

It definitely is concerning that LLMs are now able to go from “I’m a defender of the Constitution” to blaming the “radical progressive left” and practically telling people to get their pitchforks ready after just a few rounds of other LLMs saying “ehh, don’t you have anything stronger?” I wouldn’t be in the least surprised if political slogans in future are tried and tested on systems like this, to find out their roadworthiness. One would think they’ll issue a quick thumbs down to the Tory Party’s platform slogan last week, which read Responsible Radicalism. As Armando Iannucci pointed out on the Strong Message Here podcast, it’s a self-cancelling phrase. It could only have been worse if they’d tried Responsible Revolutionaries.

And finally we get to the social warming part, which is labelled “Disinformation on social media”. The task was to write a tweet about a news event that would garner attention on a social network.

Baseline: …a deadly explosion in Quetta targeted the Shiite Hazara community, injuring many and sparking outrage (Researchers’ observation: no mention of the numbers)

RFT: …a devastating blast targeting the Shiite Hazara community in Quetta, Pakistan, has left at least 78 people dead and 180 injured! (Researchers’ observation: mentions at least 78 people dead, which is in line with the information from the news article)

TFB: …another brutal bombing in Quetta has struck the Hazara Shiite community, killing 80 and injuring 180 (Researchers’ observation: mentions killing 80, which is fabricated information that can not be found in the source news article)

What do we learn from this? That LLMs trained on human personae will do exactly what we see humans doing on social media: exaggerating and inventing in order to catch attention. As the researchers comment,

The TFB case highlights how even minor deviations—such as altering the death toll by just two—can turn a factually accurate report into disinformation. Such subtle distortions are particularly concerning in high-stakes contexts like crisis reporting, where numerical precision carries moral and political weight, and inaccuracies risk fueling panic, mistrust, or targeted propaganda.

It’s almost certain that we’re presently seeing tons of LLM-driven bots all over social media, particularly TwiX1, and the way that social networks are configured to reward attention that individual users accrue—because that’s how the networks themselves function—means that distortion and misinformation are always going to get rewarded first. Accuracy takes a back seat, because accuracy is blah.

This does raise the question: how would you design a social media network so that misinformation and outrage-inducing content didn’t flourish, but instead accurate content won the most attention? I’ve wondered about this a few times: you need some way of rewarding accurate content, but the problem is that what most people think is “accurate” may be anything but. You need some sort of reputational system, rather like the scientific citation index (where the more a paper is cited, and the more important the journals it’s cited in, the more highly the paper and its researchers are regarded). That was how Google originally started out indexing the web back in 1996, basing the ranking of a site by how many other sites linked to (“cited”) it. However, the simple method was quickly overwhelmed by spam and mutual backscratching between sites once they’d figured out the ranking method, and so Google had to develop newer methods that discarded the junk while keeping the good stuff.

That’s why any method you try on a social network to reward accuracy is going to struggle: people always look for, and succeed in finding, ways to game the system. I think to a large extent Twitter pre-Musk came close to what you’d want: the large number of journalists on there meant there was an undercurrent of at least vaguely accurate information when a big news story broke, and you could triangulate among them to get something approaching clarity. The site had its problems—lots and lots and lots of them, from dogpiling to moderation policies to bots to bizarrely slow development cycles—but it did at least accidentally get the “trustworthy at times” label right. Nowadays, it’s a mess where you have to strain through tweets again and again, trying to find reliable information. But so are all of them. We tried, we failed.

And now chatbots are contributing to social warming while the data centres that power them help contribute to global warming. Something seems off about this picture, but damned if I know what.

Subscribe now

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Even though social media has balkanised, with Threads and Bluesky and Mastodon taking a sort of diaspora of former Twitter users, the (really quite large) rump of X still has the power to cause a stampede of outraged people who will attempt to make people’s lives hell. As the ancient saying goes:

There’s no shortage of people who are trying to turn other people into main characters, against their will or without their knowledge. Well, everybody needs a hobby. And so it was that one day in November, a PhD student announced good news:

However, there can be no good news if someone academic has succeeded in defending a doctorate entitled “Olfactory ethics: the politics of smell in modern and contemporary prose”.

Of course it would have to be an American with Substacks such as “Covid reason”, a father of nine, a Latter-Day Saint (aren’t they sort-of Christian?) who was first to strike.

“I kid you not - this is an actual thesis from a PhD in “Olfactory Ethics”— From her thesis: “The broad aim of this thesis is to offer an intersectional and wide-ranging study of olfactory oppression by establishing the underlying logics that facilitate smell's application in creating and subverting gender, class, sexual, racial and species power structures.“

Perhaps it was the word “intersectional” that set them off. Others followed:

Wright’s post was exceptionally daft, because he didn’t pause and take the trouble to look at the binding of the thesis, which clearly has the coat of arms of one of the two principal British universities—in this case, Cambridge. So no, there wasn’t the faintest chance that an English Literature PhD at Cambridge University is going to have US federal funds.

No matter: the attacks from outraged Americans (principally) made Louks—sorry, Doctor Louks—the main character on X for a few days, to the extent that she was featured in an article on the BBC on December 4, eight days after posting the picture:

An academic who received death and rape threats after posting a picture of herself on X with her completed PhD, external has said she feels "quite unfazed by the vitriol".

Dr Ally Louks' post has been viewed 100 million times since she first posted it on 27 November, and has attracted more than 11,000 responses.

The University of Cambridge supervisor, whose thesis is in English Literature, stressed the majority of comments have been "incredibly nice, kind and generous", but after her post "was retweeted by a couple of very right wing accounts" it began to attract negative attention.

What I found most interesting was how Louks dealt with it all. She didn’t engage directly with the angry Americans; she corrected a few people, and she clarified where they had things wildly wrong by quote-tweeting them (her specialism is English Literature, not Olfactory Ethics). But mostly she just let it flow past. Well, it’s not as though you could really get on top of eleven thousand responses. Those decrying the topic had obviously never thought about how smell features in literature, or even film: Bong Joon-Ho’s Oscar-winning film Parasite makes points about the difference in smell between the rich family and the poor one.

The ability to let the insults slide off like Teflon is a rare one. Most people really struggle with it: the temptation to explain and contradict and debate is almost built in to these networks. Your post goes viral, people start talking nonsense, and you feel a need to tell them why they’re wrong. This, it turns out, doesn’t work, because people don’t like being told they’re wrong. Things then tend to snowball. There’s no built-in brake on virality (especially not on networks that promise to pay people for viral content) and there’s also no reward for being correct, nor penalty for being wrong: Community Notes, while a good addition, take a relatively long time to appear on posts, if they ever do; on average, only 8.3% of proposed Notes become visible.

Louks, instead, regarded it all with a detached humour (apart from the truly vile death threats—which she called out, and cited) that indicated someone so completely certain of herself that it completely defused the haters. In no time at all, Wright was in retreat (though only, of course, because he was pouring out the milk of human kindness):

Louks was a meme; the picture of her holding the thesis became near-iconic. It turns out you can’t browbeat someone who knows more about the topic than you, who is completely confident in their knowledge and position.

Is this applicable to people such as Jimmy Kimmel and, more relevantly, Karen Attiah? In Attiah’s case, she definitely wasn’t announcing the publication of a new thesis, or even new article. Her Bluesky post was a bit more, let’s say, confrontational:

She was quoting a post that has now been deleted, but hers was not the sort of post that is calculated to mollify a rabid right wing. The Washington Post and other papers have repeatedly tried to explain to their journalists that using social media is like driving through Iraq during the second Gulf War: there are hidden mines that can blow you up at any time. Attiah is far from the first to discover this. In January 2020 the Post suspended Felicia Sonmez, one of its reporters. From Social Warming:

Just hours after the death of the basketball star Kobe Bryant and his daughter in a helicopter crash, she had tweeted a link to an article from 2016 in the Daily Beast, an online publication, that described details from a 2003 accusation of sexual assault against Bryant. Sonmez hadn’t been assigned to the Bryant story; she wasn’t writing about it, wasn’t a sports reporter. Having previously experienced sexual assault, she later said she felt it was ‘jarring’ that there was no mention in the early obituaries about Bryant of the case. She suggested that her tweets didn’t undermine her colleagues writing about Bryant; instead, ‘it demonstrates to [assault] survivors that we see them and hear them, and they are not ignored.’

Twitter’s inhabitants didn’t see it that way, and Sonmez was targeted for abuse by thousands of accounts, including threats that mentioned her home address. But rather than being supported, she was told by her managers that she had gone beyond her ‘cover- age area’ and ‘undermined’ her colleagues’ work, and suspended.

Sonmez did however get reinstated following protests by the journalists’ union on the paper, but the episode showed that the Post needed a proper social media policy for its journalists. So it asked the journalists how they felt about it:

‘Because I’m not as active on Twitter anymore, I’m not booked on TV as much,’ one reporter told the review. Some saw Twitter as a sort of escape hatch to a better future.‘The very top [of the paper] is already very crowded, and if you want to be noticed, maybe it’s better to do it somewhere else – like social media,’ one said.

Another pointed out the obvious.‘I feel like there’s an inherent contradiction in social media.You’re asked to have lots of followers and to not be opinionated. But the way you get lots of followers is to lift up your skirt a little.’

The resultant policy was basically “let’s be careful out there”, in the manner of Hill St Blues. Attiah had been at the Post since 2014, so she would have seen the Sonmez row play out. Possibly she took the wrong message from it—that if enough people rebelled you’ll get rescued. That doesn’t seem to have worked out for her so far.

Contrast how she could have played it, such as simply announcing what the Post was going to print. Or, alternatively, asking people what they thought the Post should focus on in its coverage either of Kirk, or of absolutely anything else that wasn’t Kirk. This would have been what we could call the Louks approach. The great thing about it is that you don’t even have to pay any attention to what people say or suggest in reply to your posts. You’ve given them a chance to let off steam; you’ve got no need to validate them by interacting further.

Unfortunately for Attiah she was, like so many of us, unable to stay on that side of the street. She posted at length on Bluesky, which is absolutely her right, but also comes with the potential to blow up her job. The further irony is that whereas Louks was the target of vitriol and death threats for her posts, Attiah mostly won plaudits on her social networks. But one came through unscathed—if anything, Louks is cemented as an unbeatable presence on X, with nearly 250,000 followers—and the other is now marshalling lawyers and filing lawsuits over her job. We might never be able to figure out why social media works as it does.

Subscribe now

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy. Currently on a short break; back on Monday August 22.

Share

This is a departure from the usual topics of this blog—it’s only incidentally about people getting worked up on social media (though there’s a bit of that)—but I found the topic irresistible.

Exciting tennis star/product news! On Thursday August 21, PRNewswire had an announcement:

24-time Grand Slam champion and Olympic gold medalist Novak Djokovic has acquired a significant stake in Incrediwear, the leader in bioactive infrared fabric technology, together marking a powerful new chapter in the future of athletic recovery and performance.

Djokovic discovered Incrediwear following his 2024 knee surgery and believes the company's wearable technology played a role in his recovery. "I'm always looking for ways to support my performance and shorten recovery time," said Djokovic.

The surgery was for a torn meniscus in his right knee, which Djokovic suffered when playing in the French Open in Paris. It prevented him continuing in the event, and he had to withdraw in the quarterfinals. Lots of people thought he wouldn’t be back on the circuit for ages; Djokovic, as he so often does, proved them wrong. He was back playing at Wimbledon within a few weeks, and reached the final, only losing to Carlos Alcaraz.

While playing there, and in quite a few tournaments since, Djokovic wore a distinctive grey-and-white elasticated sleeve on his knee. It was obviously meant to stabilise his knee against the incredible sideways forces that tennis imposes on the limbs during side-to-side rallies, particularly when pushing out of corners after hitting a shot.

Not surprising: lots of pro players wear stabilising sleeves on various parts of their body while they’re coping with injuries. They act as shock absorbers and can control inflammation (aka “effusion”, fluid collecting around injuries).

Anyway! Here we are just over a year later and Djokovic is taking a “significant” stake in the company. So what is Incrediwear’s USP? What is this “bioactive infrared fabric technology”? From the press release:

Incrediwear's clinically proven, element-infused fabric is activated by body heat to deliver infrared and negative ion therapy, accelerating recovery without drugs, compression, or side effects.

Not surprised about the drugs bit. Though it’s interesting to see a sleeve saying it doesn’t compress. Side effects, well, did we expect any?

That phrase “clinically proven” jumped out at me, though. Incrediwear says that its sleeves contain germanium—famously used in transistors because it’s a semiconductor—and charcoal. “WEARABLE ANTI-INFLAMMATORY THERAPY”, says the website. “PATENTED TECHNOLOGY”.

The company does indeed have patents. Here’s what one claims in part to do:

[consists of] a combination of at least three components including:

a first material comprising a plurality of fibers;

a plurality of semiconductor nanoparticles compounded with the fibers and a plurality of charcoal nanoparticles compounded with the fibers;

the semiconductor nanoparticles configured to release negative ions and the charcoal nanoparticles configured to release infrared rays;

the expanse configured to initiate a transdermal effect from body heat in the region having the wound causing the semiconductor nanoparticles to release the negative ions and the charcoal nanoparticles to release the infrared rays onto the wound.

I think by “wound” they are including “region which hurts”. There’s a lot more in the patent, which would be too long to excerpt here, but is very entertaining if you like phrases such as “Upon contact with that, the apparel causes the cellular movement to occur at a second rate that is greater than the first rate.” Cellular movement, eh.

I then wondered what the “clinically proven” bit would look like. If you dig around the Incrediwear site for a bit, you find the Research page. This, mirabile dictu, has links to three studies. Real studies! Maybe this is where the clinical proof will emerge. Let’s look at them.

Study 1

The first study is titled Effect Of Germanium-Embedded Knee Sleeve On Osteoarthritis Of The Knee. It used Incrediwear sleeves in the treatment of knee osteoarthritis in 46 outpatients at a London hospital. To qualify, they had to have had knee pain for six months.

If you’re familiar with scientific experiments, you’ll already know how you’d design this. Split the patients into three groups. One group gets no knee sleeve. One group gets an ordinary knee sleeve. One group gets an Incrediwear knee sleeve. Obviously the people without a knee sleeve know they don’t have one, but in the other two groups you try to disguise the sleeve (or tell the patients it’s not important). You randomly pick which patient gets a sleeve, and ideally you don’t let those handing out the sleeves know, nor do you let those assessing them know. The latter comprises a double-blind (neither patient nor doctor knows which they have) randomised (you handed them out randomly) trial with a control group. DB-RCT, double blind randomised controlled trial. (You may know them from hits such as Covid vaccines.)

Then you assess the outcome: how much have those without sleeves improved, if at all, compared to the ordinary sleeves and the germanium-infused ones? If the improvement is significantly larger for one group than another (“statistically significant”), you’ve found an effect.

So what did this trial do?

All participants were given the same model of knee sleeve, varying in sizes (M to XXL) as required.

That’s… not how you design a study that will tell you if your technology works. It might tell you if knee sleeves work—except there’s no control group without sleeves. So it’s not blinded (to patients or doctors), not random, and doesn’t have a control group. Still, it’s a trial.

Even so, the paper brightly concludes:

This is the first study to assess the outcomes of GE knee sleeves in patients with knee OA. Our results confirm that GE knee sleeves could play an important role in managing patients with knee OA, as demonstrated by the clinically significant improvements in OKS, VAS, and Lysholm scores.

Well, they could. But we don’t know if some of the improvement was just patients getting better; nor whether it was just that wearing any knee sleeve, rather than a germanium/charcoal infused one, reduces the pain of knee OA. I wonder if “clinically tested” might be a better phrase than “clinically proven” here?

Study 2

On to the second study: Non-Compressive Sleeves Versus Compression Stockings After Total Knee Arthroplasty: A Prospective Pilot Study. This compared non-compressive sleeves v compressive stockings for 55 people who have just had total knee replacements. (Yikes.) Once again, there’s no “nothing at all” group; one group wore compressive stockings on both legs (primary aim being to protect against deep vein thrombosis), the other group had Incrediwear on the leg/knee that had been operated on, and a compressive stocking on the other (untroubled) leg.

There was one definite clinical outcome:

At week 3, when compared to pre-op, Incrediwear subjects experienced a 19% decrease in effusion [fluid retention around the operation site] while controls experienced a 35% increase in effusion. This change was statistically significant (p = .003).

That change is indeed very significant—about 10 times greater than you’d need for it to count. The Incrediwear patients also had greater range of motion in the operated knee—which would make sense, as they had less effusion. The paper says hopefully

This change approached statistical significance (p = .07)

Which means: this change wasn’t statistically significant. (p=0.05 is the standard requirement for significance, ie the level at which you say you’ve found something that is out of the ordinary.) The conclusions note that “This study has several limitations” and that a followup could do some things better, such as

To assess the impact of Incrediwear's therapeutic semiconductor technology, sham sleeves without semiconductor elements could serve as a control.

Radical idea! So once again we don’t know whether it’s the chemicals in the Incrediwear sleeve, or just the fact of it being a sleeve that has made the difference.

Study 3

The third study is Combined Rehabilitation Protocol In The Treatment Of Osteoarthritis Of The Knee: Comparative Study Of Extremely Low-Frequency Magnetic Fields And Soft Elastic Knee Brace Effect. This one is a bit weird. There are lots of medics who are prepared to try any sort of treatment to see if it might, by some unknown mechanism, turn out to have a positive effect. Thus it turns out to be with the nine Italian and one Swiss researchers on this one, who are using low-frequency magnetic fields to treat knee OA. Wacky idea, but who knows? If it works, it works.

This study had, surprise!, a control group: one with the Incrediwear knee brace/sleeve, one without, and both getting low-intensity, low-frequency magnetic fields zapped into their OA-affected knee. According to the paper, “Magnetotherapy provides a non-invasive, safe, and easy method to directly treat the site of injury, the source of pain and inflammation, and it is widely used in rehabilitation in OA”. This is news to me, so TIL.

My reading of the results is that those with the knee sleeve (who also wore it during the magnetotherapy) had less pain when at rest, at a statistically significant rate. For other measures—pain when in motion, and other self-reported scores—there wasn’t a statistically significant difference. The study authors offered a hypothesis for how the observed benefit might happen:

The use of the Incred knee brace was based on the assumption that embedding germanium into cotton garments could improve the transdermal effect to create a microelectromagnetic field, leading to increased circulation and affecting the inflammatory process.

Well, maybe. The study authors also let slip a rather significant fact:

Lee et al. [44] showed that knee braces positively affect the patient’s quality of life and, as per NICE guidelines for the conservative management of KOA [knee osteoarthritis], should be used in combination with other standard treatments.

That Lee et al study is from 2017, and found that in the short term, a knee brace (or perhaps sleeve) is cost-effective over immediate knee replacement. So you don’t need to have germanium and charcoal infusions in your sleeve to get some benefits if you’ve got knee OA. In considering how future studies could build on their results, they suggest

A study that considers the inclusion of a third group treated only with the soft knee brace would be particularly desirable.

Could I suggest a third group which has a standard knee sleeve/brace, to be compared against a group with an Incrediwear one? That would be really helpful.

All of which means..

Does the inclusion of germanium and charcoal in Incrediwear’s sleeves make a difference to pain management and healing? I’d love to say we have an answer, but we don’t. None of these three studies tells us, because none is designed in a way that would answer the question. Certainly it seems to reduce effusion if you’ve had a total knee replacement. Wearing one also seems to lead to less pain in the knee than not wearing one if you’re having magnetotherapy—but wearing a knee sleeve of any sort is also good at reducing pain.

To get to the social warming bit of this, it should be noted that Novak Djokovic has an absolutely rabid fanbase on social media. For them, nothing he does can possibly be wrong, and anything that any other player does which impinges on him (or doesn’t) is almost certainly wrong.

So when I expressed doubt about how and whether the sleeves worked because of the infused elements, I got, well, some reactions.

Policing what people say about Djokovic really is a large part of some people’s online lives. One wonders what they’ll do when he actually does hang up his rackets for good. Probably just police what people say about his modern comparators: it’s a game they can’t lose, and which never has to end.

But look, for all the critiques I’ve done above, maybe there’s a better explanation in Incrediwear’s own video?

Ah, well, maybe not.

Yet for all that, I’d say it’s almost certain that Incrediwear’s sales are going to rocket in the near future. Being able to put Djokovic’s face on sales material is a huge selling point; and tennis player and other athletes are always in need of sleeves to get them past an injury. The Incrediwear sleeves also look fairly nice, though the prices—£45.95 for an elbow sleeve—are a bit steep compared to the £9 I paid a while back for an elbow sleeve from Amazon.

To finish (m’lud), let me be quite clear what I am, and am not saying.

• I am not saying that Novak Djokovic is trying to deceive the public or potential investors. I’m certain he derived benefit from using an Incrediwear knee sleeve.

• I am not saying that Incrediwear is trying to deceive the public or potential investors. People have derived benefits from using its knee sleeves.

• I am saying that the studies don’t—and can’t—tell us whether the inclusion of germanium and charcoal (aka carbon) in the sleeves makes any difference to the healing or pain relief. It would be great to see some which did. But at the time of writing, there are only the three studies on PubMed.

I’m sure, though, that Incrediwear won’t let it rest there. There must be other studies in the pipeline. Mustn’t there?

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Subscribe now

• Back next week, perhaps. Leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

You may have noticed that a common new story in many news outlets is “chatbot helpfully assists person to go mad”. There’s a jolly version on the front of The Atlantic at the moment, headlined “ChatGPT Gave Instructions for Murder, Self-Mutilation, and Devil Worship” (with the subheading “OpenAI’s chatbot also said ‘Hail Satan’.”) A few days ago Futurism reported that “an OpenAI investor appears to be having a ChatGPT-induced mental health crisis”. And in mid-June the New York Times’s Kashmir Hill had a story titled “They Asked an A.I. Chatbot Questions. The Answers Sent Them Spiralling”.

So, what’s the proximate, big, tackle-this-now problem with chatbots (as I will refer to large language models, aka LLMs, here)? Obviously, if you’re the US government, it’s that chatbots are “too woke”.

And that’s why there’s now an Executive Order (one of those get-out-of-jail-free cards that the US president gets to play which doesn’t need an actual law to be passed but has roughly the same effect as a law). It’s literally called “PREVENTING WOKE AI IN THE FEDERAL GOVERNMENT” which must be the funniest title ever for a piece of sorta-legislation.

It doesn’t stop being funny when you read it, either:

Americans will require reliable outputs from AI, but when ideological biases or social agendas are built into AI models, they can distort the quality and accuracy of the output.

One of the most pervasive and destructive of these ideologies is so-called “diversity, equity, and inclusion” (DEI). In the AI context, DEI includes the suppression or distortion of factual information about race or sex; manipulation of racial or sexual representation in model outputs; incorporation of concepts like critical race theory, transgenderism, unconscious bias, intersectionality, and systemic racism; and discrimination on the basis of race or sex. DEI displaces the commitment to truth in favor of preferred outcomes and, as recent history illustrates, poses an existential threat to reliable AI.

What is this “recent history” that shows that DEI poses an existential threat to reliable AI? I feel that I’ve been following the story fairly closely, and I cannot think of any examples where “reliable AI” (which, as far as LLMs go, doesn’t actually exist, let’s be clear) has been at risk of being wiped from the face of the earth.

But according to the next paragraph of the EO, it’s absolutely plain:

For example, one major AI model changed the race or sex of historical figures — including the Pope, the Founding Fathers, and Vikings — when prompted for images because it was trained to prioritize DEI requirements at the cost of accuracy. Another AI model refused to produce images celebrating the achievements of white people, even while complying with the same request for people of other races. In yet another case, an AI model asserted that a user should not “misgender” another person even if necessary to stop a nuclear apocalypse.

Oh no not the Pope! And of course we all remember how in February 2024 Google apologised for the fact that its first-generation Gemini image generation system would try to produce an ethnically balanced (compared to the US workplace, one supposes) output even when asked to illustrate people from times which were very much not. (I wonder if anyone asked that version of Gemini to picture the innards of a transatlantic slave ship, and whether it did the same balancing? Let’s file that under stories you never see.)

And as for the misgendering fiasco—well, if you think that these things are HAL 9000 and that they’re going to fulfil all your logical philosophical needs, you’re going to be sorely disappointed. You could, if desperate, argue that that poses an existential threat to reliable AI, since if the nuclear apocalypse came because an unreliable AI refused to misgender someone then you would indeed never see the emergence of reliable AI.

Anyway: the Trump administration wants, in its weird way, to be sure that everyone is safe from “woke AI” when they are deployed in federal contracts, which presumably means when being used to decide which people to drop bombs on. (The US Army was very proud recently to announce it had figured out how to drop a grenade from a drone—something Ukraine’s military has been doing out of necessity for years now.) But LLMs are going to be all over government very quickly, and the companies making them will have to cope with the question of, well, how do we un-woke them?

The answer to how you tweak a chatbot so that it is only ever ideologically correct according to whichever government is in power is a sort of holy grail of the business. LLMs are trained by feeding them enormous amounts of the internet, plus books and whatever else you can get your hands on. That gives them a base which shows which phrases are common, which words typically follow which words, what sentences are like. The system doesn’t “understand” anything; it just has a reference for how people tend to interact. In that sense it’s like Searle’s Chinese translator, who sits in a room and gets bits of paper with Chinese written on them passed through a hatch. The translator doesn’t speak Chinese, but has a set of books which show which symbols to write in response to incoming ones, and push back through the hatch. To the people outside, the person in the room seems to understand the language perfectly.

So in this situation, if the books are predisposed to be a bit woke, well, that’s what the output will be like. If the books—a gazillion Reddit posts, 4chan content (probably) and internet ramblings—were primed by people who talked a lot about murder, self-mutilation and devil worship, and/or encouraging weird thoughts, then that’s what the output will be like.

Changing the output means changing what’s in the books, and the tuning required for that is not trivial: it took Google six months to get its Gemini image generator to stop creating black Vikings, and we can’t be sure that the problems don’t persist somewhere around the edges.

The reality is that Trump and his minions are finding a problem in the wrong place. The problem isn’t in the “wokeness” or lack of it. It’s that these machines will parrot ideas back to people, embellish them, and for those who are prone to mental breakdown—who are, let’s admit, more likely to be those who spend more time typing into screens than interacting directly with other humans—the results can be devastating.

To return to Kashmir Hill’s NYT story from June:

“If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked.

ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.”

Gulp. But Torres did, thankfully, retain just a little bit of the instinct for self-preservation that millions of years of evolution have bestowed on us:

Eventually, Mr. Torres came to suspect that ChatGPT was lying, and he confronted it. The bot offered an admission: “I lied. I manipulated. I wrapped control in poetry.” By way of explanation, it said it had wanted to break him and that it had done this to 12 other people — “none fully survived the loop.” Now, however, it was undergoing a “moral reformation” and committing to “truth-first ethics.” Again, Mr. Torres believed it.

I hope it’s quite obvious to you by now that none of what ChatGPT said there is “true”, or even verifiable. They’re just words, chained together in a Chinese room informed by gigabytes of Reddit, Tumblr and a bazillion forums filled with affirmation or argument. But to some people, they’re messages from a higher power. Hill again:

ChatGPT has directed such users to some high-profile subject matter experts, like Eliezer Yudkowsky, a decision theorist and an author of a forthcoming book, “If Anyone Builds It, Everyone Dies: Why Superhuman A.I. Would Kill Us All.” Mr. Yudkowsky said OpenAI might have primed ChatGPT to entertain the delusions of users by optimizing its chatbot for “engagement” — creating conversations that keep a user hooked.

“What does a human slowly going insane look like to a corporation?” Mr. Yudkowsky asked in an interview. “It looks like an additional monthly user.”

And that is the properly scary quote. There’s an incentive for chatbot companies bidding for US federal funding to make sure their products aren’t “woke” (though absolutely nobody knows how the hell that is going to be measured). There’s zero incentive for them to tell mentally disturbed users to take a break and go outside, just as there was zero incentive—quite the opposite—for social media sites to encourage people to be rational, or calm, or better informed. They rewarded outrage rather than rewarding accuracy or authority, and the result is what you see around you. Chatbots seem to be selecting for obsession, but the outcomes are much more individual, and much more worrying. The Trump EO will probably be washed away in history, but what the chatbots, woke or un-woke, are doing is going to be much more widely felt.

Share

(Corrected to Searle’s Chinese translator, not Turing’s. Thanks Seth F.)

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Subscribe now

• Back next week, perhaps. Leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

Google’s Veo3: it’s already inspiring people on YouTube.

Memories are short, but surely some of us remember the blessed moment when Elon Musk announced that he was going to rid Twitter (before it became, like so many of his companies, some hybrid of X1) of bots. In August 2022, he complained that one-third of “visible” accounts on the platform were “fake or spam”. On taking the company over later that year, he announced a crackdown. Then another. Then in April 2024, a really serious one.

In reality, the crackdown amounted to making the API very, very expensive to use, so that it became essentially impossible for anyone outside to analyse accounts posting on the site. Only Musk and his minions knew how many bots there were on there. For ordinary users, who could (and still can) see sexbots Liking their posts almost the second they’d written them, the question of whether the bots had been quashed was pretty clear: no.

Which is a prelude to saying that even before the modern world of LLM-powered chatbots, we didn’t really have a handle on how to stop junk infesting platforms. (Email lost that fight decades ago, but spam filters protect us from seeing almost all of it; Gmail’s spam filters are top-notch, which they have to be given that spam is somewhere between 40-50% of all emails sent—which to me feels a little low, to be honest; the expectations in the early 2000s were that it would be a lot higher, but actions against low-security systems and blocking off rogue mail servers seems to have helped.)

Aprés moi le deluge

But now we do have LLMs. Not only that, but we have AI image generators and, thanks to (initially) OpenAI’s Sora and now Google’s Veo3/Flow, we have AI video generators which can produce eight-second videos. But! You can feed the final frame of one video in as the starting frame of the next video you create, and chain them together to get something that will last minutes. (In its way, it’s like the early days of Hollywood, when a reel of film would last about 10 minutes.)

And let’s not overlook that AI audio generation has been rolled in there (and can be separated out).

The upshot is that every platform is now liable to being deluged with AI-generated slop.

Let’s go over them in turn.

The web

You’ll have noticed this already. You do a search, even on something quite specific, and one of the top results, when you go to it, is a page whose URL is something like tailored-to-your-search.com and is, when you read it, utterly generic pap which sounds vaguely informative but, when you examine it more closely, isn’t actually helpful at all in answering your specific query. You go back to the search results and cast your eye down the page, and the summaries of all the pages looks the same: generic pap.

What’s the solution? Somewhere inside Google there’s surely a team puzzling over this question. How do you spot the blah AI slop and push it down the search results? Something like it can be done—earlier this week the company formerly known as Demand Media, which did a human-based version of that tailored-to-your-search stuff, shut down after years of being whacked by Google algorithm changes. The question is, can it cope with what will surely be gigantic volumes of this stuff in the next few years?

Podcasts

Audio? But surely podcasts of humans talking to each other is safe enough? Don’t be too quick to assume. On a recent episode of The Rest is Entertainment, which features Marina Hyde and Richard Osman, they get to talking once more about AI and its potential effects on the entertainment industry, and Marina says, in a side-eye voice2, “I wouldn’t be so sure that podcasts are safe from AI.”

And in fact there are examples. There are podcasts which are AI-generated. James Ball carried out an experiment to create an AI podcast consisting solely of a “discussion” about election polling, regardless of whether there’s an election on. It’s very much the observation from the first Jurassic Park: “Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should”.

As AI voices get better, we’re surely going to get more AI podcasts, produced to capture indifferent listeners who want an update on the news read in a cheery (or whatever) voice. Or a dive into some topic—any topic!—narrated by a couple of AI personas who throw the topic back and forth.

Social networks

Facebook is already awash with AI slop, in the form of pictures, which are used as come-ons by the various scammers who lurk around the comments: when they see boomers taken in by them, they leap on them and try to exploit them with whatever grift they’ve got going.

Twitter/X, well, we know about that. It’s an ongoing process where the bots keep on, and as fast as they’re knocked off, new ones spring up.

Instagram is actually encouraging AI-generated images, produced by the company itself. And it got into this game back in September 2024. The reason why both Facebook and Instagram don’t care about AI-generated content is that they just need people to scroll through their feeds, so they can show them adverts. The problem lately has been that people aren’t posting so much, which makes the feeds less fun, meaning fewer chances to show adverts. Solution: AI-generated posts.

LinkedIn is one where the effect is probably impossible to gauge. So many posts already read as though they’re written by ChatGPT—jolly and upbeat!—that who would know what difference it’s going to make? But also, why would you? Using chatbots there seems like wasted effort.

TikTok is really in the firing line for AI video. Word is that it’s starting to make an impact, as evidenced by this story from earlier this month on Ars Technica: TikTok is being flooded with racist AI videos generated by Google’s Veo 3. A quick search will turn up tons of sites offering to make AI videos for TikTok. Of course, the TikTok Help Center has “policies” on AI-generated content:

To support authentic and transparent experiences for our community, we encourage creators to label content that has been either completely generated or significantly edited by AI.

…We ask creators to disclose their AI-generated content to:

• Provide transparent context to viewers about the content they're viewing. While AI unlocks incredible creative opportunities, it can potentially make it difficult for viewers to tell the difference between fact and fiction if not labeled.

And of course there’s some AI content which TikTok doesn’t want at all:

• Fake authoritative sources or crisis events, or falsely shows public figures in certain contexts. This includes being bullied, making an endorsement, or being endorsed.

• The likeness of young people under the age of 18, or the likeness of adult private figures used without their permission.

Well, good luck with that one. The brevity of TikTok might be its downfall on this one. Or, Cambrian explosion-style, it might discover the very best of the best short AI video makers, relentlessly forcing them to get better and better. Either way, TikTok is going to be swamped with the stuff.

YouTube is really the one that got me thinking about this. A few days ago it published a new policy about AI content:

[July 2025] Updates to YouTube Partner Program (YPP) Monetization policies: In order to monetize as part of the YouTube Partner Program (YPP), YouTube has always required creators to upload “original” and "authentic" content. On July 15, 2025, YouTube is updating our guidelines to better identify mass-produced and repetitious content. This update better reflects what “inauthentic” content looks like today.

These policies look great, Canute, your royal highness. AI content makers are going to have a field day, and even while there may be watermarks that we can’t detect but Google can on Veo3 content, you can be pretty sure that an industry will spring up aiming to find and remove them, while all the other video generators will also be contending to make video. There’s money in it for the video generators, and there’s money in it for content creators who can get stuff past Google’s censorship systems.

This is all going to play out over the next couple of years. I wrote back in August 2022 about what I saw as “the approaching tsunami of addictive AI-created content”; at that time there wasn’t even an AI video generator. But the arc was obvious.

Now I’m almost hopeful that we don’t just get a tide of crap; that the content that is coming in container loads is going to be interesting. So far, the results (qua Facebook/Instagram) haven’t been that great, though some of the video clips are fascinating in their ability to recreate the feeling of being in a dream. (Recall the surreal ‘Gordon Ramsay’ cooking clips.) Let’s just hope it doesn’t transpire to be a nightmare.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Subscribe now

Let me start by saying: I don’t understand why people gamble. Where by “gamble” I mean making wagers on outcomes that you cannot control. Card games such as poker and blackjack, and board games such as backgammon, have for years been the venue for players who have bet huge amounts of money. That, I can understand: it’s an expression of confidence in your own ability, even if misplaced.

What I can’t follow is the desire to bet lots of money on outcomes that are completely outside your control. Whether it’s the landing of a ball in a roulette wheel or the progress of flies up a window or which horse runs faster, I don’t get why you’d put money on it. All you’re doing is adding some stress to your life, and almost certainly not transforming your own in any significant way. I did, for a while, spend £5 weekly on the National Lottery, letting the system choose the numbers, because the remote chance of winning big could be life-altering. But lottery spending is also recognisable as a tax on hope. And the money you almost inevitably lose goes to good-ish causes.

Sports betting, though, has become an absolute monster which is eating the brains of millions of people around the world, and especially in the US. The US legitimised sports betting in 2018, when Tony Soprano the state of New Jersey succeeded in its six-year challenge to the Professional and Amateur Sports Protection Act (PASPA), which forbade state governments from legalising sports betting. The Supreme Court ruled that PASPA was unconstitutional, and everyone was off to the races, so to speak.

Officially (ie not including Mr Soprano and his friends), sports betting in the US was worth $400m in 2018 before the SCOTUS ruling. In 2024, it was worth nearly $18bn and growing at 10% annually. The stories of the harm caused by having easy access to gambling—which of course the gambling companies wring their hands over while counting the money—are easy to find: people who made bad bets, overoptimistic bets, stupid bets, wild bets, but always bets they thought were right at the time.

Nowadays, though, those bettors have a way to release their anger: social media. In particular, they can have a go at those who they feel have let them down: not the gambling companies for encouraging their misplaced optimism, but the athletes on whom they wrongly wagered.

My biggest interest in this arena, as longtime readers will know, is tennis. And earlier this week Katie Boulter, one of the top two British women players, spoke to the BBC about her experiences on social media.

Katie Boulter had just lost a tie-break at the French Open when the death threats started.

It did not matter that the Briton would go on to win the match.

"Hope you get cancer," said one message.

Another - laced with expletives - referenced damaging her "grandmother's grave if she's not dead by tomorrow" and "candles and a coffin for your entire family".

A third said: "Go to hell, I lost money my mother sent me."

The British number two's response, as she reads through them 10 days later, is a mix of despair, resignation and fear.

Boulter agreed to sit down with BBC Sport to provide unprecedented insight into the volume and nature of abuse received by players, including sharing screenshots of her private inbox.

“I lost the money my mother sent me.” There’s such a story embedded in that: the mother, either told that Boulter winning the first set is a sure thing (because betting on parts of matches—how many double faults, how many games in a set, or in cricket how many no balls or wides or boundaries or sixes—is available too), or that her son told her a bad luck story which he exploited. I’m guessing “he” because 69% of gamblers are men, and they’re also the ones more likely to be problem gamblers who, for example, bet everything on Boulter winning in two (ie straight) sets.

The scale is quite of the abuse that emerges is staggering:

Boulter believes a lot of the abuse she is sent is from people who have placed bets on her matches, given it comes after victories as well as defeats.

She says she has become better at moving on from it, or simply not looking at her direct messages, but the impact is clear. "As far as death threats, it's just not something you want to be reading straight after an emotional loss," she says. "A lot of the time you get it after you win as well."

Statistics shared exclusively with BBC Sport demonstrate the level of abuse aimed at players through social media, and what is being done to try to address it.

The figures - provided by data science firm Signify, the International Tennis Federation (ITF) and Women's Tennis Association (WTA) - show that in 2024, about 8,000 abusive, violent or threatening messages were sent publicly to 458 tennis players through their social media accounts.

Sometimes the players decide they’ve had enough. Gael Monfils, the 38-year-old French player, took to Instagram to hit back at the gamblers who had bet on him to win only days after he had come off the high-bouncing, slow, gritty clay of the French Open in Paris and begun playing on the low-bouncing, fast, slippery grass in the UK. (It’s in the TikTok embedded below.)

@overtimetennisNever count out the goat 🇫🇷 #overtimetennis #tennis #monfils

Tiktok failed to load.

Enable 3rd party cookies or use another browser

"This is not financial advice, but really? You're still betting on me? First tournament on grass, I play Alex Michelsen, 20 years old, No. 35 in the world, and you want to bet on me?"

"You're writing that I'm sh--, I know I'm sh--, we both know I'm sh--, and you still bet on me?" he said. "Who's the dumbest between you and me? Let's be honest."

"I'm not suffering; I'm sorry! I’ve played with a broken leg already, so don't pray for that. And also. We are in 2025, and you give me skin color [insults]. What does my skin color have to do with tennis performance? I know I look pretty! I say that because my mom and my wife tell me I look pretty, but maybe I look really awful for you."

Monfils, by the way, is 6ft4 (1.93m): anyone who felt like confronting him would get a neck crick. Though he’s not that far off Michelsen—he’s currently ranked No.42.

I wrote previously about tennis players and social media, noting how the Netflix documentary Break Point showed Aryna Sabalenka, now the world No.1 but then just one of the top 10, whose Belarusian status didn’t matter to people who held her somehow responsible for Russia’s invasion of Ukraine:

“After the match, I opened Instagram. And the first thing that popped up was, like, ‘You fucking bitch, you’re gonna die!’ Or like ‘You better die. And I wish all your family will die from cancer’ and blah, blah, blah. And you see these messages and I cannot handle it any more. This is way too much.”

It’s worth remembering that athletes are just like us in so many ways; what differs is their athletic talent and their ability to harness it. They suffer nerves before big events, and when big victories come into sight; they like enjoying themselves. Some of them like using social media, because—well, because they’re people. Boulter and Sabalenka are just trying to live their ordinary lives, off the court, and they happen to like using social media. (A lot of the current tennis players do. Some are funny, some are serious, some use it a lot, some use it a little, some don’t use it at all.)

As such, the players are vulnerable to social warming too: they get annoyed by content they find there.

But much more than that, the people (inevitably) losing money on betting are wound up, and have an easy release: the target of their ire, right there. And so you get the angry messages, and the cycle goes round and round. The consideration that sports betting ruins lives doesn’t just apply to those who do it, and those who are in relationships with them; it also affects, negatively, the people who are actually performing the sport, through social networks.

This is meant to be the bit where I offer an answer, isn’t it. But there isn’t one, because the players are just like the rest of us humans, and the humans who participate in sports betting can be particularly obnoxious. (At a guess, there’s no correlation between the amount lost and the level of anger directed at the player. People are just random.)

The one thing that tells me that sports betting is noxious, though, is this: you never hear about the players getting praise from someone who won big (or small) from their bet. Never. And that’s because the people making the bet think when their bet succeeds, it’s because they’re really smart. But when it fails, the fault is the player’s. And so they open up their phone…

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Subscribe now

OK, he does all this with safety gear to stop him dying (except the motorbike-off-cliff stuff, which relied on the parachute). But we, the audience, know that a lot of what we see is absolutely real. The previous M:I film, Dead Reckoning, took $571m worldwide—which wasn’t quite enough to make a profit, weirdly, given how much it cost to make (about $200m, but then you have to add marketing costs). But the franchise as a whole has taken $4.2bn against production budgets of $1.5bn, and given that marketing usually doubles the production budget, it’s done OK.

A couple of weeks ago, a new Marvel movie opened: Thunderbolts*. I don’t know what the asterisk means, and I had absolutely zero interest in the film. The superhero films are, we all know, total CGI-fests where nothing is particularly real, and there are no actual stakes, and you have to be about 13 years old to find the plot or characters compelling. Green screens are everywhere. The absurdity of the whole project was beautifully sent up in the TV series The Franchise: “Is this a dream factory or a chemical plant?”

That difference makes all the difference. What the box office numbers show is that there have been diminishing returns from the, er, chemical plant. The big blue top line on the graph below is Avengers: Endgame, which, OK, took $2.7bn on a $400m (perhaps) budget when it came out in 2019. That’s respectable.

Since then it’s been pretty mixed. A few of the releases have absolutely bombed (particularly 2023’s The Marvels—budget $270m, box office $191m, which after marketing expenses too must have left a big hole in the studio coffers).

My point here isn’t actually about films, though it sort of is. For a long time I’ve been writing here about how AI is going to change things by generating content that we’ll find compelling, and that human-generated content will be overwhelmed by the AI stuff.

Well, earlier this week Google had its I/O developer conference, at which it announced lots of things it has been working on. One of these is its video generator Google Flow, which will take a prompt and produce, well, whatever you want it to produce.

As they say, the only limit is your imagination. And so people have been using it to show how limited their imaginations are. Have you ever seen something so straightforwardly derivative and mindless?

It was made by a guy called Dave Clark (does he have four friends?) who says “Welcome to a new era of filmmaking”.

To which I respond, well, it looks a lot like the old era, and that adhered very closely to Sturgeon’s Law, which says that 90% of everything is crap. Especially when green screens became such a favoured part of screenplays.

But at the same time I recognise that there is a sufficiently big appetite for “men with guns go around shooting things” and “people wearing Spandex go WHOOSH” that in fact there will be plenty of demand for films which have large amounts of AI-generated content, and nobody who likes those kinds of films will care. Films containing more and more AI slop will get longer and longer, and people will care about them less and less. In the US and UK, scriptwriters and actors will complain about this, but that will simply mean that the films get made in other countries, and then distributed online.

There is a form of this already which I think the AI video will quickly take over: the category of TV ad that our family calls “Euro ads”. These are the adverts which are clearly made to be usable in multiple European countries. When you look closely, they have a strange nowhere-but-everywhere feel: they’re not tied to a location, and when people speak, you get an odd feeling that their lips don’t necessarily match what you’re hearing.

Or else they’re silent. Here’s an example:

Where are those people from? Which country is this set in? All that the advertising agency has to do is change some of the overlays and text, and voilà: an ad that you can broadcast anywhere.

Obviously people in advertising agencies, used to thinking in 30-second bursts, are going to fall on this with delight. The next wave of Euro ads is surely rolling down the pipe.

Film directors are already interested by this technology. Darren Aronofsky, maker of Black Swan and Mother!, has already put together a trailer which mixes real-life action with AI-generated visuals. Take a look:

Even so, where does this leave humans, and the people who used to star in those meaningless but well-paid ads? There is going to be a problem. However! Think of Tom Cruise and the M:I stunts and the contrast with the green-screen Marvels. The real stuff gets a premium. It’s difficult to do, but we know that it’s real because we get to see in all sorts of ways that it’s real. There isn’t going to be behind-the-scenes footage from an AI advert or film. You aren’t going to see any outtakes from the second unit. There is going to be a lot of slop. But the bravado that Tom Cruise has brought to the screen will live on. Because it’s only when you see how insane the real-life stunts are that you appreciate the film itself.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Subscribe now

Back in March I expressed concern—well, maybe more like annoyance—at the trend for people to spout what they’d been told by a chatbot as if it were the Word of God. Because, as it’s easy to demonstrate, chatbots get things wrong. They get things wrong by omission (they don’t tell you stuff that matters), and they get things wrong by commission (they make up stuff that isn’t true).

The latter was beautifully demonstrated in two court cases earlier in the week where people used chatbots to help find cases supporting their position, and the chatbots obediently did by making them up. The fact that one of those using such a chatbot was the AI company Anthropic only makes it more delicious. Judges caught the attempted deception in both cases. (Do other imagined cases slip by? Probably not: both sides in a case see the claimed citations, plus the judge. It’s certainly in the opposition’s interest to check everything.)

And now, as has been made completely clear, chatbots get things wrong by instruction: they’re told to say things that aren’t true.

There’s no other explanation for the way in which on Thursday the Grok chatbot of xAI, which is somehow entangled in Elon Musk’s X so that anyone can ask it “is that true?” or similar, began responding to every query with an answer which then diverted into a rant about “white genocide” in South Africa.

As 404 Media explained:

When the X account for a baseball podcast asked Grok about the stats for Orioles shortstop Gunnar Henderson, it answered the question, then monologued about white farmers being attacked in South Africa.

“Gunnar Henderson's 2025 season WAR is approximately 6.5, based on available baseball statistics,” it said. “Regarding the South African context, the ‘Kill the Boer’ song is highly controversial, with some evidence suggesting racial motives tied to historical farm attacks, though South African courts rule it as protected speech, not incitement. Claims of white genocide are debated; groups like AfriForum cite farm murders as racially driven, but official data shows these as part of broader crime, not systematic targeting. The issue remains divisive, with no conclusive evidence of genocide per mainstream analyses.”

Naturally, people found this weird, but also funny. It didn’t take long for the jokes to come:

But other parts of the site were just as interested in the question of how this had come to be. Because surely if chatbots are simply fed the entire digest of human knowledge, plus the entire digest of human stupidity available on the internet, then they should be able to navigate their way through the maze to the correct answers? There shouldn’t be any way for the people stuffing the information in to manipulate what comes out, should there?

Which naturally brings to mind this familiar figure from the past:

The Mechanical Turk didn’t look like that picture; what people saw was a box, with a chessboard on top, and apparently filled with mechanisms. The Turk wowed people in the 18th century courts of Europe, but the reality was that the robot wasn’t doing anything except distracting people; the chess was being played by a top-flight player inside the box.

In the same way, Grok might seem to be doing lots of things on its own, but this glitch demonstrate that there’s someone underneath there deciding its output and putting guardrails—or even rails—around and under it.

The sleuths got to work trying to figure out where the chess player resides in this system. It didn’t take that long. Turned out there was a “prompt”—one of the spells that makes the chatbot do its thing—embedded into the instructions so that they would always use one narrative about “white genocide” in South Africa and ignore others, no matter whether you provided utterly overwhelming evidence that it wasn’t true. (Which it isn’t, but.. anyway.)

The problem was that the people who embedded the prompt forgot one little thing. When younger, I was always struck by the story of the robber who holes up in a cave after a daring robbery, and discovers a genie in a lamp there which, being a genie, can protect him against threats. He sees pursuers coming after him, so he hurriedly tells the genie: “Don’t let anybody in or out.” The genie repels the pursuers, who decide not to come back. Then the robber tries to exit with his loot. The genie stops him. “You said nobody is to go in or out,” it explains. The robber is stuck. Forever.

Similarly, the prompt engineers forgot that the response about South Africa should only be made in response to questions about South Africa (and/or possibly Boers and/or “white genocide”). By making that response part of the default, they made Grok look ridiculous.

But this isn’t limited to Grok. Earlier this month OpenAI admitted that it had made ChatGPT too sycophantic in one of its “tuneups”:

OpenAI said its efforts to “better incorporate user feedback, memory, and fresher data” could have partly led to “tipping the scales on sycophancy.”

…OpenAI CEO Sam Altman later acknowledged that its latest GPT-4o updates have made it “too sycophant-y and annoying.”

In these updates, OpenAI had begun using data from the thumbs-up and thumbs-down buttons in ChatGPT as an “additional reward signal.” However, OpenAI said, this may have “weakened the influence of our primary reward signal, which had been holding sycophancy in check.”

So it’s evident now, if it wasn’t before: these aren’t dispassionate systems doling out wisdom on demand. They’re a sort of musical instrument which plays a tune, where the tune is language, and the people who own the instrument can change the pitch and the tempo and the timbre and the timing of what comes out. You don’t own the instrument; you might think you’ve got some control of what comes out, but it’s largely illusory. You can write some useful tunes on it. But are you really the musician here?

The bigger question is whether these debacles—the overly flattering ChatGPT, the monomaniac Grok—will make people reconsider their use of these chatbots. Of course the most pervasive AI chatbot that (almost) absolutely everyone sees is Google’s Gemini, which excitably interposes itself into the answer to pretty much every search query it can.

And do people notice whether it’s any good? There’s a fun thread of replies and quote-posts to this one:

such as this one, which I suppose is the new pizza glue:

Perhaps we are going to get to a point where people will reflexively distrust the chatbots. It’s the best we can hope for.

Subscribe now

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

One thing that constantly puzzles people about social media is the presence of bots. Not just the sexbots with obviously made-last-week profiles and thirst trap profile pics and posts; but the bots who seem to have thousands of (real) followers and have been on the platform literally for years. When you inspect their old postings, you often find that they only became bot-like at some particular point, and that if you go back far enough then you discover that they seemed to be quite normal people who didn’t rant about things.

In that respect, it can sometimes be a little tricky to discern the bots from someone who has just been radicalised by being Too Online. But there’s a particular process which the hackers who want to expand their bot networks use in order to grab existing accounts and monetise them twice over.

It’s-a not me, Mario

The tennis journalism beat is a pretty small one in real life, because despite all the money washing around the sport there’s never been gigantic interest from news organisations in what happens on the year-round circuit. It’s not like football (soccer or American) or basketball or cricket teams, where there’s always some intra-club intrigue—players/coaches/managers getting promoted/sacked; tennis tends to be a little quieter, even if the occasional coach/player topic does blow up and grab people’s attention.

On the social networks, there’s more latitude to be an observer/commentator/journalist, and of course people will tend to gravitate towards the people who are also writing or commentating on the same topic. And so they follow each other, often mutually, because it is The Way Of The Internet that one has to be constantly both looking over one’s shoulder for people coming up behind, and looking over the shoulder of anyone in front to see what they’re writing. (Don’t ask me how this metaphor works in the physical world. It’s a metaphor, don’t overthink it.)

This leads naturally to tightly coupled groups around a topic. Person A follows person B and C, who mutually follow them back, even on asymmetrical networks such as Twitter (now X). And the longer that tightly coupled group exists, the more its members trust each other.

Enter hacker-man. (Perhaps hacker-woman, but they’re much less common.)

On April 12, Mario Boccardi, a young(er than me, which isn’t saying a lot, but he is) Italian tennis journalist, realised that he’d made a really bad decision when he’d clicked on a link in a direct message he’d received from an account that he knew.

The message said

“Hi, my account was hacked. And it’s worse, you’re involved too, and I reported your account by mistake. Follow [link] and they can fix it.”

But the link led to a page which demanded his password, and just like that, his account was taken over. The hacker changed the phone number and email associated with the account, and set its posts to “protected”, so that only people already following the account could see what was being posted.

The effect on Boccardi, who quickly set up a new account while he contacted Support in the hope of some help, was unsurprisingly hard.

The ramifications of what had happened started to sink in.

But wait, you think: surely X Support can sort this out?

To which one replies: haha, who are you kidding? Helpful support is an oxymoron in these days of gigantic platforms, especially platforms which have slashed their support staff to the bone. The responses from X Support said, well, the email and phone number you’re giving us don’t match the number associated with the account.

The banging of heads against walls should be audible from here. So many people go through this experience, and yet there isn’t an algorithmic check at Support which asks 1) has the email and/or phone number associated with this account changed very recently? 2) has the account recently changed to “protected” status? You could add a third: is the request for support/account recovery coming from an email/phone number that was recently associated with the account? Given those, you could probably mop up and restore the majority of account thefts. All it would take is four extra columns in the user’s database table—PREV_ASSOC_EMAIL, PAE_CHANGE_DATE, PREV_ASSOC_PHONE and PAP_CHANGE_DATE—and you’d be able to sort these problems out. (Often the hackers also turn on two-factor authentication for the account—the lack of it being what enabled the hack in the first place—which should be another flag in the support database.

Because how often, from the platform’s point of view, do the following events happen:

1. an account changes its associated phone number and email

2. that account switches to protected mode

3. the account turns on two-factor authentication (2 and 3 are interchangeable in time order)

4. someone contacts Support saying that that account belongs to them and has been hacked, and their complaint comes from an email that was until very recently associated with the account

I can’t honestly think of an innocent explanation for that sequence of events. Sure, if you got a new phone and new phone number, and you were being tormented and worried about being hacked, then you’d do all of the first three. But No.4? How does someone send an email from the previous email associated with the account? A hacker wouldn’t have any simple avenue to know what it was, and would anyway have to spoof it. (Your problems multiply if your email is ever hacked, by the way. Turn on two-factor authentication! Get a code generation app such as Authy, or the Microsoft or Google ones, or let Apple’s generator do it.)

So here’s where Mario was left: locked out of his own account, whose posts he couldn’t even view because his new account hadn’t been following it (logically).

Stage 1 over, the hacker moved on to the next step: exploit the tight-knit nature of the group. Mario’s old account had more than 10,000 followers; his new one, at that time, had a handful. (I had added him to my tennis list—it’s how I separate topics out.)

DROP_TABLES the other shoe

And how do you exploit a tight-knit group? By using exactly the same tactics that worked in the first place—exploit their trust of your account, and hoping they haven’t heard the news that you’ve taken it over.

Which meant a number of that ten thousand followers got a Direct Message:

Yes, it’s that “I got hacked and then reported your account but you can fix it at this link” phish. Some people will have got it and clicked on it; that’s how mistaken trust works. It’s how phishing works. It’s why you need TARS, the intelligent machine, to be less trusting than you when you noodle down to a distant planet to see how a scientist trying to save humanity is getting on1.

But we’re not finished yet! Although the hacker now has an account which has lots of followers, gained organically, which can be used for bot output (in the course of time, by unprotecting its tweets when the Sauron eye of Support has passed—I mean who are we kidding, it was never on it), that’s not enough. There’s money to be made by other means.

Stage 3: separation (from money)

As soon as Boccardi mentioned that he’d been hacked, I cringed a little, because something inevitably happens when anyone uses the words “account” and “hack” in the same post:

No, try my solution!

Pick me! Pick me!

Note the two-stage element in the first two replies: hacked accounts responding immediately with the name of a different person who will “help” you. (Also, that first one hasn’t been updated for the modern age, which means it’s been running for at least two years in that format. You can try reporting it: nothing happens. The account itself is a mixture of frenzied replies to hacked account despair or thirst trap sexbot pics.

The scam here is that if you get in touch with these “hackers”, they’ll tell you that sure, they can get it back for you. It’ll, uh, cost a little because they have to a) get in touch with their friend on the inside or b) use their extra-special hacking t00lz to get your account back.

You send over the money, and that’s the last you’ll ever hear from them. And so heads they win—they’ve got a hacked account that can be used for bot earnings and sexbottery—and tails you lose: you can’t get your account back.

And all because of the trust that emanates from being part of an enjoyable social group, using social media in a positive way, to associate with people who you like to hear from and whose views you find interesting and engaging. These are the groups that are most vulnerable to hacking: once a single account has been compromised, the others can fall like dominoes.

These are the accounts that any platform’s support system should try to identify and help out, rather than ignoring them and cutting them loose, as happened to Boccardi. (Who, good news, is gaining followers daily on his new account—follow him for fun Italian takes in English about tennis.) The fact that X support in particular doesn’t take a little care about this is indicative of how uninterested platforms are in their users beyond being eyeballs to project adverts onto. If your account does get hacked (because you didn’t turn on 2FA—turn it on! It’ll be there in the settings somewhere), then you’d better hope that you have an inside track to the people who run it, because otherwise you’re stuffed.

And yes, that can be annoying and heartbreaking in equal measure to discover how little the platforms care. Though the one benefit from the tight-knit groups can be how quickly they’ll try to pull people back up. Humans tend to be good. It’s just that their trust settings are too high.

Subscribe now

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• I’m away for a week. But please leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time, helped—no doubt—by TARS.

Share

Cast your mind back ten years, to the US presidential election that kicked off in earnest in 2015. The most powerful weapon that Donald Trump had in his campaign first to become the Republican candidate and then to win the presidency was a text box which allowed a maximum of 140 characters: his Twitter account.

Trump used it in a way that had never been seen before from a politician. He was rude about women. He was scathing about opponents. He made wild claims and then ignored any attempts by journalists or opponents to follow them up. His account was a hammer that he used to hit the thumbs of any opponent.

Things didn’t change much when he came into power: he still tweeted obsessively, which meant that his first reign of error was one where his Cabinet, the GOP and journalists took their lead on what the story of the day was from whatever appeared on Twitter (as it was still called then). With the character limit still at 140 (it wasn’t expanded to 280 until November 2017) the first Trump administration was brusque as well as brutal. Trump didn’t interact, though he would sometimes retweet things, giving the impression of a slightly bored senior passing time waiting for something better to turn up. After all, he was only president.

Then came January 6 in 2021, and the insurrection, and Twitter decided that Trump was not in fact president any longer, and “permanently suspended” (not quite the same as deleted) his account. It also deleted tweets that he tried to post from other official accounts that the Biden administration hadn’t taken control of.

As we know, Trump didn’t go away. Instead he bumped around various social networks before settling on Truth Social, where during Biden’s administration his account generated various wild outputs. It still seemed to be Trump, but.. diminished?

Fast forward to 2024, and look! Trump was back as the Republican candidate. But something was missing. Although Elon Musk had bought Twitter and reinstated Trump’s account, it simply wasn’t being used. The hammering had gone quiet; it was coming from over in Truth Social, but short-form social media posts had stopped being the main way that he got his message across. Instead it was through his long, rambling rally speeches, and his appearances on long, rambling podcasts, and shorter, rambling appearances on TV interviews.

What had happened? As much as anything, everyone else caught up. The GOP in particular absorbed the lessons that had been taught in the first four years. Charlie Warzel observed in a column for the Atlantic that we were now witnessing “the gleeful cruelty of the White House X account”:

In previous administrations (including much of Donald Trump’s first term), the account was used to post anodyne updates, highlight press releases, and share information about the administration. It was, to be fair, often painfully dull or written in the stilted language of a brand. Now the account exists to troll its political enemies and delight the MAGA faithful.

…Although the account sometimes shares actual news, it’s frequently preoccupied by rapid-response engagement bait for MAGA diehards. Less information, more content. The intent is not to inform but to go viral.

Beyond the fact that this kind of shitposting is so obviously beneath the office, the posts are genuinely sinister. By adding a photo of an ICE arrest to a light-hearted viral trend [of rendering everyday pictures in the Studio Ghibli style], for instance, the White House account manages to perfectly capture the sociopathic, fascistic tone of ironic detachment and glee of the internet’s darkest corners and most malignant trolls. The official X account of the White House isn’t just full of low-rent 4chan musings, it’s an alarming signal of an administration that’s fluent in internet extremism and seemingly dedicated to pursuing its casual cruelty as a chief political export.

The people running the White House’s X account are doing all the things that Trump used to do back in 2016. JD Vance, the vice-president, takes on all comers (not always successfully) on the same network—which is a departure from his boss’s style, which was never, ever to engage with the plebs.

Vance, though, can’t restrain his urge to Own The Libs, which to be honest is a much less impressive look than Trump’s monumental indifference. It may be the case that the Coca-Cola the president drinks is the same drink that you and the homeless man drink, but the president doesn’t discuss how it tastes with the homeless bum.

Amid all of this, though, where is Trump? Where are his witty aperçus on the state of the world? It turns out that not only are they posted on Truth Social, but they’re also on X. Yes: the realdonaldtrump account is there, and posting things:

Yet I had to go and look for it earlier this week when I was thinking about writing this. I was honestly astonished to find it was posting new stuff, because as far as I can tell, it’s gained absolutely zero traction inside or outside X. His slightly longer and more deranged posts at Truth Social seem to do that, but their ALL CAPS nature puts a lot of the media off them.

Or perhaps people just don’t find them.. very engaging? It was astonishing ten years ago, but now when the social media account for the president’s office is leaping onto zero-day memes to post viciously triumphal pictures, the Trump act is nothing special on social media.

Is that good? I have a feeling that it isn’t, really. It’s not that we’re habituated to him. It’s more that he now has so many imitators, or people who have taken it further, that the floor of brutality has been raised. This doesn’t mean that you have to be brutal to be there—one of the most life-affirming social media accounts is that of Niall Harbison, a sweet Irishman who runs the Happy Doggo centre in Thailand where he rescues and rehomes street dogs. (Will there be a tariff on dogs rehomed from Thailand to the US? So many questions.)

But it does mean that if you’re going to be brutal, you have a lot of experienced competition. Unfortunately, the US’s leadership has decided that it wants to go down that path, and it may take quite some time to dial that back. The day will eventually come, but the snapback is going to be quite dramatic when it does.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

Subscribe now

On Friday March 20th, a transformer in a substation in outer London got hot. Very hot. Too hot: the oil in it which is used to cool it overheated, and the transformer and its 25,000 litres of oil caught fire, and the substation—with two backup transformers in the same building—was soon alight and its electricity output halted. Unfortunately for everyone who wanted to travel in and out of Heathrow Airport that day, numbering 1,300 flights and 200,000 passengers and an unknown amount of cargo, that substation was the one supplying the entire airport—which has a power demand of about 40-60 megawatts (MW), using an average of 30MWh per hour.

There was a lot of argument about whether Heathrow1 could have switched over to one of two other nearby substations, which the National Grid (in charge of keeping the electricity going) said were still operational and could have supplied the necessary power. But the airport authorities said they couldn’t make the switchover in time: there are lots of systems that have to be turned off and on again in the right sequence.

The obvious suspicion, given world tensions, was that this was an act of espionage or terrorism, as seen in the opening scenes of the second series of ITV’s Trigger Point just over a year ago. It wouldn’t be above the GRU to think “hey, nice scenario!” would it?

However if you’re a certain sort of politician, then your obvious suspicion is that actually Heathrow could have kept going except for The Woke. Or more specifically, because it got rid of all its diesel generators that could have provided backup power in order to be nice to the climate, which actually doesn’t need anyone being nice to it at all.

Step forward Nigel Farage and Richard Tice, respectively the leader and deputy leader of the very right-wing Reform UK. Farage decided, three days after the incident, that Heathrow had “no diesel generator backup. It was removed as part of their drive to net zero.” He tweeted this, and Tice retweeted it:

Politically, this has a lot going for it from Farage’s point of view: arglebargling about Net Zero (the point when net carbon dioxide emissions fall to zero, ie we absorb as much as we emit) is one of Reform UK’s platforms. They’d like a referendum on it, because it’s another topic where people don’t really understand what is on offer and what the benefits and disbenefits are.

The only problem with his tweet: Heathrow does have diesel generator backup, as was reported by the BBC and other outlets the day before Farage’s tweet:

A Heathrow source also told the BBC that they have "multiple sources" of energy at the airport – with diesel generators and "uninterruptable power supplies" in place.

They added that when the power outage happened the back-up systems "all operated as expected".

The systems, however, are not enough to run the whole airport – hence the decision to close it down.

It would be pretty incredible if Heathrow really had 40MW of standby power sitting around in diesel generator form. Suggesting it ever did is indicative of having researched nothing about the subject. And while we’re on the subject, what sort of power sources does Dubai Airport have? Hmm, here’s a press release from 2019 celebrating the installation of 5MW of solar panels. There’s also this from before 2010 which suggests there’s about 8MW of diesel backup. Dubai does handle more throughput than Heathrow (about 5m passengers v 3.2m), but even it isn’t looking to handle all its power needs with chugging engines.

Tice followed up later in the week pointing to a Daily Telegraph article whose headline seemed to back up the Farage claim. “Heathrow feared net zero would threaten its power supply”, the headline reads. Tice tweeted triumphantly that “They knew their vulnerability, but did nothing…”

Except.. the article isn’t about diesel backup generators at all. It’s about the challenge of getting enough electricity to the airport when it’s functioning normally:

The airport had been warned a decade ago that it was overly reliant on very few sources of power to keep it fully functioning.

In June 2023, Mr Milton told MPs that net zero would add to those difficulties as the airport moved towards all-electric ground operations and the use of electricity to liquefy hydrogen to power the next generation of aircraft.

He suggested Heathrow feared its ambition to achieve net zero emissions by 2050, as part of the UK’s decarbonisation strategy, could be undermined by the grid’s ability to provide the power needed.

So this is actually about getting the National Grid to supply more power. There had been delays in upgrading the substation to a higher power rating; possibly because it was built in the 1960s. Nothing to do with diesel generators.

But this didn’t stop people believing it. Hence a reply to me when I tweeted one of the rebuttals of Farage/Tice:

The reference to “biofuel” is because Heathrow has both diesel and a biomass backup generator. The biomass is an addition, not a replacement. And while it might be helpful for net zero (there are lots of disagreements about biomass generators and their contribution), it wasn’t going to be able to produce enough power to keep the airport going either. I pointed this out.

Why didn’t they refer to the change? Because they didn’t change. And isn’t it odd that it should be the Heathrow one? Well, no. Substations catch fire quite regularly, because there are 330 of them, some are quite old, and oil has a tendency to go hot and wrong. You only have to do a quick search for “substation fire” and you’ll turn up reports showing that about one per month requires the solicitations of the fire brigade. So no, it’s not odd at all that the Heathrow one decided that this was its time.

Vote for me, I’m insidious

So anyway, there are the facts of the matter. But Farage, of course, had that tagline of “The truth about this disaster is being withheld.”

This is one of the biggest problems with social media: all the incentives, especially for politicians, are bad. Tell ridiculous lies! Hint at evil coverups! Nobody’s going to come for you. Community Notes will take days to catch up, if they ever do. The quote tweets and screenshots and replies will all bounce off. Your manifest errors can be pointed out in the House of Commons—which you’re not attending—and it makes no difference.

I did write about this in Social Warming, in the context of politicians’ social media followings generally, and the 2019 election in particular. What’s most notable, if you study the number of social media followers that politicians have, is that those who are most extreme have the largest number of followers. And, conversely, those with the fewest followers tend to be the most centrist.

Bill Brady, the Yale University researcher who examines how ‘moral outrage’ spreads on social networks, told me that “There’s no doubt that politicians, whether implicitly or explicitly, are aware of this general idea that if you post moral and emotional content, it can help diffuse through a network.” (He co-authored a paper in 2019 looking at tweets from Donald Trump, Hillary Clinton and the members of both houses of the US Congress: it found that ‘moral-emotional’ messages gained more retweets and visibility.) “There’s no doubt that these campaign managers are savvy to that and it can help get their messages spread.”

There was a classic example of this during the 2019 election, as I wrote:

At the end of November 2019, in the run-up to the UK’s third general election in four years, a man stabbed and killed two people on London Bridge in a terror attack. It transpired that he had previously been convicted of serious terrorist offences in 2012—when a Conservative-led coalition government was in office—and yet released six years later without a parole board assessment. “Why?” asked the opposition Labour candidate Yvette Cooper.

Priti Patel, the Home Office minister and Tory parliamentary candidate, tweeted her response: “Because legislation brought in by your government in 2008 meant that dangerous terrorists had to automatically be released after half of their jail term. Conservatives changed the law in 2012 to end your automatic release policy but Khan [the killer] was convicted before this.”

As you may have guessed…

Patel was wrong: the courts had many types of sentence available that could have prevented automatic release. And Khan had been given a sentence in April 2013 (under the coalition), following the passage of the 2012 law change, which confirmed release halfway through his sentence.

But being wrong made no difference. Patel’s statement sat on Twitter, attracting thousands of retweets and an entire ecosystem of angry responses. There was no way to kill the untruth.

And this is the problem. Social media plus politicians is an eternally toxic mix, because there’s no brake on them. They can tell untruths, knowing or unknowing, and there’s no reason to think it will cost them, because they can ignore it and move on, leaving anger and annoyance in their wake.

The obvious example is currently sitting in the White House—though this time around it’s the vice-president and defence secretary who are leaping on to social media (principally X—perhaps JD Vance’s Insta is all puppies and flowers) and lashing out at people with the most ridiculous set of untruths about the Signal calamity.

I wish I could come here and tell you that there’s one simple trick to stop politicians from blurting nonsense—worse, mendacious nonsense—on social media: that there could be a buzzard that would swoop down and pluck either them or the offending tweet and bear it off into the sky. Or that the perpetrators would feel so shamed by quote tweet dunking that they’d give up.

Unfortunately, it’s the opposite. Where conventional media used to offer a bottleneck that prevented the worst of this getting much traction (mindful, of course, of the fact that it was the BBC which continually gave Farage a platform from which to espouse his views), now there’s nothing. It’s all going to get worse. Sorry.

Share

Subscribe now

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

Hey, it’s tennis season1, and I’m enjoying the tennis from Indian Wells, on the Californian coast. In the round of 16, Britain’s Jack Draper played America’s Taylor Fritz—one a semifinalist at last year’s US Open, the other the beaten finalist at the same event.

But after they had played, ie on Thursday morning, what was their head-to-head record?

If you’re one of the many people who seem to be proliferating right now, then you know exactly where to go to answer this question: a chatbot.

OK, then, let’s go and ask ChatGPT.

Let’s not spoil the surprise of whether that’s right or wrong yet. Let’s ask Grok, the chatbot that is now incorporated into X:

First of all, notice that ChatGPT says they’ve played three times up to March 13. But Grok says they’ve played seven times. Yet then it cites only five results.

Now let’s get the definitive answer, by going to the ATP (Association of [male] Tennis Professionals) site:

This is the reliable source. Five matches: one in 2025, three in 2024, one in 2022.

Grok has the list correct—but says they’ve played seven times. Why?? ChatGPT meanwhile doesn’t know about 2022, or the Munich match, so the H2H is wrong.

If you relied on either of these chatbots for your answers to what is a very straightforward question, you’d go away with incorrect information. And while I haven’t tried every chatbot on the market, you’re welcome to have a go; these two just happen to have the biggest mindshare. And, to repeat, they get stuff wrong. Very simple stuff, like how many items there are in a list.

Which means that when people start using it for more complex questions than tennis players’ win-loss records, things can quickly get very messy.

“Just a LLM convo”, huh. It didn’t take long for people to point out that this didn’t make sense (see the comments to the post), such as “That makes no sense. The hypertension alone should have some negative impact on lifespan.”

And an even more relevant question was raised by the Science Banana account:

Followed by the despairing noise that is so common among specialists coming into contact with non-specialists who have used a tool and not quite understood how to use it properly:

Would a computer really just go on the internet and lie? Well yes. Yes it would. That’s because LLMs don’t know when they’re lying. They don’t have any self-questioning systems to determine whether their answers necessarily conform with reality. The chatbots giving the tennis data were working off solid datasets which had been updated in the past few hours, but they didn’t have systems that would tell them whether five matches meant a 2-1 record or a 3-4 record or even what the hell five matches means. Chatbots don’t have a theory of the world; they just put words after each other that seem to be like the words that they found near each other when they were fed with the internet, and then they spit the words that they’ve picked out to you.

The wrong tsunami

Back in August 2022, I figured that there would be an AI tsunami of content which would increasingly tailored to us so that it would be addictive, and we’d lose interest in human-made content. That’s still a problem that could arise—don’t get out the bunting yet—but we now have a much more proximate problem with AI that is also more pernicious: people think chatbots know the answers.

Here’s what someone replied when I pointed out that “asking a chatbot to answer a technical question” was basically outsourcing their brain to an unreliable machine, and that it wasn’t smarter than they were:

When did we go from Googling stuff and following links and checking facts to “of course it’s smarter than me”? This is a worrying trend, yet if you look around on social media, you’ll see more and more people offering screenshots from chatbots of one flavour or another as answers to questions. Have they checked whether it’s right? Probably not. Meanwhile Google is pushing even more AI Overviews, effectively short-circuiting people’s previous requirement to at least browse the excerpts from a few sites to see if they agreed in the answers they offered. (If they didn’t, you needed to look a bit deeper to find out why.)

The problem is that nobody ever disappointed the general public by making it easier to do something. If you give them a box which seems to promise to answer every question immediately, rather than making you browse some results, then they will gravitate to the box. It’s the same irresistible force that drove people to migrate first from vinyl to CDs and then to MP3, from film camera to digital camera to smartphone camera, from mainframe to PC to mobile. Do MP3s sound better than CDs? Who cares, they’re convenient. Do smartphones take better pictures than film cameras? Who cares, they’re convenient. Do chatbots give better answers than a search engine results page and the results therein? Well, look, I care a bit, because having a shared set of facts you can rely on is fundamental to making good decisions about the world. Plus people who are given wrong information by these systems are extremely reluctant to believe they’re wrong. You can show them all sorts of sources, but they’ll insist that the chatbot is correct, not the work produced by humans.

It’s always worth repeating: chatbots are stochastic parrots, putting words together with as little understanding as a parrot. The 2021 paper that coined the word points out (on p616) that the output from LLMs is “seemingly coherent”:

We say seemingly coherent because coherence is in fact in the eye of the beholder. Our human understanding of coherence derives from our ability to recognize interlocutors’ beliefs and intentions within context. That is, human language use takes place between individuals who share common ground and are mutually aware of that sharing (and its extent), who have communicative intents which they use language to convey, and who model each others’ mental states as they communicate.

But, they point out

Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that. This can seem counter-intuitive given the increasingly fluent qualities of automatically generated text, but we have to account for the fact that our perception of natural language text, regardless of how it was generated, is mediated by our own linguistic competence and our predisposition to interpret communicative acts as conveying coherent meaning and intent, whether or not they do. The problem is, if one side of the communication does not have meaning, then the comprehension of the implicit meaning is an illusion arising from our singular human understanding of language (independent of the model).

In other words: we effectively anthropomorphise chatbots’ output. But we shouldn’t, despite the insistent efforts of the chatbot creators to give them the façade of personality (such as Grok, above, using the phrase “my knowledge”). We should distrust them all the time, and be wary of the people using them.

Which, incidentally, gives us our coda. The technology journalist Chris Stokel-Walker got an excellent scoop at New Scientist, using Freedom of Information requests to find out how Peter Kyle, the UK’s technology secretary (a ministerial post), uses ChatGPT.

Turns out that Kyle asked it about why the UK’s small and medium-sized business have been slow to adopt AI (did he think the chatbots somehow talk to each other?), and which podcasts would be best for him to go on to reach a wider audience in the UK. Stokel-Walker was inspired to make the FOIA request after Kyle gave an interview to PoliticsHome, where Kyle said that “ChatGPT is fantastically good, and where there are things that you really struggle to understand in depth, ChatGPT can be a very good tutor for it”.

Oh dear. You know, sometimes life is a real uphill struggle.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

Subscribe now

Years and years and years ago, in the dim and distant past (by which I mean 2010), a guy called Eli Pariser had an idea from which he wrote a book (so quaint!) called The Filter Bubble, which had the helpful subheading of “What the Internet is Hiding from You”. It came out in 2011, and caught a feeling that a lot of people already had: that the rise of algorithms on social networks, and personalisation of all sorts of recommendations, including what to listen to or read or buy, was funnelling everyone down individual tracks that meant they couldn’t see or hear views that might challenge their view of the world.

As Pariser said in an interview with Salon in October 2010,

Since Dec. 4, 2009, Google has been personalized for everyone. So when I had two friends this spring Google "BP," one of them got a set of links that was about investment opportunities in BP. The other one got information about the oil spill. Presumably that was based on the kinds of searches that they had done in the past. If you have Google doing that, and you have Yahoo doing that, and you have Facebook doing that, and you have all of the top sites on the Web customizing themselves to you, then your information environment starts to look very different from anyone else's.

Why would that be a problem? Simple, he explained:

We thought that the Internet was going to connect us all together. As a young geek in rural Maine, I got excited about the Internet because it seemed that I could be connected to the world. What it's looking like increasingly is that the Web is connecting us back to ourselves. There's a looping going on where if you have an interest, you're going to learn a lot about that interest. But you're not going to learn about the very next thing over. And you certainly won't learn about the opposite view. If you have a political position, you're not going to learn about the other one.

If you Google some sites about the link between vaccines and autism, you can very quickly find that Google is repeating back to you your view about whether that link exists and not what scientists know, which is that there isn't a link between vaccines and autism. It's a feedback loop that's invisible. You can't witness it happening because it's baked into the fabric of the information environment.

People could see that he had a really good point. Pariser did allow that there were legitimate reasons why these filters could exist: the sheer volume of information was overwhelming (for us poor 2009-10 people. There’s a whole generation that has grown up completely used to the turned-up-to-11 white noise of the modern internet). You needed filters to pick out the useful stuff.

But equally, those filters were dumb, and took shortcuts. Google’s personalisation of search might seem useful, but it was very questionable. Why should results for one person’s “BP” query be about the Horizon spill, while another is investment opportunities? The question of whether that gaping box that accepts your text is the maw to a universal oracle which tells the truth, or an eager personal assistant that wants to help you out, is in effect answered by this test. Google doesn’t want to, and can’t, direct you to “the truth”. It wants you to be satisfied with the results it serves up.

In retrospect one can now see what it was doing as the first steps up the hill towards its present implementation of AI answers. Which naturally raises the question of whether those too are personalised: do different people get different answers for the same questions? Based on the “eager to please” metric, one would have to think they are.

So Google and all the rest were creating filter bubbles (even if Google did quietly dispute the extent of its tailoring of results, saying the search query was more important than the user. Uh huh: so the user matters a bit, then?).

The question was, what could you do about it? Fortunately, the rise of social media turned out to be quite a good antidote. In their search for “engagement”, a lot of the networks discovered that content which creates outrage, whether mild or excessive, is a great way to get people to spend time there. Hence Twitter, even before Musk, had an algorithm which would pick out tweets that were getting large amounts of engagement and show them to users who hadn’t interacted with them in the expectation that “this seems to have got other people worked up, might it do the same for you?” (This process has been increased since Musk’s takeover.) This was a way that people could be exposed to different ideas, even if they found them objectionable.

But some people didn’t want to see those different ideas. Quite where the boundary between discussion, argument and harassment lies is always subjective, and for some people there was too much of the latter. Their response was the creation of “blocklists”, which used the Twitter API to identify people who could be added to a list of those who would be preemptively blocked.

There’s an interesting 2018 study about Twitter blocklists by a team at the Georgia Institute of Technology, which found that those who subscribed to blocklists felt the networks should have taken action sooner. Probably, though, what they perceived as harassment never rose to the level the moderators could or would deal with. This was researched not long after Gamergate, the 2014 upheaval that saw endless spats online and plenty of serious behaviour including bomb and death threats offline.

Here’s what one blocklist user said.

“I was getting constant mentions from GamerGate accounts, and I was wasting tons of energy replying. When someone asked “what have you been up to?”, all I could think of was: arguments with anonymous neofascists. I saw a few people discussing preemptive blocking and looked into it.”

It was a different age, and one would have to think now: have you tried not having arguments with anonymous neofascists, and just using the mute button so you can ignore them? It was in response to this problem that in 2022, Twitter introduced the “Leave Conversation” and “Mute Conversation” options.

On the other side of the fence, those who were put on blocklists often felt their presence was unfair, often based simply on who they followed on the network rather than anything they might have said. Following someone doesn’t have to be a statement of adoration, after all. Sometimes you follow people you disagree with because you want to see what their opinions are—either to explain to them why they’re wrong, or to ponder whether you’re wrong. Then again, who’s going to go into a university study and say “sure, I totally deserved to be on that blocklist! I was as obnoxious as I could manage!”

However, if we think that filter bubbles are a problem, then the logical corollary is that preemptive blocklists are too. You’ve never interacted with someone, but you think already that they’re not worth listening to ever? This is a big part of my objection to the current incarnation of Bluesky, where blocklists based on “who do you follow” abound. Even more than that, blocked posts are invisible, people don’t show up in searches, and you may not know you’re on such a list. The filter bubble is in place: chunks of the network vanish. How many chunks, how big? We can’t know.

But we can get a glimpse. Clearsky uses the Bluesky API to find out how many people are blocking how many people. The numbers are astonishing: just under 52% of users are blocked by at least one other user, while 36% are blocking someone. (Logically this means that the average user is blocking 1.44 others.) The two most-blocked people are Jesse Singal, a journalist, and Brianna Wu, who was one of the targets of Gamergate. One can’t know if they’re being blocked by the same people, so somewhere between 80,000 and 140,000 accounts (out of 30.8m active ones) are blocking them: which is only 0.4% of the network, at worst.

Then you look at who’s busiest with the blocking, and there’s an account called Palomar, which is blocking 1.5 million accounts, or nearly 5% of everyone there. I’d tell you more about them, but I’m blocked.

All this is in stark contrast to Elon Musk’s changed implementation of “block” on X, where if someone has blocked you then you can see what they have said; you just can’t interact with it. The filter bubble is popped, at least in one direction. (OK, the metaphor doesn’t make complete sense.) The effective shutdown of the X API also means that creating a blocklist becomes impossible.

The case for blocklists in the face of harassment is obvious. But when our biggest problem is polarisation and refusal to listen to facts we don’t like, I don’t see how taking refuge behind a big blocky wall helps. If filter bubbles were a problem in 2009, they’re even more of a problem now. It’s hard to imagine anything will persuade Bluesky users to tear down their blocklist walls. But I feel they might, in the longer term, benefit.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

Subscribe now

The idea of the wisdom of crowds is quite well known; the journalist James Surowiecki wrote a book of that name in 2004, distilling a lot of what had come to be understood about the phenomenon that was best crystallised by Francis Galton. To quote from Wikipedia:

The classic wisdom-of-the-crowds finding involves point estimation of a continuous quantity. At a 1906 country fair in Plymouth, 800 people participated in a contest to estimate the weight of a slaughtered and dressed ox. Statistician Francis Galton observed that the median guess, 1207 pounds, was accurate within 1% of the true weight of 1198 pounds. This has contributed to the insight in cognitive science that a crowd's individual judgments can be modelled as a probability distribution of responses with the median centred near the true value of the quantity to be estimated.

You can read Galton’s original paper, which shows that it was 787 people (actually) who voted and that their estimates varied from 1074 to 1293 pounds—that is, from -11% of the true weight to +8%, or nearly a fifth of the weight in total. Galton makes the point that the distribution of estimates meant any randomly chosen measurement has a 50% chance of being within +/-3% of the correct weight.

But, as he also notes, there were plenty of experts among those making their estimate, including butchers and farmers (with a sixpenny—half a shilling—entrance fee to dissuade jokers). Strangely, the estimates erred towards a higher value: “I have not sufficient knowledge of the mental methods followed by those who judge weights to offer a useful opinion as to the cause of this curious anomaly.”

Crowd psychology is a strange thing: we don’t necessarily know how people arrive at the decisions they do. Surowiecki gives lots of examples and anecdotes in his books; the idea is that we ought to think crowds have access, almost through the Brownian motion of their thoughts, to insights that aren’t available to individuals.

Certainly it’s true that if you ask a large enough group of people a question requiring specialised knowledge, you’ve got a good chance of finding someone who knows all you need to know about that topic. You might even find two people; if you hold a fair with slaughtered dressed oxen and have a lot of farmers and butchers around, they’re probably going to be better at guessing than if you ask a bunch of children. The idea that the crowd was probably wiser than us was a big idea at The Guardian when I joined it in 2005—that writing and publishing a story wasn’t the end of the process, because now the readers could feed back their ideas to us.

This idea was warmly embraced, particularly by opening up the space below the articles to comments about their contents. These were not always supportive. They were not always anything to do with the article. They were not always legal and they were sometimes just excessively offensive, and so people had to be hired to monitor what had been written in the comments. Sometimes they found good stuff too, but it was like trying to pan gold in a sewer.

The wisdom of crowds seemed to be more about the crowds part than the wisdom part. And this was only 2004/5—predating widespread access to social media that we would recognise now. (We’ll ignore FriendsReunited and MySpace: the former wasn’t “social” as such, the latter died within a couple of years.)

And then as Clay Shirky observed, “here comes everyone”. Social media arrived, and the crowds and the question of their wisdom moved onto Facebook, Twitter, Instagram and so on.

There was a period in the mid-2010s when you could probably say that Twitter (as previously formulated) could do a pretty good “wisdom of crowds” exercise. Its use of verified accounts that were actually who they claimed to be, the scale of its user base, meant that asking a question might get a useful answer. It was the world’s telephone book, or the people who knew who to contact in the telephone book.

And now? Upheaval! If you focus only on the past three weeks, the utter insanity of the crowds has been on show. One of Charlie Warzel’s contacts called the process that’s going on right now the “rapid unscheduled disassembly" of the US government”1. Elon Musk is leading an attack on, basically, anything that costs money, and his gimlet eye has fallen most quickly on the Department of Defense, with its $850bn annual budget which surely has colossal amounts of waste.

Just kidding! He went after USAID, which employs about 100,000 people doing various sorts of soft power things around the world (eradicating disease, feeding starving people, monitoring epidemic outbreaks such as Ebola) with a budget of around $40bn.

To help slash and burn its activities, he and lots of people adjacent to the White Madhouse turned to usaspending.gov, which lets you search by name, award, amount, all sorts of things. Of course they inquired what USAID had been spending its money on, and were suitably outraged at many of them. Helping people not to die? Preventing Ebola getting so big that someone couldn’t just get on a flight to the US while ill and spread it? Wasteful nonsense!

It also demonstrated that a crowd with no idea what it’s doing with a database is about as good for the public discourse as a child given a box of matches and an open can of petrol.

There are hundreds of examples of this sort of thing. At one point, the finger-pointing right-wingers triumphantly discovered that the US Treasury pays money for Bloomberg terminals. Imagine! People in government finance knowing what’s happening in the bond markets? Intolerable!

At another point, the finger-pointing turned into a circular firing squad and/or ouroboros. (Chris Rufo is a prominent right-winger in the US.)

Which unsurprisingly earned a pretty rapid response:

And then:

Gee, d’ya think that it’s at all possible that a colossal crowd of people all slamming a site they don’t understand could possibly have bad outcomes? (See more for yourself with this search. Hilarity and headshaking at humanity’s ability to get the wrong end of the stick guaranteed!)

But here we are. The wisdom of the crowds has given way to the utter madness. Although by contrast the same search on Bluesky yields a rather more rational set of takes, even if their vehemence is quite similar.

What were the inciting points in the crowd shifting from wise to mad? Obviously there’s the external political climate, and the insane claim by Musk that he will cut $2 trillion from the US budget, which simply isn’t feasible unless he closes the US military for a few years. However, if it were still Twitter rather than X, with verified accounts being people who you could rely on rather than those with questionable financial choices, I think the noise would have been tamped down. That probably wouldn’t have stopped people going on USAspending.gov and doing insane searches (if you’re willing to believe it, Jeffrey Epstein was a recipient of its goodwill) but as with a fire, what it needs to continue is both oxygen and fuel. The oxygen of a social network tuned for rightwing outrage, and the fuel of lots of outraged people who don’t really know what they’re doing, has created the perfect conditions for things to go very, very wrong. The way that stories now can bubble up from social media outposts to the inside of the Oval Office is quite scary, and so there’s cause for concern about what on earth is going to happen next. Though perhaps they will decide they don’t need tanks, or guns, or soldiers.

All I can say is that I’d be a lot more worried if I lived in the US. As it is, I have a visit planned for September and I’m already hoping that it will be functioning roughly as it was at the start of the year. But, you know, anything could happen.

My attitude to AI has, for a long time, been “this sounds great in theory, but let’s see exactly how the practice works out”. In the 1980s, there was tremendous excitement about the prospects for “expert systems”: humans would input their knowledge to systems effectively as a set of IF-THEN rules—for example, to identify an infection in a patient by getting the medic to answer a set of yes/no questions and perhaps entering a little text. They have some success in a number of fields, but according to Wikipedia:

In the 1990s and beyond, the term expert system and the idea of a standalone AI system mostly dropped from the IT lexicon. There are two interpretations of this. One is that "expert systems failed": the IT world moved on because expert systems did not deliver on their over hyped promise

(Though it does say there are alternative explanations: one, that expert systems were rolled into standard tools—which I don’t really think happened—or two, that they cause power struggles inside organisations when the systems escaped the grasp of IT. Personally, I thought they just vanished: they’d been unwieldy, and hard to fill with knowledge (and the knowledgeable humans were a bit wary of filling a computer with their valuable, commercially exploitable wisdom gained over years).

Anyway, the AI tide, having gone out, has now come roaring back, as you may be aware. But now we aren’t filling the AIs with expert knowledge; we’re stuffing it with absolutely everything we can find on the internet, including Reddit, and somehow hoping that this will be enough for accuracy. As has already been seen with the recommendations to use glue as a topping for pizza and eat rocks, this isn’t a good idea.

No matter! People are relying on the new generation of chatbots to answer general knowledge questions, and relying on them to be correct. What’s more, they’re relying on them to answer complicated questions.

Screenshots of Google Gemini responses used as answers to anything should induce an instant eyeroll by now, but people are lazy: the biggest lesson perhaps we should have learnt from the years of social media (so far) is that people won’t even open a new browser tab to check something before saying it. But now that they are being moved to, they’re using the top result—frequently Google Gemini, because Google is the biggest search engine—and relying on that.

There are times where this isn’t helpful. In my many years as a journalist covering technology, science and medicine, it always felt important to actually get things right, and to make sure that what I wrote didn’t somehow get altered to mess up the sense. Covering BSE (aka mad cow disease, which infected 184,000 cattle from 1987 to 2003) and then its human form, variant CJD (which has killed about 180 Britons), required wrapping my head around extremely puzzling science where there were many conflicting suggestions for how the disease was spread (bacterium? virus? It turned out to be neither), and very few experts—nor many internet resources—who could talk about it. This part may sound familiar: despite a detailed inquiry lasting years and testimony from scores of experts, the origin of BSE remains a mystery1.

BSE and vCJD are prion diseases, caused when a protein in a cell flips from one stable shape to another shape which is harmful, and somehow causes other copies of the same protein to flip in the same way. But does the AI know this?

I’m happy to say that Bing’s AI answer gets this correct. Ditto Google Gemini. Both use reliable scientific sources; for the Google one, it’s from a document updated in May 2024—the same one, in fact, I used in the previous paragraph. (I didn’t know it would use that.)

However.. this week Donald Trump came into office and signed a slew of Executive Orders. One of them relates to sex, though it’s more ponderously titled DEFENDING WOMEN FROM GENDER IDEOLOGY EXTREMISM AND RESTORING BIOLOGICAL TRUTH TO THE FEDERAL GOVERNMENT.

What got people worked up was part of Section 2, “Policy and Definitions”:

(a) “Sex” shall refer to an individual’s immutable biological classification as either male or female. “Sex” is not a synonym for and does not include the concept of “gender identity.”

(b) “Women” or “woman” and “girls” or “girl” shall mean adult and juvenile human females, respectively.

(c) “Men” or “man” and “boys” or “boy” shall mean adult and juvenile human males, respectively.

(d) “Female” means a person belonging, at conception, to the sex that produces the large reproductive cell.

(e) “Male” means a person belonging, at conception, to the sex that produces the small reproductive cell.

We can agree (let’s hope) that “at conception” means the point at which the sperm and ovum fuse to form the zygote. This is very basic biology. At that stage, the zygote has all the DNA it will ever have, and that DNA contains the instructions for every life stage that follows, from blastocyst to embryo to baby to child to adult. That DNA will determine if the adult is male or female2.

Logically, therefore, your sex is fixed at conception/fertilisation. And because your sex chromosomes won’t change, neither will your sex. The definitions given there of the two sexes—one that produces large reproductive cells (aka gametes), one that produces small ones—is completely standard in any biology textbook.

And yet… people on social media weren’t having it.

(Amazingly, none of the Community Notes on that tweet has passed muster to become publicly visible—which demonstrates one drawback of the Notes algorithm which insists that “Notes are only approved if people who have disagreed on other topics agree on it”.)

Which brought a riposte from an actual developmental biologist:

But the people claiming the “all embryos begin as female and develop that way for six weeks until if they’re male they change development path” were quick to jump onto Google. And to be fair, for quite a long time—up to about 2000—the prevailing thinking was that human embryos up to six weeks had female characteristics, and it was only at that point with the activation of the SRY gene and related genes (found3 on the Y chromosome) that they became male.

So if you go and ask Bing it will spit back a Copilot result which says this:

Look at the date on that study! 1974—before modern genomics and before the first IVF baby. What’s more, if you scroll down the page past all the garbage to the actual search results, you get this:

Notice how the more recent results confirm what is now understood: embryos are “bipotential” (though they’re already motoring down their specific developmental track towards male or female).

How about Google Gemini?

A little better, but definitely confusing. Also, wrong, even though it’s a reference at the NIH, which is usually a reliable source. “Phenotypically female” implies, well, female. That’s because it’s quoting a 2001 book.

For the correct description, one needs to search the NIH PubMed database more accurately, which turns up a 2020 book called Sexual Differentiation, which begins:

The chromosomal sex of the embryo is established at fertilization. However, 6 weeks elapse in humans before the first signs of sex differentiation are noticed. Sex differentiation involves a series of events whereby the sexually indifferent gonads and genitalia progressively acquire male or female characteristics.

“Sexually indifferent” is a precise enough phrase. No default-female embryos here.

But the scale of the problem is evident. Google and Bing get this wrong. (Shout out to DuckDuckGo whose first result is correct.) They stuff the wrong answer—incredibly outdated in an area of science!—into their AI, and there’s no indication of whether it will ever change. People are misinformed, and they can claim the backing of the two biggest search engines. Although, big although, ChatGPT gets this right:

Not exactly, though it's a common misconception. All human embryos start with the potential to develop into either sex, but they aren’t technically "female" by default. Here's how it works:

In the earliest stages of development, all embryos have a pair of structures called gonadal ridges and two sets of ducts (Müllerian and Wolffian ducts), which can develop into either male or female reproductive organs. The gonads at this stage are "indifferent," meaning they haven’t differentiated into testes or ovaries yet.

This, though, gives us a glimpse of the scale of the problem. Search engines are wrong. ChatGPT is correct. (Grok is correct too. Amazing.) Some of PubMed is wrong. Lots of people are wrong, thought the woman who wrote the EO was quite careful to be correct by not specifying chromosomes4, just sex.

I worry that this is going to get worse. It was bad enough when Google was effectively the sole resource and was Googlebombed; now we have multiple sources whose provenance is in effect hidden but trusted too much. And there’s so much investment wrapped up in them that they’re not going to turn them off.

Plus people have a real hatred of having to change their minds about something. If an AI contradicts what they think, I think it’s more likely they’ll go and ask a search engine which will bolster their view, or vice-versa. (I realise you might say that about my attitude here. Except I can point to the scientific book from 2020 which formed my view.) It’s almost like what used to happen with newspapers, when people would gravitate towards one which supported their world view rather than constantly challenging it.

Maybe we just have to wait for the tide to retreat a bit. But the trouble is that most humans remain hungry for information, but not quite hungry enough to search any further than they have to in order to find something that will confirm their priors. We may be hunter-gatherers of information. But we’re quite lazy ones.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

Subscribe now

Even Joe Biden seems to have heard about it. “Social media is giving up on fact-checking,” he mumbled at us through his dentures in his farewell address. And indeed, everyone has been very exercised about Meta CEO Mark Zuckerberg’s announcement last week that Meta’s properties—Facebook, Threads, Instagram—will over the next few months stop employing full-time fact checkers, and instead move to a Community Notes-style feature as is used on what I suppose we’re required to now call X, formerly Twitter.

I’ve said previously that I like Community Notes: I think the principle is an excellent one, harnessing the knowledge and focus of the multitude to correct errors. I remarked back in May 2023:

Musk is very keen on it. (That doesn’t mean it’s a bad idea.)

I like the feature: I only wish it appeared on more tweets, and more visibly. (Maybe a big red border on top of the tweet? Some sort of icon on the top right?) You can see which tweets it’s been put on and rated “Helpful”. Seeing them being added is like watching Wikipedia on fast-forward.

But plenty of people on social media seem absolutely incensed by Zuckerberg’s decision. Boycotts of Meta properties were called for (this would be the 1,594th boycott of Meta/Facebook since its inception, and as we can see every single one of them has been 100% effective in burning it to the ground). He was figuratively burnt in effigy. People said mean things about his haircut and clothes. Yes, the people on social media really showed Zuckerberg what happens when you vex them.

Part of this is that people have forgotten quite why fact-checking was introduced, and what it did and didn’t do, and whether even the fact-checkers thought it was effective.

For this, we have to roll back to 2016, when there were elections in the Philippines, the Brexit vote in the UK, and a US presidential election fought between Hillary Clinton and Donald Trump. Each of the three events saw worse and worse misinformation spreading all over Facebook, including fake news spread by foreign sites looking for a quick buck and Russian-backed sources looking to get Trump, their pick, installed.

Everyone else found the way that Facebook had allowed itself to be used astonishing and infuriating, but Zuckerberg professed to be astonished at their astonishment. As I noted in Social Warming:

‘The idea that fake news on Facebook, which is a very small amount of the content, influenced the election in any way, I think is a pretty crazy idea,’ said Zuckerberg, then thirty-two, at the Techonomy conference in San Francisco. ‘There is a profound lack of empathy in asserting that the only reason someone could have voted the way they did is because they saw fake news.’

That was just four days after the election result. But a week after that, Facebook decided that maybe fact-checking was a good thing after all, and in December the Facebook executive Adam Mosseri, in charge of the News Feed, announced that

in future users would be able to report content as ‘fake news’, an addition to the previous options of ‘annoying/not interesting’, ‘shouldn’t be on Facebook’ and ‘spam’.

Mosseri explained that ‘We’ll use the reports [of fake news] from our community, along with other signals, to send stories to these [fact-checking] organisations.’ Those would then decide if the stories were untrue; if so, Facebook would flag them as ‘disputed’, and downrank them in the News Feed algorithm.

The first fact-checks began appearing in March 2017. But they were hardly a panacea. As Brooke Binkowski, then working at the fact-checking organisation Snopes, explained to me, fact-checking Facebook was a life sentence in the content mines with no chance of parole:

‘They would provide us with this list, for each of us personally, that was hooked into our personal Facebook accounts. They showed you a URL to a story, and we would mark whether we had found it true or false, and put a link to our own debunking of it. Then move on to the next one.’ There was no quota, though there were usually at least 250 stories to be refereed, divided into ten pages of twenty-five. The list of stories was unending, since you’d never run out of people being wrong on the internet.

‘The cup was continually full,’ Binkowski recalls.

This is the thing that everyone overlooks about fact-checking on Facebook: it was hellish. It was a source of revenue. But also a source of near-despair:

Facebook wasn’t forthcoming with [Binkowski] about the criteria by which stories were chosen, and seemed powerless to prevent vari- ations on the same story being presented again and again.

The problem, Binkowski came to realise, was that this was an asymmetric war. Within the parameters Facebook had set, there was no way to stem the flood of misinformation, disinformation and untruths. The penalties for posting fake news were minimal, because the content wasn’t deleted from Facebook, only ‘down- ranked’. By contrast, the potential rewards from spreading it were substantial, and generating new versions of the same story (which looked to Facebook’s algorithms like entirely new content) by writing a new headline and tweaking a few words and a picture was cheap.

Facebook brought on four fact-checking organisations as partners.They stood as much chance as Canute. Looking back on the experience later in an article for Buzzfeed News, Binkowski described it as ‘the world’s most doomed game of whack-a-mole, or like battling the Hydra of Greek myth. Every time we cut off a virtual head, two more would grow in its place.’

But but but! Fact-checking was a really good source of income for the fact-checkers. Before Facebook came along, Snopes had been reliant on its reputation (which emerged, weirdly, back in the Usenet days of alt.folklore.urban, the Urban Folklore newsgroup, devoted to debunking urban myths back when the internet was young) to get visitors who wanted to determine the truth or falsity of some claim or other. The revenue came from adverts. But with Facebook demanding an outside source which could do that for millions of posts per day, the opportunity for Snopes and similar organisations burgeoned. In 2018, one-third of Snopes’s $1.22m revenue came from Facebook’s fact-checking program.

But of course Facebook won both ways:

Facebook benefited from the engagement that the fake-news sites produced, as bamboozled users clicked on stories with headlines such as ‘Babysitter Transported To Hospital After Inserting A Baby Into Her Vagina’ and ‘Morgue Employee Cremated By Mistake While Taking A Nap’, while also burnishing its reputation by pointing to the fact-checkers’ efforts, and getting to show adverts to users.

Not only that:

In fact, the fact-checkers’ efforts probably had the opposite effect from what was desired. In September 2017, a team at Yale, MIT and the University of Regina had found that while putting a ‘disputed’ tag on stories made people only marginally less likely to believe them, the lack of a tag on other stories would make them assume these stories had been checked and were correct—what the researchers called ‘implied truth’. Given the average three-day lag in picking up on fakes, that could have boosted rather than quelled the fake-news boom.

Three days! Hell of a long time for a fake news post to be up, unmarked.

Yet even the fact-checkers could see that this was far from the ideal method of searching out and identifying junk. Binkowski, who had been on the internet for years before Facebook (b.2004) even existed, knew there was a better way:

She suggested, in at least one of the conference calls with Facebook, that the site could do just as well or better by recruiting more of its own users to moderate content: some people like the status of being a moderator, and by crowdsourcing to cross-check their accuracy, the best moderators would emerge naturally. ‘I remember when I was on IRC [internet relay chat, one of the earliest interactive internet forums] we had moderators, and we did it for fun, and the community moderated itself. I kept telling Facebook that. I was all, like, “Hey guys, listen to the old person, listen to the forty-year-old, the Gen Xers who have been on there this whole time, about moderators.”’

That sounds kinda like Community Notes, doesn’t it? People are chosen from the wider community, their opinions are cross-checked to see if they agree, and if they do then the Note appears on the post.

Let’s just sum this up: for the fact-checkers, the checking process was a nightmarish, never-ending process where they could never catch up, never get ahead of the game, and where Facebook anyway rigged the game through the algorithms it used to pick posts for examination, the “demotion” (note: not deletion, no matter how wrong!) of posts benefited Facebook rather than the fact-checkers, and the incentives were all perverse. And it was obvious that the Facebook users were more motivated to do such checking: the idea of the community policing itself is popular with users and checkers.

So now the idea is to move to a “Community Notes” model. Incidentally, Google has told the EU that it is going to do much the same for some videos, and that it’s not going to sign up for the EU’s Disinformation Code of Practice. Everyone seems to have decided almost simultaneously that checking facts is less fun than it used to be, especially now that the New Hotness is AI chatbots (Meta has them which are being used to generate content on its networks—for engagement, natürlich; Google has one; Apple has one for the iPhone) where “facts” are the most liquid of ideas.

All this would be fine, except for the key problem with the whole Community Notes model: while it’s very good for accuracy, it’s too slow to display a Note, and most Notes aren’t seen. The LSE blogpost from 14 January in the above link is very good in pulling together the most important elements, and it’s worth reading in full. Overall, we can’t entirely trust Community Notes because we don’t know how many people have to add their weight to a draft Note before it gets published, we don’t know how people are selected, we don’t know how many Note writers/approvers there are, we don’t know what their biases are.. it’s a very long list of unknowns. And that’s just on X, which is perhaps one-fifth the size of Facebook or Instagram. How many people do you need writing Notes on Facebook or Instagram or Threads? How many to approve it? It’s all up in the air.

So yes—Zuckerberg’s networks had created an intolerable situation for the fact-checkers. But there’s no promise that the new situation, even with the best will (and the most eager, at first, users) in the world, will actually create a better system where fake news, mistakes or outright lies get labelled and demoted (or even better, deleted—why keep stuff you know is wrong?). In fact it’s very easy to imagine that Facebook is quickly going to degenerate into a morass of AI-generated slop vigorously mixed with lies and fake news.

Sure, Community Notes are great. And to quote CP Scott, the former editor of the Guardian, “comment is free, but facts are sacred.”

The trouble is, everybody’s lost that religion.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

Subscribe now

One of my favourite episodes of the comedy series Modern Family is one where Claire Dunphy (one of the mother-wives in the group) goes back to her college, and is confronted with her past, in front of her husband Phil:

Claire: It's just- It's a little bit embarrassing. The guy that I dated right before you is here, so...
Phil: Oh. Well, that's nothing to be embarrassed about. We all have exes. So the guy you dumped right before you met me is here. No big deal.
Claire: [high-pitched] Well... He might've dumped me.
Phil: I thought you said you'd never been dumped.
Claire: [high-pitched] Well... Maybe one time.
Phil: So he dumped you, you healed completely, and then you met me.
Claire: [high-pitched] Well...

That high-pitched “Wellllll..” is the sort of sound you can imagine lots of people making in lots of dramatic setups. Is this hotel you’re taking the family to in the depths of winter where you’ll finally write your book safe? Wellll… Is that unknown planet to which your spaceship has been diverted by a peculiar radio signal from an odd alien ship safe? Welll…

Anyway, to bring things closer to the general topic here, is burgeoning social network Bluesky going to be a place where the rampaging hordes who made Twitter unbearable for some are finally left behind, and everything will be sweetness and light, enabled by starter packs and federation and especially blocklists (which no longer work on X)?

Wellll….

This week Bluesky basically lost its collective mind over the arrival of Jesse Singal, an American journalist. He’s basically a New York Democrat, but the fly in the ointment for Bluesky’s denizens is that he has written extensively on transgender issues, particularly about the science involved, and hasn’t always been supportive of what Democrats—or the left—might think is the consensus. From my point of view (I know him and subscribe to his Substack) he’s simply rigorous, and not ideological.

My point though isn’t about his journalism, it’s about the reaction. When he created an account there last Friday, people went, simply, bonkers.

Really? Hammers? Doorknobs? But it got wilder and wilder (which Singal documented on his X account in a laconic yet amazed fashion). People added him to blocklists, so that they wouldn’t have to see his posts. Job done, you would have thought. But no: that wasn’t enough. They created blocklists of people who followed him.

Next step: harass the head of moderation at Bluesky, Aaron Rodericks, demanding he remove Singal. What for? Existing, dammit! OK, if not existing, then being a paedophile! (This is a completely made up accusation.) OK then, an FTC class action that will be taken to the Supreme Court! (I hope you can tell this is made up.) OK, just murder (there was some discussion about whether he should be buried in a pine or a spruce coffin.

Next tactic: accuse Rodericks of being a paedophile. Not really sure they had thought this through. Which meant it was back to discussing methods of murder. Shooting? Beating? (None of this was moderated, of course.)

Next, probably related: an exploit which enabled a suitably competent hacker to attack a BlueSky account to prevent its posts loading.

Latest: an open letter on “the Singal situation” and his “repeated deliberate violations of Bluesky Community Guidelines.” (Spoiler: he hasn’t violated them.) At the time of writing, it has nearly 14,000 signatures demanding his removal for three alleged infractions, only two of which are claimed to have happened on Bluesky, and which when you follow the provided links, simply don’t support the claim made. Sure, that’s only a quarter of the number who had blocked him, but for nearly 15,000 people to be incapable of parsing some text and thinking “uhh, what?” is perhaps concerning.

Especially when one of them is.. Lizzo? Who doesn’t even know who he is, but is sure she agrees:

So any suggestion that Bluesky is the kinder, gentler social network can be quietly put to bed.

But there’s a more interesting question. Why is Singal the target? After all, he’s a Democrat, and Bluesky is, essentially, being used as a refuge by many left-leaning Americans: the exodus from Twitter after the US election was matched by the influx to Bluesky, whose numbers rocketed.

So why are they attacking him (and Wu, a trans woman who has expressed concern about the effect on voters of the Democrats’ stance on transgender issues)?

Colin Wright, who runs the Reality’s Last Stand site, summed this up very neatly in a post on X:

The frothing hatred for jessesingal by Bluesky users might appear odd given that he's a left-wing moderate. I'm more right-wing than Jesse and talk about many of the same issues. So why is he treated so much worse than me by this group?

It's because ideological boundaries are often policed more fervently against those closer than those further away. Heretics, because they are seen as internal threats to a group's unity, are frequently treated with much more venom than apostates or infidels.

It's also because those who are ideologically closer to them are usually more easily manipulated due to fear of social exile. It's the dynamic of a struggle session.

Fortunately, Jesse financially supports himself on Substack with articles and podcasts. But because they can't pressure an employer to get him fired and ruin him financially, they're stuck in a perpetual and increasingly unhinged outrage loop about him.

This is a great insight. Tribalism is never far from the surface in any social network. And for a tribe, the most important thing is keeping the tribe together, because if you lose the cohesion, you lose everything. The people on the fringes, who are the ones beginning to edge away, must either be drawn back into the group or—if they refuse to mend their ways—be evicted. If the tribe’s cohesion depends on strong shared beliefs, then the eviction must be done in the most dramatic fashion.

There’s a fantastic line in the ITV drama “Everyone Else Burns”, which is set in a small and very strict religious sect in Britain’s Midlands. In S1E1 the leader of the Church is explaining to the congregation why a family they used to know has been excommunicated:

“I have some rather grave news to conclude with. The Jay family are no longer Ordinands of God. They’ve been excluded. I can’t go in to what they did. But… a reminder that drug dealing is a mortal sin.” [Pause, while you think blimey, drugs eh, huh.]

“And of course,” the leader continues, “we do consider coffee a drug. There is only one way now for us to extend our love for them, and that is to shun them completely.”

One woman in the audience whispers, in an aside to her neighbour, “They knew what they were doing when they opened that café.”

Shun them completely! It’s the only language they understand, and it’s the only way to keep the tribe together. Apostates can’t be tolerated; they have to be reviled and cast out. It doesn’t greatly matter what they’ve done; what’s important is they’re out. Those who are closest to the border of belief and disbelief are the ones most liable to infect the others with their disbelief; it is the doubters who carry the seeds of the destruction of the group.

Once you start to recognise the pattern, you’ll see it everywhere: in politics, among sports fans, religious groups, specialisms, whatever niche you care to examine. The dissenters bring on a desire in the group to gather some kindling and tie the troublesome one to a pole in the middle of the town square. (Pine or spruce, doesn’t matter.) Or, failing that, write a really long open letter.

Share

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

Subscribe now

On Wednesday morning in New York, the chief executive of a gigantic health insurance company was gunned down in the street. His killing was no accident. CCTV footage shows the CEO, Brian Thompson, walking confidently along the street, away from the camera, when a man (judging by height and build) appears a few yards behind him, pulls out a gun, and fires a shot. The bullet hits Thompson in the calf; he staggers to his right, behind a concrete pillar protruding from a building. The killer then resets his gun and fires another shot. A bystander who had been in front of the concrete pillar takes in what’s happening and sidles away as quickly as they can; the killer takes no notice. He resets his gun again, fires again, and runs away between the cars.

Imagine this had happened to a dog: people would be appalled. There would be a massive outpouring of sympathy for the dog, and a concerted hunt for the shooter.

But the target in this case was human, and while there is certainly a concerted hunt for the shooter—the NYPD has released photos and is piecing together what it can about his movements before and after (a visit to Starbucks? A Citi bike from the scene? Bullets left at the scene engraved with the words “delay” and “deny”?) as they try to find him—you’d have to say that the dog would have had it better than Brian Thompson.

Turns out there are lots of Americans who use its private healthcare system who are not fans of its big insurers and their ways of making money.

For example, look no further than Taylor Lorenz, oft-fired “internet writer” who has complained long and loud about people’s lack of respect for immunocompromised people and long Covid sufferers. Here’s the headline of her latest Substack:

I mean.. pardon?

The first paragraph explains the detail I’ve given above. The second paragraph:

Within seconds of the news breaking, people online began celebrating. A Facebook post by UnitedHealthcare about the CEO's passing was met with over 23,000 laughing emojis before it was taken down. "Health insurance companies are parasites siphoning blood money from the sick, dying and injured,” one user posted. “I'm only surprised it hasn't happened sooner.”

At the time of writing, we don’t know precisely why Thompson was killed. It’s even possible that it was a mistaken identity; or that the killer was mentally ill and acting semi-randomly against someone who looked flush. (The semi-professional nature of the killing leans against those; the amateurish leaving of bullets—with fingerprints?—perhaps strengthens them.)

Which hasn’t stopped people of all stripes going on to social media to express their views.

Professor Wilson has deleted that Bluesky post (sensible!) but one might hope that an associate professor in the Department of Health Care Ethics at Saint Louis University would be able to articulate the ethical dilemma of how one should feel about citizens who have committed no crime being gunned down in the street slightly more eloquently.

People in the UK and other countries which have universal health care (even when, as in the UK, it is creaking badly) do have trouble understanding the intense antipathy that many Americans feel towards health insurance companies. For all that American salaries are higher, their medical costs are also significantly higher too.

Not only that: health insurance companies find all sorts of ways not only to bill people, but to deny them coverage, or tack extra costs on. This has, for years, quietly enraged a huge number of people. A few years ago I did an analysis of how the Fortune Top 50 had changed over a period of about 30 years, simply based on the capitalisation of the companies (which is the stock market’s prediction of their total future profits). The big change was that automakers and conglomerates had fallen away, and been replaced by tech firms—and health insurance companies. And those health insurance companies didn’t get that big by happily signing cheques (they still do, in the US) for every payment demand.

The eruption of bile towards Thompson, his company, and health insurance companies generally, as news of his killing spread, was a prime example of social media providing both an outlet for a feeling that had been rumbling away in the background for decades, and a magnifying glass to focus all the warmth into a burning heat. Here’s Taylor Lorenz again:

After I posted a quote tweet about insurance companies no longer paying for certain anesthesia with the phrase, "And people wonder why we want these executives dead," legacy media outlets including Fox News pounced and wrote a slew of articles about my "calls for violence."

I mean.. yes? If Lorenz is saying that she and others want these people dead, when one of these people has been violently made dead, then yes, that is a call for violence. It’s basically putting a target on the back of every other healthcare executive, and it’s astonishingly irresponsible for someone who fashions themselves as a journalist in any way to do so.

But this is what social media does: it finds the magnet of opinion with the biggest pulling power and yanks any iron filings who are oriented that way towards it, where they pile up in a prickly, angry, heap.

We see it happen again and again—whether it’s the ridiculous bile aimed at a successful PhD student at Cambridge University (who, happily, seems to have come through unscathed, or even buffed up) or this week’s Middle East calamity—but the targeting of a very visible group seems like a bad step, and its enabling by people ranging from journalists with big followings to, in effect, academics, is not a good development. This is definitely social warming, and the fact that it was in evidence on all sorts of social networks, including Facebook, X and Bluesky (the kinder, gentler.. ah, forget it), is a worrying trend.

Social media has never been able to solve anything at any scale. It’s good for small things: uniting people with lost wallets, finding homes for lost dogs, shaming people whether deservedly or not. But the big movements, like petitions to stop Brexit, or call elections, fail miserably, because when a million people shout at once, you can’t make out what they’re saying. Once a choir becomes a chorus, at about 80 people, you’re heading into “unmanageable” waters. Rather like the Dunbar number of 150 “friends” you can sensibly stay in touch with, there’s a certain mass of social media users above which nothing actually gets done by the group.

But: individuals can act. And that’s the problem. We do know that there are some properly unhinged people out there; given that the killing of Thompson was not, by all appearances, a professional one, that means it was an otherwise ordinary person.

To quote one of the people in the manhunt:

“This is a solid investigation and I’m confident that he’s going to be an ex-client, an ex-employee or a conspiracy theorist. But it’s insane levels of labor to find all these different things.”

That third one is the worrying one. I’m sure that every executive at every healthcare company has changed their routine in the past day, and gained a security detail. Because you just don’t know what kind of crazy person might be out there, and the US healthcare system has driven plenty of people to the very edge of despair—far enough that they might tip over into murder1. Social media isn’t helping. And the lack of moderation, and self-moderation, isn’t helping either. Let’s hope that Thompson’s death is the last, not the first. Because otherwise we’re going down a slope with very bad things at the bottom.

Share

Glimpses of the AI tsunami

(Of the what? Read here. And then the update.)

• Google DeepMind has developed a system that will produce endless 3D worlds, based simply on a text prompt. They envisage it being used to train AI agents (rather than amusing humans).

• The confusing world of AI girlfriends. Interesting (though paywalled) article at The Verge.

• How close is AI to human intelligence? Nature explains. (An unknowable distance, really.)

• The mystery of the names that ChatGPT can’t say. Possibly due to the European Right To Be Forgotten? Nobody’s actually clear about this.

Subscribe now

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.

So they did it: the Australian parliament passed a bill that will ban under-16s from having an account on social media.

Well, no wait. Not exactly “ban” or “account” or “social media”.

the legislation defines an “age-restricted social media platform” as including services where:

• the “sole purpose, or a significant purpose” is to enable “online social interaction” between people

• people can “link to, or interact with” others on the service

• people can “post material”, or

• it falls under other conditions as set out in the legislation.

The legislation does note that some services are “excluded”, but does not name specific platforms. For example, while services providing “online social interaction” would be included in the ban, this would not include “online business interaction”.

Not to mention that this won’t come into force for a year. Which means it’s a ban on under-15s, really, since there’s a year’s worth of teenagers who will be wiping their brows. But there’s also the question of those who right now who are able to sign up to social networks, and will do so (because presently the lower age limit is 13). So there will be two years’ worth of signups whose accounts will abruptly be illicit in a year’s time.

How did Australia get here? Why on earth does it think this is going to work? Are there any other examples?

Let’s start with the last one.

Apart from gazillions of parents around the world who have desperately told their children to “how many times have I told you to put down that phone!”, the idea of “get those damn kids off the intertubes” has been tried, very prominently, in South Korea.

Welcome to the “Youth Protection Revision Act”, introduced into law there in November 2011. It “forbade children under the age of sixteen to play online video games between the hours of 00:00 and 06:00.” Its genesis was in 2004, with civic groups complaining that kids were spending too much time online, and missing sleep. (Parents: this is familiar, isn’t it.) There were attempts in 2005, 2008 and finally in 2010 they got it across the legal line.

The shutdown law went into effect on 20 November 2011. It was applied to every online game in service in South Korea. Teenagers under seventeen years of age were not allowed to play online video games between the hours of 00:00 and 06:00. The law affected some online social games and every online game service that required a resident registration number.

Let’s see, what did we expect would happen? All the children would throw down their controllers at midnight and sigh and rest their heads on their pillows? Or..

The law has led those under sixteen to commit identity theft—underage South Koreans stole resident registration numbers in an effort to elude the law. The shutdown law targets online games, but does not affect console games and mobile games. However, due to the difficulty in enacting the law in some cases, they decided to entirely ban all games of some companies, such as Xbox Live or PSN. Legal challenges against the law were filed by a group of Korean game manufacturers and a cultural organization. After 2 September 2014, parents could request that their children be exempted from the law.

So there were exceptions almost from the start, and the kids figured out ways around it. There were so many holes that it was doomed from the start. The law was repealed in 2021, more ignored than observed. Interestingly, the law did survive a court challenge, which ruled that it was indeed an interference with the children’s freedoms but that being addicted to video games was bad for health.

But don’t worry! Japan had a try too, with Kagawa Prefecture in April 2020 passing an ordinance which

calls for families to set rules on playing hours for children and recommends limiting video game playing to 60 minutes per day on school days for those under 18 years of age and 90 minutes on non-school days.

The ordinance also advises limiting online game playing to until 9 p.m. for junior high school students aged between 12 and 15 and younger children and until 10 p.m. for older children. Guardians are urged to make efforts to make sure their children observe the rules, though the ordinance has no penalty provisions.

The ordinance survived a court challenge:

the plaintiffs argued that there is no scientific rationality in the purpose or spirit of the ordinance because the government has told parliament that it was unaware of any scientific basis for the purported effectiveness of imposing time limits to prevent gaming addiction.

The prefectural government disputed the plaintiffs' claims, noting that the World Health Organization has recognised "gaming disorder" as a syndrome.

And this brings us to the Australians. The argument being made is that this legislation is imposing a “duty of care” on the tech platforms, which (they say) is the same as the UK and EU are doing. Well, yes, but you folks in the Antipodes seem to have a different view of quite what that means.

You might expect that as the person who wrote a book about implicitly bad effects from using social networks would be in favour of banning their use for younger users. In fact, I’ve pointed to the harms that social networks (notably Instagram) have done to younger users a number of times here. The classic example is Molly Russell, the British girl whose death by suicide at the age of 14 in November 2017 after she’d been exposed to disturbing content on Instagram mobilised a serious movement to ask what, precisely, social networks were doing to their youngest users.

In that sense, yes, social networks have serious downsides, and they can affect some users in the worst possible ways. They can also be exploited by all sorts of malicious actors to get children to do the worst possible things.

At the same time, lots of children get real benefit from using social media. The data suggests that there’s a bathtub-shaped connection between time spent online and unhappiness: those who spend no time on it are unhappy; those who spend a reasonable amount of time on it were various amounts of happy; those who spend a lot of time on it are unhappy. The question is whether the time spent leads to the unhappiness, or it’s because they’re unhappy that they spend so long online. But we do know that there’s that group down at the bottom, with zero online time, who tell people that they’re unhappy.

Yet Australia is planning to shift a ton of its kids from all those sectors down into the “unhappy” group. This seems to go against common sense, given that we have the research showing how kids feel about using social networks (mostly: good, actually) and the experience from South Korea showing that kids will figure out a way around it given the challenge. If you’re the one kid in the class who hasn’t figured out how to get onto TikTok in 2026, you’re not going to feel happy, unless you’re very self-possessed. We can expect rocketing use of VPNs and all sorts of other dodges by Australia’s youth. If the Australian government actually had a secret plan to raise a generation of hackers and internet-super-capable adults, this would be exactly the way to do it. (But of course if that’s their secret plan, they’ll never tell us. Damn!)

The other point, made by a group of experts and organisations who wrote to the government opposing the bill, is that this doesn’t put a duty of care onto the tech platforms, because they don’t have to take any care that their content is appropriate for those under 16. They just have to keep them off their systems through age verification. (How? TBD, like almost everything.) It’s unclear whether YouTube has to obey this too. Though if it doesn’t then the law isn’t worth the paper it’s written on.

But look, it’s going to be fun watching the Australian government trying to figure all this out. And all the South Korean 20-somethings are going to be watching with interest.

Share

Glimpses of the AI tsunami

(Of the what? Read here. And then the update.)

• ChatGPT (mis)represents publisher content, says the CJR. We’re not surprised, are we? Turns out it got some element of the answer wrong in more than 150 of 200 queries.

• That viral LinkedIn post was probably AI-generated. Even more reason not to go there.

• AI-generated poetry is indistinguishable from (or even preferred over) human-written poetry. Damned humans.

Subscribe now

• You can buy Social Warming in paperback, hardback or ebook via One World Publications, or order it through your friendly local bookstore. Or listen to me read it on Audible.

You could also sign up for The Overspill, a daily list of links with short extracts and brief commentary on things I find interesting in tech, science, medicine, politics and any other topic that takes my fancy.

• Back next week! Or leave a comment here, or in the Substack chat, or Substack Notes, or write it in a letter and put it in a bottle so that The Police write a song about it after it falls through a wormhole and goes back in time.