Social media brain rot for AI

I recently read a super interesting, yet slightly terrifying study, by researchers at the University of Texas at Austin, Texas A&M, and Purdue, which shows that Large Language Models (LLMs) can suffer permanent cognitive damage when they're trained on low quality, high engagement social media posts.

The term low quality, high engagement social media posts, pretty much just means posts which are not factually correct, based on real data or verified, but receive a lot of validation in the form of engagement. The types of echo chamber posts we often see when "doom scrolling", which receive lots of likes, comments and shares, and often perpetuate misinformation or conspiracy theories.

The concerns

Using “viral” content from posts on X, which we all know is widely regarded as a bit of a cesspit these days, researchers found that models trained exclusively on content that was low in quality but high in engagement, reduced in accuracy from 74.9% to 57.2%. Their long context understanding was also affected, reducing from 84.4% to 52.3%.

What’s really concerning, is that the models began “thought skipping”. This means they cut corners when attempting to use their reasoning capabilities. Through monitoring, researchers found that they also started to score higher for narcissism and psychopathy traits, and score lower on “agreeableness”. In essence, the behaviour and personality of the LLM was being altered or warped, by the content it was reading online.

Attempts to fix the LLMs by retraining them with high quality data didn’t work. The viral posts appear to cause permanent damage to the LLMs internal knowledge-base, and any attempts to fine-tune it doesn’t seem to be able to completely reverse it.

The damage is very… human

What’s fascinating, or perhaps horrifying, is that LLMs have been designed to be so… human… that they seem to even experience the same brain rot and reasoning regression we’re seeing in people, who are often being manipulated and conditioned using the same awful content. And, attempts to re-train or "rehabilitate" a damaged LLM appears to also have a weirdly similar outcome as it does in humans.

As somebody who has been through a lot of therapy, it feels eerily close to home for me. Like us, it appears an LLM can be re-trained, to a point. But, the damage can't be completely reversed. It can learn to work around the trauma to some degree, but it's always there. It's insidious, and continues to have an impact on any interactions going forwards, especially when under pressure or attempting to cut corners.

The researchers warn we’re potentially headed towards a “Zombie Internet”, where LLMs are training on dodgy content, generated by other LLMs which are already compromised. Creating a sort of, super-echo-chamber, and fully embracing the term “viral content”.

Final thoughts

When I worked at Elastic Security, it was forshadowed that there would come a time when LLMs would be used to generate thousands of pages of misinformation, post it out into the world, then inadvertantly use it's own poor content to re-train, verify information, or have it's behaviour manipulated in the future. A sort of, dormant, long-game attempt at prompt injection.

So yeah, I guess the long and the short of it is, we need to be more vigilant than ever when it comes to thinking critically and verifying information! Oh, and X continues to be cesspit, and a terrible source of information, for both people and machines!

Thanks,
Craig

Sources

## Further reading

Published:
Read time:
2 minutes
Tags:
ai

Browse all tags