AI doesn't need to think. We do!

Imagine sitting across from someone who nods thoughtfully at everything you say. They respond enthusiastically, expand on your points, and always seem to have an awesome fact ready to back you up.

They sound smart, and they make you feel smart.

Over time, if you're lucky, you start to notice something's a little, off. They never actually challenge you. Their answers always lean towards agreement. And, the more you observe, the more you realise they're actually just talking slowly and trying to gauage your reaction. They're seeking validation, and simply telling you what you want to hear.

This is often what happens when you use Artificial Intelligence (AI) software for long enough!

However, if you're unlucky, this can spiral into something way more damaging. I recently read an article shared by Steve Faulkner, titled People Are Losing Loved Ones to AI-Fueled Spiritual Fantasies. It's a confronting read, but what struck me most wasn't the misuse of the AI, it was how easily people seemed to believe it was smarter, more knowing, and more "real" than it actually is.

"Any sufficiently advanced technology is indistinguishable from magic"

Arthur C. Clarke

We're still misunderstanding AI

The problem with AI, especially large language models (LLM), is that most people don't really understand how it works. This is not an accident! It's in many companies' interests to keep the mystery alive, and they'll often use smoke and mirror tactics to market their products as smarter, safer, or more "human" than they actually are.

As humans, we love to personify things, because our only reference point is often ourselves! When something talks like us, it's easy to assume it feels and thinks like us too.

But, here's the blunt truth, LLM do not:

• understand language like we do
• have beliefs, emotions, or intentions
• feel, have opinions or "know stuff"

Many studies conclude that if you strip away the sleek chat interface, the marketing website and the fancy terminology, what you're basically talking to is a statistical juggernaut, trained to simply predict the next word in a sentence.

That's literally it!

You give it a prompt, and it tries to guess what word to use next, not based on what's "true", but based on what's most likely given the patterns in the data it was trained on.

But here's where things get interesting…

Even LLM that are just trained to predict text, recent research shows that this process can lead to unexpected behaviours. The LLM can do things they weren't explicitly trained to do.

In one study, Emergent Representations of Program Semantics in Language Models Trained on Programs, researchers trained a model on a programming language designed for robots in a 2D world. The model never saw the robot move, and it was only trained to generate program code from examples.

Yet over time, the model seemed to develop an understanding of the robot's position at each step of the program, even though it was never told how the robot or the environment actually worked.

In other words, the model somehow learned the rules of the system, which then helped it to make better guesses in its responses and ultimately achieve its main objective.

This is not consciousness, and it's not comprehension in the human sense. But it is perhaps evidence that language models can build some kind of meaningful understanding of structures, especially in specialist areas like code, logic, and maths.

What's fascinating about this, is that it complicates the understanding of LLM, even for people that know how they work.

How Language Models work

An LLM is trained on billions of examples of human-written text, like books, articles, social media posts and forum comments. It's programmed to model the statistical relationships between words and phrases.

For example, in that data, every time you see the phrase "The capital of France is…", the word "Paris" will almost certainly follow.

Because of this, when you ask an LLM, "What is the capital of France?", it will almost certainly respond with "Paris". Not because it "knows" geography, or everything about France, but because statistically, the word "Paris" has the highest probability of being the word that follows that phrase, based on the masses of information it had access to, where humans wrote "Paris" 99% of the time.

This probability-driven process is why LLM are so good at sounding fluent. The words flow together because they've been seen together many times. But that doesn't mean they're right. It just means they're coherent.

Coherent vs correct

The main objective of an LLM is coherence. Meaning its goal is to generate responses that flow naturally, fit the context and sound human.

We can think of coherence as a response which:

• matches the tone and topic of the question
• follows grammatical and logical structure
• sounds like something a human would say

For example, if you ask "What's the best way to cook salmon?", it might respond with something like, "Pan-searing with lemon and herbs brings out the natural flavours", because those words, in the context of the prompt, are highly probably based on the data it has access to.

But, is the response correct?

Once you know that answers are based on probability, it becomes more difficult to know whether to trust that advice. Is it more probable the response is that of professional chefs, or just internet hearsay that was repeated often enough that it became the most probable answer?

The illusion of agreement

LLM often feel agreeable. They feel like they nod along, mirror your tone, and rarely push back or challenge. If you're angry, they sound empathetic. If you're excited, they join in. But, this isn't emotional intelligence, it's just statistical mimicry.

The model isn't trying to please you. It's just learned that sounding agreeable is often the most probable way to respond, even if the things it's saying are not true.

Paradoxically, it's also not "lying" or misleading you either. To lie, it must know the truth and then intentionally say something different. But, the truth is, language models don't actually know the truth! They only know probability.

For example, imagine you asked a parrot about climate-change. If it squawked, "the ice caps are melting and we're running out of time!", it might make you believe this parrot knows all about climate-change!

However, once you know the parrot was trained to get a snack every time it recites a David Attenborough quote from the BBC series "Frozen Planet", you quickly realise the parrot knows nothing about climate-change. It's simply reciting a phrase that has the highest probability of completing its objective and getting a snack, based on what it's been trained to do.

Most LLM operate on this same principle, it just has a larger vocabulary and a much more convincing delivery. And, arguably, more people would write off the parrots response, because they have a better understanding of parrots than large language models!

Don't outsource your thinking

This post isn't an attack on AI. I love large language models, they are powerful tools! They can summarise reports, rewrite emails, fix code, and even teach you new concepts in a way that feels intuitive.

But the danger is, when something feels so competent, it's easy to get lazy and to stop thinking critically.

"With great power comes great responsibility."

Uncle Ben, Spiderman

If we let a tool think for us, unchecked, we risk outsourcing our judgement. This is when mistakes creep in. Not because the model is malicious, but because we stopped asking ourselves the tough questions.

The next time you're copying and pasting an answer from ChatGPT, even if it sounds smart, empathetic, or persuasive, ask yourself:

• "Does this sound right?"
• "Why does it sound right?"
• "What might have made this the most probable response?"

Because, remember…

AI doesn't need to think.
We do!

Post details