Oh, Will Laplace's Demon Descend Upon This Earth?

Wittgenstein once said, "The limits of my language are the limits of my world."

What can be perceived can be described in language.

Conversely, what can be described in language can be perceived.

Within the paradigm that has prevailed until now, this proposition holds sway. At least, that's how I see it.

Communication requires language as a common medium.

We need SYMBOLIC signifiers capable of carrying opinions, thoughts, and the like,

and these signifiers must imply a signified that can be recognized as meaning.

Here, regarding "language" as the medium of cognition,

I'd like to define it, in my own way, on two levels.

Language consists of the Symbolic "signifier" and the Semantic "signified."

(Of course, any interpretation of the signified must also be expressed through a signifier.)

This is an extremely important point.

Because it means humans can perceive the signified that lies beyond the signifier.

In other words, we are beings who inherently possess something like a "signifier interpreter" model,

which means it should be possible to model this.

( Of course, even if humans "understand" the signified, that doesn't mean it can be fully "formalized." )

Most of today's LLMs

operate within a vast vector space that forms a semantic distribution,

converting embedded signatures into natural language as output.

When deep learning first rose to prominence with the "AlphaGo shock" of the 2016 match against 9-dan Go player Lee Sedol,

games like Gomoku and various others could have their solutions constructed as algorithms,

but the world of Go was thought to be so vast

that no winning algorithm could ever be derived for it — and that human belief was shattered in that moment.

At that time, humanity felt afraid.

It seemed like a day might come when AI would rule over humans.

Yet deep learning, for all its impressiveness, is actually quite simple:

if you train, on an enormous scale, on the correlation between the sequence of stones placed in Go and the win rate,

you can determine the win rate depending on where the next black (or white) stone is placed. This is precisely the vector space.

Move 78 in Lee Sedol's fourth match? That was a gray zone within the vector space.

It was a kind of bug — faced with that strange move, AlphaGo could no longer calculate a win rate.

In other words, AlphaGo didn't understand Go;

it merely possessed an "extremely, extremely vast" answer key for which move yields the highest win rate.

If you memorized all the answers to a math workbook, you could solve every problem in that workbook.

But that hardly counts as understanding mathematics, because you couldn't answer a problem for which there's no answer key.

Still, suppose you learned the answers to a set of workbooks so vast in scope that it's beyond imagination,

the belief that you could then solve every problem out there in the world —

that is precisely deep learning, and it can be called the upper bound of what is computable.

It wouldn't be an exaggeration to say that today's LLMs represent the limit of natural language processing technology built on vector spaces.

GPT := Generative Pre-trained Transformer

It means a [Generative - Pre-trained - Transformer] model.

Quite literally, in order to construct a vast vector space,

it was pre-trained on enormous amounts of data, just like AlphaGo's deep learning.

And this model operates much like AlphaGo's win-rate prediction algorithm.

The Transformer model is built so that, when our natural language is broken into tokens and given as input, it constructs the sequence that completes it.

Through this, it's made to look as though it is "generating" speech.

There are various techniques built into the model, such as hyperparameters, LRHF, and so on.

In particular, LRHF is designed to assign greater weight to responses that people have "liked."

Through this reward system, the model comes to be endowed with a kind of personality.

This is connected to why, these days, a single AI model can't handle every kind of task, and why specialized models exist instead.

The nature of the training data, the personality types of the people using it, and so on —

it's no exaggeration to say that a variety of contexts and orientations (statistical biases) influence the personality type of an AI model.

In the end, AI is

a modeling of the statistical distribution of the "signifiers" that "beings capable of interpreting the signified" have contextually produced —

that's what it means.

Rather than saying this granted it the ability to interpret context,

it would be more accurate to say, "In this context, expressing it this way is 'statistically' appropriate."

So then, driven by the belief that training on an extremely vast scale might let it know everything,

AI developers are focused on improving model performance.

(Let's set aside optimization techniques like RAG (Retrieval-Augmented Generation) or input parameter quantization for this discussion.)

If AI's performance were pushed to its absolute limit, could it truly know the answer to everything?

(We have to think on an astronomical scale here — just as the distance to the Andromeda galaxy, roughly 2.5 million light-years, is a scale we can't truly picture.)

Have you heard of the concept of Laplace's Demon?

AI developers sometimes seem as though they're striving to summon this demon into our world.

At least, that's how it appears amid the current AI frenzy.

Have you ever heard of the Honeywell Kitchen Computer?

It was a $10,600 machine released in 1969 for managing recipes,

and to this day, no evidence has ever been found that a single unit was sold. Lol.

It wasn't that the technology was bad;

people simply misjudged

what they would actually do with a computer.

Today's AI boom seems to show a worrying parallel.

Astronomical GPU costs, a grand vision of automation,

yet the actual use cases remain unclear.

Until concrete AI use cases emerge,

I remain skeptical of this frenzy.

What has happened to the internet world since AI arrived?

It's overflowing with mass-produced shorts and mass-produced blog posts,

and even though this is already a dated meme,

isn't it flooded with things like "Tung Tung Tung Sahur"?

Now, information has begun to be filled with identity-less lumps of AI-generated data.

Forbes has even run an article about exactly this.

Even if we manage to solve practical problems like the ones above,

even if we find the perfect use case and justify every dollar poured into GPUs,

I believe a deeper philosophical barrier still remains.

As I see it, at its core,

this entire endeavor is self-contradictory.

To begin with, humans

cannot understand or define a system on a higher order than themselves.

According to Gödel's incompleteness theorems,

"even a sufficiently powerful formal system cannot prove its own consistency."

To put it in a more intuitive metaphor,

it touches on the same paradox as the barber who shaves only those who do not shave themselves — a barber who, as it turns out, cannot shave his own head.

In other words, a creation (AI/system), by its own logic alone,

cannot fully define or transcend the dimension of its creator (human/the external).

Just as summoning an otherworldly demon lord into this world is the stuff of fantasy,

I think this delightful imagining is also science-fiction fantasy — at least, for now.