They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters.
So yeah they generate probable text, sure. Where they get the probabilities is a very good research problem.
> E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters.
Broadly agreed, but there's no "representation"...the model has no memory, let alone a "concept" of a chessboard. It's just trained on a bunch of textual replays of chess games, and this works well enough for a game with finite probability space.
Likewise, I have asked generic LLMs to generate "novel" drugs that solve particular problems, because their training sets have included lots of examples of drug molecules in string form (aka SMILES strings). This works far less well (because chemical space is much larger than a chess game), and the "novel" drugs usually end up looking like some mishmash of existing chemicals for the same problem. This can be a useful tool, but nobody is confusing it for a human brain doing problem solving.
People are easily misled by the purported intelligence of these things because they assume that common examples of human intelligence are less probable than they really are. Our languages and games and stories are pretty regular, all things considered. Expand the probability space to something truly vast (even images work for this), and you can easily see the stochastic parrot emerge.
The fact that tweaking parameters which appear to store the board makes it play according to the tweaked numbers instead of what was passed to it the context (i.e. working memory) directly contradicts your assertion that LLMs have no memory. The context is their memory.
I can’t comment on your drug generation task - they aren’t magic, if the training didn’t result in a working drug model in the billions of params you’ll get garbage output, not very surprising.
My point boils down to the ‘what’s more likely’ question: magical stochastic parrots which just by accident manage to create understandable and coherent responses to unreasonably large set of questions or the magic is actually some kind of a world model, or multiple, inside the blob of numbers, outputs of which steer the probabilities, just as this confirmed chess example. My bet is on learned world models because I’m not convinced there’s magic in our physical world.
If you want to call context "memory", then sure, but that's not what anyone means when they say the word. We don't build our world model fresh with every sentence someone says to us, nor do we have to communicate our complete knowledge of conversational state to another human by repeating the entire prior conversation with every new exchange. It's obviously different in a fundamental way.
> My bet is on learned world models because I’m not convinced there’s magic in our physical world.
You don't need to bet, and it has nothing to do with "magic". They quite literally have no ability to have a "world model" -- it's just a text generator, producing tokens. There's no working set memory, other than the text you pass into it. It should hopefully be obvious to you that when you write, you're not simply emitting one word at a time. You have a complete mental model of whatever you're discussing, stored in working memory, and it's persistent. We also update that model with every interaction we have.
The point of my post was that as soon as you take on a harder problem than simulating language, the lack of intelligence slaps you in the face. It turns out that understandable, coherent free-text responses is not magic, and the surprising result is that human language is regular enough that you can statistically simulate "intelligence" with a few hundred million free parameters.
>We don't build our world model fresh with every sentence someone says to us
Neither do LLMs. The state for the current text perhaps. Definitely not the entire world model(s) which is learnt from the training process and stored in its weights.
>They quite literally have no ability to have a "world model"
You keep repeating this so let's get one thing straight. You're wrong. You're just wrong. I'm not trying to convince you of my opinion. This has been empirically observed and tested multiple times.
You're literally saying absolute nonsense with a high level of confidence. When an LLM does this, somehow it's a "hallucination". Why are you different ?
You keep saying LLMs just produce tokens as if it was trivial, whereas it’s the key result that those models produce tokens at all (on that we seem to agree, at least) and how they do that is more or less unknown. You have a hidden assumption somewhere that the mechanism doesn’t matter, but in fact the opposite is true: it’s the only thing that matters. Yes the model is static and doesn’t update weights all the time, so what? I’m not saying it’s a sentient being, I’m saying it has world models discovered and embedded in the parameters and a working (as in, writeable, short term) memory.
They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters.
So yeah they generate probable text, sure. Where they get the probabilities is a very good research problem.