If you start studying basically any field that isn't computer science you will in fact discover that the world is rife with randomness, and that the dreams of a Laplace or Bentham are probably unrealizable, even if we can get extremely close (but of course, if you constrain behavior in advance through laws and restraints, you've already made the job significantly easier).
Thinking that reality runs like a clock is literally a centuries outdated view of reality.
> Game theory is inevitable.
Because game theory is just math, the study of how independent actors react to incentives.
That's not how mathematics works. "it's just math therefore it's a true theory of everything" is silly.
We cannot forget that mathematics is all about models, models which, by definition, do not account for even remotely close to all the information involved in predicting what will actually occur in reality. Game Theory is a theory about a particular class of mathematical structures. You cannot reduce all of existence to just this class of structures, and if you think you can, you'd better be ready to write a thesis on it.
Couple that with the inherent unpredictability of human beings, and I'm sorry but your Laplacean dreams will be crushed.
The idea that "it's math so it's inevitable" is a fallacy. Even if you are a hardcore mathematical Platonist you should still recognize that mathematics is a kind of incomplete picture of the real, not its essence.
In fact, the various incompleteness theorems illustrate directly, in Mathematic's own terms, that the idea that a mathematical perspective or any logical system could perfectly account for all of reality is doomed from the start.
Confusion of effect for cause. Unconscious or effortless processing by the brain is usually way more accurate and reliable than conscious processing, but outside of being "gifted" you only get to consistent unconscious processing after years of training and conscious practice that ingrain muscle memory etc.
Yes. And the article is a perfect example of the dangerous sort of automation bias that people will increasingly slide into when it comes to LLMs. I realize Karpathy is sort of incentivized toward this bias given his career, but he doesn't even spend a single sentence even so much as suggesting that the results would need further inspection, or that they might be inaccurate.
The LLM is consulted like a perfect oracle, flawless in its ability to perform a task, and it's left at that. Its results are presented totally uncritically.
For this project, of course, the stakes are nil. But how long until this unfounded trust in LLMs works its way into high stakes problems? The reign of deterministic machines for the past few centuries has ingrained a trust in the reliability of machines in us that should be suspended when dealing with an inherently stochastic device like an LLM.
> The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
Speculative fiction about superintelligences aside, an obvious harm to underestimating the LLM's capabilities is that we could effectively be enslaving moral agents if we fail to correctly classify them as such.
When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
Language and society constrains the way we use words, but when you speak, are you "predicting"? Science allows human beings to predict various outcomes with varying degrees of success, but much of our experience of the world does not entail predicting things.
How confident are you that the abstractions "search" and "thinking" as applied to the neurological biological machine called the human brain, nervous system, and sensorium and the machine called an LLM are really equatable? On what do you base your confidence in their equivalence?
Does an equivalence of observable behavior imply an ontological equivalence? How does Heisenberg's famous principle complicate this when we consider the role observer's play in founding their own observations? How much of your confidence is based on biased notions rather than direct evidence?
The critics are right to raise these arguments. Companies with a tremendous amount of power are claiming these tools do more than they are actually capable of and they actively mislead consumers in this manner.
> When you have a thought, are you "predicting the next thing"
Yes. This is the core claim of the Free Energy Principle[0], from the most-cited neuroscientist alive. Predictive processing isn't AI hype - it's the dominant theoretical framework in computational neuroscience for ~15 years now.
> much of our experience of the world does not entail predicting things
Introspection isn't evidence about computational architecture. You don't experience your V1 doing edge detection either.
> How confident are you that the abstractions "search" and "thinking"... are really equatable?
This isn't about confidence, it's about whether you're engaging with the actual literature. Active inference[1] argues cognition IS prediction and action in service of minimizing surprise. Disagree if you want, but you're disagreeing with Friston, not OpenAI marketing.
> How does Heisenberg's famous principle complicate this
It doesn't. Quantum uncertainty at subatomic scales has no demonstrated relevance to cognitive architecture. This is vibes.
> Companies... are claiming these tools do more than they are actually capable of
Possibly true! But "is cognition fundamentally predictive" is a question about brains, not LLMs. You've accidentally dismissed mainstream neuroscience while trying to critique AI hype.
Thanks for the links! I'll have to dig into this more for sure. Looking at the bulleted summary, I'm not sure your argument is sufficiently nuanced or being made in good faith.
The article argues that the brain "predicts" acts of perception in order to minimize surprise. First of all, very few people mean to talk about these unconscious operations of the brain when they claim they are "thinking". Most people have not read enough neuroscience literature to have such a definition. Instead, they tend to mean "self-conscious activity" when they say "thinking". Thinking, the way the term is used in the vernacular, usually implies some amount of self-reflexivity. This is why we have the term "intuition" as opposed to thinking after all. From a neuronal perspective, intuition is still thinking, but most people don't think (ha) of the word thinking to encompass this, and companies know that.
It is clear to me, as it is to everyone one the planet, that when OpenAI for example claims that ChatGPT "thinks" they want consumers to make the leap to cognitive equivalence at the level of self-conscious thought, abstract logical reasoning, long-term learning, and autonomy. These machines are designed such that they do not even learn and retain/embed new information past their training date. That already disqualifies them from strong equivalence to human beings, who are able to rework their own tendencies toward prediction in a meta cognitive fashion by incorporating new information.
How does the free energy principle align with system dynamics and the concept of emergence? Yes, our brain might want to optimize for lack of surprise, but that does not mean it can fully avoid emergent or chaotic behavior stemming from the incredibly complex dynamics of the linked neurons?
FEP doesn't conflict with complex dynamics, it's a mathematical framework for explaining how self-organizing behavior arises from simpler variational principles. That's what makes it a theory rather than a label.
The thing you're doing here has a name: using "emergence" as a semantic stopsign. "The system is complex, therefore emergence, therefore we can't really say" feels like it's adding something, but try removing the word and see if the sentence loses information.
"Neurons are complex and might exhibit chaotic behavior" - okay, and? What next? That's the phenomenon to be explained, not an explanation.
This was articulated pretty well 18 years ago [0].
This essay completely misunderstands how the notion of emergence gained prominence and how people tend to actually use it. It's a straw man that itself devolves into a circular argument "embrace a reductionist epistemology because you should embrace a reductionist epistemology".
It doesn't even meaningfully engage with the historical literature that established the term, etc. If you want to actually understand why the term gained prominence, check out the work of Edgar Morin.
> can you confidently classify all mental activity that you experience as "predicting the next thing"? [...] On what do you base your confidence in their equivalence?
To my understanding, bloaf's claim was only that the ability to predict seems a requirement of acting intentionally and thus that LLMs may "end up being a component in a system which actually does think" - not necessarily that all thought is prediction or that an LLM would be the entire system.
I'd personally go further and claim that correctly generating the next token is already a sufficiently general task to embed pretty much any intellectual capability. To complete `2360 + 8352 * 4 = ` for unseen problems is to be capable of arithmetic, for instance.
> When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
So notice that my original claim was "prediction is fundamental to our ability to act with intent" and now your demand is to prove that "prediction is fundamental to all mental activity."
That's a subtle but dishonest rhetorical shift to make me have to defend a much broader claim, which I have no desire to do.
> Language and society constrains the way we use words, but when you speak, are you "predicting"?
Yes, and necessarily so. One of the main objections that dualists use to argue that our mental processes must be immaterial is this [0]:
* If our mental processes are physical, then there cannot be an ultimate metaphysical truth-of-the-matter about the meaning of those processes.
* If there is no ultimate metaphysical truth-of-the-matter about what those processes mean, then everything they do and produce are similarly devoid of meaning.
* Asserting a non-dualist mind therefore implies your words are meaningless, a self-defeating assertion.
The simple answer to this dualist argument is precisely captured by this concept of prediction. There is no need to assert some kind of underlying magical meaning to be able to communicate. Instead, we need only say that in the relevant circumstances, our minds are capable of predicting what impact words will have on the receiver and choosing them accordingly. Since we humans don't have access to each other's minds, we must not learn these impacts from some kind of psychic mind-to-mind sense, but simply from observing the impacts of the words we choose on other parties; something that LLMs are currently (at least somewhat) capable of observing.
The defenders and the critics around LLM anthropomorphism are both wrong.
The defenders are right insofar as the (very loose) anthropomorphizing language used around LLMs is justifiable to the extent that human beings also rely on disorder and stochastic processes for creativity. The critics are right insofar as equating these machines to humans is preposterous and mostly relies on significantly diminishing our notion of what "human" means.
Both sides fail to meet the reality that LLMs are their own thing, with their own peculiar behaviors and place in the world. They are not human and they are somewhat more than previous software and the way we engage with it.
However, the defenders are less defensible insofar as their take is mostly used to dissimulate in efforts to make the tech sound more impressive than it actually is. The critics at least have the interests of consumers and their full education in mind—their position is one that properly equips consumers to use these tools with an appropriate amount of caution and scrutiny. The defenders generally want to defend an overreaching use of metaphor to help drive sales.
Luckily for us, technologies like SQL made similar promises (for more limited domains) and C suites couldn't be bothered to learn that stuff either.
Ultimately they are mostly just clueless, so we will either end up with legions of way shittier companies than we have today (because we let them get away with offloading a bunch of work to tools they rms int understand and accepting low quality output) or we will eventually realize the continued importance of human expertise.
Or even solving problems that business need to solve, generally speaking.
This complete misunderstand of what software engineering even is is the major reason so many engineers are fed up with the clueless leaders foisting AI tools upon their orgs because they apparently lack the critical reasoning skills to be able to distinguish marketing speak from reality.
Yeah, unfortunately Marx was right about people not realizing the problem, too. The proletariat drowns in false consciousness :(
In reality, the US is finally waking up to the fact that the "golden age" of capitalism in the US was built upon the lite socialism of the New Deal, and that all the bs economic opinions the average american has subscribed to over the past few decades was completely just propaganda and anyone with half a brain cell could see from miles away that since reagonomics we've had nothing but a system that leads to gross accumulation to the top and to the top alone and this is a sure fire way (variable maximization) in any complex system to produce instability and eventual collapse.
If you start studying basically any field that isn't computer science you will in fact discover that the world is rife with randomness, and that the dreams of a Laplace or Bentham are probably unrealizable, even if we can get extremely close (but of course, if you constrain behavior in advance through laws and restraints, you've already made the job significantly easier).
Thinking that reality runs like a clock is literally a centuries outdated view of reality.
reply