If someone handed you an envelope containing a hidden question, and your life depended on a correct answer, would you rather pick a random person out of the phone book or an LLM to answer it?
On one hand, LLMs are often idiots. On the other hand, so are people.
That's not at all analogous to what I'm talking about. The comparison would be picking an LLM or a random person out of the phone book to, say, operate a vending machine... and we already know LLMs are unable to do that, given the results of Vending-Bench.
More than 10% of the global population is illiterate. Even in first world countries, numeracy rates are 75-80%. I think you overestimate how many people could pass the benchmark.
Edit - rereading, my comment sounds far too combative. I mean it only as an observation that AI is catching up quickly vs what we manage to teach humans generally. Soon, if not already, LLMs will be “better educated” than the average global citizen.
And yet, I would be completely confident that an average illiterate person could pass the Vending-Bench test indefinitely if you gave them interfaces that don't depend on the written word (phone calls, abacuses, piles of blocks, whatever), and that the "smartest" LLM in the world couldn't. It's not about level of education, beyond the bare minimum needed to have any kind of mental model of the world.
I'd learn as much as I could about what the nature of the question would be beforehand and pay a human with a great track record of handing such questions.
On one hand, LLMs are often idiots. On the other hand, so are people.