OpenAI didn’t even exist when Adam was invented! The contact email on the paper is from 2017.
OpenAI has made 1 major contribution - the auto regressive decoder. You could argue also their productisation of RL for LLMs has been highly influential even though they strictly didn’t invent it.
Plus all the things they haven't yet published... there must be something, because nobody has yet come close to the performance of GPT-4 8 months later, despite many people trying.
Wouldn’t Google be able to catch up easily if that was really OpenAI’s only advantage? Google has a lot more resources to throw at RL. It seems like there must be some other secret sauce for OpenAI to continue maintaining such a huge lead.
I believe they are using much more synthetic data than others. That's the secret sauce besides their larger training set and model. Synthetic data makes the model more consistent.
For all of the novel interest in the tech and service side, I can't understand why so many overlook the value of the datasets. I've been promoting them as potential products to license, similar to textbook publishing, and a bit reminiscent of how Google used its search appliances. Automating methods of creation will be an immensely valuable service.
I've already introduced methods of instruct formatting within our org, because as the architectures change, what we learn from performing the work to organize and account for our internal knowledge will continuously pay off.
Nobody has tried to come close to the performance of 4 and failed.
Palm-2 was never intended to match 4 (Gemini is). Llama 2 was never intended to match 4 either. Same with Claude 2. Nobody has matched GPT-4's training compute and failed to reach it.
OpenAI has made 1 major contribution - the auto regressive decoder. You could argue also their productisation of RL for LLMs has been highly influential even though they strictly didn’t invent it.