Maybe those original Home Sapiens migrants just went to Neanderthal territory, maybe established "homo sapiens republics" (different culture, technological practices), and eventually got absorbed into the lasting main Neanderthal culture.

Just like it happens to migrants in our time, in just a few years, not thousands.

It could be that after 600.000 years, Neanderthals got somehow behind minimal species fitness required to endure lasting species survival, and they were - this time - absorbed by the Homo Sapiens.