Im thinking a lot of those issues might be related to “smart” scraping which parses JavaScript. Could lean in to the bot and just make it easier for them to scrape by removing JavaScript from the websites.

I realize this is somewhat off-topic, but the big companies kind of destroyed the internet with all the JavaScript frameworks and whatnot.