I was thinking about jamming along these lines, but the problem is that it's a game of whack-a-mole -- you have to keep up on what bots are active (robots.txt doesn't really help here, and focusing on Common Crawl is insufficient).
My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.
> So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation?
Yes, although I've been blocking googlebot for years, so nobody will get to my sites through google anyway.
> I don't see how it could be considered a moral act.
I'm curious about this -- why do you think this is in any way an immoral act?
If a naïve human comes across the site, they'll quickly realize that it's not useful and move on. No harm done. How does morality enter into it?
Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?
>If a naïve human comes across the site, they'll quickly realize that it's not useful and move on
I do not share your confidence.
>Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?
My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.