Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Doesn't Wikipedia offer full tarballs?

This would imaginably put some downward pressure on scraper volume.



From the first paragraph in my comment:

> You'd see them taking some additional steps to combat this. They haven't. Their CDN handles it just fine. They don't even both telling AI bots to just download the tarballs they specifically make available for this exact use case.

Yes, they do. But they aren't in a rush to tell AI companies this, because again, this is not actually a super meaningful amount of traffic increase for them.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: