Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

As soon as you get your TLS certificate you get bombarded with scraping. You don't need someone to "point a scraper at you".

What matters most is usually how much there is to scrape. If you have like 5 pages that's nothing. For forum like websites where each thread, each user profile, etc. gets scraped that's when traffic increases. I just let them have at it with no issues though, computers are fast.

 help



That's really weird. My experience is quite different: I have several subdomains and all of them have TLS certs and I haven't (yet) seen this (thankfully). Either that, or my server is masking it. The weird thing is that my server is an OVH dedicated box that doesn't exactly have top-tier specs, so I have no idea what's going on there. Very weird indeed.

Probably you don't have much to scrape?

I mean... It may be that most of the things I run aren't really scrape-able. I run Matrix (which requires authentication), an XWiki instance, Zulip, Terraria, Forgejo, Nextcloud, a Mastodon server... Most of those require auth behind my Kanidm instance to actually do anything. Well and most of them have APIs that are much better than "scrape the universe".

If you run the site on a custom port, scrapers won't find it?



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: