Do they actually do anything to circumvent paywalls or do websites just whitelis... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		dogma1138 34 days ago \| parent \| context \| favorite \| on: FBI orders domain registrar to reveal who runs mys... Do they actually do anything to circumvent paywalls or do websites just whitelist their crawlers?

walletdrainer 34 days ago [–]

Websites don’t whitelist their crawlers, they maintain custom bypasses for a wide variety of websites.

If the websites were inclined to whitelist these crawlers, they’d also whitelist archive.org which is actually easy to whitelist. Archive.is is not

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact