How do you control for the faked user agents scrapers use? A bad scraper will be... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		jeroenhd on Dec 17, 2022 \| parent \| context \| favorite \| on: A self-updating list of the most current useragent... How do you control for the faked user agents scrapers use? A bad scraper will be much more common in your logs than a well-behaved browser, after all.

stayml on Dec 17, 2022 [–]

I filter out any user agents that are invalid, but there's no way to see which are real or faked. The access logs include the useragent of every single site visitor - not only errors/bad actors.

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact