But whats the point of it downloading the robots.txt file if it isn't going to h... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		afhof on July 15, 2011 \| parent \| context \| favorite \| on: How to make fun of Google Bot (PicoLisp Wiki) But whats the point of it downloading the robots.txt file if it isn't going to honor it?

robtoo on July 15, 2011 [–]

Presumably, one cluster (or whatever google calls them) is honoring it, while a different cluster isn't.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact