I am interested in doing this. Do you have any information on how to find out wh...

jacquesm · on Jan 2, 2013

here are a few lists to get you started:

http://proxy-ip-list.com/download/proxy-list-port-3128.txt http://proxy-ip-list.com/download/free-usa-proxy-ip.txt http://www.proxylists.net/http_highanon.txt http://www.proxylists.net/socks4.txt http://www.proxylists.net/socks5.txt http://www.stopforumspam.com/downloads/listed_ip_90.zip

I've got my own db of hosting facilities which I made by taking 100M urls and doing a lookup on the hostname, then saving the IP found in a db. This gives you some level of confidence that a certain class 'C' is used for hosting.

ck2 · on Jan 2, 2013

It's also possible to build such a list by watching for static ips that do more than "x" requests and queuing rdns on them.

Google is easy to identify this way, even with a spoofed user agent (which they do a lot now).

But this technique is not possible with EC2 because Amazon refuses to make a public database of what customer is using what.

jacquesm · on Jan 2, 2013

> Google is easy to identify this way, even with a spoofed user agent (which they do a lot now).

That's part of their page-cloaking detection code.