Now that I added user-agent: BDFetch, disallow / to my robots.txt, all the BDFetch bot gets is robots.txt. However, some people Brand Dimensions is now browsing my blog:
72.14.164.134 - - [08/Jun/2009:13:58:26 -0400] "GET /blog HTTP/1.1" 301 314 "-" "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)"
72.14.164.133 - - [08/Jun/2009:13:58:26 -0400] "GET /blog/ HTTP/1.1" 200 57901 "-" "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)"
72.14.164.196 - - [08/Jun/2009:13:58:42 -0400] "GET /blog/2009/06/comcast-is-collecting-data-on.html HTTP/1.1" 200 16335 "-" "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)"
A little research reveals that they have a class C block of IPs:
CustName: Brandimensions Inc.
Address: 5090 Explorer Drive
Address: Suite 203
City: Mississauga
StateProv: ON
PostalCode: L4W-4T9
Country: CA
RegDate: 2008-06-25
Updated: 2008-06-25
NetRange: 72.14.164.0 - 72.14.164.255
CIDR: 72.14.164.0/24
Thus blocking 72.14.164.0/24 at the firewall will prevent them from seeing anything.
Leave a comment