I just noticed some interesting entries in my logs from a new bot: BDFetch. Brand Dimensions is a company that collects information from the Internet, looking for bad things people say about Comcast's poor service. Apparently, they're conducting private investigations for U.S. clients from Canada by collecting files on everyone that says something about Comcast online. Personally, I'd rather not be investigated by a Canadian company in a state that requires licensing for such activities.
Here are the entries from my web server's access log. Clearly, they're looking only at content that mentions Comcast. I mention other brand names, but they're not interested in them.
72.14.164.176 - - [05/Jun/2009:16:49:17 -0400] "GET /robots.txt HTTP/1.1" 200 289 "www.brandimensions.com" "BDFetch"
72.14.164.150 - - [05/Jun/2009:16:49:39 -0400] "GET /blog/2008/11/comcast-strikes-back.html HTTP/1.1" 200 17006 "www.brandimensions.com" "BDFetch"
Since then, I have two new lines in my robots.txt file:
User-agent: BDFetch
Disallow: /
If that doesn't work, I'm going to cut off the 72.14.164.0/24 network at my firewall.
Wondering if Brand Dimension is watching you? Here's the grep command to find them:
grep BDFetch access_log
You'll need access to your web log. Also, remember that grep and unix are case-sensitive.
Leave a comment