Information Systems Grad School

Great article by Microsoft on their internal monitoring structure and organization.

After a year-and-a-half and over 1,000 miles on the Breezer Uptown 8, I thought I'd write a review. There is a dearth of material on commuter bicycles and bicycling. The trade press is advertiser-focused and almost all of their bicycle reviews are non-critical and improve as the price increases. Most bicycle stores don't focus on the commuter cycle market segment. If one store started doing it in or around DC, I suspect they'd get a hardcore following pretty quick. There's a rush-hour on the CCT even in the winter. (Minneapolis claims 3,000 winter bicycle commuters -- they plow 50 miles of bicycle trails there.)

First: my ride. On days that I ride (I'm not a 5-day-a-week rider yet), I ride seven miles (mostly downhill) into work in 35 minutes. That same ride home (mostly uphill) takes about 45 minutes. I carry a laptop and a full set of work clothes each way in panniers on a rack.

Overall, the Breezer Uptown 8 is a great bicycle. However, there are some things to nit-pick about, mostly because commuter bicycles are relatively new in the United States. Nobody reviews them because they're not sexy.

First off, the Uptown is comfortable. I ride in an upright position. The seatpost is suspended, and overall it's a pleasure to ride. However, it's not a fast bike. It's got full fenders and a fully-enclosed chaincase. The only place I get splashed in rain is my feet.

The built-in Shimano front hub generator and lights are great. The headlight is pretty bright. I did get a Niterider light because the trail is not lit and pitch black. The rear light connections are weak and the light failed after going over a few bumps. Some new wire fixed that. The generator does add a little drag when it turns itself on. It's great having lights without ever having to worry about a charger or batteries. I have two Planet Bike flashies (one works consisently) mounted on my panniers. That worked out especially well after I wrecked and my black laptop pannier fell off and onto the dark, unlit trail at night.

The stock tires, Schwalbe City, are heavy. I replaced the front with a Conti 1" which works fine. The rear tire I haven't replaced because the rear wheel is a pain to take off. You need to shift to 4th gear, remove the tiny screws in the chaincase, and then use a 15mm wrench. Not something to do on the trail after dark.

You can see photos of my nitpicks here.

I bought the bike at Bikes at Vienna in Vienna, and get some help with maintenance from Griffin Cycle in Bethesda. There's nothing wrong with the service at Bikes at Viennna, it's just that Griffin Cycles is close to home.

I updated the airspace KML files again -- skipping one release. The latest covers from December 17, 2009 throughFebruary 11, 2010. You can find them in the archive here.

I loved my Dell 1750 server. It has plenty of power and a great 3Ware RAID card, two 500 GB RAID-1 drives, 4 GB RAM, and ran Red Hat Linux. I bought it from the Dell Outlet site several years ago when my condo fee included electric. Since then I have moved. I pay my own electric bill, and my 1750 consumes 150 watts at idle. When I publish with Movable Type or Gallery, power consumption exceeds 200 watts. I pay $0.150845894 per kilowatt-hour. (That's summing the separate generation, transmission, distribution, demand-side surcharge, and adding the gross-receipts tax. Pepco doens't make it easy to figure out what you're paying.) With a thirty-day month, that's 108 kwh, which comes to $16.29135658. That's $16/month, just for idling. And that doesn't use the noise of the server in my office or the additional AC required in summer.

That may seem expensive, but it's far cheaper than getting that much server capacity at Rackspace. However, I don't need that much capacity. I can trade processor power for power savings, keep the disk space and RAID card, and switch to an Atom-based server. My current FreePBX Atom server runs at 40 watts with an analog card powering two FXS modules. I bet I can match that on a new server. I'd get the new Supermicro Atom Server, but it has space for only one 3.5 inch hard drive. I need two. Thus I'll be using another miniITX case. In the meantime, everything's running on another old Dell tower box.

How did I move it? First I though reinstalling all the software from scratch would be a good idea. I'd get a nice clean, efficient build. But that took way too long, and I'd have to re-customize my templates and tweaks. I had MySQL backups running for a while, why not start testing the restores? I used rsync:

rsync -avz /var/www/ -e ssh:user@mynewserver /var/www/

(Note: Please study rsync syntax. Those / at the ends make a big difference.) It worked. Next I had to tweak the new httpd.conf file. I couldn't just copy the old one, because I was using the latest Apache version. But I could use almost all of the old file. I just needed to adjust the modules it loaded, because several have changed names.

Then I restored the databases:

mysql -u root -p

enter your password. (You ARE using a PW for MySQL root, aren't you?)

mysql> create database mynewdb

then

mysql> quit;

then

$ mysql -u root -p [mynewdb] < [backupfile.sql]

But that generally does not restore your user privs on the db. Back to mysql:

mysql -u root -p

then

mysql> use mynewdb;

mysql> GRANT ALL PRIVILEGES ON *.* TO 'myuser'@'localhost' 

IDENTIFIED BY 'pAssW0rd' WITH GRANT OPTION;

mysql> flush privileges;

Don't forget the above step or you'll need to restart MySQL to get it to work.

mysql> quit;

Then double check that the user and pass from above match your config files.

Finally, test your applications. Gallery2 and MovableType worked fine. Your milage may vary. My office is almost silent now.

Working with Bluecoat files in the raw can be time-consuming. Findstr and grep only work so fast. Windows grep is slow. I know SQL syntax OK, so I tend to dump logfiles into databases to analyze them for activity. There are certainly other ways to do it, such as using a reporting tool for Bluecoat. (Splunk's free Bluecoat application, e.g.).

Theoretically, Bluecoat logfiles are the same as W3C web server log files that logparser can consume via the -i:W3C directive.

You can see the fields in a Bluecoat log below.

#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id

For some reason, Bluecoat leaves two spaces between cs(Referrer) and sc-Status, so all the columns to the right of sc(Referrer) past that will be one off. BlueCoat also leaves spaces in cs-categories and surrounds them with quotation marks, so you need to specify -dQuotes:on. Logparser doesn't have a quick and easy way to handle the double-spaces issue, so I wrote a VB Script to handle it. (VBScript is pretty quick at text handling and it's much faster than using search and replace in WordPad or Notepad on a 500-1000 MB File.)

Here's the VBScript:
'start

Set objFSO = CreateObject("Scripting.FileSystemObject")
'change this line to wherever you want to read the input from.
Set objTextFile = objFSO.OpenTextFile("c:\myBluecoatlog.log",1)
Set objNewFile = objFSO.CreateTextFile("c:\myCleanBlueCoatlog.log")
Do Until objTextFile.AtEndOfStream

myString = objTextFile.Readline
objNewFile.WriteLine(Replace (myString, " ", " "))
Loop
'end vbscript
Here's the logparser file:
-------------------start
SELECT TO_LOCALTIME(TO_TIMESTAMP(date, time)) AS date,
time-taken,
c-ip,
cs-username,
cs-auth-group,
x-exception-id,
sc-filter-result,
cs-categories,
cs(Referer) AS Referer,
sc-status AS scStatus,
s-action,
cs-method,
rs(Content-Type) AS ContentType,
cs-uri-scheme,
cs-host,
cs-uri-port,
cs-uri-path,
cs-uri-query,
cs-uri-extension,
cs(User-Agent) AS UserAgent,
s-ip,
sc-bytes,
cs-bytes,
x-virus-id

INTO BlueCoat4
FROM c:\myCleanBlueCoatlog.log
------------------end
And here's the command line for logparser. (Save the logparser file as c:\scripts\log\bluecoat.sql)

logparser file:c:\scripts\log\bluecoat.sql -i:W3C -o:SQL -server:sqlservername -database:BLUECOAT -createtable:ON -dQuotes:ON

Statistics:
-----------

Elements processed: 613076
Elements output: 613076
Execution time: 241.20 seconds (00:04:1.20)
About 2500 lines/sec. Processor utilization is almost zero for SQL and logparser, so it's all about disk time.

The above is from a file that's 310,935,417 bytes large. That means BlueCoat logs are about 507 bytes per line, or 0.5k per line before compression. The last time I checked BlueCoat gz compression, it was about 15% of the original file size. Compressed, the line would cost you 76 bytes.

Usually, installing an application from source on Linux/Solaris/BSD is easy:

1. ./configure --help (Always look at the help to see the options. It makes a difference if, for instance, you compile php without support for MySQL.)


2. ./configure


3. make


4. make install

However, with Darkice, it's prerequisites are numerous, and Darkice's configure doesn't find it's prereqs if they're installed in the standard locations. I've done this twice so far without documenting how I did it, so this time, I'm writing it down.

Here's my configure line:
./configure --with-vorbis-prefix=/usr/local/ --with-lame-prefix=/usr/local/lib/ --with-twolame=prefix=/usr/local/lib/ --with-faac-prefix=/usr/local/lib/

Then you'll get this when you launch darkice:
darkice: error while loading shared libraries: libmp3lame.so.0: cannot open shared object file: No such file or directory

So you need to link that, and when you link that you'll get the next error, so here are both:
ln -s /usr/local/lib/libmp3lame.so.0 /usr/lib/libmp3lame.so.0
ln -s /usr/local/lib/libfaac.so.0 /usr/lib/libfaac.so.0.
If you're wondering what links you're missing, try
ldd /usr/local/bin/darkice
If one of the links to the libraries reads "missing" then that's the one you need to link.

Yum install darkice might work for you, but then again, if you need all the features, it probably won't.
Prereq links are below. Generally ./configure, make, make install works well with all of them, but you really want to track exactly where each lib gets installed -- usually /usr/local/lib/.
Lame
Twolame
libogg
libvorbis
faac

Preqrequisite for faac or twolame -- I forget which:
libsndfile
Prereqs I didn't neeed:
Alsa
Jack

Case 1: You Google me and click on my page

Yes, I'm using google as a verb. If you Google me and click on one of my pages, my web server logs the information:
1.2.3.4 - - [01/Oct/2009:10:23:41 -0400] "GET / HTTP/1.1" 200 7186 "http://www.google.com/search?hl=en&source=hp&q=larry+s&aq=f&aqi=g10&oq=&fp=7d15299a959dbb33" "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)"

As you can see, I get your IP address, a date, an offset to Universal Time (-0400), a verb (GET, in this case / means my default site page), a status code (200=OK), and a referrer. From the referrer, I can tell you Googled me with the phrase "larry s". Finally, I also get some information about the browser you used, Firefox, and the operating system, Windows XP with service pack 2. There's a chance you may have used a anonymizing proxy, but I'd still get an entry. (Generally, Anonymizer says "TuringOS," so I know it's them.)

Case 2: You Google me and don't click on my page.

That's more difficult but not impossible, because I have a Google AdWords account. I bought my own name as a keyword. Google AdWords works by selling keywords for search insertion. It's an open market, with the second-highest bidder winning in a dutch auction that is Google's revenue machine. When you buy a keyword, you get two measures back from Google:

1. how many impressions it got (viewing)


2. how many clickthroughs it got. (someone clicks on the ad)

A keyword ad's success is measured by the ratio of impressions to clickthroughs. The more clickthroughs per impression, the better. So if you don't click on my ad link, which I have made irresistable by promising dirt on me, I still know that someone Googled me, because the impression counter increments with each search. If you click on a regular page on my server rather than the keyword ad (Google calls this "organic"), we're also back to case one above. If you don't click on any of my links, I don't get any of the details from case one.

And that's how I know that you Googled me. If you're wondering if you've been Googled, but don't have a web site with logs you can comb through, or don't want to set up a Google AdWords account, try Google's external keyword tool. Just don't forget to un-check the synonyms box.