BOT House|RELOADED - Part I

At long last, BOT House is finally reincarnated.  And on a much perkier Intel box with a newer, 64-bit version of Debian.  The new, official name is BOT House|RELOADED or BH|R for short.

So much hardware has crashed and burned this year it's hard to keep teack of it all.  First, it was the proxy project box.  It died of—what else—hard drive failure.  It was a strange setup in the first place: a (hardware) "RAID Nuthin" array spread across an IDE and a SATA drive.

And to complete the nightmare... on LVM.

But there were backups.  I re-installed everything on an external USB/IDE drive temporarily just to keep things running.  Later I bought a pair of 3.5T SATA drives & a new RAID card.  I mirrored the drives and plan to use it as the main backup for all this crap I shit out.

But before I finally got around to taking it off the USB, UPS's started to shit themselves.  Power failures have been brutal this year.  It used to be all I had to worry about were a few minor brown-outs during the beginning of "air conditioner season", but this year multi-day blackouts were far too common for my comfort level.  Two UPS's died.  I replaced the batteries in one and upgraded another from 350VA to 1000VA.

A few weeks after the derecho hit and knocked us out for four fucking days (two off, one on, two more off) it was getting a little windy outside, so I decided to check the Weather Channel for a forecast.  I turned on the TV, tuned in, and no sooner than they said "... high winds approaching our area..." the entire house went dark and stayed that way for another forty-eight hours.

A couple of weeks later, there I was, minding my own business and limping away on the USB drive in the proxy project box when one day, after mowing the lawn, I sat down and searched for images of Mossberg shotguns (for this story) on Google.

And... nothing happened.

Then I hear this "click click" sound coming from BOT House.  I switch to the console.  The last thing I saw was a message that said "Replace UPS battery" before I tried to reboot it.

It didn't reboot.  It just went click click click...

I spent the rest of that afternoon recreating the router & firewall on a bootable USB version of the Backtrack5 LiveCD and ran that for a couple of months before buying all the new hardware—computers, UPS's, hard drives—for everything and re-engineering the whole DinkNet NOC from the bottom up.

The things I do for you kids!

The Year Of Proxies

That's right, the List is now a year old, going on 13 months.

March 15 (the Dinkster's birthday) was the actual one year mark, but I was busy that day, meeting the Fockers (the parents of Rinky Dink's True Love, Twinky Dink). It was a busy day and I never got around to it. That and the other blogs, the new UT server, and a half dozen other projects kept me from marking the date.

I only mention it now because I've been up since about 3AM moving the server that runs the publication of the List and all its varied and sundry subfunctions.

That was fun. Right now, the new server (a VM - virtual machine) is re-doing the 4AM run and it's going quite nicely. But it was a rough road getting there, but it had to happen because I hit a solid brick wall with the old VM. Ubuntu, in their infinite wisdom, stopped supporting "Feisty Fawn" (a.k.a. Ubuntu 7.04).

That in itself was a major pain in the ass. No more security updates. No more new software packages. And no more OLD software packages. After I moved the UT servers to a common code base by putting everything on an NFS (Network File System) share, I wanted to leverage the information stored there for the proxy site VM. But I couldn't install the software! ARRRGH!

So right now, this is the THIRD incarnation of the new proxy site server, an Xubuntu 8.04 platform. It is the THIRD because I neglected the First Rule of Virtual Machines: snapshot! snapshot! snapshot!

For the uninitiated, snapshotting a VM freezes the configuration so you can roll back to a known good state. But... I was in a hurry. In order to make the new box 100% like the old box, I made a list of all the Debian software packages that weren't in the new and did a shotgun upgrade to synchronize them.

Consequently, I ended up screwing the hard disk configuration by getting packages that required LVM (Logical Volume Manager).

I hate LVM. I has been a major pain in my ass ever since I set the old VM up. It's a very powerful, complex package that no one needs. Sure, you can extend volumes when space gets tight, but it's easier and faster to use gparted (Gnome Partition Editor). LVM was on the top of my Shit List and it had to go.

But it kept coming back. On the third incarnation I had the presence of mind to take the snapshot and, of course, I never needed it. I finally had the list pared down to packages that I needed and didn't rely on LVM (stuff like Rhino and gocr and a lot of other little packages my scripts rely on). I restored the database (yes, I do make backups these days), twiddle MySQL's run-time parameters, and did a few test runs. These died because they relied I utilities I wrote and forgot about. I moved those packages over and it ran like a top.

And it's running like a top as we speak. The 4AM run is almost done. When it finishes I'll shut the VM down, patch the Windows XP box it's running on, and reboot.

There should be a new list by noon.