BOT House|RELOADED - Part I

At long last, BOT House is finally reincarnated.  And on a much perkier Intel box with a newer, 64-bit version of Debian.  The new, official name is BOT House|RELOADED or BH|R for short.

So much hardware has crashed and burned this year it's hard to keep teack of it all.  First, it was the proxy project box.  It died of—what else—hard drive failure.  It was a strange setup in the first place: a (hardware) "RAID Nuthin" array spread across an IDE and a SATA drive.

And to complete the nightmare... on LVM.

But there were backups.  I re-installed everything on an external USB/IDE drive temporarily just to keep things running.  Later I bought a pair of 3.5T SATA drives & a new RAID card.  I mirrored the drives and plan to use it as the main backup for all this crap I shit out.

But before I finally got around to taking it off the USB, UPS's started to shit themselves.  Power failures have been brutal this year.  It used to be all I had to worry about were a few minor brown-outs during the beginning of "air conditioner season", but this year multi-day blackouts were far too common for my comfort level.  Two UPS's died.  I replaced the batteries in one and upgraded another from 350VA to 1000VA.

A few weeks after the derecho hit and knocked us out for four fucking days (two off, one on, two more off) it was getting a little windy outside, so I decided to check the Weather Channel for a forecast.  I turned on the TV, tuned in, and no sooner than they said "... high winds approaching our area..." the entire house went dark and stayed that way for another forty-eight hours.

A couple of weeks later, there I was, minding my own business and limping away on the USB drive in the proxy project box when one day, after mowing the lawn, I sat down and searched for images of Mossberg shotguns (for this story) on Google.

And... nothing happened.

Then I hear this "click click" sound coming from BOT House.  I switch to the console.  The last thing I saw was a message that said "Replace UPS battery" before I tried to reboot it.

It didn't reboot.  It just went click click click...

I spent the rest of that afternoon recreating the router & firewall on a bootable USB version of the Backtrack5 LiveCD and ran that for a couple of months before buying all the new hardware—computers, UPS's, hard drives—for everything and re-engineering the whole DinkNet NOC from the bottom up.

The things I do for you kids!

UPS: The Autopsy

I performed a few more experiments with the dead UPS and noted that after it had sat for a few days I could get four or five software self-tests out of it as long as there was zero load on the battery.

Then it dawned on me that I didn't have a dead UPS. I had a dead battery.

After that I decided to put its dead carcass on the slab and rip its guts out.

I was quite surprised to find that its insides were spotless. For having rested for four years in the same spot I was astonished that it wasn't full of cat hair, parakeet feathers, dust mites, and all the other detritus that collects inside the many computers hooked up to DinkNet.

Four years! I have to clean out the computers every six months or the fans get clogged!

I pulled the battery out, which was a much more difficult operation than it should have been. The two APC UPS's I have both have easy-access battery compartments, but this thing required a near complete disassembly to remove the battery.

Once I got the battery (actually two batteries taped together and wired in series) out, I pulled the part number (CB1270 for those who care) and looked it up on the 'Net. An equivalent replacement was forty bucks and the manufacturer clearly stated it would last three to five years.

So... there was never anything wrong with the UPS at all and four years is about what you'd expect to get out of a battery anyway.

Anyway, I ordered a replacement battery. This UPS will be dedicated to keeping the Proxy Project up and running, since that particular box runs 24x7 and takes a lot of abuse whenever the power blinks out. Its an XP box, my favorite UT99 platform, but its primary purpose is to run a Linux VM for proxy testing and database operations. Since I despise the Windows software Powercom distributes with these things, I'll probably run NUT on the VM to monitor it and WinNUT on XP for shutting down the physical box.

That's really an odd way of doing power management, but it's the Hinky Way!

UPS Post-mortem

It's dead, Jim!
We had another power outage today! Did you notice? Of course not, because everything's working as intended now.

For giggles, I plugged in the old UPS and hooked it up to a power-unprotected Windows XP box. I installed the GAWD-AWFUL-UGLY software (see below) and did a self-test on the UPS.


The result?

IT DIED. THE FUCKER CHOKED. IT PUKED ALL OVER ITSELF.

And everything went dark.

I had to go back to the Old Blog to see when I installed this thing. It was officially announced in October 2005, so it lasted for four years and change. Now, it's junk.

TOXIC JUNK, since it has a lead-acid battery inside it. I suppose if I were a drunken electrician (like my dear old bald-headed Daddy, Drinky Dink) I could fix this thing, but now I have to take it somewhere and probably pay somebody GOOD MONEY to throw it in the lake for me.

For the record, I would never buy one of these again, although... ummm... I already have bought one (a different model) for my "domain services" box. That one appears to be working fine, but I do feel like I'm waiting for the other shoe to drop.

If/when I buy another UPS, it's definitely going to be an APC.

UPS Replaced!

We had yet another power outage today! EXPIV was seriously fuckered when the power came back on, just like last time.

Thing is, I bought a new UPS yesterday. I just neglected to set it up.

So, I set it up. And while BOT House was offline I gave it a much needed cleaning. It was caked with dust and cat hair. Very nasty.

Instead of going with a cheap Chinese knock-off, this time I bought an APC Smart UPS (which was probably made in China anyway but what the fuck).

The last UPS had a serial port connection. It was probably the last of its kind because everything is USB these days. So I had some fucking around to do. EXPIV has had a USB UPS since it was EXPIII so I just copied the config files over. It didn't work! In fact it seems the UPS on EXPIV has never worked, which explains why it was always fucked up whenever the power went out. So I had two boxes to fix.

I fixed EXPIV first, because, like I said, it was fucked up. After I un-fucked it (it turns out mounting a JFS root partition with errors=ro is very bad idea - it fixes itself at boot but never remounts read/write), I set about getting the UPS to run right. When I proceeded to fix BOT House I discovered I was using two different versions of Network UPS Tools (NUT). The old version worked fine with the new UPS, but the new version of NUT seemed more capable, even though EXPIV's UPS (also an APC) has less features.

So I bit the bullet and built the new version for BOT House.

The dust has settled and everything's working fine now. We should be good to go for the next power outage/brownout.

Famous last words.

Powerloss

The juice went down yesterday and again today.

Yesterday, the power company dropped by, unannounced, to install a "smart grid meter". Of course, they had to cut the power "momentarily".

Now it appears my UPS may be fried, because it didn't make it through that outage.

Today the power went out all by itself, and the UPS didn't come back on again.

That's three times in less than a week.