BOT House|RELOADED - Part I

At long last, BOT House is finally reincarnated.  And on a much perkier Intel box with a newer, 64-bit version of Debian.  The new, official name is BOT House|RELOADED or BH|R for short.

So much hardware has crashed and burned this year it's hard to keep teack of it all.  First, it was the proxy project box.  It died of—what else—hard drive failure.  It was a strange setup in the first place: a (hardware) "RAID Nuthin" array spread across an IDE and a SATA drive.

And to complete the nightmare... on LVM.

But there were backups.  I re-installed everything on an external USB/IDE drive temporarily just to keep things running.  Later I bought a pair of 3.5T SATA drives & a new RAID card.  I mirrored the drives and plan to use it as the main backup for all this crap I shit out.

But before I finally got around to taking it off the USB, UPS's started to shit themselves.  Power failures have been brutal this year.  It used to be all I had to worry about were a few minor brown-outs during the beginning of "air conditioner season", but this year multi-day blackouts were far too common for my comfort level.  Two UPS's died.  I replaced the batteries in one and upgraded another from 350VA to 1000VA.

A few weeks after the derecho hit and knocked us out for four fucking days (two off, one on, two more off) it was getting a little windy outside, so I decided to check the Weather Channel for a forecast.  I turned on the TV, tuned in, and no sooner than they said "... high winds approaching our area..." the entire house went dark and stayed that way for another forty-eight hours.

A couple of weeks later, there I was, minding my own business and limping away on the USB drive in the proxy project box when one day, after mowing the lawn, I sat down and searched for images of Mossberg shotguns (for this story) on Google.

And... nothing happened.

Then I hear this "click click" sound coming from BOT House.  I switch to the console.  The last thing I saw was a message that said "Replace UPS battery" before I tried to reboot it.

It didn't reboot.  It just went click click click...

I spent the rest of that afternoon recreating the router & firewall on a bootable USB version of the Backtrack5 LiveCD and ran that for a couple of months before buying all the new hardware—computers, UPS's, hard drives—for everything and re-engineering the whole DinkNet NOC from the bottom up.

The things I do for you kids!

Five Hour FAIL

Today, on the Autumnal Equinox, a powerful storm blew through the area during rush hour. Driving through it was a serious BITCH, and when Pinky & I finally got home, the power was out and the house was dark.

According to the one analog clock in this house, it shut down a 5PM. It finally came back on after 10PM.

On the bright(-ish) side, all four UPSes shut down perfectly, even the new APC 1500, which had been giving me fits lately. Everything came back up without incident.

But, as expected for a long outage, the IP address changed, so you'll have to update your UT99 favorites accordingly.

And speaking of UT99, I may be replacing one of the servers in the near future, probably the 64bit dual core system, which is the hottest box I have right now. Unfortunately it is also severely underutilized.

But that plan is subject to change without notice and I have so many things going on right now it may be quite some time before I do anything at all (but I do have the replacement box right now).

Stay tuned.

Pussyfooting Does It AGAIN!!!

Last night at about 7:45PM she shut down EXP IV for the second time in three days!

She wasn't as graceful this time. First I heard the sound of shit getting knocked over before the tell tale "beep" of the UPS off-button.

This was obviously a malicious, deliberate act! Cyberwar in my own Fambly Room!!!

I have since implemented countermeasures to keep it from happening again. A cardboard barricade, secured with spare paving bricks, is the first obstacle. If she gets past that, she has to knock the keyboard out of the way.

Power Outage 09/23/09

Back to normal now.

I had to pull the plug prematurely on EXP4, since it has such a wimpy UPS. BOT House stayed up for about 45 minutes before it shut itself down.

Of course, we ended up with a new IP address, so all your favorites are fucked.

I've had a lot of good luck this year with the power situation (unlike last fall's 48 hour outage after the remnants of Hurricane Ike blew through town). I did notice a crew with a backhoe a couple of blocks away when I came home from work, so I'm going to blame it on them. They have their path all marked out with those little plastic flags and they have a way to go until they're done. Hopefully they won't make the same mistake twice.

MURDER/SUICIDE > UT99

Yesterday I was poking around in my old posts at the Murder/Suicide Blog when I stumbled across something I wasn't aware of.

It's actually getting traffic.

And what really punched me in the gut was that people are actually posting comments to the cut&paste news stories I drop in there day after day. REAL people. Not linkbaiters. Not blackhat SEO SPAM bots.

Real. Fucking. People.

Not only are they real people, but they're people who in many cases knew or worked with the victims and went the the funerals. People pouring their hearts out looking for meaning, reasons, and prayers.

It stopped me in my tracks, put me in a real weird mood, and made me realize I will never look at that blog in the same light as before.

Happy Horseshit

Well, you eventually discover exactly why these things go on "Price Blow-Out" sale.

Caveat Emptor and all that crap.

But this was a new one for me. Never seen anything quite like it before.

I did install the Windows 7 beta. That went really well. It looks like Vista and runs like XP, just like everyone says it does. Setup is incredibly fast on a new system. Reboot, plug in your serialz and there you have it. Very nice.

Whoop-dee-freakin-doo.

So this morning I called in sick. Because I was sick (troof!). Had an emergency dentist appointment (I've been having a lot of those lately). Now I have the rest of the day to hack around with this box.

I put Debian 4.0r6 back on it. Runs great, as usual. I do a few updates, reboot, and... why is this thing taking so long to boot up?

I check my network config. It tells me my NIC is now /dev/eth8.

ETH-fucking-EIGHT? WTF is this? This is the kind of crap that happens when you go moving your NIC around to different slots, but I haven't done that. It's a built-in NIC, for fuck's sake!

I know enough to take a look at /etc/udev/rules.d/z25_persistent-net.rules, and there they are, all NINE (0-8) incarnations of my single NIC.

And each and every one had a different MAC address.

What the motherfucking FUCK is this crap?

I wipe the file out and reboot (I've learned this from moving VMs around - they always change MAC values when you go to another host). Sure enough another new MAC address.

Then I remembered a post I read in BugTraq or somewhere about NVidia motherboards shipping with identical MAC addresses. Could this be how they fixed that problem? Flash the BIOS to change the MAC on every reboot?

Who the fuck knows? Damned Chinese junk!

I fixed it by changing eth0's line in z25_persistent-net.rules to read:

ATTRS{address}=="00:00:6c:?*"

... since the first three bytes were always correct. It's a work-around and I'm screwed if I ever put another RTL NIC in that box (all I have are RTL NICs right now), but that's not in the cards at this point.

I shouldn't have to deal with this HORSESHIT, but I bought the box so I'm going to live with it. HOWEVER, the possibility exists that the MAC address could change at any time, which would be very, very bad (screwing up ARP tables and such), so I'm going to have to keep a close eye on this thing.

Thus begins the strange tale of EXP4 Revision 2.

UPDATE: turns out it was Linux all along. Check out this crap:

forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.56.
ACPI: PCI Interrupt Link [APCH] enabled at IRQ 22
GSI 16 sharing vector 0xC9 and IRQ 16
ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [APCH] -> GSI 22 (level, low) -> IRQ 201
PCI: Setting latency timer of device 0000:00:07.0 to 64
forcedeth: using HIGHDMA
0000:00:07.0: Invalid Mac address detected: cd:8a:b3:4d:e0:00
Please complain to your hardware vendor. Switching to a random MAC.