Friday, December 28, 2007

BOT House Disaster Recovery & Business Continuity Planning

That would be a really great idea. My current DR/BC Plan is this:
  1. Shit pance.
  2. Plug in the junk Microsoft ISA 2006 server
  3. Get a new IP address
  4. Re-hack the DNS names (got a bunch)
  5. Fire up a Linux VM for miscellaneous services
  6. Figure out what to do next
I'm still on the last step.

The day after I released the Websense advisory (I now OWN Google searches for "Websense Policy Bypass") was routine. I checked my gmail and got up to pack my little brown bag lunchie. When I came back, BH's hard drive light was flashing like crazy and the box was making a peculiar, high speed snicketta-snicketta-snicketta sound, which turned out to be a death rattle.

A reboot confirmed that the box was hosed.

A fortune would have it, I had already performed my daily bowel function, so I skipped Step 1. But there wasn't enough time to get into Step 2 so that waited until later that evening.

By the end of the day (I hate that expression) - that was a Thursday - I was up to Step 6. I tried to salavge something from the hard drive (a Hitachi, if you care) but it was hosed. I may eventually get around to trying the "freeze method" to get the data off but I believe it's a lost cause (I had some luck with that earlier this year on another box).

On Friday I picked up a 250G drive at lunch and that evening I started rebuilding BOT House. EXPERIMENTAL II was up but it would be a week before BOT House was resurrected.

When it finally was back up I decided to run UTPure on it. That was probably a mistake, considering the BH philosophy has always been "NO DOWNLOADS REQUIRED".

And apparently people don't like it.

By the week of Christmas everything (except UTClassicPack]I[ Online) was back up, but it was (and is now) still running through the ISA server. But since I took that week off and because I have a number of... uh... ummm... heh... covert connections back to my workplace (a risky proposition for people not in the security business) I really can't move everything back to the BH box until I get back to work after the first of the year.

But I have been busy.

The Map has been rehacked and is now working better than ever. It now shows the player locations on both BH and EXP II and it appears the caching problem is gone. The trick to that was deleting the data file on the Web server before uploading the new one. One side effect is that it is occasionally blank between data uploads. But, hey, it's working. Even through proxies.

FireFox used to have a terrible memory leak with that map, but it seems that was fixed in one of the many updates and patches they released in the last 12-15 months.

I also bought a cheap UPS (Uninterruptible Power Source) for EXP II. It's only a 350VA, but it should help keep EXP II up during the many power sags we get around here from May-June and you just might get 10 extra minutes of play during an outage.

The things I do for you kids!

Anyway, lookout for an IP change during the first week of January.

No comments:

Post a Comment