BOT House|RELOADED - Part I

At long last, BOT House is finally reincarnated.  And on a much perkier Intel box with a newer, 64-bit version of Debian.  The new, official name is BOT House|RELOADED or BH|R for short.

So much hardware has crashed and burned this year it's hard to keep teack of it all.  First, it was the proxy project box.  It died of—what else—hard drive failure.  It was a strange setup in the first place: a (hardware) "RAID Nuthin" array spread across an IDE and a SATA drive.

And to complete the nightmare... on LVM.

But there were backups.  I re-installed everything on an external USB/IDE drive temporarily just to keep things running.  Later I bought a pair of 3.5T SATA drives & a new RAID card.  I mirrored the drives and plan to use it as the main backup for all this crap I shit out.

But before I finally got around to taking it off the USB, UPS's started to shit themselves.  Power failures have been brutal this year.  It used to be all I had to worry about were a few minor brown-outs during the beginning of "air conditioner season", but this year multi-day blackouts were far too common for my comfort level.  Two UPS's died.  I replaced the batteries in one and upgraded another from 350VA to 1000VA.

A few weeks after the derecho hit and knocked us out for four fucking days (two off, one on, two more off) it was getting a little windy outside, so I decided to check the Weather Channel for a forecast.  I turned on the TV, tuned in, and no sooner than they said "... high winds approaching our area..." the entire house went dark and stayed that way for another forty-eight hours.

A couple of weeks later, there I was, minding my own business and limping away on the USB drive in the proxy project box when one day, after mowing the lawn, I sat down and searched for images of Mossberg shotguns (for this story) on Google.

And... nothing happened.

Then I hear this "click click" sound coming from BOT House.  I switch to the console.  The last thing I saw was a message that said "Replace UPS battery" before I tried to reboot it.

It didn't reboot.  It just went click click click...

I spent the rest of that afternoon recreating the router & firewall on a bootable USB version of the Backtrack5 LiveCD and ran that for a couple of months before buying all the new hardware—computers, UPS's, hard drives—for everything and re-engineering the whole DinkNet NOC from the bottom up.

The things I do for you kids!

BOT House Meltdown 07/14/2011

I've been waiting for this day.  It's been a bad year for hardware.  Boxes have been dropping like flies.  The 'House itself is over six years old, and it's the oldest running box on DinkNet at this point (the oldest runnable box was shut down on Windependence Day).

I've always had a Bad Feeling about this box.  It has a non-standard power supply with a teensy-weensy fan in it.  Not an easy thing to swap out on a moment's notice.  The CPU has a funky cooling system.  The possibility of a meltdown has always been there.

And tonight it happened.

What the heck, six years is a good run around here.

Apologies to everyone who was playing around 10PM EDT.  It's up and running with the case open now, but I may be doing some maintenance this weekend, which will take everything, including the Proxy List, down.

So go outside and play.

It's Always Something!

After I published my first Websense hack back in 2007, my hard drive died. It took out BOT House for a couple of months (just the box itself, the UT servers were jacked around to other boxes and VMs).

The second time, RoadRunner died.

Now, on the day after the third Websense hack, RoadRunner decides to re-engineer its entire network.

They moved the DHCP server and re-IP'd the entire network in my area sometime after 2AM this morning. As a result, everything died.

For the average user, moving the DHCP server should never be an issue, but in RR's case, it's on a 10.0.0.0/8 network. The problem is, with my unique setup, I already have a route for 10.0.0.0/8, so the router/firewall (which is where BOT House lives) was never able to renew its address, knocking it offline.

Which is just as well because BH only expects to lose its address after an extended power outage. It's hacked around so that everything - startup scripts, routes, system variables, etc. - gets changed after a reboot with a new IP. No reboot, no IP change.

And naturally, once I rebooted BH to set everything right again the damned thing decided to check all the disks, which took forever.

The servers are indeed back up, but you will need to update your UT99 favorites, if that's how you connect.

AND ANOTHER THING...

For the first time in four years this place attracted comment SPAM in the Websense Redux article, which was THE WORST PLACE they could have picked. I don't want to moderate the comments, but if it keeps up I'll have no choice.

10 Days of "Auto AutoAdjust"

And no one has complained yet!

I must say it adds a bit of a challenge. And that challenge is staying less than 10 points ahead! That, and finding a good place to hide when it's time to KICK OUT THE JAMBS!

IN OTHER NEWS, it seems like the stars and planets have aligned for some sort of cosmic spotlight on the old Dinkster. Normally, for me at least, when I get that feeling it's reminiscent of the archetypal wide-eyed convict with hands spread and back against the prison wall.

Generally speaking, not a good feeling.

This time around I think things are going to be different. There are some endings and beginnings on the horizon and all of them are good. Not only is there a light over at the Frankenstein Place there's one at the end of the tunnel as well. Last week brought a Triple Whammy of Good Things, all centered around the same subject, a subject on which I just happen to be a Subject Matter Expert (SME, for you non-PMPs).

And although I don't want to get specific just yet, suffice to say this may finally be my ticket out of the Salt Mines.

UT Map Switcher In Production

I got bored at work today and wrote the map switcher for Ban-O-Matic.

Now, after ten minutes of idle time (seems to be a lot of that lately), the servers will automatically switch to a random map.

Also, in the case of BOT House and EXP4, an update with the new map name will be sent to Twitter. These may or may not show up depending on whether or not I used my hourly tweet allotment.

FWIW, it looks like I may be upgrading the kernel on BOT House this weekend, since it went so well on EXP4. That means there will undoubtedly be an IP address change in the near future and your bookmarks will be fucked.

There are still a few tests I'd like to do first, but it looks inevitable.

I really hate changing IPs, but the upgrade is worth it.

Script Haxx

Upgrading to 2.6.31 went so fast Saturday morning I was at a loss for things to do over the weekend. Before shutting down the servers to rebuild the kernel I noticed the scripts were running wild, so after I fixed the other problems with the video and iptables, I attacked the scripts.

All of this crap, Chat-O-Matic, Ban-O-Matic, the Map, and now this Twitter Shit is all built off data that's scraped from the UT Web Admin interface with bash scripts. There has always been one big problem with approach: it's really hard for all those scripts to tell when a game is over.

The main issue is when the last player leaves when a new game starts but before before the server is populated with bots.

This isn't a problem when the last player leaves an active game. A bot will simply take his place and the bots will finish the game off themselves, which is good since another player could jump in.

But the scripts get seriously confused when there are only, say, one or two bots wondering around waiting for players that will never show up. The game has to be reset for everything to work again. This has been a problem for a long time, and I usually ignored it because it usually fixes itself as soon as another game started.

Usually.

Sometimes you just have to SIGKILL all the games and game sub-processes and start over.

So I put the "Game Over" functionality into Ban-O-Matic. Now, if there's nothing but bots playing for more than two minutes, the game is restarted (future expansion: switch to a random map). B-O-M was a good place for this, since it deals exclusively with real players and ignores bots completely.

Plus, aside from banning bad players, B-O-M has very little else to do.

On top of that, I optimized a ton of other scripts and put together a script monitoring system.

With the new kernel and optimized scripts the place is running better than ever.

Twitterfication Update/Busman's Holiday

Since BOT House and EXPIV were put on Twitter the two servers have generated almost 40,000 tweets.

I haven't managed to control things enough to avoid the 150 tweets per hour maximum, so the servers get cut off quite a bit. Usually in about an hour they're back, chattering away.

If you have a Twitter account, it's not a good idead to "follow" the servers because they'll stomp over everyone else you're following.

At one point, BOT House had over 700 followers (mostly bots, I suppose) but overnight that number dropped to about 200 and has been there ever since. EXPIV is lagging behind with about 140 followers.

This whole exercize has me looking over the Chat-O-Matic code to see how I can liven things up, because, quite frankly, the chatterbots tend to say the same things over and over and over again. Plus there's that need to throttle things back so I don't get banned every hour.

I will be playing a bit more in the next two weeks since I'm on vacation. There will be a break in the action because my wife (Pinky Dink) and myself will be attending the kid's (Rinky Dink's) wedding to his True Love (the future Mrs. Twinky Dink).

Aside from the wedding festivities, this will be Yet Another Busman's Holiday. Sometime after the wedding I'm going to consolidate www.mrhinkydink.com and proxyobsession.net, mostly just to put everything into one account and partly to move the proxy material over to ProxyObsession and perhaps - just perhaps - to make some parking meter change off the content. Traffic to the Proxy List has tripled since last year and although it seems to have peaked for the time being I think it will continue to grow.

Some disruption to UT may be involved, but it shouldn't last for more than a few minutes.

I Hate Ads

Advertising is private sector propaganda. It's vile. It's evil.

I avoid it whenever possible.

Especially on the Internet. I've been running a Squid proxy at home for over ten years, primarily to use Ad Zap to eliminate the popups, banners, and ShockWave Flash crap all mainstream Web sites would like to shove down our pipes.

For example, here's a screenshot of ComputerWorld on a slow advertising day. Click for a larger view..

Again, this was captured on a weekend. Durig the week, when there is much more traffic from corporate proles such as myself, it is one hundred times worse.

Minimum.

Here is the same exact page when viewed through my Squid prozy with Ad Zap...

Much better.

One thing the screen captures don't show is the fact that all three of the ads are animated. Animation sucks primarily because the bandwidth required chokes a remote desktop connection. I do most of my browsing at work through an encrypted pipe back to my house, which uses the 785kBps (**SLOW**) uplink. Any animation sucks the lifeblood out of that link.

There are several other ways to accomplish the same thing. I had been using FireFox with NoScript exclusively for the past eight months, but I switched to Google Chrome because it is much faster than FireFox.

Blazingly fast. Un-fucking-believably fast.

Blogably fast! TROOF! But maybe some other time. We're talking about ads today.

Since there's no NoScript plug-in for Chrome, I had to go the Ad Zap route. As it's distributed in its stock configuration, it kills maybe 90% of all adds, banners, popups, etc.

But I want no less than 100%!!!!

The big problem I had with Ad Zap was the small town TV and radio news sites across the US. They tend to use more obscure Web advertising companies, or they use their own home-brew methods. These aren't included in Ad Zap out of the box, so you have to roll your own rules. This is simply a matter of setting a few variables and editing a text file or two.

The result is very clean and very safe Web surfing. Once you start blocking ads, you will never want to go back.

Never.

I have mentioned before I'm the Network Nazi at work. I run a commercial content-blocking software package - that will remain nameless - and I also block ads on my own corporate account. I've been doing so for a very long time with this package. I am so used to not seeing ads that I'm totally appalled whenever I have to use someone else's computer.

Why do people put up with that crap?

Lean Sock Puppets

sock puppet (noun) : a name or identity used online to deceive others and that is often used to direct praise or attention to oneself. (From Merriam-Webster's Open Dictionary)

Anyone who has spent any hard time in a corporation or bureaucracy knows that there are certain management fads that make the rounds. There is no escape. It's going to happen and it usually happens for a reason.

For our previous CIO, whose job it was to reorganize the IT department it was Who Moved My Cheese.

Back then, if you were a middle manager and didn't have a copy of the book prominently displayed somewhere in your loser cruiser or on your desktop, you could end up being branded as some sort of corporate insurgent. It was everywhere. The most faithful always carried a copy with them to meetings to spread the faith whether they believed it (or read it) or not.

The latest CIO is proselytizing The 7 Habits of Highly Effective People.

Ugh. Not again. The less said about that nonsense, the better. That crap has been making the rounds for almost twenty years now.

But this isn't about cheese or habits or CIOs. It's about IT Security and the CSO.

A few months back, the CSO sent us this link (don't bother reading it, it is crap) to an article on "Lean Security" and noted we would be hearing more about it in the months to come.

Uh-oh. In my mind that meant "here come the budget cuts, kiddies!"

I dutifully read the article and before going beyond the second paragraph I got an erie feeling of déjà vu.

I knew I had read it before, but something was different. The subject had come up in the late 90s, but back then it was "Lean IT". As I read on I became convinced that the author had simply recycled the "Lean IT" article by searching for "IT" and replacing it with "Security".

It was an astounding epiphany. "What now?" I asked myself. Was this going to be the Next Big Thing industry-wide? In order to answer that I created - what else? - a Google News Alert for "Lean Security".

The were very few hits over the next few months, and nearly all of them pointed back to the same article the CSO had provided a link to. That settled the "industry-wide" question in my own mind. There was no buzz. Anywhere.

The CSO never mentioned it again.

This brings us to the sock puppets. It turns out there's a Lean Enterprise Institute. They've been responsible for distributing this crud for over 11 years. It figures. They even have a Lean Forum, and that's where my most recent Google Alert came from. It pointed to this thread, which goes like this:

Sock Puppet #1: I've read several articles of applying LEAN principles to security operations. Can anyone suggest additional readings? Just a really interesting concept for me. Thanks!

Sock Puppet #2: It makes sense. Security is just another process, with a specified outcome.

Sock Puppet #3: I'd be happy to discuss this topic with you. I have co-authored the original article on applying lean principles to security and will continue a series of columns in Security Technology & Design Magazine for the next 10 months.

Oh, brother. Nothing stands out like a self-serving clusterfuck.