Power Outage 09/23/09

Back to normal now.

I had to pull the plug prematurely on EXP4, since it has such a wimpy UPS. BOT House stayed up for about 45 minutes before it shut itself down.

Of course, we ended up with a new IP address, so all your favorites are fucked.

I've had a lot of good luck this year with the power situation (unlike last fall's 48 hour outage after the remnants of Hurricane Ike blew through town). I did notice a crew with a backhoe a couple of blocks away when I came home from work, so I'm going to blame it on them. They have their path all marked out with those little plastic flags and they have a way to go until they're done. Hopefully they won't make the same mistake twice.

BOT House 2.6.31 Upgrade COMPLETE

BOT House has been upgraded to kernel 2.6.31 and once again I got it on the first shot. BAM! Nailed it. Plus the new iptables & ipset utils were installed.

BOT House had been up for 215 days, so I was absolutely positive RoadRunner would dish out a new IP but as luck would have it the old one stuck. This was a big surprise considering Linux decided it had to check all the disks on reboot (I hate that - should've gone with JFS instead of EXT3 on that box), delaying the reboot even longer.

Something happened during the switch-over from the old iptables to the new because our old buddy StinkFly snuck in. Once I reset the firewall rules he was back in Ban Land.

It plays well. Hope you enjoy it.

UT Map Switcher In Production

I got bored at work today and wrote the map switcher for Ban-O-Matic.

Now, after ten minutes of idle time (seems to be a lot of that lately), the servers will automatically switch to a random map.

Also, in the case of BOT House and EXP4, an update with the new map name will be sent to Twitter. These may or may not show up depending on whether or not I used my hourly tweet allotment.

FWIW, it looks like I may be upgrading the kernel on BOT House this weekend, since it went so well on EXP4. That means there will undoubtedly be an IP address change in the near future and your bookmarks will be fucked.

There are still a few tests I'd like to do first, but it looks inevitable.

I really hate changing IPs, but the upgrade is worth it.

Script Haxx

Upgrading to 2.6.31 went so fast Saturday morning I was at a loss for things to do over the weekend. Before shutting down the servers to rebuild the kernel I noticed the scripts were running wild, so after I fixed the other problems with the video and iptables, I attacked the scripts.

All of this crap, Chat-O-Matic, Ban-O-Matic, the Map, and now this Twitter Shit is all built off data that's scraped from the UT Web Admin interface with bash scripts. There has always been one big problem with approach: it's really hard for all those scripts to tell when a game is over.

The main issue is when the last player leaves when a new game starts but before before the server is populated with bots.

This isn't a problem when the last player leaves an active game. A bot will simply take his place and the bots will finish the game off themselves, which is good since another player could jump in.

But the scripts get seriously confused when there are only, say, one or two bots wondering around waiting for players that will never show up. The game has to be reset for everything to work again. This has been a problem for a long time, and I usually ignored it because it usually fixes itself as soon as another game started.

Usually.

Sometimes you just have to SIGKILL all the games and game sub-processes and start over.

So I put the "Game Over" functionality into Ban-O-Matic. Now, if there's nothing but bots playing for more than two minutes, the game is restarted (future expansion: switch to a random map). B-O-M was a good place for this, since it deals exclusively with real players and ignores bots completely.

Plus, aside from banning bad players, B-O-M has very little else to do.

On top of that, I optimized a ton of other scripts and put together a script monitoring system.

With the new kernel and optimized scripts the place is running better than ever.

Kernel 2.6.31 x86_64 on EXP4

Best kernel compile EVAH.

It worked on the first compile and booted up clean for the first time this morning just before 5AM.

In my experience that's damned near unheard of. In fact I did the build over ssh while the system was headless - no keyboard, no mouse, and no video - which is just asking for trouble.

I had to rebuild the nVidia driver to get X up and running so I could get back on the system with VNC. That ran incredibly smoothly with the latest x86_64 release (thanks, nVidia! And, yes, I still hate ATI!). Then ipatbles and ipset had to be brought up to the latest versions. I got everything just right and played a test round on BITCH House.

Smooth as silk. Slicker'n snot on a doorknob.

So I rerouted EXP4 back to the AMD64 box, rebooted yet again, and was back in business before 6:30AM.

I'll be playing today to see how it runs. See you around.

RESOLVED: EXP4 is Experimental

Well, shucks, folks. It turns out my custom-built Linux kernel (based on version 2.6.28.7) was completely fux0r3d.

After fucking around with grub for way too long, during which I found a longstanding Bug That I Should Have Known ABout But Didn't, I set the box back to the stock Debian kernel ("2.6.18-6-amd64") and rebooted. Then I jumped in for a few quick rounds at Bitch House, since EXP4 is currently running on the "new" iPaq (which I have rechristened "iPunk").

The son of a bitch ran like it should. No little red network doobie flashing, no more getting killed by bots that aren't even pointing at me.

Mea culpa, mea culpa, me maxima culpa.

Fuck. Me.

Right now UT is the only thing that is running right on that box. The old kernel killed X/Windows outright, but at least ssh still works so I can get in & twiddle the bits (it runs headless on DinkNet - no vid, no keyboard).

Iptables is also fucked, or so I assume. You have to compile it against your running kernel, so right now its functionality is suspect. Ban-O-Matic is probably fucked. The latest incarnation of BOM relied on ipset, which doesn't come stock on the older kernels. Instead of using ipset, I'll fall back to the old method of letting the iptables rules grow. And grow. This is how it's set up on iPunk right now. Thank you StinkFly for helping test it out (poor schmuck, he's coming in to get banned on a third IP address as I write this).

BOT House runs an earlier kernel, v2.6.28.5, so it was not affected. And OF COURSE at the moment we're between releases for the Next Big Thing™. Kernel 2.6.30 is "stable" (right) but 2.6.31 is supposed to cure cancer (among other things) when it finally comes out.

Not that BOT House doesn't have its own problems. I discovered just before we left town to go to Rinky Dink's wedding that it doesn't work with those little gspca USB webcams I'm so fond of, but that's a minor issue at this point.

So I have all these issues to work out but right now I find myself with a severe shortage of Free Time™. Work has been a serious bitch for the last couple of weeks. I'll blog about that when I have some time to get get pissed off, but I have to keep reminding myself I'm lucky to have a job in this crappy economy.

I will probably leave EXP4 on iPunk for the time being, since both BITCH House and Classic3 get little traffic. That means if the power goes out here (so far we've been very lucky on that front), EXP4 goes down until I do something about it. BITCH & C3 will go up and down as I work on the new kernel. There's no sense sticking with the stable release so I'll hack away at one of the 2.6.31 Release Candidates until I get it just right.

Experimenting with EXP IV

Last week I bought another old PC from the junk computer store, just like the one on the left there (except mine has a CD drive).

What a piece of CRAP, eh?

I always liked those old iPaqs. Back around the turn of the century, they were pretty futuristic. I have a fond place in my heart for them, and since the price was a mere $39.95, I figured what the heck, take it home and have some fun.

It came with Windows 2000 Professional, so I wiped the drive and installed Xubuntu 9.04 on it. Runs Xubu like a top. 1gHz PIII & 512MB of RAM with a 20G hard drive. It's not going to blow anybody's doors off, but it's a nice little "Mail & Web" PC.

But I have one of those already so I decided to play around with it.

I moved a copy of EXP IV over to it, called it "EXP V", made sure it wasn't advertising itself on the Web, and fired it up just for funsies.

And the little bitch pissed me off! It played UT much better than my Hot Shit AMD64x2 with four times the RAM and processing power.

So now I'm mad and I need to know WTF is the deal with the AMD box. I'm not sure where to start but I think a good place to begin is the NIC, although I've been down that road before.

But it BUGS THE HELL out of me. Something has to be done.