Twitter Muzzles BOT House, EXP4

Twitter updates from BOT House and EXPIV stopped abruptly on September 1st when Twitter stopped supporting BASIC authentication.

This is both good and bad.

Bad because I have no idea how to use their new authentication model. I'm sure I can hack it out sooner or later, but I have other weenies to roast at the moment.

Busy, busy, busy!

Good because, face it, BASIC authentication sucks ass. It's so barely encrypted (in BASE64) that it might as well be clear text. It's a method to be avoided at all costs. But, it's also a good quick hack, something I'm very good at.

So for now, no more Twitter updates from BH or EXP4.

You can still tweet me if you feel like it.

UPDATE!

09/13/2010 — BOT House is back on Twitter!

Pussyfooting Does It AGAIN!!!

Last night at about 7:45PM she shut down EXP IV for the second time in three days!

She wasn't as graceful this time. First I heard the sound of shit getting knocked over before the tell tale "beep" of the UPS off-button.

This was obviously a malicious, deliberate act! Cyberwar in my own Fambly Room!!!

I have since implemented countermeasures to keep it from happening again. A cardboard barricade, secured with spare paving bricks, is the first obstacle. If she gets past that, she has to knock the keyboard out of the way.

My Cat Just Shut Down EXP IV

Jesus.

The UPS is at floor level and evidently she just stepped on the power switch.

Seriously.

To make matters worse it appears to be hung. It's starting up read-only.

This is NOT the first time this has happened, but it is a first for the cat, that's for sure!

Bear with me while I bring it back to life.

UPDATE!

WHAT AN INCREDIBLE PAIN IN THE ASS!!!

Since Miss Kitty kicked the plug, /etc/mtab was not deleted so the system thought all the drives were mounted. The "remount" option would not take and a shutdown with "-F" would not check the partitions on reboot. I had to force a check on the root system while it was mounted, which is normally a HUGE no-no. But it's a JFS partition so evidently that doesn't matter (try it with ext3 and see what happens).

So I finally mounted it read/write, deleted /etc/mtab, and brought the system back up.

Now I have to figure out where to put the UPS so the cat won't step on it again.

The Experiment Continues...

After experiencing the bizarreness of EXPIV for almost a year now, I think I finally have the bugs figured out.

I added a utility called schedutils that lets you set the processor affinity for a given process. EXPIV now runs exclusively on CPU #0 while BITCH House and Classic3 share CPU #1. This has made a big difference as far as I can tell from this end.

I was so disappointed with EXP's performance I was going to install VMWare GSX server and run the games on a 32bit virtual machine. Unfortunately, that is way too much work if I want to stick with the current kernel. That was when the processor affinity idea dawned on me. Sure enough, Debian had a tool, but it didn't come stock with the distribution.

And speaking of Debian, they just announced they would be discontinuing security updates for Debian r4. That's a real shame because DEBIAN 5 STILL CAN'T RUN VNC4 WORTH A SHIT.

Pissed? A little. I can probably squeeze a couple more years out of 4.0 and roll my own security updates, but up to a point it becomes a losing battle with all the code dependencies.

And it's a shame Ubuntu 9.10 turned out to be a steaming pile of shit or I'd be looking into moving in that direction.

Google Pollution via Twitter+UT99

Try this search for ZSnDYcunt on Google.

Great lulz.

Of course, ZSnDYcunt (ZSnDY for short) is a bot on EXPIV. Only the first two Google hits are auto-tweets from the server, the rest are various other (sometimes bizarre) sites that have picked up on the tweets from EXPIV for whatever reason.

Real interesting. Just goes to show there's a lot of crap on the Web. I'm glad to know at least some of it is mine!

ipset v4.1

On November 10th, ipset v4.0 was released.

Or maybe it escaped, because less than 24 hours later ipset v4.1 came out.

So it's been two weeks and ipset v4.2 is nowhere to be seen, so I finally bit the bullet and installed ipset v4.1 on EXP IV.

You may recall the upgrade to kernel 2.6.31 on EXP IV was declared a huge success at the time. Well, that was back in September. As the days and weeks went by I started noticing some performance issues. It was something like LAG, but different. Things would freeze - but you could still move - then the world would snap back into place and you'd be somewhere else.

Usually dead.

Annoying, but - for the most part - playable.

The BOT House upgrade was a disaster and resulted in a complete rollback, so I knew somewhere, something wasn't right.

I suspected iptables, but EXP IV uses a very small subset of iptables, namely, ipset. And the issues that plagued BOT House never showed up on EXP IV, so I let it roll.

Until now.

I've only played a few games but it seems OK. I also blew out the Ban List once again, for testing purposes (only on EXP IV - the BOT House Ban List was untouched).

If this upgrade pans out, I will probably revisit the 2.6.31 upgrade on BOT House next weekend.

I'm sure ipset v4.2 will be out by then.

Script Haxx

Upgrading to 2.6.31 went so fast Saturday morning I was at a loss for things to do over the weekend. Before shutting down the servers to rebuild the kernel I noticed the scripts were running wild, so after I fixed the other problems with the video and iptables, I attacked the scripts.

All of this crap, Chat-O-Matic, Ban-O-Matic, the Map, and now this Twitter Shit is all built off data that's scraped from the UT Web Admin interface with bash scripts. There has always been one big problem with approach: it's really hard for all those scripts to tell when a game is over.

The main issue is when the last player leaves when a new game starts but before before the server is populated with bots.

This isn't a problem when the last player leaves an active game. A bot will simply take his place and the bots will finish the game off themselves, which is good since another player could jump in.

But the scripts get seriously confused when there are only, say, one or two bots wondering around waiting for players that will never show up. The game has to be reset for everything to work again. This has been a problem for a long time, and I usually ignored it because it usually fixes itself as soon as another game started.

Usually.

Sometimes you just have to SIGKILL all the games and game sub-processes and start over.

So I put the "Game Over" functionality into Ban-O-Matic. Now, if there's nothing but bots playing for more than two minutes, the game is restarted (future expansion: switch to a random map). B-O-M was a good place for this, since it deals exclusively with real players and ignores bots completely.

Plus, aside from banning bad players, B-O-M has very little else to do.

On top of that, I optimized a ton of other scripts and put together a script monitoring system.

With the new kernel and optimized scripts the place is running better than ever.

Kernel 2.6.31 x86_64 on EXP4

Best kernel compile EVAH.

It worked on the first compile and booted up clean for the first time this morning just before 5AM.

In my experience that's damned near unheard of. In fact I did the build over ssh while the system was headless - no keyboard, no mouse, and no video - which is just asking for trouble.

I had to rebuild the nVidia driver to get X up and running so I could get back on the system with VNC. That ran incredibly smoothly with the latest x86_64 release (thanks, nVidia! And, yes, I still hate ATI!). Then ipatbles and ipset had to be brought up to the latest versions. I got everything just right and played a test round on BITCH House.

Smooth as silk. Slicker'n snot on a doorknob.

So I rerouted EXP4 back to the AMD64 box, rebooted yet again, and was back in business before 6:30AM.

I'll be playing today to see how it runs. See you around.

RESOLVED: EXP4 is Experimental

Well, shucks, folks. It turns out my custom-built Linux kernel (based on version 2.6.28.7) was completely fux0r3d.

After fucking around with grub for way too long, during which I found a longstanding Bug That I Should Have Known ABout But Didn't, I set the box back to the stock Debian kernel ("2.6.18-6-amd64") and rebooted. Then I jumped in for a few quick rounds at Bitch House, since EXP4 is currently running on the "new" iPaq (which I have rechristened "iPunk").

The son of a bitch ran like it should. No little red network doobie flashing, no more getting killed by bots that aren't even pointing at me.

Mea culpa, mea culpa, me maxima culpa.

Fuck. Me.

Right now UT is the only thing that is running right on that box. The old kernel killed X/Windows outright, but at least ssh still works so I can get in & twiddle the bits (it runs headless on DinkNet - no vid, no keyboard).

Iptables is also fucked, or so I assume. You have to compile it against your running kernel, so right now its functionality is suspect. Ban-O-Matic is probably fucked. The latest incarnation of BOM relied on ipset, which doesn't come stock on the older kernels. Instead of using ipset, I'll fall back to the old method of letting the iptables rules grow. And grow. This is how it's set up on iPunk right now. Thank you StinkFly for helping test it out (poor schmuck, he's coming in to get banned on a third IP address as I write this).

BOT House runs an earlier kernel, v2.6.28.5, so it was not affected. And OF COURSE at the moment we're between releases for the Next Big Thing™. Kernel 2.6.30 is "stable" (right) but 2.6.31 is supposed to cure cancer (among other things) when it finally comes out.

Not that BOT House doesn't have its own problems. I discovered just before we left town to go to Rinky Dink's wedding that it doesn't work with those little gspca USB webcams I'm so fond of, but that's a minor issue at this point.

So I have all these issues to work out but right now I find myself with a severe shortage of Free Time™. Work has been a serious bitch for the last couple of weeks. I'll blog about that when I have some time to get get pissed off, but I have to keep reminding myself I'm lucky to have a job in this crappy economy.

I will probably leave EXP4 on iPunk for the time being, since both BITCH House and Classic3 get little traffic. That means if the power goes out here (so far we've been very lucky on that front), EXP4 goes down until I do something about it. BITCH & C3 will go up and down as I work on the new kernel. There's no sense sticking with the stable release so I'll hack away at one of the 2.6.31 Release Candidates until I get it just right.

Experimenting with EXP IV

Last week I bought another old PC from the junk computer store, just like the one on the left there (except mine has a CD drive).

What a piece of CRAP, eh?

I always liked those old iPaqs. Back around the turn of the century, they were pretty futuristic. I have a fond place in my heart for them, and since the price was a mere $39.95, I figured what the heck, take it home and have some fun.

It came with Windows 2000 Professional, so I wiped the drive and installed Xubuntu 9.04 on it. Runs Xubu like a top. 1gHz PIII & 512MB of RAM with a 20G hard drive. It's not going to blow anybody's doors off, but it's a nice little "Mail & Web" PC.

But I have one of those already so I decided to play around with it.

I moved a copy of EXP IV over to it, called it "EXP V", made sure it wasn't advertising itself on the Web, and fired it up just for funsies.

And the little bitch pissed me off! It played UT much better than my Hot Shit AMD64x2 with four times the RAM and processing power.

So now I'm mad and I need to know WTF is the deal with the AMD box. I'm not sure where to start but I think a good place to begin is the NIC, although I've been down that road before.

But it BUGS THE HELL out of me. Something has to be done.

EXP IV Has Been Twitterized!

Like you didn't see that coming.

I promise to stop right there.

Getting it all together today, I think I went over the 150 tweets/hour/IP because everything on both servers stopped working and spontaneously started back up again a little later. To get around this limit, I'm running EXP IV's tweets through an anonymous proxy in Indonesia.

Anyway, you can follow EXP IV here. You can also just peek over there to the right if you want because I changed the Twitter updates feed to EXP IV.

Enjoy!

Nali BITCHes

I stopped by the Unreal Admin Page today, something I rarely do anymore. Somehow, one thing led to another and I bumbled across the NaliWeaponsIIX package. Evidently it's been around for about five years, but it was new to me so I figured what the heck - give it a shot.

After all, I haven't done anything new and exciting with the servers in a long time.

So I put it on BITCH House.

Does anybody play UT2004 anymore?

Yeah, I didn't think so. Anyway, I stumbled across my cdkey while looking through some old files on the Proxy Project system and found out what I was doing wrong. I'd bring it online if there was any interest, but the truth is no one really played it much when it was "popular".

Alien Arena

I have found this game mildly amusing. Even bothered to recompile it for AMD64, so it can go on the new box any time. I played it online a couple of times but there was not much action on the few servers that were out there.

EXP IV Move

This didn't make much difference, although one thing definitely improved. Before the move any network activity caused the system to report high (90%) network utilization. Now it's dropped down to something like "normal". Still, it didn't help.

However, dicking around with the system today I happened to notice the "hinky" account (that's me) was getting flooded with email from the cron process, causing disk utilization to hit the roof.

It made no sense. It was cron activity from root, and nowhere in root's config or in crontab was there any directive to send email to "hinky"! This is not the first time I've had issues with email filling the disk up and I thought I had it hammered. Now, it's fixed (famous last words) and nobody is getting any email at all on the system.

And I don't mean the "nobody" account.

A UT Experiment This Weekend

I ran across the "linux32" command the other day, which is supposed to force 32 bit programs to run as 32 bit programs in a 64 bit environment.

From the online descriptions I can't tell if it actually does anything astounding (or at all), but I started up all three ( EXP4, BITCH House, & Classic]i[ ) servers on the AMD64 box with this thing anyway.

Play was smooth, but... what the fuck... it's always smooth for me!

We'll see what happens.

I've been playing as PIG_VIRUS for a couple of weeks now.

See you online.

Happy Fragging!

NIC Death/Resurrection

Today I moved EXP IV to its Final Resting Place (FRP), hooked it up to its woefully underpowered UPS, re-installed NUT (Network UPS Tools), played a quick game, and forgot about it.

About four hours later I noticed the sucker dropped off the network.

I guess it took that "Final Resting Place" concept too literally.

The FRP, where EXP III used to live, is at the end of the longest wire on the network. It leaves the switch, travels across the fireplace mantel, and drops down to the other side of the fambly room. Maybe twenty five or thirty feet max. Apparently, the built-in NVidia NIC was unhappy with that situation, because I plugged my laptop into that wire and it was happy.

So I added a RealTek NIC (RTL 8129C) and edited the udev settings to tell the OS eth0 was now eth1 and eth1 was now eth0 and rebooted.

Maybe this had something to do with the reported lag. Maybe it will be better. Maybe it will die again.

I don't know. If you see any improvement drop me a note.

DAMN NICE BOX!

Although not exactly bug-free, EXP4 went online yesterday as (sorta) planned.

It plays exceptionally well. The only problems I've noticed are at the beginning when you first log on. Play is a little choppy, but it smooths out fast and remains smooth game after game.

That was the best $139.95 (+S&H) I've spent in a long time. I"m slightly pissed Debian 5.0 ("Lenny") sucked as badly as it did, but I have a Deb 5 VM and I'm going to keep it updated to see if they ever get around to fixing that two year old Gnome vs. VNC4 problem. Since that is a fight between two third parties, they (Debian? VNC? Gnome?) may never get around to fixing it. That was a major disappointment.

As (almost) expected, kernel.org put out Yet Another Revision (2.6.28.7) on Friday. I would have let it slide but there were enough major issues to cause concern. I got that all compiled and ready to run just before putting EXP4 online. But I didn't have the time to upgrade the old server BOT House lives on. That will likely happen before next weekend.

Ban-O-Matic is broken at the moment so I've been banning our buddy MISERABLE_S.O.B. by hand whenever he changes his IP. I'd ban his entire ISP but they have a pretty large CIDR block (I'd call it a Class B+). I'm confident B-O-M's problems are trivial so I'll be taking a look at it while I'm at work tomorrow (I do some of my best work at work - don't tell my boss).

Today I've been hacking around with the old EXP III box. I upgraded it to 2.6.28.7 and I'm trying to get those awful Marvell wireless cards to run on it (no luck, although there is now a native loadable module for some Marvell NICs - ones I don't have, apparently).

I've also been hacking around with TrueCrypt, which is an awesome file & disk encryption package that was undoubtedly written by a team of paranoid schizophrenics. They have all the angles on encryption hammered down, with an emphasis on plausible deniability.

This is a program designed from the ground up for people who need to hide what they're doing.

Of course, I have a professional interest (that's my story and I'm sticking to it).

I've been spending long hours digging through my old hard drives, wiping then and encrypting them so I can throw them away without worrying about dumpster-diving tweakers stealing my personal information. Although, to be fair to the tweakers, the credit card processing industry itself is doing a great job losin my personal info in massive security breaches as it is.

Truecrypt takes about two hours to prepare a lousy 40G hard drive (at least it takes that long on he old EXP III box), so it's been quite a hassle. And I still have drives that spin but won't format and are therefore technically susceptible to advanced forensic techniques. Plus I seem to have more SCSI drives than I remember ever buying (vintage mid-90s drives in onsie, twosie, and foursie gigabyte capacities).

And - silly me - I built the EXP III box without SCSI support. Considering it takes three hours to build a kernel from scratch (a little less when you've pre-built it), that probably won't happen soon.

In the end I may just end up taking a hammer to them all, but for the ones that work this is my last chance to see if there's any long lost, forgotten data that I may still need on them.

TrueCrypt was made for Windows. It's different on Linux and there are a few ducks to put in a row before you start using it. The information is out there on the Web but I haven't seen a single respectable "HOWTO" on the subject yet. I will probably end up writing one for my own benefit since that's the only way I can remember all this crap.

If I do you'll find it here, so stay tuned.

But no promises!

Hinky Is A Twit

Not sure why, but I decided to get a Twitter account. I will be documenting (they call it "tweeting" - how cute!) what's left over of this monster called the EXP4 Migration.

I got on Twitter before they got hacked. The plan was to integrate UT with Twitter - somehow, some way. It's very simple to use with curl and bash scripts (which is probably how they got hacked in the first place - I'm not sure where that stands now).

I have absolutely ZERO FOLLOWERS.

You could be the first!

Check it out.

EXP4 - First Incarnation

The boxes were sitting on the front porch when I got home from work today. I unpacked everything, slapped it together, found an old hard drive and a CD ROM, and did a quick default install of Debian 4.0r6/AMD64.

It feels very nice. Every bit as smooth and responsive as the MythTV box I built last year. In fact I'm using it right now to write this blog entry.

The hardware leaves a few things to be desired. The power supply has one fan and no airflow through the bottom. The power cables are almost too short. The main just reaches the power connection on the motherboard. It's very quiet as is, but it will definitely need another fan.

The case was too big, but I used the MiniATX case I had on deck. Just had to move the power supply from the new case into it. I still may use the new case for some old hardware, but that is going to be a big screwdriver project for some other day.

There are still some configuration decisions to be made, so I probably won't have this thing running until next weekemd. During the transition I may run EXP /// on the VM I built for the proof of concept. I may also get completely distracted and throw Windows 7 on this thing just for giggles.

You never know.