EXP /// Risen From The Ashes

I dragged the old NetVista out of its resting place and experimented with dropping connections when players request a local file.

It doesn't look like it's gonna happen, boys and girls. Although there is a unique byte sequence at the beginning of every UT mod file, the server apparently doesn't start sending data exactly from byte zero. Sometimes it does, sometimes it doesn't. But mostly it doesn't. Scratch that idea (but I do have a Plan B).

This old box didn't take well to sitting in a corner for the past year. Booting it was iffy. Sometimes it took six power on/off cycles to come up, so on Christmas Morning, in true Busman's Holiday style, I replaced the boot drive, which was a tiny 6G Maxtor with a manufacturing date of 10/30/1998.

Whoa. No wonder it had problems booting.

I replaced it with a 20G Maxtor that had "GOOD" written on it in Sharpie marker. That's the only way I can keep track of these things anymore (I have a huge box of "BAD" Sharpie'd CD ROM drives in the basement. I'd throw them out but the spiders think they're upscale condos).

I used Clonezilla to copy the drive. It was the first time I've ever used it and was quite pleased withe the result, although the user interface leaves much to be desired.

I moved the /var partition to the "new" drive and put the box back together. Then just for the Hell of it I upgraded the kernel to 2.6.32.2 (yes, while you weren't looking they up-revved it twice). Re-compiling the kernel killed about four hours.

Now it runs better than ever, with no sign of the DMA issues BOT House had, even though I raided its RAM to beef up the old iPaq I've been twiddling around with (it suffered a horrible fate when I attempted to upgrade it to Ubuntu 9.10, BTW).

The only complaint I had with the new kernel was that damned Linux penguin (Tux) appeared out of nowhere on the boot screen. I've always hated that little rat bastard ever since they made it the official logo (this is blasphemy, BTW) and I had one Hell of a time getting rid of the little fucker. It happens there's a little known kernel default option - logo.nologo - that shuts him off, although I've never had to use it on any other box.

So that got me wondering if I could put my own logo on the boot screen. It's possible, but you have to recompile the kernel to make it happen. I thought it would be cute to have a little Hinky head in there, but I just couldn't seem to get a good 80x80 pixel version in the required 16 color (not 16 bit color) console format. But I wasted hours playing around with it.

HOURS.

So anyway, that was my Christmas. I hope you had more fun than I did.

More BOT House Twitter Pollution

I was somewhat amused to find BOT House's tweets echoed on this real estate page for Henrietta, Florida.



I captured the page for posterity.

70+ Hours On 2.6.32

I am declaring myself the victor in this epic battle against kernel 2.6.32!

Disabling DMA on all IDE interfaces did the trick. If BOT House was a file server I'd probably be pissed at the resulting loss in performance, but it's not, so I'm a happy camper. And if there has been a drop in performance I haven't noticed it (nor do I have a baseline to compare any tests against - tests which I haven't performed anyway).

UT Files seems to work OK. There is one music file in Classic ]I[ on one map (formerly) in BITCH House (DM-Clementine) that had issues with the remote versions, but it is a very fast redirector.

Stunningly fast.

In fact, it's almost too good to be true. I don't see it lasting for a long time, although the guy has over 1000 UT servers as clients. But, he depends (???) on donations, so how long can that last? The cheapest plan at his hosting provider is $75/mo ($900/year).

Maybe he makes the Big Bucks with ads (I wouldn't know, I haven't seen an Internet ad in years). Regardless, I have no Plan B if this guy goes down, which bugs the HELL out of me.

With that in mind I'm going to take another look at trashing the connections of players who download directly from my servers. I didn't find anything the first time around, but I have devised a better approach this time, using one of the many VMs I have with UT99 support built-in (or perhaps the old EXP III server itself - it hasn't been powered up in months and the last time it was powered up it was having "issues"). So that's back in the Master Plan.

I still plan to consolidate the GoDaddy servers onto their buggy Linux platform. That issue is really weird. Sometimes it redirects, other times it doesn't. To make matters worse, the log files available to me don't seem to bother to report a 302 redirect when it happens. This must be some kind of weird load-balancing hardware that they haven't ported to their "beefy" Windows servers yet (hence the "not a good long-term solution" comment by "Just Jonathan" of GoDaddy fame). Internally, something appears to be juggling IP addresses around. Although the IP of the server externally is 97.74.26.128, the logs show that the last octet is changing frequently. BUT the different address is not in the redirect responses. WTF is up with that?

ATTENTION: "Just Jonathan" at GoDaddy

I told you I'd have the #1 Google Search for "just jonathan godaddy" on Sunday and now I have it.

Enjoy.

You bastard!

Fuck GoDaddy and "Just Jonathan"

And the horse they road in on.

I went to The Unreal Admin Page originally to rant about GoDaddy and their suck-ass Tech Support but I ended up just browsing through the forums. While there, I found someone promoting an Unreal mod redirection service, ut-files.com.

Since they had all the files required for Classic ]I[ Online, I set them up as the redirector. I'm not really comfortable with this decision, because I can't stand relying on someone else to always to the Right Thing, another reason I set up the GoDaddy account originally. But since GoDaddy has demonstrated they are incapable of doing the Right Thing anyway... well, why not use ut-files.com?

I contemplated this switch for some time, but in the end the deciding factor was speed. UT Files turned out to be three times faster than GoDaddy's junky servers. And, there's no "Terms of Service" agreement at UT Files (or at least none that I can find so far).

So I opened an account and I'll be uploading the files they're missing that are required to run BITCH House (not many, actually), at which point I'll move it over. BOT House has nothing special besides FuckIdlers, my custom-hacked version of KickIdlers.

And apparently everything needed by the Too Many Mods server is there as well.

There's another part of the puzzle I'm looking into: dropping the connection when a player requests a file from poor, bandwidth-limited me instead of the redirection service (naturally, via iptables). If I can nail that one down, you can kiss lag good-bye forever.

UPTIME: Forty-Five Hours

This is the longest-running BOT House 2.6.32 kernel yet.

I was going to bounce it this morning "just because" but I think I'll let it run. Besides, Pinky Dink has the day off and I can have her bounce it if worse comes to worse.

I am looking into some 3rd party hosting options for the UT mods and in the meantime I've been twiddling file attributes at proxyobsession.net to see if I can get around the 302 redirects reliably, regardless of whether I'm violating GoDaddy's Terms of Service or not. That seems to be working, but it "seemed to be working" last time I messed around with it, too.

During all that messing around I discovered a few mod files were missing, so I uploaded them. One was BP4Handler7C.u.uz, which is part of UTPure. It's a file everyone needs when they play BH. Since it was missing it was probably the cause of some short-lived lag on the server.

Anyway, it's Monday morning. Gotta run.

GoDaddy SCREWS THE POOCH AGAIN!

As if I don't have enough problems...

I just got off the phone with "Just Jonathan" of GoDaddy. He's not getting a favorable Customer Satisfaction survey from me.

Here's the scoop: as I mentioned earlier, I have two hosting accounts at GoDaddy. A new, Linux-based account and good old reliable mrhinkydink.com, which is Windows.

Up since 2005, mrhinkydink.com is where you get your UT mods from when you play on any of my UT servers. Since Day One, that was the reason I opened the account, and it has worked well for that purpose. However, getting it hosted on Windows was a mistake in the first place. Now, I want to put everything on the Linux server.

Sadly, it can't handle UT mods!

When you ask for a mod, your UT console sends an HTTP request - a regular, "nothing special" HTTP request like millions of other http requests made every microsecond - for the file. The Web server then sends you that file, no questions asked. Sounds easy? Maybe it's too easy. GoDaddy's Linux servers send UT a "302 Moved Temporarily" redirect back to UT, which it dies not understand.

For instance, if you ask for SkeletalChars.u.uz in the "utmods" folder, their Linux (Apache) server responds like this:

HTTP 302 Moved Temporarily
Location: /utmods/SkeletalChars.u.uz?29e15220

... whereas the Windows (IIS6) server responds:

HTTP 200 OK

...and sends the file. Notice the appended "?29e15220"? What the HELL is that? UT doesn't understand and drops back to downloading it from my server. I don't have the bandwidth for that. I never have had the bandwidth for that, which is why I got the GoDaddy account in the first place.

I explain the situation to "Just Jonathan", who puts me on hold while he goes to talk to Someone Who Should Know About These Things.

"Just Jonathan" comes back to tell me that BY SERVING UT MODS I'M VIOLATING THE TERMS OF THE SERVICE AGREEMENT and they don't have to support my issue. He says it's not a good "long term solution" (although I've been doing it for ALMOST FIVE YEARS NOW on a GODDAMNED WINDOWS BOX I NEVER WANTED IN THE FIRST PLACE) and if I really want to do this kind of thing I need to buy a dedicated hosting account.

Some "Tech Support" that is. Can't fix a problem? Upsell the sucker to a dedicated account and let him do his own fucking tech support.

UnFUCKINGbelievable.

Losing Battle?

Although uptime is now 20 hours - the best since the initial install last weekend - I am not optimistic about 2.6.32 on BH.

I jumped the gun yesterday and re-compiled for SMP & Hyperthreading. And I did a quick test to see if I could lose the SiS5513 IDE driver (which has ZERO native options for disabling DMA) and run with the generic EIDE driver. No such luck. That experiment left me with an unbootable system (easily repaired, though).

I also considered the possibility that the drive is going bad. Since it supports S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) natively I downloaded a few tools to see what the drive had to say about itself. It appears to be happy and healthy, although the "uptime" value is blatantly wrong. It says it's been up for 191 days, but in reality that drive has been running 24x7 for almost two years now (do the math). If that's wrong, the health report may be bogus as well.

The thought occurred to me to check a Release Candidate (RC) version of 2.6.33, but I ran across this little bit of bad news in the ChangeLog...

The current Kconfig text for CONFIG_IDE doesn't give a hint to users that this subsystem is currently in maintenance mode and isn't actively developed.

Yes, kiddies, IDE is nearly extinct. This motherboard doesn't do SATA and since it only has a grand total of ONE PCI slot (currently occupied by the second NIC), SATA isn't going to happen. But the thought occurred to me that someone, somewhere must make a combination SATA/Ethernet card so I started searching and actually found one.

One.

And it's no longer in production, probably because it could do SATA or Ethernet, but not both at the same time.

So right now I'm waiting on the next crash. In preparation for that I think I'll fire up a Deb 4 VM and see what I can get out of the latest stock Debian kernel.

So much for THAT theory...

And now for something completely different.

I had set up netconsole to monitor BH's problem remotely. Netconsole sends debugging messages across the network to another system so you can see what's happening up to the moment of a crash.

And here we have it:

hda: ide_dma_sff_timer_expiry: DMA status (0x20)
hda: DMA timeout retryhda: timeout waiting for DMA
hda: DMA timeout: status=0x58 { DriveReady SeekComplete DataRequest }
hda: possibly failed opcode: 0x35
hda: status error: status=0x58 { DriveReady SeekComplete DataRequest }
hda: possibly failed opcode: 0x35
hda: drive not ready for command
ide0: reset: success

It's not a NIC driver issue after all. It's the drive hanging. Or rather, some kind of DMA issue with the drive and the controller.

So it's back to the drawing board.

But right now I have shut off DMA on the drive manually, so I expect things to be OK while I re-hack the kernel.

Looking Good...

... so far everything seems O.K.

Uptime is over 20 hours. If this works out, I'm going to re-hack the EXPIV kernel and get rid of the 8139cp driver on it as well. Even though there aren't any visible problems there, better to get rid of it if it's not needed.

If BOT House makes it until Sunday, I'm going to re-hack the BOT House kernel to re-enable SMP and Hyperthreading, since it was never an issue in the past.

I took some time to browse the code for both RTL8139 drivers to see if there are any obvious clues to problems like the ones I have been having. Nothing in there jumps out at me. They seem to share a lot of common code, but at first blush the 8139too driver appears to be not quite as complex as the 8139cp driver.

But who knows?

Last night after I took out the 8139cp driver, I had to wait out a bunch of players. It took over THREE HOURS for everyone to leave, but it was worth the wait. Since I didn't have anything to do but wait, I jumped in and played too. It was a good crowd until Koga showed up and started beating everyone's asses. She managed to clear everyone out pretty quick.

I knew she was good for something.

One More Time

Well, those were all great ideas. Or so I thought.

BH still hung, with the same symptoms. Back to the legendary drawing board.

Since time immemorial (the 90s), when I had a stack of 10mB NE2000 NIC clones, I learned a Wonderful Thing about Linux and NIC driver modules, specifically, of course, NE2000 NICs.

If you had a system with two NE2000s, without fail the kernel would only detect the first one at boot time (and, yes I tried boot time parameters). If you had one NE2000 and one SMC NIC, the kernel would detect them both. I eventually found that if you compiled the NE2000 driver into the kernel that both NICs would be detected. So that's what I did.

Time went by and the stack of NE2000s went into the landfill to be replaced by a stack of RealTek 100mB NICs (RTL8139s). I found the same issue and ran across another: not all RTL8139s were created equal. Some liked one driver (8139cp) and some liked another (8139too). I ended up building both drivers into the kernel because for one thing, you can't tell which NIC will like which driver just by looking at them (I have at least one, now marked with a Big Black "X", that doesn't like either).

This may not have been a good idea, although once again this is exactly how EXPIV is set up, except EXPIV has only one NIC.

So tonight I rebuilt the kernel with only the 8139too driver, which seems to be the preferred way to go from what I've been reading on the Web.

I am not optimistic, but one of the symptoms appears to be that the CPU is spending 100% of its cycles on IO when the system hangs, at least according to the Gnome panel system applet that runs on the desktop. That, combined with the odd network behavior BH shows when it's hung, leads me to believe the drivers may be confused about which one is in charge.

Maybe I'm grasping at straws at this point, but time will tell.

This Shit Is Killing Me

As you may have realized by now, the glowing reports about all being fine with the kernel upgrade on BOT House were premature.

BH is now choked. It's not taking new connections. Oddly enough the old connections are working fine. Odder still, anything that passes through BH (which, among other things, is the firewall here on DinkNet) is hunky-dory.

In fact I'm passing through BH right now to blog about this happy horseshit.

BH survived the initial upgrade for about 36 hours. When it died Monday morning (when I was absolutely unprepared to do anything about it), everything else on the network was humming along fine. I bounced it and things were, again, fine. Sometime during the day it choked again, but my connections from work to home stayed up. But I could do nothing on BH remotely.

This really pissed me off, but it made me all the more determined to figure out WHAT THE FUCKING FUCK IS WRONG WITH THIS FUCKING BOX.

Monday night when I bounced it again I started to simplify the firewall rules. And I installed conntrackd to do some statistics on the firewall.

Tuesday morning, all was well. I zipped off to work, nearly getting killed in the process (long story - almost got run off the road but I swerved clear and the car that almost hit me hit someone else and they both ended up crashing into the restraining wall), sat down at my desk and at about 10AM everything died again.

Plus, my connection from work went with it. I could not reconnect, but, again, everything going through BH was fine, even new connections. It was starting to look more and more like a firewall issue.

So Tuesday evening I took a closer look at everything. I shut down the proxy server on BH, which left ssh and nfs as the only services on the box (besides UT, that is).

I played a few EXCELLENT rounds of UT on BH (people started hitting it as soon as it was back up) and hit the sack at about 11:30PM.

I woke up the next morning (Wednesday) to find the box fucked again. It appeared everything choked right after midnight.

In fact it was becoming clear that every time it choked, it was at xx:02 AM or PM, which is meaningful since that is when the proxy project box does its all of its dirty work (this particular system continues to crank away while BH is down, BTW).

So I bounced the box, went off to work, and the thing dies once again on the hour of 10AM plus change.

This time around I had set up an alternate, pass-through ssh connection so I don't get locked out like Tuesday. It tunnels through BH directly to the box running EXP IV, which still shows no adverse reaction to the same kernel upgrade (apples to oranges? Same everything except it's a 64bit AMD dual core and BH is a 32bit Intel single core... hmmm...).

So that's where we were at Wednesday. Down.

When I got home Wednesday, I noted the time of the last conntrack log (once again "on the hour"), rebooted, and sat down to generate another 2.6.32 kernel image, which takes about two hours with the Debian make-kpkg tool.

This time around I took out SMP (Symmetrical Multi Processing) and Hyperthreading support (actually, hyperthreading simply disappeared as an option after SMP was removed). It is a single CPU box, after all, but it is a P4 and SMP support never seemed to matter in kernels past. While it was cranking away at the code I hit BH to see how that affected performance of UT, since building the kernel was chewing up most of the CPU cycles. No problems there.

Once the kernel & modules were built and installed, I had to rebuild iptables, ipset, conntrackd support and tools/libs, and reboot one more time.

Now, on Thursday (12/172009), BH has been running for a little less than twelve hours. Whether it will keep running is anyone's guess. I am optimistic that removing SMP was the way to go, since EXP IV, a true multicore system, has had zero problems with this kernel version.

And now I can get back to my other projects, like messing around with the TOO MANY MODS Windows UT99 server (I will be taking requests, so if you have some favorite maps or other UT99 extensions, let me know).

One slightly positive outcome of all this is that I leveraged a private (at the moment) Websense hack to get the pass-through connection back to EXP IV to work through the corporate proxy. It's an extremely small and elegant hack for completely bypassing monitoring and filtering that I've been working on for a few months now. I have contacted Websense but they seem to be ignoring me. It may be a unique flaw in our environment (I suspect it could be the Microsoft ISA servers), but my testing facilities are limited. We have a Websense upgrade scheduled for January and if the hack survives the upgrade I plan on shoving it up Websense's ass.

Or, I may just keep it in my private toolbox.

Classic ]I[ Files Moved

Since August I've been running two hosting accounts at GoDaddy, www.mrhinkydink.com and proxyobsession.net. Proxy Obsession is WordPress, which I really like. Mr. Hinky Dink is (ugh) some kind of FrontPage compatible Windows box, which was a mistake from Day One.

It seems I clicked the "back" button when I was signing up and the account defaulted back to Windows (I wanted Linux). But since back then (2006) all it was doing was holding UT mod files I didn't bother with moving it to Linux like I wanted in the first place.

Anyway, I want to change all that and move everything to the Linux box and consolidate the two accounts/domains.

With that in mind today I moved the Classic ]I[ files to the Linux box. Soon I hope to have everything moved, but it's a lot of files and transferring them really sucks up the UT bandwidth. Downloading them isn't bad, but uploading is a KILLER.

And it turns out you can't simply transfer files between boxes at GoDaddy. You have to download/upload to move things around. Luckily the Proxy Obsession box has SSH/SFTP, so I can trim the speed down to a rate (~64kBps) that still lets people play UT without (much) lag. Naturally, it takes longer but I have the time to waste.

The upgrades appear to have gone without incident. It's now been over 24 hours and everything is hunk-dory so far.

Let's hope it stays that way.

UPDATE 13:40 EST

The Classic ]I[ test worked well, so I'm moving everything else over to Proxy Obsession. But what an incredible pain in the ass this has been!

The fact that it's slow doesn't bother me (it's been an hour and a half so far and I'm only halfway done), but GoDaddy keeps dropping the SSH connection on me. At least WinSCP is smart enough to pick up where it left off, but Jesus Fuck this is annoying. I hope it's not a sign of things to come. If it's OK with the small transfers (the Proxy List, the Map, etc.) I'll be a happy camper.

If this goes well, I may put up a server I use locally. It's called "Too Many Mods" and - you guessed it - it has a lot of mods. It's also a Windows server, which could get ugly, but I've been wanting to experiment with it. All the scripts & shit still work, since they get all their data from the Web interface anyway. It runs on the box that runs the Linux VM that generates the Proxy List. A few more CPU cycles shouldn't be an issue.

FWIW, since I wrote this I ran across a "green ghost" in EXP IV. It might have something to do with the upgrade, but it went away after the game was over. I think I remember those green artifacts from when the new EXP IV box went online earlier this year, but I'm not sure.

What A Pain In The Ass

Linux kernel 2.6.32 + ipset v4.1 + iptables 1.4.6 are now current on BOT House and EXP IV.

Just like last time, the upgrade went without a hitch on EXP IV. Everything is smooth as silk. No issues at all. Period. Well, maybe one. Once again, the NVidia driver had to be replaced because of the kernel upgrade, but that too compiled on the first shot and installed without a problem.

BOT House, on the other hand, would not cooperate.

Sure, the kernel compiled on the first attempt, but then the fun began. The NFS (Network File System) server choked on reboot. This was not good because EXP IV gets all its scripts from an NFS share on BOT House. This turned out to be the Debian NFS start-up script. Fixed. Rebooted. Then iptables wouldn't come up. It suddenly didn't like the order it was started in, so I changed that. Bingo.

BOT House took about four and a half hours to upgrade, two and a half for the kernel build and another two hacking around with the side issues. Now if it stays up without a problem for 24 hours I will be a happy fella. The upgrade to 2.6.31 in September was fine until the next day, when the whole system ground to a halt. I think I did something silly during the configure back then, so this time around I used the stock Debian config. Only time will tell.

Anyway, I'll be playing on and off the whole weekend. See you there.

iptables-1.4.6

Here we go again.

EXP IV now has the latest, allegedly greatest, version of both ipset and iptables. And of course kernel 2.6.31, which is now (naturally) .01 revision behind the times. Or .8 revisions, depending on how you look at it (it went through 7 revisions during its lifetime, the last being 2.6.31.7).

BOT House had some serious issues with 2.6.31 and I have been threatening to try to upgrade it again for some time now. This looks like a good weekend to do so. It's been running a stock Debian kernel (2.6.28.5) without ipsets for a long time now and it bugs the HELL out of me.

None of this should impact playing, but there's always the possibility of the IP address of the server changing.

Keep that in mind if you can't connect.

Google Pollution via Twitter+UT99

Try this search for ZSnDYcunt on Google.

Great lulz.

Of course, ZSnDYcunt (ZSnDY for short) is a bot on EXPIV. Only the first two Google hits are auto-tweets from the server, the rest are various other (sometimes bizarre) sites that have picked up on the tweets from EXPIV for whatever reason.

Real interesting. Just goes to show there's a lot of crap on the Web. I'm glad to know at least some of it is mine!

ipset v4.1

On November 10th, ipset v4.0 was released.

Or maybe it escaped, because less than 24 hours later ipset v4.1 came out.

So it's been two weeks and ipset v4.2 is nowhere to be seen, so I finally bit the bullet and installed ipset v4.1 on EXP IV.

You may recall the upgrade to kernel 2.6.31 on EXP IV was declared a huge success at the time. Well, that was back in September. As the days and weeks went by I started noticing some performance issues. It was something like LAG, but different. Things would freeze - but you could still move - then the world would snap back into place and you'd be somewhere else.

Usually dead.

Annoying, but - for the most part - playable.

The BOT House upgrade was a disaster and resulted in a complete rollback, so I knew somewhere, something wasn't right.

I suspected iptables, but EXP IV uses a very small subset of iptables, namely, ipset. And the issues that plagued BOT House never showed up on EXP IV, so I let it roll.

Until now.

I've only played a few games but it seems OK. I also blew out the Ban List once again, for testing purposes (only on EXP IV - the BOT House Ban List was untouched).

If this upgrade pans out, I will probably revisit the 2.6.31 upgrade on BOT House next weekend.

I'm sure ipset v4.2 will be out by then.

Microsoft Security Essentials Sucks

Microsoft's latest foray into consumer security, Microsoft Security Essentials (MSE), does not live up to Microsoft's advertising fluff.

If you read the LARGE PRINT on the MSE download page, you will see, quite plainly stated, that the product offers "automatic updates".

As it turns out, this is not the case.

If you install this software you will notice your virus & spyware definitions (under the "Update" tab of the application) are often days out of date.

If you look at the Update tab closer, you will find this sentence:

"You should always keep these definitions up to date to help protect your computer against the latest threats."

WH-WH-WHAT?

I have to do this? What happened to the "Automatic Updates"?

Not surprisingly, I'm not the only Microsoft customer who's pissed off about this state of affairs. See this thread at Microsoft Answers for a good laugh.

You can't trust Microsoft to do anything right and you definitely can't trust their advertising bullshit.

I Hate ATI - FINAL EDITION

After a very long hiatus I decided to lose 45 minutes of my life and finally upgrade the drivers for my Radeon X1300 on my "old" (circa 2003 - JESUS! That is old!) Windows XP box.

So I sashayed over to ATI/AMD and looked around for the driver-of-the-month.

To my astonishment, the driver was three months old!

Not only that, it was classified as a legacy driver.

Yes, Windows XP is on its last legs, boys and girls. I suppose I could upgrade it but I very rarely use the Old Girl for anything except running Virtual Machines. It currently hosts the VM for the Proxy Project, which runs 24x7, but it still runs UT99 better than my dual AMD64 Windows 7 laptop (recently upgraded from Vista).

Anyway, I downloaded the last ATI driver and played some UT today. It ran nicely, but it made me nostalgic for the Good Old Days back when a 3.4mHz "single core" P4 was a hot system.

And now I can't even complain about the drivers anymore.

RoadRunner Suckage

The cable went dead last night around 11:30PM. It's back up now (5:15AM) and all the servers are running.

I need coffee. I haven't finished my first cup yet.

Damned glad it's Friday.

Power Outage 09/23/09

Back to normal now.

I had to pull the plug prematurely on EXP4, since it has such a wimpy UPS. BOT House stayed up for about 45 minutes before it shut itself down.

Of course, we ended up with a new IP address, so all your favorites are fucked.

I've had a lot of good luck this year with the power situation (unlike last fall's 48 hour outage after the remnants of Hurricane Ike blew through town). I did notice a crew with a backhoe a couple of blocks away when I came home from work, so I'm going to blame it on them. They have their path all marked out with those little plastic flags and they have a way to go until they're done. Hopefully they won't make the same mistake twice.

BOT House 2.6.31 Upgrade COMPLETE

BOT House has been upgraded to kernel 2.6.31 and once again I got it on the first shot. BAM! Nailed it. Plus the new iptables & ipset utils were installed.

BOT House had been up for 215 days, so I was absolutely positive RoadRunner would dish out a new IP but as luck would have it the old one stuck. This was a big surprise considering Linux decided it had to check all the disks on reboot (I hate that - should've gone with JFS instead of EXT3 on that box), delaying the reboot even longer.

Something happened during the switch-over from the old iptables to the new because our old buddy StinkFly snuck in. Once I reset the firewall rules he was back in Ban Land.

It plays well. Hope you enjoy it.

UT Map Switcher In Production

I got bored at work today and wrote the map switcher for Ban-O-Matic.

Now, after ten minutes of idle time (seems to be a lot of that lately), the servers will automatically switch to a random map.

Also, in the case of BOT House and EXP4, an update with the new map name will be sent to Twitter. These may or may not show up depending on whether or not I used my hourly tweet allotment.

FWIW, it looks like I may be upgrading the kernel on BOT House this weekend, since it went so well on EXP4. That means there will undoubtedly be an IP address change in the near future and your bookmarks will be fucked.

There are still a few tests I'd like to do first, but it looks inevitable.

I really hate changing IPs, but the upgrade is worth it.

Script Haxx

Upgrading to 2.6.31 went so fast Saturday morning I was at a loss for things to do over the weekend. Before shutting down the servers to rebuild the kernel I noticed the scripts were running wild, so after I fixed the other problems with the video and iptables, I attacked the scripts.

All of this crap, Chat-O-Matic, Ban-O-Matic, the Map, and now this Twitter Shit is all built off data that's scraped from the UT Web Admin interface with bash scripts. There has always been one big problem with approach: it's really hard for all those scripts to tell when a game is over.

The main issue is when the last player leaves when a new game starts but before before the server is populated with bots.

This isn't a problem when the last player leaves an active game. A bot will simply take his place and the bots will finish the game off themselves, which is good since another player could jump in.

But the scripts get seriously confused when there are only, say, one or two bots wondering around waiting for players that will never show up. The game has to be reset for everything to work again. This has been a problem for a long time, and I usually ignored it because it usually fixes itself as soon as another game started.

Usually.

Sometimes you just have to SIGKILL all the games and game sub-processes and start over.

So I put the "Game Over" functionality into Ban-O-Matic. Now, if there's nothing but bots playing for more than two minutes, the game is restarted (future expansion: switch to a random map). B-O-M was a good place for this, since it deals exclusively with real players and ignores bots completely.

Plus, aside from banning bad players, B-O-M has very little else to do.

On top of that, I optimized a ton of other scripts and put together a script monitoring system.

With the new kernel and optimized scripts the place is running better than ever.

Kernel 2.6.31 x86_64 on EXP4

Best kernel compile EVAH.

It worked on the first compile and booted up clean for the first time this morning just before 5AM.

In my experience that's damned near unheard of. In fact I did the build over ssh while the system was headless - no keyboard, no mouse, and no video - which is just asking for trouble.

I had to rebuild the nVidia driver to get X up and running so I could get back on the system with VNC. That ran incredibly smoothly with the latest x86_64 release (thanks, nVidia! And, yes, I still hate ATI!). Then ipatbles and ipset had to be brought up to the latest versions. I got everything just right and played a test round on BITCH House.

Smooth as silk. Slicker'n snot on a doorknob.

So I rerouted EXP4 back to the AMD64 box, rebooted yet again, and was back in business before 6:30AM.

I'll be playing today to see how it runs. See you around.

RESOLVED: EXP4 is Experimental

Well, shucks, folks. It turns out my custom-built Linux kernel (based on version 2.6.28.7) was completely fux0r3d.

After fucking around with grub for way too long, during which I found a longstanding Bug That I Should Have Known ABout But Didn't, I set the box back to the stock Debian kernel ("2.6.18-6-amd64") and rebooted. Then I jumped in for a few quick rounds at Bitch House, since EXP4 is currently running on the "new" iPaq (which I have rechristened "iPunk").

The son of a bitch ran like it should. No little red network doobie flashing, no more getting killed by bots that aren't even pointing at me.

Mea culpa, mea culpa, me maxima culpa.

Fuck. Me.

Right now UT is the only thing that is running right on that box. The old kernel killed X/Windows outright, but at least ssh still works so I can get in & twiddle the bits (it runs headless on DinkNet - no vid, no keyboard).

Iptables is also fucked, or so I assume. You have to compile it against your running kernel, so right now its functionality is suspect. Ban-O-Matic is probably fucked. The latest incarnation of BOM relied on ipset, which doesn't come stock on the older kernels. Instead of using ipset, I'll fall back to the old method of letting the iptables rules grow. And grow. This is how it's set up on iPunk right now. Thank you StinkFly for helping test it out (poor schmuck, he's coming in to get banned on a third IP address as I write this).

BOT House runs an earlier kernel, v2.6.28.5, so it was not affected. And OF COURSE at the moment we're between releases for the Next Big Thing™. Kernel 2.6.30 is "stable" (right) but 2.6.31 is supposed to cure cancer (among other things) when it finally comes out.

Not that BOT House doesn't have its own problems. I discovered just before we left town to go to Rinky Dink's wedding that it doesn't work with those little gspca USB webcams I'm so fond of, but that's a minor issue at this point.

So I have all these issues to work out but right now I find myself with a severe shortage of Free Time™. Work has been a serious bitch for the last couple of weeks. I'll blog about that when I have some time to get get pissed off, but I have to keep reminding myself I'm lucky to have a job in this crappy economy.

I will probably leave EXP4 on iPunk for the time being, since both BITCH House and Classic3 get little traffic. That means if the power goes out here (so far we've been very lucky on that front), EXP4 goes down until I do something about it. BITCH & C3 will go up and down as I work on the new kernel. There's no sense sticking with the stable release so I'll hack away at one of the 2.6.31 Release Candidates until I get it just right.

Experimenting with EXP IV

Last week I bought another old PC from the junk computer store, just like the one on the left there (except mine has a CD drive).

What a piece of CRAP, eh?

I always liked those old iPaqs. Back around the turn of the century, they were pretty futuristic. I have a fond place in my heart for them, and since the price was a mere $39.95, I figured what the heck, take it home and have some fun.

It came with Windows 2000 Professional, so I wiped the drive and installed Xubuntu 9.04 on it. Runs Xubu like a top. 1gHz PIII & 512MB of RAM with a 20G hard drive. It's not going to blow anybody's doors off, but it's a nice little "Mail & Web" PC.

But I have one of those already so I decided to play around with it.

I moved a copy of EXP IV over to it, called it "EXP V", made sure it wasn't advertising itself on the Web, and fired it up just for funsies.

And the little bitch pissed me off! It played UT much better than my Hot Shit AMD64x2 with four times the RAM and processing power.

So now I'm mad and I need to know WTF is the deal with the AMD box. I'm not sure where to start but I think a good place to begin is the NIC, although I've been down that road before.

But it BUGS THE HELL out of me. Something has to be done.

GENERAL AMNESTY Aug. 2009

I cleared out the "BADUSERS" firewall IP set today. There were over 1000 IP addresses in both tables (2:1 in favor of EXP4), so it seemed like a good place to start over again.

So, watch for the Bad Guys. They're not hard to spot, especially when they start sliding all over the place.

This is the last day of my two week Busman's Holiday. I never got around to doing half the things I wanted to do, but that's what holidays are for.

I did mess around with the files from that time lapse movie I made a few months ago. The hardest part was actually finding the original files (they were stuck on a seldom used Linux VM). The plan was to make a better quality video, and I finally did, but in the end the file was multiple gigabytes in size. Just too large to do anything at all with.

While Pinky and I were out of town at Rinky & Twinky's wedding I had another small video project going on. We hired a Cat Lady to come by the house twice a day to feed the cats and 'keets.

She seemed like a Nice Old Gal but the bitch ripped us off. She came once a day and I have the whole thing on video (three cameras total were running). If that's the worst of what she did, fine (we haven't counted the silverware yet). After discussing it, Pinky & I decided to let it slide and simply not hire her again next time we go globe-trotting.

So it's back to work on Monday.

Fuck.

Twitterfication Update/Busman's Holiday

Since BOT House and EXPIV were put on Twitter the two servers have generated almost 40,000 tweets.

I haven't managed to control things enough to avoid the 150 tweets per hour maximum, so the servers get cut off quite a bit. Usually in about an hour they're back, chattering away.

If you have a Twitter account, it's not a good idead to "follow" the servers because they'll stomp over everyone else you're following.

At one point, BOT House had over 700 followers (mostly bots, I suppose) but overnight that number dropped to about 200 and has been there ever since. EXPIV is lagging behind with about 140 followers.

This whole exercize has me looking over the Chat-O-Matic code to see how I can liven things up, because, quite frankly, the chatterbots tend to say the same things over and over and over again. Plus there's that need to throttle things back so I don't get banned every hour.

I will be playing a bit more in the next two weeks since I'm on vacation. There will be a break in the action because my wife (Pinky Dink) and myself will be attending the kid's (Rinky Dink's) wedding to his True Love (the future Mrs. Twinky Dink).

Aside from the wedding festivities, this will be Yet Another Busman's Holiday. Sometime after the wedding I'm going to consolidate www.mrhinkydink.com and proxyobsession.net, mostly just to put everything into one account and partly to move the proxy material over to ProxyObsession and perhaps - just perhaps - to make some parking meter change off the content. Traffic to the Proxy List has tripled since last year and although it seems to have peaked for the time being I think it will continue to grow.

Some disruption to UT may be involved, but it shouldn't last for more than a few minutes.

MURDER/SUICIDE > UT99

Yesterday I was poking around in my old posts at the Murder/Suicide Blog when I stumbled across something I wasn't aware of.

It's actually getting traffic.

And what really punched me in the gut was that people are actually posting comments to the cut&paste news stories I drop in there day after day. REAL people. Not linkbaiters. Not blackhat SEO SPAM bots.

Real. Fucking. People.

Not only are they real people, but they're people who in many cases knew or worked with the victims and went the the funerals. People pouring their hearts out looking for meaning, reasons, and prayers.

It stopped me in my tracks, put me in a real weird mood, and made me realize I will never look at that blog in the same light as before.

EXP IV Has Been Twitterized!

Like you didn't see that coming.

I promise to stop right there.

Getting it all together today, I think I went over the 150 tweets/hour/IP because everything on both servers stopped working and spontaneously started back up again a little later. To get around this limit, I'm running EXP IV's tweets through an anonymous proxy in Indonesia.

Anyway, you can follow EXP IV here. You can also just peek over there to the right if you want because I changed the Twitter updates feed to EXP IV.

Enjoy!

BOT House Has Been TWITTERFIED!

I have no fucking idea why, but I finally Twitterfied BOT House. Now, everything you say when you chat at BOT House is sent to Twitter.

It took about three hours total to hack it all together and it was a lot of fun when it finally started working. It's fundamentally just Chat-O-Matic with a Twitter feed. It works exactly the same way.

Unfortunately, there is a 150 Tweets per hour maximum. Hopefully they won't get pissed and ban me.

The original idea behind Twitterfication was to get the converstion inside UT to be reflected on The Map (which has been working flawlessly for months now). It would be a little difficult to do with all of the servers, but it is doable.

Check it out and let me know what you think.

Nali BITCHes

I stopped by the Unreal Admin Page today, something I rarely do anymore. Somehow, one thing led to another and I bumbled across the NaliWeaponsIIX package. Evidently it's been around for about five years, but it was new to me so I figured what the heck - give it a shot.

After all, I haven't done anything new and exciting with the servers in a long time.

So I put it on BITCH House.

Does anybody play UT2004 anymore?

Yeah, I didn't think so. Anyway, I stumbled across my cdkey while looking through some old files on the Proxy Project system and found out what I was doing wrong. I'd bring it online if there was any interest, but the truth is no one really played it much when it was "popular".

Alien Arena

I have found this game mildly amusing. Even bothered to recompile it for AMD64, so it can go on the new box any time. I played it online a couple of times but there was not much action on the few servers that were out there.

EXP IV Move

This didn't make much difference, although one thing definitely improved. Before the move any network activity caused the system to report high (90%) network utilization. Now it's dropped down to something like "normal". Still, it didn't help.

However, dicking around with the system today I happened to notice the "hinky" account (that's me) was getting flooded with email from the cron process, causing disk utilization to hit the roof.

It made no sense. It was cron activity from root, and nowhere in root's config or in crontab was there any directive to send email to "hinky"! This is not the first time I've had issues with email filling the disk up and I thought I had it hammered. Now, it's fixed (famous last words) and nobody is getting any email at all on the system.

And I don't mean the "nobody" account.

EXP IV Has Been Moved

The EXP IV box used to be the longest link on HinkyNet. I had a very long ethernet cable from the Old Days (1995 or 1996) that I used to link it in. The said cable spent many years in the trunk of the Hinkymobile and was used as an emergency measure back in my consultant days, which are now almost ten years behind us.

As a result of the use, abuse, and neglect of this cable, it was quite kinky and probably not a good choice to put "in production". And since I'm a Cheap Sun of a Bitch I was damned if I'd go out and actually buy a decent cable for this purpose.

So I moved the whole damned box across the room, as well as the UPS. I put it on a cable that is about six inches away from being too short. Hopefully this tightens up the whole "collision domain" and complaints of LAG will, if not disappear, be minimized.

But you never know.

Yet Another Dark Ass Blog

Every two-bit security hack, has been, and wannabee (like me) has a dark blog. It's edgy. A throwback to those old h@><0r days of dark blue type on a black background and a shitload of pr0n links.

Although, right now, doing a quick Googie against BlogSpot I can't find any. Not one.

Are they turning away from the Dark Side? Could be. They have, after all embraced (barf) FaceBook and Twitter. Every man Jack.

Which brings me to my latest creation: mrhinkydink.wordpress.com

A couple of weeks ago I mentioned my issues with Google Pages. Well, I got tired of waiting for them to shit or get off the pot, so I moved the Proxy Project Notes lock, stock, and proverbial barrel over to WordPress.

Three hours of cut & paste.

When I was finished I noticed they had an import feature.

Dang.

Lucky for me it didn't support Google Pages. Lucky in that I didn't feel like a complete idiot after finding that feature.

But I made a mental note that they do BlogSpot pages because they pissed me off last week. They flagged Murder/Suicide as a "SPAM blog" and threatened to delete it!

How RUDE!

I may use my newfound WordPress skillz to move it over and see what it looks like. Besides, BlogSpot has some idiosyncracies that piss me off (I'm sure WordPress has a few but I haven't discovered those yet).

After all that cutting & pasting I spent another three hours archiving the Google Pages site "as is" on GoDaddy. Now that I have two backups, Google can go ahead and move that site over at will, like they threatened in the first place. If I like the results - and I do believe I won't - I'll leave it.

In the meantime I'll keep hacking away at WordPress.

Why I Love StreetView

View Larger Map
See the lightpole on the left? It's a killer lightpole.

Some poor woman waiting for a bus got killed when it hit her on the head after a truck knocked it down.

See the bus shelter? Well it was removed sometime between the time the StreetView was recorded and the woman got killed.

Pissed off the local residents (see the KDKA News video).

If that bus shelter had still been there it might not have been fatal.

Awesome stuff.

A UT Experiment This Weekend

I ran across the "linux32" command the other day, which is supposed to force 32 bit programs to run as 32 bit programs in a 64 bit environment.

From the online descriptions I can't tell if it actually does anything astounding (or at all), but I started up all three ( EXP4, BITCH House, & Classic]i[ ) servers on the AMD64 box with this thing anyway.

Play was smooth, but... what the fuck... it's always smooth for me!

We'll see what happens.

I've been playing as PIG_VIRUS for a couple of weeks now.

See you online.

Happy Fragging!

MySQL ODBC Hell

You can roll your eyes and pass on this one. I don't care.

Like most people, I started using databases after I ran out of rows in Excel. But I passed on Microsft Access. Never cared for it. Can't stand it.

Now, I'm a database abuser, still stuck in that old Excel mindset. What I call a "database" is what most knowledgeable people would call a "Big Fucking Table" (or "BFT"). I cram everything I can into a table and use SQL queries to get what I want out of it. It's a very brute force approach.

The Proxy List is a good example of this. 1.75 million rows of junk gathered over the past year to make a crappy Web page. I do basically the same thing at work and lately I ran into a brick wall with a MySQL BFT that pulls data from multiple MSDE BFTs.

MSDE is the Lite version of Microsoft SQL Server (MSSQL).

The point of this exercise is to mash together a bunch of Little Big Fucking Tables (LBFTs) into one Really Big Fucking Table (RBFT). The actual, real life solution would be to just make one RBFT in the first place, using MSSQL instead of MSDE. The reality is I can't justify the licensing costs for MSSQL so I have to use MySQL instead.

Plus there is a lot of cruft in the LBFTs that I don't need in the RBFT (I don't want a Really Really Big Fucking Table, or RRBFT, because I don't have the disk space). I can ignore all that by writing an SQL query.

So I made a Distributed Transaction Services (DTS) package that used the MySQL ODBC driver to pump the data between the LBFTs and my RBFT.

It worked fine for over a month, or about five and a half million rows. Then it stopped, complaining about running out of memory.

Naturally, I Googled the error.

I found that a lot of people were running up against this problem. I tried every fix posted in every link I could find - usually twiddling settings in the MySQL ODBC driver - but nothing worked.

So in desperation I fired up Wireshark to see what was going on over the network between the three boxes - the MSDE server with the data, the MSSQL server with the DTS package, and the MySQL server with the RBFT - in this process, and I found the answer.

And it was pure stupidity.

In the first step, the ODBC driver pulled the entire RBFT over the network from the MySQL box to the MSSQL server with the DTS package. It would then run out of memory and die.

Well, DUH!

The actual first step of the DTS package is "Get the data from the MSDE server". The second step is "Put the data on the MySQL server". At least, this is how the DTS package is designed.

How it works is something completely different.

With that discovery, I added a temporary, empty BFT to the MySQL database, changed the DTS package to use that instead of the RBFT, and scheduled a cron job to move the temporary BFT to the RBFT. After the move, all the rows in the temp BFT are deleted, cleaning it out for the next run.

No more ODBC "out of memory" issues. For now.

I'm not sure who is to blame here. After all, it is the ODBC driver that ran out of memory. That is a fact. But it is very tempting to point the finger at Microsoft, although they're just the middleman in this transaction. Did the DTS coordinator ask the ODBC driver for the entire table? Or does the ODBC driver always need a full copy of the data it's going to work on, whether you use it in a DTS package or as a data source for an Excel spreadsheet? Is this an issue with ODBC drivers in general?

I don't know. I do know I'm not interested in doing the research to find out the answers to these questions. I just want this shit to work.

When I Googled the problem, very few people were reporting any success with any of the posted solutions.

And nobody ever said "use a temporary table, dumbass".

Bahrain: A Year Later

Last year, I reported on a flood of open proxies in Bahrain.

Starting in April, picking up in May, and bursting at the seems in June and July, I found hundreds of open proxies listed by the various proxy lists I raid hourly (I never actually scan for proxies. I collect everything I can find online - usually through Google - test, and report).

Bahrain was awash with open proxies. Then, in August, it stopped.

In the end I had tested almost 17,000 proxies and found more than 2200 open proxies. On closer inspection, most appeared to be public access points or some type of consumer grade router. Nearly all were owned (notice I didn't say "pwned") by Bahrain Telecomm.

One by one, the open proxies eventually went dark and that was that. There were a couple later in the Fall but nothing like June and July of '08.

Today I ran across this article, which states:

Security company Trend Micro, has recently warned that Internet community in Bahrain is at high threat from the rising cybercrime. The security firm has sterilized over a million infected PCs throughout the Middle East during Q1 2009, out of which, 159,228 were located in Bahrain...

Security researchers believe that high Internet use in Bahrain is the prime reason behind the increase in the nation's cybercrime. As far as the figures by Trend Micro are concerned, Internet usage in the country has increased 525%, which indicates that currently 34.8% Bahrainis are susceptible to cybercrime.

Fancy that.

Laptop PWN3D!

It took three tries and as usual OpenVPN stability was an issue, but I finally stole my own laptop without ever touching it.

It took almost thirteen hours, but that's within the limits of an unattended laptop in a "secure" location. Not everyone takes their laptop home, and if you work in an environment like I do, nobody likes to log off or reboot because it takes at the very least twenty minutes for your system to get back to normal (our specific problem is Outlook - it really has a hard time waking up in the morning).

Granted, the hard disk only had 20G of data on it. A bigger drive would have taken more time, bleeding into working hours and increasing the likelihood of an OpenVPN interruption, but as a Proof of Concept (PoC) the results are valid.

It would have taken four hours had the VMware vConverter taken full advantage of my cable connection. It never went over 585kBps for the duration of the transfer.

The first two attempts never went over 400. On those runs I was using vConverter 3.x. I upgraded to 4.x before the final run. I have a feeling, which I can't prove, that the free versions of the VMware Infrastructure tools might be crippleware. There is no reason for it not to have taken full advantage of my pipe. I have gotten the full bandwidth in other file transfer exercises between home and work and the CPU utilization on the source and destination systems was minimal-to-nothing.

As an added bonus, it turns out that nothing on the network even noticed that gigabytes of data were being sent out to the Internet for three full days! No alarms went off. No red flags went up. It didn't even show up in the reports generated every day by the Microsoft ISA (Internet Security & Acceleration) servers that "control and monitor" access to the Internet.

Unbelievable! Especially considering it was me who set those reports up (and I wasn't even trying to hide anything).

The skeptics (I among them) will say, "Sure, you had admin access to the machine, what is so special about this 'hack'?"

That, my friends, is the whole point of this PoC. The environment I work in has 50+ "DesktopSupport" personnel that have admin access to every PC in our multi-campus WAN. Some of these people are complete, utter bozos who have been known to do idiotic things like Google for "flash upgrade" and then complain because the file they downloaded from a Ukrainian Web site gets pounced on by the anti-virus.

They are not too bright. Maybe that was an upper management decision. I could see the logic in that, but in my opinion stupid people are dangerous.

The problem is the smart ones, and the smart ones who act dumb (the dumb ones who act smart usually blow their own cover anyway).

This group of support personnel should be split up to support the different campuses, but with sick days, vacations, and scheduling conflicts it's just easier to give them access to everything.

Luckily, almost no one trusts them. But there is the "out of sight, out of mind" problem.

That aside, the Really Scary Issue - in my own mind - is my Big Shot Boss, the Chief Security Officer, cannot seem to grasp the power they have. Sure, the guy's at 35,000 feet and everyone looks like ants, but he's been out of the trenches for so long he doesn't realize what people can do with the access they have been handed on a silver platter.

He doesn't know that, by utilizing the tools built-in to Windows, these jokers can slurp up any file on any hard drive on any desktop across the Enterprise, deleting the security logs as they exit. If those logs were turned on, which they're not.

To him, and the rest of his ilk, the security problems we face are all about servers. Nobody cares that the desktop is an accident waiting to happen. When the desktop is pwn3d, the servers, the network, and the data will surely follow.

It's never the other way around.

I'm Stealing A Laptop Today!

Don't get excited. I haven't gone over to the Dark Side.

Yet.

Besides, it's my own laptop. That is, it's the laptop my employer has issued to me. And I'm not taking it home in my lunch box. This time.

I'm stealing it virtually!

You see, nobody steals laptops for the hardware anymore. It's all about the data. With the right access level, laptops, or any computer, can be stolen without ever busting a lock or leaving a fingerprint.

All with free tools easily available over the Internet. I'm not talking about "hacking tools" - you have to be brave to use that crap these days because you never know what might be hiding in them - I'm talking about legitimate software distributed by legitimate companies. In this case, VMware.

VMware distributes a nice little tool called the VMware vCenter Converter which allows you, among other things, to turn a real nuts and bolts box into a virtual machine.

Which is exactly what I'm doing now. As I type this, the bits and bytes of the hard disk in my laptop are flying over the Internet to a VMware server in my family room. When it's all over I will have an exact copy of my laptop, minus the hardware of course.

This is really No Big Deal. Anyone with the right amount of access can do this surreptitiously in your IT environment, cut the image to a USB thumb drive and take it home to hack at their leisure. Or sell to the highest bidder.

The trick is in doing it over the Internet. If I had a 32G USB drive I'd probably do it that way, but I don't. What I do have is a cable modem and three covert channels back to the office.

Plus an aging Linux box that I talked a former Boss into letting me install on the corporate network over eight years ago. If I had my way, Linux would only be allowed under the strictest security policy possible - it's just too damned powerful for mere mortals.

The biggest problem to overcome is establishing a common network share for the corporate and VMware boxes. That is accomplished with OpenVPN, the BEST damned Open Source SSL VPN on the planet.

That is covert channel #1. Channels 2 and 3 are port-forwarding SSH tunnels that connect back to HinkyNet over the corporate proxy. One of the SSH channels is established with a Cygwin service running on my corp workstation. The other is a bash script on a Debian VM that runs on the VMware GSX server on my workstation. All three will reconnect if the workstation is bounced and there is enough redundancy so that if any two of them go down the third can be used to bring the other two back up.

In practice, OpenVPN is the hardest to keep running, but that is due to the security limitations in our environment (many of which are of my own doing).

And because of that issue, I'm on my second attempt at this Proof of Concept exercise. I started yesterday and got 15 gigs downloaded before the OpenVPN connection crashed at 5AM this morning. I can also do this just as easily over SSH tunnels, but that would require using the VMware 2.0 server on my MythTV box, which currently has too much disk space dedicated to unwatched recordings of "Terminator" and "Life on Mars"!

The first time through is always a learning experience.

But the point remains: given enough time and enough access and the right tools, an insider can walk away with your company's entire IT infrastructure. I'm already looking into what can be done about this with the tools our company already has (like everyone else in this economy we're not spending cash we don't have). VMware and virtualization in general is so hot, no one is looking into the security implications these tools bring with them.

Or at least they're not publishing.

StinkFly BANNED!

I got a complaint! Here it is...

If you are the guy cheating on your maps, known as STINKYFLY..then I am talking to a degenerate. If you are not him, then I suggest you please remove him. He is flagrantly winning games by cheating on your maps. Scores like 35 to 5 when nobody else is even close and they are good players. Show your class and integrity and kick his childish, cheating ass off. He spoils the game.

Well, jeepers. I never did like that guy anyway. He's one of "those" players who tends to make everybody leave, all at once. But I never caught him in flagrante delicto, if you know what I mean.

And I try to refrain from banning people who just happen to piss me off (except for Zodiac). I'm not a good player, even though I've been fragging away for the last ten years (my, how time flies) I just never got the hang of it (I still can't strafe for the life of me). I'm so bad sometimes it seems like everybody's cheating.

So I refrain from banning people - as much as possible - until somebody complains.

But I love to do it, because I've put a lot of time and effort into Ban-O-Matic and tying it into the firewall rules. And because it works so well (when it works - there have been no lack of problems with it over the years).

And since I migrated everything to the new server and put all the scripts on a network share it's easier than it's ever been!

Plus I get no small amount of satisfaction of watching the banned players trying to play. They sort of float across the map, if they can move at all, and they bitch and moan and curse their fate. Or they get paranoid and blame it on a Denial of Service attack from another player.

That's just precious.

Anyway, Stinky's gone. I may immortalize him as a bot ("StinkyFly" seems like a nice subtle twist, but "ShitFly would work, too).

But be forewarned, a lot of times they come crying back to Hinky, promising to play nice and never ever cheat again. And being the kind hearted old fool I am, I take pity on their worthless basement boy existence and usually let them back on (except for Zodiac).

So anyway, please feel free to complain! I do listen and I like playing UT with a good crowd.

Eliminate the ninnies and the twits!

www.mrhinkydink.net

I got the news today that Google Page Creator (GPC) is going to die this June. This is a shame because I was quite fond of it and have been using it for the last two years to document a number of (usually silly) things, including documenting the progress of The Proxy List. Sometime in June it's going to be moved over to Google Sites.

Luckily I already squatted on Google Sites about a year ago. At about the same time, I registered mrhinkydink.net (the dot-com version is the site at GoDaddy that holds the utmods and the Proxy List).

So anyway, since mrhinkydink.net wasn't really doing anything, I pointed www.mrhinkydink.net to my little corner of Google Sites.

Nothing much is there right now. To be brutally honest, I don't like Google Sites that much. It seems limited compared to GPC, which was an awful hack (with infuriating quirks) to begin with but I got finally used to it (and GPC is much faster to working on with Google Chrome than either FireFox or Internet Explorer).

Since Google is forcing this move on me I'm starting to put backup copies of my GPC pages on the GoDaddy site, just in case. And I plan on investigating other venues, so the hinkydink.net name may yet be transferred again. Maybe to WordPress.

That plan could change if Google decides to improve Sites, but I'm not counting on that happening.

I Hate Ads

Advertising is private sector propaganda. It's vile. It's evil.

I avoid it whenever possible.

Especially on the Internet. I've been running a Squid proxy at home for over ten years, primarily to use Ad Zap to eliminate the popups, banners, and ShockWave Flash crap all mainstream Web sites would like to shove down our pipes.

For example, here's a screenshot of ComputerWorld on a slow advertising day. Click for a larger view..

Again, this was captured on a weekend. Durig the week, when there is much more traffic from corporate proles such as myself, it is one hundred times worse.

Minimum.

Here is the same exact page when viewed through my Squid prozy with Ad Zap...

Much better.

One thing the screen captures don't show is the fact that all three of the ads are animated. Animation sucks primarily because the bandwidth required chokes a remote desktop connection. I do most of my browsing at work through an encrypted pipe back to my house, which uses the 785kBps (**SLOW**) uplink. Any animation sucks the lifeblood out of that link.

There are several other ways to accomplish the same thing. I had been using FireFox with NoScript exclusively for the past eight months, but I switched to Google Chrome because it is much faster than FireFox.

Blazingly fast. Un-fucking-believably fast.

Blogably fast! TROOF! But maybe some other time. We're talking about ads today.

Since there's no NoScript plug-in for Chrome, I had to go the Ad Zap route. As it's distributed in its stock configuration, it kills maybe 90% of all adds, banners, popups, etc.

But I want no less than 100%!!!!

The big problem I had with Ad Zap was the small town TV and radio news sites across the US. They tend to use more obscure Web advertising companies, or they use their own home-brew methods. These aren't included in Ad Zap out of the box, so you have to roll your own rules. This is simply a matter of setting a few variables and editing a text file or two.

The result is very clean and very safe Web surfing. Once you start blocking ads, you will never want to go back.

Never.

I have mentioned before I'm the Network Nazi at work. I run a commercial content-blocking software package - that will remain nameless - and I also block ads on my own corporate account. I've been doing so for a very long time with this package. I am so used to not seeing ads that I'm totally appalled whenever I have to use someone else's computer.

Why do people put up with that crap?

The Year Of Proxies

That's right, the List is now a year old, going on 13 months.

March 15 (the Dinkster's birthday) was the actual one year mark, but I was busy that day, meeting the Fockers (the parents of Rinky Dink's True Love, Twinky Dink). It was a busy day and I never got around to it. That and the other blogs, the new UT server, and a half dozen other projects kept me from marking the date.

I only mention it now because I've been up since about 3AM moving the server that runs the publication of the List and all its varied and sundry subfunctions.

That was fun. Right now, the new server (a VM - virtual machine) is re-doing the 4AM run and it's going quite nicely. But it was a rough road getting there, but it had to happen because I hit a solid brick wall with the old VM. Ubuntu, in their infinite wisdom, stopped supporting "Feisty Fawn" (a.k.a. Ubuntu 7.04).

That in itself was a major pain in the ass. No more security updates. No more new software packages. And no more OLD software packages. After I moved the UT servers to a common code base by putting everything on an NFS (Network File System) share, I wanted to leverage the information stored there for the proxy site VM. But I couldn't install the software! ARRRGH!

So right now, this is the THIRD incarnation of the new proxy site server, an Xubuntu 8.04 platform. It is the THIRD because I neglected the First Rule of Virtual Machines: snapshot! snapshot! snapshot!

For the uninitiated, snapshotting a VM freezes the configuration so you can roll back to a known good state. But... I was in a hurry. In order to make the new box 100% like the old box, I made a list of all the Debian software packages that weren't in the new and did a shotgun upgrade to synchronize them.

Consequently, I ended up screwing the hard disk configuration by getting packages that required LVM (Logical Volume Manager).

I hate LVM. I has been a major pain in my ass ever since I set the old VM up. It's a very powerful, complex package that no one needs. Sure, you can extend volumes when space gets tight, but it's easier and faster to use gparted (Gnome Partition Editor). LVM was on the top of my Shit List and it had to go.

But it kept coming back. On the third incarnation I had the presence of mind to take the snapshot and, of course, I never needed it. I finally had the list pared down to packages that I needed and didn't rely on LVM (stuff like Rhino and gocr and a lot of other little packages my scripts rely on). I restored the database (yes, I do make backups these days), twiddle MySQL's run-time parameters, and did a few test runs. These died because they relied I utilities I wrote and forgot about. I moved those packages over and it ran like a top.

And it's running like a top as we speak. The 4AM run is almost done. When it finishes I'll shut the VM down, patch the Windows XP box it's running on, and reboot.

There should be a new list by noon.

NIC Death/Resurrection

Today I moved EXP IV to its Final Resting Place (FRP), hooked it up to its woefully underpowered UPS, re-installed NUT (Network UPS Tools), played a quick game, and forgot about it.

About four hours later I noticed the sucker dropped off the network.

I guess it took that "Final Resting Place" concept too literally.

The FRP, where EXP III used to live, is at the end of the longest wire on the network. It leaves the switch, travels across the fireplace mantel, and drops down to the other side of the fambly room. Maybe twenty five or thirty feet max. Apparently, the built-in NVidia NIC was unhappy with that situation, because I plugged my laptop into that wire and it was happy.

So I added a RealTek NIC (RTL 8129C) and edited the udev settings to tell the OS eth0 was now eth1 and eth1 was now eth0 and rebooted.

Maybe this had something to do with the reported lag. Maybe it will be better. Maybe it will die again.

I don't know. If you see any improvement drop me a note.

Screwing The Pooch, Linux-Style

My dear wife, Pinky Dink, has been running Ubuntu 6.04 LTS (LTS="Long Term Support") for close to three years now. She has been quite pleased with it but I've been bugging her about upgrading to 8.04 LTS for months.

But she's seen how my "upgrades" go (the long saga of EXP IV being the most recent) and wanted nothing to do with it. So we compromised for a while. 6.04 came stock with FireFox 1.x, which is dreadfully out of date. I found a hack on the Web and upgraded here to 2.x, which is only woefully out of date. The problem there is the upgrade was out-of-band, so she stopped getting updates to FireFox or important plug-ins like Flash (which is full of vulnerabilities and has been hacked to death lately).

Not a good situation.

So I finally talked her into it and she let me do it.

I anticipated the SCSI issue. Maybe you haven't noticed, but IDE hard drives are obsolete. SATA is now king and to simplify things, Linux now treats all drives as SCSI drives. If you upgrade to a new kernel and your root drive is on /dev/hda1 (the old IDE drive designation), the kernel will choke on boot because it thinks it should be /dev/sda1 (SCSI). The way around that is to either use LVM (Logical Volume Manager - which I despise) or to label your drives before you upgrade, which is exactly what I did, having learned that the hard way over several VM Linux upgrades and the EXP III/IV debacle. Use e2label for ext2/3/4 drives and mkswap /dev/whatever -L yourlabel for the swap partition. Then edit /etc/fstab with LABEL=yourlabel where /dev/hdax used to be. Reboot to make sure it works. If it doesn't... call me. Leave a message.

I knew there were going to be other issues. VNC4 is a wash, as I mentioned before. I was prepared for that. The alternative has always been Cygwin/X, which is slower and suffers from a lack of a dependable clipboard, among other things.

What I didn't expect was problems with her wireless drivers. When I first set that box up back in '06, the RaLink drivers had to be built from alpha or beta source and had to be rebuilt every time there was a kernel upgrade.

But over the years, the RaLink drivers were mainlined into the Linux kernel and that problem disappeared.

Turns out, it's back. With a vengeance. And it's apparently due to some "philosophical" differences the developers at Ubuntu have with the driver. For details, see this exasperated posting from some poor schmuck who can't get Ubuntu to admit they have a problem. I followed his lead, downloaded the source code, and blacklisted the stock Ubuntu drivers. Wireless is fine again, but I will be fucked on the next kernel upgrade.

One problem down.

The next was the remoting issue. Pinky's box is in the kitchen, but I always used VNC4 to hook up to it and do patching and housekeeping (which is a really bad idea - if your session gets trashed during an upgrade you can leave the box in an "undefined" state - but I'm prepared to take that risk because I'm too lazy to get up out of my chair).

As I said, I expected to fall back on Cygwin/X. Silly me.

Cygwin, for those who may not know, is a set of Linux utilities that run on top of Windows. Cygwin/X is an X server that allows you to run graphical programs or full graphical desktop sessions on another Unix box across the network while you're logged in to Windows.

This is very nice, considering it's free when most third party X servers (such as Hummingbird Exceed) cost hundreds of dollars.

The problem is, with all due respect to the Cygwin developers, they keep fucking the god damned thing up.

Upgrading Cygwin is a crap shoot. Something is always guaranteed to be broken.

So when I tried to connect to Pinkie's box and the X screen kept bouncing, blinking on and off, I knew I was in for more fun. I crossed my fingers, girded my loins, and upgraded Cygwin.

It worked. Or perhaps I should say I simply haven't found what's broken in it yet.

Now I have the box where I want it to be (almost). In fact I'm posting from it now, wirelessly and over a network Cygwin/X session! I'm 90% pleased with the way things turned out and this is still the best Linux box I ever built.

Ban-O-Matic is BACK

And it's up to revision 3 at this point. I made several seriously schweet improvements that should be ported over to the other scripts.

Sorry that took so damned long.

An unintended casualty came in the form of _gabriel_chile_. He was an innocent bystander. He simply chose the wrong name. I'll have to straighten that out.

Sorry to make you put up with all those idiots for so long.

And speaking of idiots, I noticed Ranger was back. He's not in the KillFile but he used to be. The sucker is just too damned good.

I've had a few complaints about the new server concerning lag. It's hard to tell from here, but it appears to be OK. There may have been some other issues. Time Warner Telecom (TWT) got hit with a DNS attack last week.

How would TWT's problems affect us?

Well, it goes like this: Some poor schmuck on TWT logs in to the server while TWT's DNS servers are getting bombed. UT isn't affected by the DNS attack because there is no name resolution involved. No problem there, but as soon as this schmuck needs to download one of our UT mods, if he can't resolve www.mrhinkydink.com (the high speed server with all the mods), he's going to pull that mod off the local server (here in my family room) and subsequently consume most of the upstream bandwidth, which would seriously suck for anyone else who happens to be playing.

I'm sure that's clear as mud, but it is a possibility. We get TWT players here all the time and the local mod downloads have been a problem forever.

I did some other things to make the server more responsive. All the game binaries have had their CPU priorities upped.

Anyway, that's that for now. I hope everything's working out for you.

I Hate February

And I'm glad as Hell it's almost over.

I believe I have Seasonal Affective Disorder, a.k.a. "S.A.D." In fact, I'm pretty damned sure I've got it, although I've never been formally diagnosed (I stay away from the medical profession in general and avoid shrinks like the plague).

It takes a while to "get it". That is, to understand that you have it. You reach a certain age, let's say ten years out from High School, and you look back and you discover all your worst "life decisions" were made in February or January. Wrecked cars, lost jobs, broken relationships, chronic health issues are all clustered around those damned Winter months. Once you realize the risks, you're prepared, even if there's not a Hell of a lot you can do about it.

I've been taking St. John's Wort for the last couple of years. It seems to help. I used to take it starting in October - it takes a month to go systemic ("kick in") - and ending in May, but the zombie-like withdrawal would generally last through June, making June every bit as painful as February.

So I take it year 'round now.

And, I drag a full spectrum light to work. I shine it on my mouse hand for a couple of hours a day in the morning (twenty minutes a day on the back of the knees - I kid you not - has shown to be effective). I also try to get as much natural sunlight as possible, even though sunny days are relatively rare in February.

One of the things that helps a lot is realizing life can be a lot worse than just being down in the dumps and irritable. Compiling the Murder/Suicide Blog for the last three months has had a sobering effect. You see these news stories, with photographs of yellow police tape around snow-covered houses, and you can't help but wonder if S.A.D. didn't have an effect on those people (the economy notwithstanding).

Especially considering February has been a killer month for murder-suicides (there were three double-murder suicides in Ohio in the span of a little more than a week this month), although it has been absolutely in line with the statistics (you can expect at least two a day regardless of how happy or sad people are - it simply happens that often).

Anyway, March 1st is less than fifteen minutes away. Pinky Dink's garden is already starting to sprout, the days are getting longer, and this crappy month is over.

It gives me a warm fuzzy just thinking about it.

DAMN NICE BOX!

Although not exactly bug-free, EXP4 went online yesterday as (sorta) planned.

It plays exceptionally well. The only problems I've noticed are at the beginning when you first log on. Play is a little choppy, but it smooths out fast and remains smooth game after game.

That was the best $139.95 (+S&H) I've spent in a long time. I"m slightly pissed Debian 5.0 ("Lenny") sucked as badly as it did, but I have a Deb 5 VM and I'm going to keep it updated to see if they ever get around to fixing that two year old Gnome vs. VNC4 problem. Since that is a fight between two third parties, they (Debian? VNC? Gnome?) may never get around to fixing it. That was a major disappointment.

As (almost) expected, kernel.org put out Yet Another Revision (2.6.28.7) on Friday. I would have let it slide but there were enough major issues to cause concern. I got that all compiled and ready to run just before putting EXP4 online. But I didn't have the time to upgrade the old server BOT House lives on. That will likely happen before next weekend.

Ban-O-Matic is broken at the moment so I've been banning our buddy MISERABLE_S.O.B. by hand whenever he changes his IP. I'd ban his entire ISP but they have a pretty large CIDR block (I'd call it a Class B+). I'm confident B-O-M's problems are trivial so I'll be taking a look at it while I'm at work tomorrow (I do some of my best work at work - don't tell my boss).

Today I've been hacking around with the old EXP III box. I upgraded it to 2.6.28.7 and I'm trying to get those awful Marvell wireless cards to run on it (no luck, although there is now a native loadable module for some Marvell NICs - ones I don't have, apparently).

I've also been hacking around with TrueCrypt, which is an awesome file & disk encryption package that was undoubtedly written by a team of paranoid schizophrenics. They have all the angles on encryption hammered down, with an emphasis on plausible deniability.

This is a program designed from the ground up for people who need to hide what they're doing.

Of course, I have a professional interest (that's my story and I'm sticking to it).

I've been spending long hours digging through my old hard drives, wiping then and encrypting them so I can throw them away without worrying about dumpster-diving tweakers stealing my personal information. Although, to be fair to the tweakers, the credit card processing industry itself is doing a great job losin my personal info in massive security breaches as it is.

Truecrypt takes about two hours to prepare a lousy 40G hard drive (at least it takes that long on he old EXP III box), so it's been quite a hassle. And I still have drives that spin but won't format and are therefore technically susceptible to advanced forensic techniques. Plus I seem to have more SCSI drives than I remember ever buying (vintage mid-90s drives in onsie, twosie, and foursie gigabyte capacities).

And - silly me - I built the EXP III box without SCSI support. Considering it takes three hours to build a kernel from scratch (a little less when you've pre-built it), that probably won't happen soon.

In the end I may just end up taking a hammer to them all, but for the ones that work this is my last chance to see if there's any long lost, forgotten data that I may still need on them.

TrueCrypt was made for Windows. It's different on Linux and there are a few ducks to put in a row before you start using it. The information is out there on the Web but I haven't seen a single respectable "HOWTO" on the subject yet. I will probably end up writing one for my own benefit since that's the only way I can remember all this crap.

If I do you'll find it here, so stay tuned.

But no promises!

Final Touches On EXP4

Damn I'm good.

My problems with the Map code were all of my own making. I had originally used the sample API code to pull the latitude and longitude out of the GeoIP database. That turned out to be a really bad thing to do.

I used that code because you could give it a text file of IP addresses and it would spit out all the data. The tool they give you, geoiplookup, only does a single IP at a time. But I took a look at the geoiplookup code and realized I could hack it to do the same thing as the sample code.

It was slick and quick.

I hacked it to check to see if the "IP address" on the command line was actually a file. No file, assume it's an address. If it is a file, loop through the lines inside it. Maybe ten lines of code max.

Since I started doing this Chat-O-Matic crud I've always had a heck of a time keeping the code consistent between multiple machines. I either had to copy everything over or edit everything twice. I usually ended up editing everything twice because the original EXP #1 was on Slackware, which has the irritating habit of putting everything that should be in /[s]bin into /usr/local/[s]bin.

Or maybe I did that when I upgraded bash. Could be. It was two years ago. I don't really remember.

All that is fixed and the code now lives on a NFS share on the BOT House server, so all machines run the same exact code, which has been cleaned up considerably.

The ipset code is all finalized as well. I'm still using my hacked version of Debian's ancient (kernel 2.4.x "sarge") iptables init.d script. Why they dropped that script I'll never know. They give you nothing in 2.6.x kernels. I imagine they expect you to use one of firewall packages they ship with the distro (I never liked any of them).

The ipset code is running on BOT House as we speak. That's 132 lines of firewall crud (banned IP addresses) wrapped into one line. There may or may not be a noitceable performance increase, but it feels good just to get that mess cleaned up.

The last piece was the UPS (Uniterruptible Power Supply) configuration, which I simply copied from the EXP/// NetVista box and edited.

This turns out to be the major flaw in my plan. The NetVista's UPS is way underpowered for the new system. It will survive brownouts (the biggest problem around here), but I doubt if there is ten minutes of standby time.

Anyway, it's all there. I will need to do a few edits on the UT game files but that's all minor stuff. Now it's a matter of putting the box in its final resting place.

That will happen Saturday.

Be there.