Power Outage 6/28/2008

It rained yesterday. Thunder, lightning, then gentle summer showers on and off until the sun came out.

And then some nutjob took out a telephone pole (what an archaic sounding phrase - I suppose they're "power poles" these days). It was brutal. The power bounced hard about four times and then everything that wasn't on a UPS went dark.

The juice was out for two hours. It drained all three UPS's. When it started flowing again my DHCP lease had expired and BOT House & company got a new IP address:

75.179.182.137

Write that down (and remember, port 32777 for BH and port 38777 for EX///).

An IP change is ALWAYS a major pain in the ass. I don't use Dynamic DNS so I have to log on to GoDaddy's DNS servers and change... everything. And for as much as I've been ragging on them for the past few weeks, I have to say their DNS runs better than it ever used to be. The changes were damn near instantaneous and all my... ahem... covert connections at work found their way back home within a few minutes.

Stunning work, GD.

Back to Business

I have decided to stop boring you all with proxy business so I have split off all the project notes here. There are links to the pertinent blog postings on the project so that if you're really interested you can go back and review.

It's time to get back to other stuff. First off, check out this shit (click the pic for a larger view):

The joint has been jumpin' lately. I have never seen so many folks banging away at the UT servers. There are probably two reasons for this:

• The RoadRunner Turbo upgrade in March
• The Map has never worked so well before

GoDaddy finally fixed the ftp issue (for now at least), but they gave no explanation. I would have bothered them for one but I was getting very tired of dealing with them. The best tech support, in my opinion, is no tech support at all. Just make it run right and leave me alone, fellas.

I leave the Map on the screen when I kick back in the evening and watch TV or read a book (I'm currently reading a biography of W.C. Fields if you give a shit). Every now and then I look up and it seems like the entire world is killing each other in my fambly room. It's quite extraordinary.

I'd like to do a time lapse video of the Map but I haven't figured out how to do that yet, short of a real time capture and iteratively doubling the speed in Windows MovieMaker. I think it would look pretty cool.

Be that as it may, I have a lot of bloggable things going on. Our security group recently got re-orged at work and that has been loads of fun. Our new Chief Security Officer is very Web 2.0 and he commands us to blog on our intranet SharePoint server (big woof) at work. I'm already slacking off on that piece. My "editorial style" is definitely NSFW ("not safe for work") so I think I'll do it my way here and then tone it down for the retards at work.

Like anybody actually reads that shit.

Or this shit, for that matter.

WTF is going on in Bahrain?

View Larger Map

If you've been following the Proxy List since it went online (and I know you haven't since all the hits I've been getting are from Google - but what the heck, I'm usually just talking to myself here anyway) you may have noticed that Bahraini proxies take up about 30% of the list.

All of the lists I poach show the same thing. Somebody is scanning the fuck out of Bahrain.

I've done a small random sampling/reality check and found that the proxies are, indeed, answering. None of them are ping-able, but that's not surprising since a lot of Web sites started following Microsoft's lead when they shut off ICMP to their servers in the late 90s (can you say "Ping of Death", boys and girls?).

All the addresses I've checked belong to Bahrain Telecomm. None of them have DNS names.

And the few I've scanned have only port 80 open. Because there is no server banner, my initial, expert evaluation is:

• Bahrain Telecomm is new at the ISP business
• They have no clue what they're doing

The alternate view, which I consider less likely, is that their customers have been hacked. No, these are probably access points or cable modems or, simply, some sort of distributed cache setup for their client base.

In the process of reality-checking my reality check, I have started to get some "403 Access Denied" responses from addresses that were working yesterday. The Bahraini proxy surge may be over soon.

[ OK, quick Smurf joke:
Q. What do you call a Smurf from Manama?
A. Bahraini Smurf ]

I have seen these proxy surges dozens of times. I used to use Proxy4Free back in the heyday of proxy lists (roughly 2001-2005, since then it's been relatively useless). They would have pages and pages and pages of Chinese and/or Brazilian proxies (port 6588 was big in Brazil for some reason) that, by the time I got to them, were all offline.

Those ISPs obviously discovered the error of their ways and fixed everything. You can't blame that kind of massive idiocy on the end user (well, you can and they probably did).

You may have also noted a slew of Japanese "proxies" at the end of the list. These have been reality-checked as well, and they're all junk. They will disappear sometime on June 30th, when the Master Reality Check process kicks off next (it runs on Monday, Wednesday, and Friday). After a typical Master Reality Check the list will go from ~450-500 proxies down to ~350.

For some unknown reason those Japanese sites are proxy judge pages (here is a random sample - it won't bite), so they look like proxies to my algorithm. I have found a way to distinguish them from the genuine article and will be implementing that this weekend.

Putting a Fork in the Proxy List

Boys and girls, it's finished. The Proxy List is now in maintenance mode. I may add a few extra Hinky Links to polish it off but for now it's running like a well-oiled machine.

What started out as 1010 proxies dwindled down to 389 after I discovered I had a serious duplicate problem. I meditated on that issue Sunday, and wrapped it up on Monday. Since then the number has crawled up to 420 proxies as the harvester bash droids continue to mindlessly skim the public proxy lists.

And... I must admit to doing something a little evil:

Yes, boys and girls, it's the dreaded extreme-dm.com Web bug. I warned you about this back in March.

mea culpa, mea culpa, mea maxima culpa

I did it for one main reason: I'm one cheap son of a bitch. I want to see how much traffic the list gets but I don't want to pay for the extra service on GoDaddy. $3.95 per month is killing me as it is. That's almost a gallon of gas!

I could have hacked out a HinkyBug on my own (and frankly I'm still thinking about that) but the only Web server I have on DinkNet is an extremely slow piece of crap and the uplink bandwidth is required for other things, primarily the UT99 servers.

With the extreme-dm.com bug, all it costs me is a few extra SPAM emails per day (grumble grumble).

And it's already showing some interesting results. I've only got a handfull of hits, but Google is treating me well, especially overseas. I'm not sure why that is, but most of the Google referrer hits are not domestic. In fact, it intrigues me and it's keeping me awake at night.

If you're interested, go to the page and click on the bug at the bottom. It will show you everything.

In order to try to "pump the page" a joined a few Yahoo Groups that specialize in proxies. God, what a vast wasteland that is. The first group I joined was run by some guy calling himself "Baron von Assmunch" that just uses it to pump his Web site. The next one was years and years worth of newsgroup SPAM. So was the next. And the next.

What a joke.

I did find one or two, dropped my link, and lost all interest whatsoever in Yahoo Groups.

I'm going to see what kind of exposure I can get elsewhere.

So anyway, it's there, it's stable, and in the spirit of "eating my own dog food" I am using one of the listed proxies myself (speedy little sucker running SQUID in Belgium) for a few "special" sites.

No, it's not what you're thinking.

You filthy-minded bastard.

SHRINKY DINKS!!!

I don't often do advertising. In fact I despise advertising, having worked and slaved in the advertising industry for more than a decade in a previous life.

But if you have kids or just like to have fun doing creative stuff you should check out my cousin Shrinky's Web site. He has a seriously cool product. While you're there don't forget to check out the video.

I'm not much into merchandising either, but Shrinky's been trying to talk me into offering HinkyDink Shrinky Dinks. Branded keychains, luggage tags, and doo-dads mostly. That might just be a winner, but there could be copyright issues since I've been using Great Grandpa Winky's logo as my own for some time now (I even used his theme song in the infamous Websense Policy Bypass video, as you may recall).

My legal problems aside, Shrinky's got a cool product. And what the heck, he's family.

Check it out!

Busman's Holiday

Ahhhh... vacation time!

I have the next five days off to do whatever I damn well please. I started off this morning at 4:30AM by hacking around with the Proxy Page.

We are almost there, boys and girls!

I have hacked it around to make it a multi-page list. Looks sweet. Right now I'm working on the master update loop, which checks all the known good proxies to make sure they're still online. Since they can disappear pretty quickly it's imperative to make sure the list is up to date.

I have also finagled a speed rating this time around, so look forward to that. Anything with a 30 second or greater response time is going to be thrown into the bit bucket. I'm considering listing them from fastest to slowest to make sure you don't have to look too hard to find the proxy that's right for you.

I have also included another proxy type, "Undefined".

A lot of these placed I steal proxies from don't bother to do a thorough check. Not every proxy in a list is necessarily a proxy server. Sometimes it's just a Web server. Sometimes it used to be a proxy, but today it's not. Any proxy that responds but can't be identified as CoDeeN, Transparent, Anonymous, or High Anonymous will be classified "Undefined".

We'll see how that goes. I'll be re-re-checking those manually and if a significant number are junk, they'll just be excluded from the list.

So far, from what I've seen (and I'm getting fewer than I expected) it looks like the "Undefined" sites, for the most part, redirect you to a cheesy search page for a click-thru or two. Dirty rat bastards!

For now I'm going to freeze the page and let the updates catch up. Plus I'm going to suspend "harvesting" activities until that happens, since it really whacks the VM this stuff is running on. The next time the page is updated you should get all the new goodies.

Hacking all this crap together with bash scripts and ftp uploads is definitely an exercise in Bad Web Development, but I go with what I know. I suppose I could learn ASP or LAMP or .Net development, but I'm old, I'm stuck in may ways, and it works. After a fashion.

In other news, I never heard back from those dilrods at GoDaddy. I guess they're still scratching their tiny little pinheads on their server problem. Perhaps their plan is "Ignore it and it'll go away".

If so, they don't know me very well.

Incident ID: 4111284 Reloaded

The other evening I was admiring my World Domination Map and I was just about ready to stretch out on the couch for a little Hinky snooze when BAM! The map went blank.

I pulled up an ssh session to BOT House and did a quick tcpdump to see what was going on, and sure enough that worthless, fucking "426 Connection closed; transfer aborted." error had elevated its grotesque pate.

I took a quick peek at the Proxy List page (which is also uploaded via ftp) and it was chopped in half.

Since the tech heads at GoDaddy had brushed me off with a "Call us if it happens again" the last time, I picked up the phone and punched their number up.

It took me a little longer than I would've liked to prove to the frontline support person that I wasn't nuts, but he put me on hold while he escalated the issue to the neckbeards in the backroom.

I must say I do like GoDaddy's "on hold" background music. It was Squirrel Nut Zippers' Put a Lid on It and a few other cuts from the "Hot" CD. Great stuff. Love that shit.

How did they know?

Anyway the guy finally came back and said the neckbeards observed the problem first hand and that they'd get back to me in "24 to 48 hours".

That was about 47 hours ago.

This whole incident has opened my eyes. Very occasionally there is some serious... dare I say it... lag on the UT servers. I've put a lot of time and effort into taming it. The biggest problem in the past was with users who play the old demo version of UT99. The number of maps they have is limited and when a map that everyone should have already had came up, that demo version would request the map from BOT House, not from the GoDaddy server (the same box that is chewing up my files).

In other words, Fat File + Skinny Pipe = LAG.

I solved that problem by putting all the standard maps (and a lot of other things) on the GoDaddy box about a year ago so that no one would ever need to pull files directly from BOT House. But now the lag comes back occasionally and we have this intermittent file-chewing problem. Unrelated? I'm starting to think not. I think perhaps the GoDaddy server chokes every now and then and the clients are falling back to BOT House for mods, maps, etc. thus clogging the skinny pipe we all play over.

Be that as it may, I am anxiously awaiting their response, which I still believe will be "Microsoft says it sucks to be you".

In other news, I've done a few minor upgrades to the Proxy Page. Found some really nice flag icons, added a "Date Found" column, and learned a thing or two more than I wanted to know about "non-breaking hyphens".

I have also added my Standard Disclaimer to the Hinky Links section on the right. It's hosted on GooglePages, which I have recently taken a liking to. If you've never used it, check it out. It's a great way for the HTML Impaired (such as myself) to hack together a decent looking Web page with very little fuss. Highly recommended.

Hinky's CoDeeN-Free Proxy List

Classifying the proxies went much faster than I had anticipated, although there are a few details and maybe one bug left to hammer out.

The result, after 90 days of hacking on this thing, was a total of 1010 usable proxies out of a total of 215,000 stolen from various lists on the Web.

I decided not to list the CoDeeN/PlanetLab proxies, although there were almost 2500 of those in good working order. First, they don't like outsiders. Second, they're more useless than transparent proxies if anonymity if your goal. Third, there's that whole "we cooperate with Law Enforcement Agencies" thing. Ew.

And fourth, they're only used to pad proxy lists anyway.

I upped the output to the 200 most recent (non-CoDeeN) proxies and changed the refresh to once every two hours. You will find a new page after every even hour (except 4AM) EDT. On the odd hours it scans the Web for more. At 4AM it does the "Big Run", which leeches off all the most active proxy lists. As such, the 6AM posting might not make it until 6:15-6:30, since the page-making process synchronizes the "gold" list with the "raw" list every time it runs.

I have enough data to make this a ten page list, with 100 proxies to a page and I think that will be the next evolution.

Stay tuned.

Tiny Little Flags

Today I added Tiny Little Flags to the Proxy Page.

I painstakingly researched them all and created them one by one in Windows Paint.

Right.

And if you believe that I have some floodplain land in Iowa you may be interested in.

The process of classifying all the "known good" proxies trudges on. Many have died since they were first added to the database so it is taking its sweet time because the dead ones have to timeout after 30 seconds. The project started on the Ides of March and at the moment we're up to April 21st. It's going to take some time to catch up. After that I need hack out a method to synchronize the tables (that is actually halfway finished; I only need to automate it). Once that job is finished I can put the new page up.

And speaking of Iowa, back in the 80s I seriously considered a job in Cedar Rapids. After agonizing over it for a few days, I turned it down. Good thing I did. The industry became obsolete about ten years ago. Had I taken that job, today I'd be unemployed and under water.

Funny how life works out.

Proxy List/GoDaddy Update 06/14/2008

Very soon after I put up the Proxy List I became disappointed with it. Although it's running fine (with caveats), I regretted not classifying the proxies I had collected into Transparent, Anonymous, High Anonymoity, etc.

I took a random sampling and re-tested. The results were dismal. A few were already offline, and although that happens a lot, they were just barely hours old.

As if that wasn't bad enough it turned out there was an extraordinary number of CoDeeN proxies popping up.

As I mentioned before, I never had much luck with CoDeeN servers. Whenever I used one it was down. That state of affairs has changed. All over the globe, they're taking requests and working just fine, thank you. Some are a little persnickety and will give you this kind of response:

You are trying to use a node of the CoDeeN CDN Network. Your IP address is not recognized as a valid PlanetLab address, so your request rate is being limited.

Very nice of them to note that. Other nodes don't seem to care, but they greet you with a warning that all your activities are being logged and that the system does not work with Anonymizer and that they're happy to cooperate with LEAs (Law Enforcement Agencies), so don't fuck around, buddy!

After ten seconds they forward your request.

Obviously if you have a privacy agenda of some sort (whatever your motivations may be), you'd probablys want to stay away from CoDeeN proxy servers.

There are some interesting side effects to connecting to a CoDeeN server. They note your IP address and browser USER-AGENT string and if they see you again, you don't get the warning. As a result, if you're using a proxy judge to test a proxy server, it goes right on through and for all intents and purposes the CoDeeN server looks like a "High Anonymity" server (no HTTP-VIA or X-FORWARDED-FOR headers).

That could be a Very Bad Thing, depending on how you use their network.

This was all very new to me, probably because for years (until I found my own) I used ProxyDetect.com as a proxy judge. Can you say "PWN3D!" boys and girls?

It's a good thing I never had an Evil Agenda. I do this for educational purposes, dontchaknow.

It was kind of a trick to convince CoDeeN they never saw you before, but I found a workaround that ferrets them out without a question (the easy thing to do is look for the string "planet" in the DNS hostname, but that's only 95-ish% reliable).

Now, I'm trudging through all the proxies I've harvested since March and re-testing every one of them. The Proxy List should be re-done in a week or so. Until then, use with caution.

As for GoDaddy, they are still chewing up ftp transfers and they are still clueless as to why it's happening. It started happening with the Proxy List yesterday morning. The page was cut in half, the file truncated on the upload. I pinged them almost immediately and sent them a screen capture of the transfer and "426" response received from their server (see Incident ID: 4042264).

Their response? You'll love this one...

Unfortunately we are unable to duplicate any FTP problems as you suggested as we have successfully connected to your account and also successfully performed a file upload test. If you are still experiencing problems, please respond with a screen capture of any errors you may be receiving, or the behavior of the application that is causing difficulty. Also, please provide a detailed description of the issue at hand, and steps that we can take to reproduce this issue.

Duh. I sent them a screen capture. I gave them a detailed description. I told them it was intermittent.

This has gone out of the realm of Tech Support and into Customer Satisfaction territory.

MrHinkyDink's Proxy List Finally Online

At long last the Proxy List is LIVE!

I started this in mid-March. Now, according to the database, I have almost 10,000 "good" proxies to share with the world (out of over 200,000 total IP addresses).

It took so long because I don't know Jack Shit about HTML. In the end, I stole somebody else's page and hacked it for my own use.

In typical Dink-O-Matic style, the daily update is made by a bash script that queries the database, mashes the page together, and spits it out to the site at GoDaddy. It is set to update at 8AM and 4PM (EST), 24x7, with the 150 most recently stolen proxies.

Whether they're "Elite" or anonymous or transparent is left as an exercise for the user. I'll get around to that sooner or later, but the smart money's on "later".

And remember, Kids, don't use a proxy unless you know what you're doing. Know the laws of your country. It may be illegal. An evil country probably has evil proxies, so choose wisely.

Incident ID: 4042264

"Due to its complex nature, your issue has been relayed to our Advanced Technical Support Team. Our most skilled technicians will be working to resolve your issue quickly and completely. You will be notified promptly upon resolution."

Great. Just... great.

Like I mentioned last time, the map likes to make a liar out of me whenever I say it's working great. And it is working great. When it works. And it works all the time. But now it seems I have run into Someone Else's Problem.

It turns out that "Someone Else" is probably the Usual Suspect.

Randomly last Sunday, I would look at the map and it would be blank. Not whitespace blank, not 404 blank, but markerless. That should never happen. Even when no one is on the server, the last uploaded data file should still be there and at least one marker should be displayed until another player shows up.

Here's what is supposed to happen: once every 30 seconds or so a bash script wakes up, checks who's playing, their score, ping and which server they're playing on through the UT Webadmin interface. Then it looks up the latitude and longitude of the player's IP address, and sends all this information in a tiny, tidy XML file, via FTP, to the Web server at http://www.mrhinkydink.com/ where it is displayed through the map using the Google Maps API.

It turns out the weakest link in that strategy is the FTP protocol itself. Or rather, the FTP server at the other end.

As I dug deeper into this issue I found that the XML file was getting stomped on by the server. It was a zero byte sized file every time the map was markerless. So I ran tcpdump to watch the file transfer. It was working, the file was being uploaded and it wasn't a zero byte file (in fact there is logic in the script that won't even send a zero byte file).

The server was responding like this:

426 Connection closed; Transfer aborted.

In non-technical terms this means "FUCK YOU I DON'T WANT YOUR GODDAMNED FILE".

That was last Sunday. I hacked things around on my end and nothing helped. I monitored this all week and discovered it seems to happen at about 20 minutes after the hour and persists for five to thirty minutes. Then it goes back to normal.

Last night I opened Incident ID: 4042264 with GoDaddy. But I think I've already determined the root cause, and it will come as no surprise to anyone.

Microsoft.

Fuck. We're doomed.


UPDATE

Here is GoDaddy's somewhat worthless response:

Dear Sir/Madam,

Thank you for contacting Hosting Support. We have resolved the issue with the FTP on your hosting account.

We were able to connect both internally and externally from our network to the hosting account. You should now be able to connect to your account. We apologize for any confusion regarding this issue.

Please contact us if you have any further issues.

Regards,
Ben A.
Hosting SupportHosting Operations

This is fairly typical tech support horseshit. "We can do it so it's fixed!" They completely misunderstood the issue, probably on purpose. Ignore it and it will go away.

I wonder who they think is confused?