mythtvbackend
In January of this year I decided to blow my Christmas Bonus on a new PC for my Winter Project, MythTV. If I had it to do all over again, I probably wouldn't have bought the hardware I did, namely the Asus Pundit P-1 AH2 shown below.
Actually, in general, it's not such a bad box. But for MythTV, specifically Mythbuntu (8.04), it sucks. The video is fine, the performance is fine, but the built-in audio is just not sympatico.I "fixed" that by buying a run-of-the mill SoundBlaster card for thirty-ish bucks. That doesn't break the bank, but together with the PC-HDTV card I bought it does max out the number of PCI slots this little guy comes with. I don't like that. There should always be a spare slot.
Fast forward nine months later and we're still in "Winter Project" mode. I couldn't bring this thing into production until it was wife-proof. First, "mythtv-backend", the service that runs everything, simply refused to start up automatically most of the time. I just couldn't see myself on the phone with my wife...
"OK, honey, just open a shell and type 'sudo /etc/init.d/mythtvbackend start', plug in the root password and hit Enter."
That's just not acceptable. Pinky understands On/Off buttons. She thinks a "shell" is some kind of macaroni.
As it turned out, that issue was due to DHCP. Once the IP address was hard-coded, the backend process started every time. That was just one problem.
/tmp
Not long after that issue went away, the box started taking forever to boot up. The pretty splash screen would disappear and the screen would stay at...
Activating swap drive swap
... forever. At first it just took a long time to boot and then it took longer and longer and longer. Hitting Ctrl-Alt-Del would wake it up. I put up with this for a few months, since I was hardly ever shutting the box down anyway, keeping it up 24x7 to help with the Proxy Project.
And that turned out to be The Problem. The code used the /tmp partition heavily. There were literally hundrends of thousands of tiny temporary files I neglected to delete. The system was hanging on the /etc/init.d/bootclean script, which cleans out the /tmp folder every time the system restarts. There were simply too many files to delete in a reasonable amount of time.
In fact I couldn't delete them, so I reformatted the /tmp partition. Then I had to mess around with the permissions to make it world-writable again.
"rm -f /tmp/*" will not work if you ever find yourself in this situation, due to limitations of the command line interface. Since most people don't create a dedicated /tmp partition, if this ever happens to you, Google "Linux bulk file delete".
In retrospect, this issue may have been self-inflicted. I made all the partitions, except the root partition, JFS (Journaling File System) because I had read that JFS is the perfect filesystem for MythTV because it can delete very large files very fast. In the early days (1998-2000ish) JFS was known to have problems with multiple small files, but that issue was reported to be fixed. Perhaps it wasn't. If this happens again I'll reformat the /tmp partition as ext3.
/lo
This one came up at work after a massive power outage. All my Linux boxes (none of which are "production") were flaky after the power came back on. One by one I got them all back into line except for our WikiMedia test server. MediaWiki ran, but it ran incredibly slow. Also, the vnc service was broken. But I had other things to do so I ignored it until The Boss (our new CSO) called to complain about MediaWiki being slow.
The MediaWiki issue was related to memcached, a performance enhancement that caches Web pages to memory. For security reasons, memcached listens for connections on the local interface (/dev/lo, also known as localhost or 127.0.0.1) because it's not the kind of service you want on the network listening for connections from just anywhere.
The problem was /lo was not coming up after reboot. Odd. "ifconfig lo up" fixed everything.
So that's that. Linux and I have had a love/hate relationship going on for over fourteen years now and I thought I knew it inside out. You learn something new every day, and if you don't or if you can't, it's time to look for another line of work.
And that turned out to be The Problem. The code used the /tmp partition heavily. There were literally hundrends of thousands of tiny temporary files I neglected to delete. The system was hanging on the /etc/init.d/bootclean script, which cleans out the /tmp folder every time the system restarts. There were simply too many files to delete in a reasonable amount of time.
In fact I couldn't delete them, so I reformatted the /tmp partition. Then I had to mess around with the permissions to make it world-writable again.
"rm -f /tmp/*" will not work if you ever find yourself in this situation, due to limitations of the command line interface. Since most people don't create a dedicated /tmp partition, if this ever happens to you, Google "Linux bulk file delete".
In retrospect, this issue may have been self-inflicted. I made all the partitions, except the root partition, JFS (Journaling File System) because I had read that JFS is the perfect filesystem for MythTV because it can delete very large files very fast. In the early days (1998-2000ish) JFS was known to have problems with multiple small files, but that issue was reported to be fixed. Perhaps it wasn't. If this happens again I'll reformat the /tmp partition as ext3.
/lo
This one came up at work after a massive power outage. All my Linux boxes (none of which are "production") were flaky after the power came back on. One by one I got them all back into line except for our WikiMedia test server. MediaWiki ran, but it ran incredibly slow. Also, the vnc service was broken. But I had other things to do so I ignored it until The Boss (our new CSO) called to complain about MediaWiki being slow.
The MediaWiki issue was related to memcached, a performance enhancement that caches Web pages to memory. For security reasons, memcached listens for connections on the local interface (/dev/lo, also known as localhost or 127.0.0.1) because it's not the kind of service you want on the network listening for connections from just anywhere.
The problem was /lo was not coming up after reboot. Odd. "ifconfig lo up" fixed everything.
So that's that. Linux and I have had a love/hate relationship going on for over fourteen years now and I thought I knew it inside out. You learn something new every day, and if you don't or if you can't, it's time to look for another line of work.
