whee, server death!

So, the web server died last weekend, presumably sometime after 3:45pm, because that's the last time anything was recorded in the log.

It died in a rather interesting way, though. Instead of just freezing up or rebooting itself, it just started acting extremely odd. Images and the like on the webpage were broken, files were missing when I checked over FTP, and PHP scripts wouldn't run at all.

Apache would still serve pages though. Severely broken and incomplete pages, but pages nonetheless.

"Huh," I thought. "Gonna be a fun morning at work on Monday. Hope there's no data loss."

Anyway, I get to work and turn on the monitor on the server. Looks like ext3 barfed on something and is now filling the console with error messages. Keyboard doesn't respond, can't do a clean boot despite many attempts.

Left with no other choice, I pressed the big red button on the front and did a hard-reset.

Then, it rebooted, I heard the hard drive spin back up, and... booted perfectly as if nothing had happened.

This incident right here has turned me into a believer of journaling filesystems. Although ext3 is a kind of a kludgy journaling system slapped on top of ext2, it still saved my ass this morning.

So, I logged in and checked all the files for any corruption, and didn't find any. So I brought all the services back up. As far as I could tell, the hard drive spun itself down for some reason and didn't respond to the IDE controller telling it to wake back up.

This didn't manage to kill all processes, though. While sshd couldn't cope and died silently, pure-ftpd and apache lived on. Apache couldn't fork, however, so past a certain number of connections it just stopped taking them. And it could only serve data that was still cached in the buffers. And PHP wasn't in memory, so no PHP code could run. I don't think any scripts were, either.

The hard drive just going to sleep out of the blue doesn't bode well for the hard drive itself, though, so I think I'm going to have to look into getting a backup or replacement of some sort.

So, yeah, there was my interesting server death story. Or something.