If anyone visits this blog on a regular basis, then you’ve certainly noticed the issues the site/server was having lately. Something was causing the server to go crazy and would tie up processes on the server and slow things down to the point of a crawl. Well, last friday, when I had the host reboot the server, the disk had finally failed. I ended up having to have them boot the server with a live cd and I ran FSCK on the drive.

Apparently that was a big mistake. Whatever kind of gremlins had attacked the server, they devoured the ext3 Journal so that when FSCK ran, it couldn’t find it. It ended up dumping everything in to lost+found on /dev/hda3. Well, the server, at that point was toast. I had the host put up a new box and mount the old drive as a slave. Fortunately, it looks like everything was in lost+found and i was able to restore the databases (for the most part) and the files for the sites on this server, so not too much was lost.

I did spend most of the weekend in what appeared to be a coma fueled by some of the best diet pills you could find, though really it was pizza and caffeine that helped me get through the mess.

One thing that I am thankful for is the fact that within the lost+found directory, the directory structure was mostly in tact. If you haven’t looked in your lost and found folder on a unix file system, I suggest you check it out some time. The files are moved in there and renamed with their inode (so a file that used to be called joe.txt could now be #1132432). You can do some other reading on this @ TLDP and I suggest you do.

I was able to find the mysql files I needed to restore the databases (and all except the ones for *THIS* site were in usable condition) and the folders with the sites in them for the sites. I do have one site that is going to take a little longer to restore back to normal as it seems to have split the folders out in to more than one directory for some reason (which leads me to believe that it was this directory that caused the issues!) but as everything is there, its going to be a matter of figuring out which galleries are which….

This leads me to my next point. BACK UP ALWAYS! Us crons, use programs, whatever. Just back up and do it often. Never assume your host has a back up (because they don’t). Personally, I’m looking at backup ninja (thanks jacquelyn!) to handle it for me!

Anyways, the site is slowly being put back together and it looks like everything except the old image links are working….

Popularity: 1% [?]