Planned Server Downtime - Wednesday, October 24th

Well, as some of you noticed, the site finally came back online around 11 PM Eastern time. Go easy on me. It was only about 16 hours longer than expected.

First of all, my apologies about that. I'm terribly sorry for the extended downtime.

Basically, what I needed to do (for various reasons) was do a re-install of the operating system. My plan was to get a temporary backup store at the datacenter, rsync the data over there, re-install the OS, then scp the files over to the reconfigured original server. Typically this is NOT the way you do this on a production server. Typically you get a new server, install the new operating system on the new server, then migrate the sites over there. Doing so this way can be done with virtually no downtime. But you end up spending a months worth of cost for the second server. Since this isn't a production server (It only runs personal stuff for me, and FBB-related sites), I figured I could do this in the wee-hours of the morning and get away with it. Due to both my own mistakes and problems out of my control, this didn't exactly happen.

I woke up this morning, rsync'd the data over. No problem. Call up the data center, have them boot the server off the re-installation CD. Well, it took them longer than expected (about a friggin hour), during which times the sites couldn't run (because if content's updated, you gotta re-sync the data over to the backup location). So I wait. Finally they get back to me, and I quickly re-install. Submit the request to restart the server into the newly installed operating system. Again I wait an hour. Keep in mind these are 5 minutes jobs for even poorly trained technicians. When they come back and let me know my "server was online", they forgot to set sshd to start on the default runlevel. You can't even login to linux a server without that.

So I call them back and tell them to startup sshd. Unfortunately at this time it's already to leave for work. I drive to work, get there (it's about an hour drive), and yay, sshd is started. At that time our network at the office began having problems. Being the systems administrator, it means more fun for me. I won't bore you with the details, but it basically involved setting up our own internal BIND and dhcpd servers for the office. Fun times.

So it's now 3 PM, and I haven't even setup apache, PHP or MySQL yet, let alone transfer the content over. With what's going on at work, I haven't had more than 5 minutes to spend on it since 7 am. So in my spare moments at work I setup PHP, Apache and MySQL on the re-installed server. 6 pm rolls around and I transfer the files over as I leave to go home.

I get home, files are finally done, and I finish correcting the permissions, and we return online.

I had called up the datacenter earlier in the week and explained to them what I was going to be doing. They originally told me they would have an administrator on duty so when I called up to get it booted off an install CD/reboot the server into the new environment, they could do that right when they got off the phone. The fact that they weren't able to do that, and essentially wasted 3 hours of my time this morning has me really miffed. I used to work for a webhosting company, and had physical access to my machine. Having to really on them isn't something I like, and I may end up looking for a new place fo colocate my machine (don't worry, if I do it'll be done the "right" way as I described above. Which means no downtime). Once it got to 7 am and I had to get ready to leave for work, combined with the problems I had at my job, and I knew it was gonna be a long one.

My apologies about that. Hope nobody missed Fast Break Basketball too much during the day. You have my word that the site will be online every second for the rest of 2007.

	Author	Topic: Planned Server Downtime - Wednesday, October 24th (Read 4505 times)
0 Members and 4 Guests are viewing this topic.