SSBA

League Information => League Announcements => Topic started by: NetsGM on October 17, 2007, 02:26:37 PM



Title: Planned Server Downtime - Wednesday, October 24th
Post by: NetsGM on October 17, 2007, 02:26:37 PM
Hey guys,
I'm performing some upgrades to the server in the early morning (around 4 am GMT-5) this coming Wednesday.  Server will probably be offline for around 2-3 hours or so.  Just giving you a heads up.


Title: Re: Planned Server Downtime - Monday, October 22nd
Post by: KingsGM-old on October 17, 2007, 02:40:48 PM
Server will probably be offline for around 2-3 hours or so. 

You want to kill Tom?


Title: Re: Planned Server Downtime - Monday, October 22nd
Post by: HornetsGM on October 17, 2007, 03:09:15 PM
Server will probably be offline for around 2-3 hours or so. 

You want to kill Tom?

He's already planned ahead to make sure Tom is sedated during this time.


Title: Re: Planned Server Downtime - Monday, October 22nd
Post by: CelticsGM on October 17, 2007, 03:52:53 PM
i will most likely use one of the 12 backup sites, or, even better ...


















































... sleep.


Title: Re: Planned Server Downtime - Monday, October 22nd
Post by: PacersGM on October 18, 2007, 03:20:43 AM
i will most likely use one of the 12 backup sites, or, even better ...
... sleep.

well typical TOM stays up all night and sleeps during work :lol:


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: NetsGM on October 18, 2007, 07:00:30 AM
Ack,  Sorry guys.  This has been re-scheduled for Wednesday (24th), same time.  Please let the rbsl/djsl guys know.


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: NetsGM on October 25, 2007, 04:32:45 AM
Well, as some of you noticed, the site finally came back online around 11 PM Eastern time.  Go easy on me.  It was only about 16 hours longer than expected.

First of all, my apologies about that.  I'm terribly sorry for the extended downtime.

Basically, what I needed to do (for various reasons) was do a re-install of the operating system.  My plan was to get a temporary backup store at the datacenter, rsync the data over there, re-install the OS, then scp the files over to the reconfigured original server.  Typically this is NOT the way you do this on a production server.  Typically you get a new server, install the new operating system on the new server, then migrate the sites over there.  Doing so this way can be done with virtually no downtime.  But you end up spending a months worth of cost for the second server.  Since this isn't a production server (It only runs personal stuff for me, and FBB-related sites), I figured I could do this in the wee-hours of the morning and get away with it.  Due to both my own mistakes and problems out of my control, this didn't exactly happen.

I woke up this morning, rsync'd the data over.  No problem.  Call up the data center, have them boot the server off the re-installation CD.  Well, it took them longer than expected (about a friggin hour), during which times the sites couldn't run (because if content's updated, you gotta re-sync the data over to the backup location).  So I wait.  Finally they get back to me, and I quickly re-install.  Submit the request to restart the server into the newly installed operating system.  Again I wait an hour.  Keep in mind these are 5 minutes jobs for even poorly trained technicians.  When they come back and let me know my "server was online", they forgot to set sshd to start on the default runlevel.  You can't even login to linux a server without that.

So I call them back and tell them to startup sshd.  Unfortunately at this time it's already to leave for work.  I drive to work, get there (it's about an hour drive), and yay, sshd is started.  At that time our network at the office began having problems.  Being the systems administrator, it means more fun for me.  I won't bore you with the details, but it basically involved setting up our own internal BIND and dhcpd servers for the office.  Fun times.

So it's now 3 PM, and I haven't even setup apache, PHP or MySQL yet, let alone transfer the content over.  With what's going on at work, I haven't had more than 5 minutes to spend on it since 7 am.  So in my spare moments at work I setup PHP, Apache and MySQL on the re-installed server.  6 pm rolls around and I transfer the files over as I leave to go home.

I get home, files are finally done, and I finish correcting the permissions, and we return online.

I had called up the datacenter earlier in the week and explained to them what I was going to be doing.  They originally told me they would have an administrator on duty so when I called up to get it booted off an install CD/reboot the server into the new environment, they could do that right when they got off the phone.  The fact that they weren't able to do that, and essentially wasted 3 hours of my time this morning has me really miffed.  I used to work for a webhosting company, and had physical access to my machine.  Having to really on them isn't something I like, and I may end up looking for a new place fo colocate my machine (don't worry, if I do it'll be done the "right" way as I described above.  Which means no downtime).  Once it got to 7 am and I had to get ready to leave for work, combined with the problems I had at my job, and I knew it was gonna be a long one.

My apologies about that.  Hope nobody missed Fast Break Basketball too much during the day.  You have my word that the site will be online every second for the rest of 2007.


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: CelticsGM on October 25, 2007, 04:49:39 AM
100% availability.
:lol2:


if i promise more than 99.99 i get killed by our sales staff ...


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: KingsGM-old on October 25, 2007, 05:53:45 AM
Are there any survivors in Austria?


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: CelticsGM on October 25, 2007, 06:22:46 AM
Vienna seems ok (apart from some bad weather conditions), Graz i'm not so sure. Guess the commish is in troubles ...


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: NetsGM on October 25, 2007, 07:38:55 AM
100% availability.
:lol2:


if i promise more than 99.99 i get killed by our sales staff ...

Well, you see, I'm promising 100% availability to people who don't pay anything.  So if I fail to meet that, I can't be held liable :)


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: CelticsGM on October 25, 2007, 07:42:00 AM
well, most people not from this business wouldn't know the difference between 99.99% and 100% anyway ...

again, great job you're doing with the host servers for all these leagues.


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: RaptorsGM on October 25, 2007, 05:16:41 PM
We barely survived in canada


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: HornetsGM on October 25, 2007, 07:44:30 PM
USA here, East Coast... We made it, but we had to eat the Nets GM. RIP.


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: GrizzliesGM on October 25, 2007, 08:25:51 PM
I didn't eat anyone here, but I did almost make my computer explode pressing refresh soooooooooooooo many times!


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: CelticsGM on October 25, 2007, 09:01:24 PM
that's it. you owe us a few F5 key's, bods ...
:-)


Title: Re: Planned Server Downtime - Wednesday, October 24th
Post by: NetsGM on October 25, 2007, 11:18:46 PM
USA here, East Coast... We made it, but we had to eat the Nets GM. RIP.
You and your whole team of pansies can eat me  :moon: