NetNation Power Outage
written by John Chow
There was a power failure at the NetNation data center that took down this blog and all my other sites for about 45 minutes. No explanation was given for the outage. Their call support center is located in Austin Texas and the only thing they could tell me was “The Vancouver data center experienced a power outage in roll 17 and 18. They’re working on getting power back up now.”
Somehow, I don’t quite understand how a state-of-the-art data center with three backup systems can lose power. This is something I will definitely need to bring up with my sale rep the next time I see him at a Vancouver networking event.
Luckily, it looks like the sudden lost of power didn’t have any negative effects on the servers. All seems to be working at peak levels. However, this outage is bugging me. I’m running a bunch of scenarios in my head, wondering how to take down three backup systems. I can’t come up with a way to do it. Can you?
Find out what I am doing right now by following me on Twitter! If you like this post then please consider subscribing to my full feed RSS. You can also subscribe by Email and have new posts sent directly to your inbox.
(No Ratings Yet)
Loading ...
Here's A Few More Related Posts
Intern bumping into a certain plug?
Makes sure to keep these notes for when renewal of your hosting contract comes up.
-But they may just site their SLA if any…
On high availability systems; there is only one sure way to test if the backsup system is properly setup…. so as you can imagine, only once they realize that someone left out an important detail in the backup process they are already getting calls from their customers.
Actually my employer had a big outage in the Houston area today knocking quite a bit offline. Just saw an update saying everything was restored and that it was related to a fiber cut in the area.
Would be interesting to know if that is the true reason of NetNation going offline - because it certainly does seem peculiar that three backup systems would fail. On the other hand, they should have diverse network connectivity as well.
One Year of 365 days has 525,600 minutes. An outage of 45 minutes is way less than 0.01%.
But you are right. It seems hard to lose power when everything is backed up with 3 backup systems in place.
During the big outage here in northern germany about 2 weeks ago I only learned from someone in some marketing forums what had happened and later found that not only my house, but also my servers (located in 2 different datacenters) were not affected at all.
“short circuit”, at least that is the thing what our datacenter answered after a large power failure.
I guess that’s the %.01 of the %99.99 uptime that they probably promise?
John,
As a data centre opterator, I can tell you that there are a lot of reasons why you were effected by this power outage. Common ones would be: PDU failure, UPS failure, circuit overload.
If your server that you host with has one power supply, you will always have a single point of failure. Check that you have 2 power supplies, connected to two circuits, connected to two different UPS systems. Most hosts provide this as part of the their ‘business’ plans.
Even better, get a cluster set up with redunant load balancers, web servers and database servers and separate network connections to your host’s core.
Regards,
Richard.
Whew, got worried about the TTZ Media ads.
That’s because Austin sucks.
Someone probably tripped over the cord
You should look at a dns failover solution on an offsite server so that if it was longer then 45 minutes of down time your site will be available to your viewers.
Most of his servers do have 2 power supplies.
Gary: One time that did happen somewhat, the tech was installing a new server and the power cord came unplugged. That was before he had dual power supplies…
While not always power-related, any major site outage is a anxious moment for a webmaster/business.
A guy I know was doing some contract coding work for an ISP/webhost, and was actually needing to work at a terminal within the datacenter’s chilly main server room.
Suddenly the ISPs chief engineer comes sprinting in, waving his arms around! His cellphone alerts going crazy!
“Everything is down! ARGHH Whats happened?!”
They look down, and see that this guy had accidentally knocked a cable out behind the terminal while sitting down.
Oooops!
John - Do you maintain your own servers or do they?
Was the power out again?
Gary - No, NetNation had to change a circuit on our rack. That was why I lost power the last time. The circuit was faulty. The sites were down for about 15 min while they did this.
David - I have an admin who looks after the servers.