NetNation Power Outage
There was a power failure at the NetNation data center that took down this blog and all my other sites for about 45 minutes. No explanation was given for the outage. Their call support center is located in Austin Texas and the only thing they could tell me was “The Vancouver data center experienced a power outage in roll 17 and 18. They’re working on getting power back up now.”
Somehow, I don’t quite understand how a state-of-the-art data center with three backup systems can lose power. This is something I will definitely need to bring up with my sale rep the next time I see him at a Vancouver networking event.
Luckily, it looks like the sudden lost of power didn’t have any negative effects on the servers. All seems to be working at peak levels. However, this outage is bugging me. I’m running a bunch of scenarios in my head, wondering how to take down three backup systems. I can’t come up with a way to do it. Can you?
- Posted in Ramblings, The Net
- 15 comments what's your take?
Sorry, the comment form is closed at this time.
Loading ...
Intern bumping into a certain plug?
Reply to this commentMakes sure to keep these notes for when renewal of your hosting contract comes up.
-But they may just site their SLA if any…
On high availability systems; there is only one sure way to test if the backsup system is properly setup…. so as you can imagine, only once they realize that someone left out an important detail in the backup process they are already getting calls from their customers.
Reply to this commentActually my employer had a big outage in the Houston area today knocking quite a bit offline. Just saw an update saying everything was restored and that it was related to a fiber cut in the area.
Would be interesting to know if that is the true reason of NetNation going offline - because it certainly does seem peculiar that three backup systems would fail. On the other hand, they should have diverse network connectivity as well.
Reply to this commentOne Year of 365 days has 525,600 minutes. An outage of 45 minutes is way less than 0.01%.
Reply to this commentBut you are right. It seems hard to lose power when everything is backed up with 3 backup systems in place.
During the big outage here in northern germany about 2 weeks ago I only learned from someone in some marketing forums what had happened and later found that not only my house, but also my servers (located in 2 different datacenters) were not affected at all.
“short circuit”, at least that is the thing what our datacenter answered after a large power failure.
Reply to this commentI guess that’s the %.01 of the %99.99 uptime that they probably promise?
Reply to this commentJohn,
As a data centre opterator, I can tell you that there are a lot of reasons why you were effected by this power outage. Common ones would be: PDU failure, UPS failure, circuit overload.
If your server that you host with has one power supply, you will always have a single point of failure. Check that you have 2 power supplies, connected to two circuits, connected to two different UPS systems. Most hosts provide this as part of the their ‘business’ plans.
Even better, get a cluster set up with redunant load balancers, web servers and database servers and separate network connections to your host’s core.
Regards,
Reply to this commentRichard.
Whew, got worried about the TTZ Media ads.
Reply to this commentThat’s because Austin sucks.
Reply to this commentSomeone probably tripped over the cord
You should look at a dns failover solution on an offsite server so that if it was longer then 45 minutes of down time your site will be available to your viewers.
Reply to this commentMost of his servers do have 2 power supplies.
Gary: One time that did happen somewhat, the tech was installing a new server and the power cord came unplugged. That was before he had dual power supplies…
Reply to this commentWhile not always power-related, any major site outage is a anxious moment for a webmaster/business.
A guy I know was doing some contract coding work for an ISP/webhost, and was actually needing to work at a terminal within the datacenter’s chilly main server room.
Suddenly the ISPs chief engineer comes sprinting in, waving his arms around! His cellphone alerts going crazy!
“Everything is down! ARGHH Whats happened?!”
They look down, and see that this guy had accidentally knocked a cable out behind the terminal while sitting down.
Oooops!
Reply to this commentJohn - Do you maintain your own servers or do they?
Reply to this commentWas the power out again?
Reply to this commentGary - No, NetNation had to change a circuit on our rack. That was why I lost power the last time. The circuit was faulty. The sites were down for about 15 min while they did this.
David - I have an admin who looks after the servers.
Reply to this comment