Yesterday, starting at 10:35am PDT, the data center hosting our servers lost utility power. This happens occasionally and the data center is prepared for it. They have a bank of batteries to keep it running while the emergency generators come to life and stabilize and then power is transferred to the generators. Unfortunately, the transfer switch to do that failed (the General Manager told me that “it blew up”!) and the bulk of the data center lost power around 10:40am. They managed to bypass the transfer switch manually and got over to the generators around 11am and get back on the network about 11:10am.
One of our servers survived the power outage just fine and it was back online at that time. The other server, however, had to be manually restarted and then checked for file system damage so it did not come online until shortly after noon.
The data center stayed on generator power until around 9pm last night when they successfully transitioned back to utility power (with a newly installed transfer switch!).
We apologize for the outage and know how frustrating it is to not have your websites available. It is very unusual for this data center to have any significant outage and, after having talked to the GM, I feel they did everything they could to minimize the duration and get us all back online as fast as possible.
When outages do occur, we will post immediate updates to our Twitter account, @synotac, and we are also available via our main line at 503-517-2116.
