Article from SearchCloudProvider on how certain data center providers fared during Hurricane Sandy, particularly with regard to implementation of disaster recovery plans.
Good news: AIS operates data centers in San Diego and Phoenix, sufficiently far apart to neutralize risks / threats from any one single event, be it weather, earthquake, terrorism, or locusts.
Emphasis in red added by me.
Brian Wood, VP Marketing
———–
Hurricane Sandy: How Did Providers’ Data Center DR Plans Do?
Natural disasters are a common enemy of the data center. Hurricane Sandy recently tested the limits of East Coast-based cloud providers and colocation and hosting facilities after the “superstorm” ravaged parts of New Jersey, New York, Virginia and New England — leaving power outages, flooding and infrastructure damage in its wake.
Regardless of their geographic location, it’s critical for cloud and hosting providers to have data center disaster recovery (DR) and business continuity plans and procedures in place to ensure uptime for customers, especially when a disaster strikes.
Although many regions ravaged by the storm continue to battle power outages and fuel shortages, some cloud providers and service providers managed to avoid a service disruption during and after the hurricane. Three of them — The Telx Group Inc., Integrity Virtual IT Inc. and CentriLogic Inc. — shared how their data center DR and business continuity plans kept their services online and their customers happy.
Telx weathers the storm, thanks to data center DR prep
Telx, a New York City-based data center operator and colocation provider, has two facilities in the New York metropolitan area and two New Jersey locations; another New Jersey data center is currently under construction. All four data centers suffered a loss of utility power and were forced onto generator power during the hurricane.
Anticipating an outage, Telx had transitioned its NJR2 facility — a larger data center in Clifton, N.J. — to generator power before the storm and remained on it for eight hours, said Ron Sterbenz, vice president of marketing at Telx. After the two electrical substations the NJR2 facility uses came back online, utility power eventually stabilized. Telx’s smaller data center in Weehawken, N.J., NJR1, remained on generator power for a full day.
Both of the company’s Manhattan facilities (NYC1, at 60 Hudson St. in Tribeca, and NYC2, at 111 8th Ave. in Chelsea) went onto generator power and remained on the generators until utility power was restored, Sterbenz said. The NYC2 facility experienced slight cooling issues when the the building’s management team reduced air-conditioning throughout the building. Telx had prepared for the storm by bringing in fans for spot cooling, however, and employees were able to open the windows after the rain had subsided.
Telx also topped off each data center’s fuel supply before the storm, called in additional employees to work during the storm, and stocked up on nonperishables to feed employees who might be working longer shifts, Sterbenz said.
Despite every data center operating solely on generator power for at least a portion of the storm and despite the overheating concerns in its NYC2 facility, Telx reported no outages. “We were adequately prepared and kept our customers up and running during the hurricane,” Sterbenz said.
Integrity Virtual IT gets proactive, CentriLogic focuses on people
Meanwhile, some conventional IT service providers in the region relied on cloud providers to keep them going. Integrity Virtual IT, an IT outsourcing company in Silver Spring, Md., uses hypervisor-based data replication products from Israel-based DR-as-a-Service provider Zerto to replicate data across Integrity’s data centers in Reston, Va., and Chicago.
While the Virginia data center didn’t lose power or default to its generators during the storm, Integrity offered customers that already replicate data between its two sites to move primary hosting to the Chicago facility before the storm, said Ron Offer, CEO at Integrity.
Affigent LLC, a Herndon, Va.-based technology consulting firm for government customers and an Integrity customer, opted to switch to the Chicago data center as a proactive measure for the storm’s duration. “When we realized we were going to be in the center of the storm, we made the decision to flip our primary site to Chicago,” said Matthew Friedman, business operations director at Affigent. The process of data replication between the two sites took less than one hour, and it is transparent to Affigent’s government customers. “We aren’t in the position to experience an outage or go through a disruptive replication process that lasts hours,” Friedman said.
CentriLogic, a cloud, hosting and managed service provider with a data center in Rochester, N.Y., did not experience outages due to the hurricane, but it approached the storm as a test of how well a provider’s DR plans continue to support not only infrastructure and failover capabilities, but also people and communications, said Robert Offley, president and CEO. “You can have your computers up and running, but [providers] need to make sure their customers can remotely access the system and their employees can provide that continued support,” he noted.
Lessons learned about data center DR: Design facilities wisely
Although there is little that can be done to data center infrastructure when a natural disaster or storm looms, cloud providers and data center operators should be stocked up on enough fuel for the generators and food for emergency staff, said Rachel Dines, senior analyst for infrastructure and operations at Forrester Research Inc.
Food and fuel treat the symptoms, however, not the disease. Many data center operators keep generators in the basement of their building, the area that’s the most susceptible to flooding during a hurricane. Keeping critical infrastructure components out of the basement and first-floor flood zones is one of the first steps providers can take in protecting customers from downtime, Dines said.
For managed services or hosting providers especially, failover is crucial to keeping customers online. “For some of the customers located in the New York data centers that were impacted by significant downtime — like Datagram customers — many were at least able to fail over to another geographic location,” Dines said.
Even with failover in place, some customers experienced both primary and secondary sites going down as a result of Hurricane Sandy. Consequently, cloud providers should consider geographic diversity when choosing data center sites, Dines said. “Providers need to think about the risk profile of the area they are building [data centers] in, and get their secondary site into a different region than the primary site,” she said.
