Pat Maio of the North County Times posted this story featuring AIS and some of our clients.

The corporate world dodged a big one last week when the power went out —– not just here in San Diego County, but everywhere from coal-fired plants in the Midwest to accounting firms extending across the United States.

This is because corporations have shifted their sensitive computing data to the cloud computing environment, or hidden places where banks of leased computers can be kept with enough space, power, cooling and security.

One of the biggest companies to provide cloud-type services locally is San Diego-based American Internet Services, which operates out of an old, Cold War-era missile factory once owned by General Dynamics along Lightwave Avenue.

The cloud computing environment is where computer services are offered outside a company’s main business for storage of critical data, and where the data can be kept safe on interconnected servers and pulled up when needed. Having power to run the servers and keep companies connected is important.

When the local electrical grid collapsed on Thursday, Tim Caulfield, president and chief executive officer of American Internet Services crossed his fingers and hoped his data center services network would work without a hitch.

“All of the electricity we needed was being generated by us,” said Caulfied when the power went out. “Nobody does what we do. There are some other providers locally, but we are the largest.”

To help keep businesses connected throughout the outage he relied on banks of state-of-the-art batteries (not car batteries) and diesel powered generators, otherwise loss of sensitive financial data and productivity could have skyrocketed higher than the $118 million estimate recently touted by the La Jolla-based National University System Institute for Policy Research.  It could have hit hundreds of millions of dollars in lost productivity, said Ken Carter, executive vice president for data center infrastructure with AIS.

The Thursday outage interrupted power for 12 hours for more than 5 million people in San Diego County, southern Orange County, western Arizona and northern Baja California.

The biggest headache for Caulfield’s San Diego company was making sure he had enough deliveries of diesel fuel lined up to fill the labyrinth of electrical generators behind its buildings. The fuel is needed to keep its five data centers humming.

When the outage happened, the company prepared for 24 to 48 hours of darkness —- though the blackout ended up lasting about 12 hours at their location. Fuel tanks had already been topped, and more deliveries of fuel were on the way. The center has an above-ground, 10,000-gallon fuel tank in the rear of its Lightwave complex, and two 4,000-gallon fuel tanks underneath the power boxes of four generators each.

Larry Goldenhersh, founder, president and CEO of Carlsbad-based Enviance Inc., is especially thankful that Caulfield’s data service operation held together. Enviance processes 1 trillion data points a month for its clients.

“The promise that the cloud makes is it is always on. In order to honor our promise to be a provider of Internet services, we need to make sure our partners deliver on their promises,” Goldenhersh said.

Enviance collects air, water, waste and fuel emission data under contract for 30 percent of the electric and gas utilities in the United States, the Army’s strategic commands, oil refiners and others that need someone to keep track of voluminous environmental permitting requirements.

Privately held Enviance, which is a $20 million in annual revenue company, counts among its clients big corporations such as American Electric Power, Chevron, DuPont, Pacific Gas & Electric, Sempra Energy and Southern California Edison.

American Electric operates 49 coal-fired power plants in the Midwest. Enviance uses software to connect computer systems to measure greenhouse gas carbon dioxide emissions from the stacks of the plants.

“We track greenhouse emissions around the country,” Goldenhersh said.

When the outage hit, Goldenhersh was holding a management meeting with top executives. One of the senior vice presidents was asked to go out to his car in the parking lot and turn on the radio to find out what was going on.

“The other first thing we did was speak with AIS to make sure that the generators had fuel and the switchover was happening and that there’d be no service loss,” he said. “We also had a Verizon wireless card to get to the Web remotely to make sure our systems at (AIS) were up.”

Robert Chandler, president and CEO of El Cajon-based Cloud9 Real Time, a cloud computing company with thousands of accounting firms as customers, relied on AIS to keep its bank of servers and infrastructure stored at AIS’ Lightwave Avenue computing center operating during the outage.

“When the electricity went out, that would have brought all of our clients to their knees,” Chandler said. “The cloud did not collapse.”

AIS holds about 52 percent of the data center provider market in San Diego County with more than 600 customers ranging in size from start-up companies to Fortune 100, according to Caulfield.

A data center provider such as AIS rents out computer servers for businesses to use for housing everything from sensitive financial transactions and sales data, to emails or whatever. It also can take possession of a firm’s servers and provide the space, power and cooling to provide the same services, either on a backup basis or provide the whole package.

Collectively, the five AIS data centers in San Diego County are huge energy consumers. The Lightwave Avenue complex was consuming 2.56 megawatts of electricity when the power went out, according to Caulfield. A megawatt is enough to supply about 650 homes.

The systems at AIS immediately detected the outage, and within milliseconds transferred the power generation of its five centers to a uninterruptable power supply of batteries, which can maintain the load for up to 45 minutes. After the batteries complete their job, the load is then transferred to power generators, which can run indefinitely as long as there is available fuel.

“Essentially, most of it happened without human intervention,” said Caulfield. “We had no customer impact other than being concerned when the power went off.”

AIS also kept its clients connected to the Internet through multiple hosting vendors. Some of the vendors also are rivals, including Herndon, Va.-based XO Communications; Broomfield, Colo.-based Level 3 Communications Inc.; Wayne, Pa.-based SunGard Data Systems; and U.K.-based Red IT Ltd.

###

Photo Credit: Ken Carter, vice president of operations and infrastructure at the AIS offices in Kearny Mesa and behind him are enclosed generators capable of holding a load up to 72 hours on full power. MISAEL VIRGEN | mvirgen@nctimes.com