Taming The Power Crisis

On May 7, as an unusually hot Northern California afternoon gave way to a balmy evening, one of Exodus Communications Inc.'s greatest fears came to fruition. During a power outage attributed to the heat spell, a component failed at an Exodus data center. The Web-hosting company's backup power system failed to kick in, interrupting the flow of data to an unspecified number of customers, one of which was Yahoo Inc.

While such an occurrence hardly qualifies as a national crisis, it does highlight the vulnerability of technology-dependent companies at a time when they can't count on a consistent power supply. The power woes that have plagued the nation--and especially California--for the past several months are forcing companies to look closely at their electricity infrastructure. More than ever, power outages threaten to cut off critical services and customer access, and those who choose to ignore their backup power systems do so at their peril.

For Exodus and other hosting companies, the ability to guarantee round-the-clock power to clients' systems is an essential ingredient of success. That's why the May 7 outage was looked upon as a red flag. "It's really caused us to learn about our systems," says K.C. Mares, Exodus' director of electric energy and utilities. "We've taken a lot of precautions to ensure that an event like that doesn't happen again."

In particular, Exodus took a long, hard look at a part of its data-center operation that rarely gets noticed by anyone other than the engineers who maintain it: namely, the uninterruptible power supply, or UPS, system.

Worth The Investment
UPS systems offer a battery-powered emergency power supply that automatically kicks in to temporarily protect data during even the briefest of outages, giving way to diesel or turbine generators during longer outages. The cost can range from under $100, for a unit to protect a home office computer, to as much as $6 million, for an elaborate data-center implementation. (For tips on choosing a UPS system, see the sidebar, "Reducing Reliance On Your UPS System.")

Even without a lengthy power outage, the investment is likely to be worthwhile. Mares points out that it only takes a quarter of a second without power for a computer to go down, resulting in possible data loss or corruption. In other words, while the rolling blackouts in California are of great concern, a normal day's power blips pose a constant threat to stand-alone PCs and full-blown IT operations alike. And, during the hot summer months, those blips come with increasing regularity.

That's why UPS systems are designed to serve another purpose that is just as crucial as backing up power supplies: cleaning the power that your electric utility provides. According to Mares, by running normal power flow through UPS systems, Exodus' 43 data centers receive power that's 100 to 1,000 times more reliable than what arrives from the local utility.

Despite such obvious benefits, Farah Saeed, a power supply and battery analyst for Frost & Sullivan, says too many companies overlook this important function when determining the need for a UPS system. "There's a perception that [uninterruptible power supply systems] can only help when there's a blackout," she says. "That's not the case."

Greg Dolence, corporate facilities manager for Progressive Insurance, is well aware of the impact of such blips in the flow of electricity. Dolence says that every minute Progressive's call-center reps wait for computers to reboot after a brief surge translates to time that customer service grinds to a halt. The growing potential for such problems has Progressive considering replacing the UPS system it installed in 1995. "We have equipment we can never have down," says Dolence. "It's critical for us." (For guidelines on UPS redundancy, see the sidebar, "A UPS Redundancy Primer.")

Not Ambulance Chasers
While it might seem that the national spotlight that's been shining on the power crisis would fuel demand for UPS technology, that hasn't been the case. In fact, Saeed's research indicates that the market's boom preceded the crisis. After growing by 20% in 1999 and 17% in 2000, sales of UPS systems are expected to grow just 6% or 7% this year, to $6.4 billion, and Saeed says most of that growth will come from demand in Third World nations where power supplies are spotty at best.

Such data may explain why even UPS vendors themselves aren't viewing the nation's power woes as an opportunity. "It is not our policy to be ambulance chasers," says Peter Klanian, worldwide strategy director for American Power Conversion Corp. "[The power crisis] will increase the visibility of power protection, but certainly we don't cheer when there are rolling power outages."

Market saturation may be the biggest reason the crisis isn't fueling UPS sales. Saeed and other experts say that in the United States, those for whom power is most critical--most notably data centers, which are credited with driving the growth of the past two years--tend to have state-of-the-art UPS systems in place. Meanwhile, smaller companies that would undoubtedly benefit from having a dependable backup power system either aren't aware of the need or refuse to justify the expense.

Not all vendors share Saeed's lukewarm analysis. Daniel Doimo, vice president of marketing for MGE UPS Systems, says he's expecting the growing awareness of the need for UPS systems during the hot summer months to yield increased demand among small businesses and home offices.

"People will learn the hard way that having power protection is critical," Doimo says. "We don't see it yet, but our bid activity is much higher than last year."

Is It Time To Upgrade?
Some organizations were alerted to the UPS technology's importance before the power supply problems surfaced. Santa Clara University, for instance, hadn't placed much stock in backup power equipment--until the Web changed that outlook. Two years ago, with the Internet becoming a more important component of the school's IT infrastructure, university officials determined that its communications systems--including telephones, E-mail, and Web applications--were vulnerable.

Carl Fussell, co-director of information technology, says the old setup was so fragmented that it was difficult to determine what equipment would be impacted by an outage. Fussell and his staff embarked on a $750,000 upgrade of the university's antiquated UPS system and long-term backup power supply. With the California power crisis, the upgrade turned out to be well timed. The project is just wrapping up now; the only remaining need is a new cooling system to protect the UPS equipment.

But even those who supposedly should be in the know sometimes fall behind. Ed Harrell, vice president of engineering for data-center builder DataSphere Inc., says that's usually a function of the age of the data center. "We find a lot of older centers that are using older, less-reliable technology," says Harrell. "New data centers typically have more knowledge of what's in the market currently."

There are older data centers, however, that have kept up-to-date when it comes to ensuring reliable backup power sources. Charlie Potter, a staff engineer who helps to oversee three major data centers run by the portion of Verizon Communications that was once GTE, says none of those facilities has experienced an outage of any length in more than six years.

Potter says each of the three data centers--in Tampa, Fla., Sacramento, Calif., and Fort Wayne, Ind.--used to have multiple UPS systems connected to a single breaker. Today, each system has its own dedicated breaker, greatly reducing the chances of failure, and a fifth system has been added to each center to jump in should any of the primary backups fail. Additionally, Potter's crew has taken every opportunity to replace older equipment and add battery power to augment the time a center can run on its UPS systems.

The way Potter sees it, there's no reason that any data center can't easily ensure its system doesn't have any holes. It's a simple matter of making sure there are multiple points of failure so an outage in one component doesn't take down others around it.

"You have to start at your PCs and go all the way back to the power utility and find your single points of failure," he says. "That's where you need to spend your spare time."

Still, an engineer performing a single task out of sequence can render such diligence moot. Despite what the best technology has to offer, certain things simply remain out of our control. Says Potter, "Ninety-nine percent of outages are caused by people."