Business & Finance
12:45 PM

The Red Shift Theory

It's Sun's thesis about the explosive growth in demand for raw computing power--but is it more than the utility computing model warmed over?


An example of a red-shift company is Twitter, an instant-update social networking service that lets users post brief messages to the Web from cell phones or PCs. The number of Twitter users, now at more than 50,000, posting 30,000 telegraphic updates a day, is doubling every two to three weeks, according to Obvious Corp., the startup that launched the service in July 2006.

Hoping to grow into that neighborhood is GigaVox Media, a company that provides an Internet-based production and distribution platform for podcasters and video bloggers. "Like all startups, we envision ourselves as the next YouTube," founder Doug Kaye says with a laugh. What does that mean in terms of processing requirements? "I don't know," admits Kaye. "But we have to have that level of scalability."

To ensure that GigaVox does, Kaye has chosen Amazon Web Services, an on-tap application, storage, and computing infrastructure. That's one way red-shift companies are adapting--some, like Amazon and Google, are building supersized IT infrastructures, while others, like GigaVox, are tapping into them.

Red shift presents more complex quandaries for large companies. Papadopoulos began to understand the phenomenon, he says, in conversations over the last 18 months. "I would hear this conflict directly from customers, who would first say, 'I've got this really poor utilization--only 10% or 15% of our data center capability is really being used,'" he recalls. "And then I'd hear, 'I'm running out of data center space and power. I don't know how I'm going to continue to manage the growth,' and so on. On the one hand, the infrastructure is being poorly utilized, yet you're bursting at the seams."

The aha! moment came when Papadopoulos realized that there are really two different application sets driving computing demand: one consisting mostly of newer Web-facing applications driving exponential growth in both user demand and computing requirements; the other comprising back-end systems that are growing at more historical rates. "All this is really about which side of Moore's Law you're on," Papadopoulos says. "If your applications are growing faster than Moore's Law, you've got a fundamental set of issues about scale and power. If they're growing slower than Moore's Law, you've got all kinds of opportunities around consolidation."

It's important to emphasize that these rates of growth have seldom been seen in the computing world, even in the early days of the Internet boom. In August, for example, a little-known startup called Yoomba, which offers a peer-to-peer application that lets users place voice-over-IP calls or send instant messages to e-mail addresses automatically, announced that it had signed up 500,000 people in less than a month.

And it's not just user numbers that have IT infrastructures at Web 2.0 companies bursting at the seams. Research firm ComScore says Americans watched more than 8.3 billion video streams online in the month of May--nearly a video per day for every man, woman, and child in the United States. The largest provider, of course, was YouTube, which served up some 1.7 billion videos during that month--a crushing burden for any storage and networking system, even one as expansive as that of Google, YouTube's owner.

Mainstream companies also are feeling the processing pinch. Arizona Federal Credit Union, a midsize financial institution that services 244,000 consumers and small-business employees, is growing about 25% a year. In itself, that doesn't outpace Moore's Law. (In its original form, Moore's Law said that the number of transistors on a chip would double every two years, which is an annual growth rate of about 42%.) Expanding faster, though, is Arizona Federal's credit card division, a data-processing-intensive business that's growing at "an exponential rate" in terms of cards outstanding and transactions per card, says CIO James Phillips. Putting a strategy in place for that kind of growth has been Phillips' primary task since he joined the company in 2005.

Virtualization software let Phillips reduce the number of servers in Arizona Federal's data center to 61 from 87 over the last year, but that sort of consolidation won't keep up with the requirements he foresees in the next few years. "We will continue to expand our data center, but as it reaches maximum capacity we've got to make some long-range decisions," Phillips says. Over time, that could mean moving up to 30% of Arizona Federal's IT infrastructure to hosted services accessed over the Internet, he says.

What if you're much, much bigger? What if you're Visa, one of the world's largest transaction processors?

Visa's overall transactions are increasing at around 20% a year, which doesn't sound like a red-shift rate until you consider the base from which it's growing. The company processes some 6,800 transactions per second during the peak holiday season, and the nature of those transactions is growing more complex. Visa has spent five years and hundreds of millions of dollars annually modernizing and upgrading its transaction system, known as Visa Integrated Payments. Last year, the company opened a data center in the central United States that will process upwards of $1 trillion in transactions this year.

The rapid increase in computing demand is coming not only in Visa's real-time transaction authorization and clearing system but also in what Peter Ciurea, senior VP at Inovant, Visa's IT division, calls open systems. Those include post-transaction analyses, such as monthly aggregate reports for small businesses, in the form of Web services for Visa customers.

Those systems, Ciurea says, entail a "much more database-intensive workload," with a different processing model. And they're driving a shift in Visa's two main U.S. data centers to server virtualization and an internally developed distributed, grid-computing architecture. "In our core transactions business, we see one type of growth, but when you go to these value-added services, now you have to store that data," Ciurea says. "Storage costs are increasing faster than the actual processing costs." The result is predictable: "We need to invest in more and more processing power. My expectation is that we'll see a higher acceleration to that growth once we move to more of a grid model," he says.

That acceleration is being felt across many industries, including pharmaceuticals, where the Food and Drug Administration is pushing new quality-assurance tools known as Process Analytical Technology that have dramatically increased the need for data collection and processing, and energy, where the search for oil in ever deeper and more remote locations, particularly under the ocean, is generating terabytes of geological data that require minute analysis. It's also easy to spot in the software-as-a-service business, where industry pioneer Salesforce has seen the demands on its systems soar. According to Steve Fisher, senior VP for Salesforce's platform division, its "transactions" (i.e., API calls to its database) increased from about 500 million per quarter three years ago to 5.4 billion per quarter this year.

"Over the past four or five years, people have been talking about the commoditization of computing, as if all the innovation were over," Papadopoulos says. "Sure, general-purpose computing is a commodity. But designing really efficient systems to handle these kinds of workloads, and getting productive with the data center software and its management--those are anything but solved problems."

2 of 4
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
10/10/2011 | 11:51:23 AM
re: The Red Shift Theory
yes i like your theory on quantem theory.
Register for InformationWeek Newsletters
White Papers
Current Issue
2016 InformationWeek Elite 100
Our 28th annual ranking of the leading US users of business technology.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on for the week of June 19, 2016. We'll be talking with the editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.