Yahoo Pipes is a way to mash up RSS feeds and Web pages into combined information, apply rules and filters, and publish it. When it launched Feb. 7, 2007, the Pipes API worked fine in principle. But as "Daniel" said from the audience at a San Francisco panel this week, Pipes wasn't ready to scale.The gathering was a panel at the GigaOm network offices in San Francisco, an online publisher that produces blogs on several key niches, including Apple and cloud computing. The question before the panel was: what should a startup do, build out its own data center or contract with an infrastructure service provider?

The somewhat predictable answer from a panel of service providers was that startups with intentions of growing fast should find a service provider to their liking and let the service handle the infrastructure issues. With a service, whether an application building and deployment site, such as EngineYard, or a server infrastructure provider, such as Joyent, the service provider will handle the issues of fast growth better and spread the expense across multiple users.

Jason Hoffman, CTO of Joyent, cited the possibility of the TechCrunch or Slashdot effect on a new company, where its notice and commended in a popular opinion broker site, then crashes as traffic swarms to the newcomer. You can avoid missing out on all those prospective customers, he advised. "If you think you are good at building infrastructure, then go ahead, build it yourself. But I think I'm pretty good at building infrastructure," he said.

Tom Morini, CTO of EngineYard, the Ruby on Rails deployment platform, declined to go along with the consensus, however. "I'd like to dispute the myth that the TechCrunch (a popular blogging site) or SlashDot effect is a great surprise to startups. In many cases, they can predict when they'll see a surge in traffic and are well-prepared for it.

At that point, Daniel, who was probably Daniel Raffel, a member of the original Yahoo Pipes team, interrupted the proceedings from the back of the crowd. "We thought we'd experience a gradual ramp up over the course of a year. We were just a small team," he said of the developers who produced the popular mash-up service. With the February 2007 launch of the new service, their Pipes site was overwhelmed and couldn't come close to satisfying the download demand, he recounted.

"I kept running into people over the next year, who, when they heard I worked on Pipes, would say, 'Yeah, we went there, but it was down." The moral of his story: some people are going to be caught before they're ready to address the scalability issue, if they do their own infrastructure.

Yahoo has since announced it is building a private cloud for more elastic, internal operations and external services.