InformationWeek

Loudcloud Inc., an Internet infrastructure service provider founded by former Netscape executive Marc Andreessen, offers a range of services designed to speed Web downloads such as highly available storage that can boost access to a disk. Loudcloud also has a load-testing service that can mimic increasing levels of network stress; it's called Stresscloud but is known internally as the Death Star. "We point the Death Star at a mimic site and then we turn up the knob slowly," says Scott Dunlap, Loudcloud's VP of product management. "We pay attention to the entire infrastructure to see where it's breaking." Such a service is invaluable early in the development cycle, Dunlap says. "How the site performs at low levels rarely indicates how it will perform at higher levels."

Loudcloud designs Web sites that have redundant equipment to eliminate any single point of failure; it has a lot of equipment on hot standby, so there's little or no disruption of service when servers or devices go down. A typical site has redundant networking gear such as firewalls, load balancers, and routers; redundant Web servers, database servers, and clustered application servers; and redundant Internet connections, according to Dunlap.

To address spikes in traffic along the network, LoudCloud has a system called Opsware that determines where capacity is required and automatically adds capacity where it's needed. "We have idle machines ready at all times--Opsware can configure them in minutes, and we can quickly address the spike," says Dunlap. "The trick is to find out how to most efficiently address a spike in traffic. Throwing hardware at the problem only works if you know exactly what to throw, and where to throw it."

One way to understand a site's problems and how to best resolve them is via testing, and more companies are hiring third-party service companies to probe for weak links. Such a move can pay big dividends, says Michael Budowski, CEO of Ourbeginning.com Inc., an online stationery resource center in St. Petersburg, Fla. Budowski decided to make a splash this year and take a big gamble: He spent about $5 million to run several commercials during this year's Super Bowl telecast. He knew that such a move could attract enough attention to overwhelm his site, so Budowski and his team completely reorganized and rearchitected the Ourbeginning.com site to handle the traffic. He started by replacing his servers. "Getting ready for the Super Bowl, performance was everything," says Budowski. "We upgraded to Sun boxes because of the performance. We had used Windows NT boxes before and they performed well, but we realized the end user wants the optimum performance."

Ourbeginning.com now has seven Sun Quad Processor Servers running Solaris and three Windows NT Dual Processor Servers at dual locations in Virginia and Orlando, Fla., running Lotus Domino Release 5 and NetObjects 5.0. The company uses 10-Mbps burstable connections in Virginia and 3-Mbps burstable connections in Orlando, Fla. The rated speed for the connections is the minimum amount of bandwidth allocated to the servers; the burstable aspect provides additional bandwidth if the site has a very active day or an unusual peak period. The total bandwidth available at the facility is more than 115 Mbps, with many redundant paths into a Verio hosting facility. While 10 Mbps isn't fast for a LAN connection, it's extremely fast for a wide area network link, Budowski says.

The site had 366,000 unique visitors per month, at an average user session length of almost 16 minutes. That translates to 189,000 hits per day. The servers can handle up to 4,000 connections per second, compared to 200 per second before upgrading for the Super Bowl. The site also uses Resonate's Central Dispatch and Global Dispatch for load balancing and to handle failovers, according to Budowski. To handle such numbers, Budowski impressed upon his development team the importance of designing their network with load balancing in mind. "We had less than two weeks to have the operating system installed, the whole thing configured, the backbone installed, the load balancing figured out, as well as completely redoing the site," he says. "You have to make sure that's all optimized to work together. It was a very tense time."

As a final precaution, Ourbeginning.com turned its site over for stress tests to Mercury Interactive Corp., a testing service that can simulate a large group of users. The night before the Super Bowl, Mercury Interactive found a defective connection that would have crashed the company's Web site had it been visited by large numbers of users. "That was 11:30 on Saturday night," says Budowski. "The problem was a matter of throughput on the LAN side of the setup. As we tested the site and pushed the connection count higher, we started to see network degradation. Once the problem was identified, the network latency subsided and the throughput was up to the level of expectation."

When Send.com's Web site slows down and the problem doesn't seem to be in the site or the network, the company turns to an Internet diagnostic service from Keynote Systems, says Ken Surdan, VP of technology for the Waltham, Mass., E-retailer of high-quality gifts. The service continually accesses client Web sites from points worldwide and keeps a tally on the time it takes to connect, as well as how long it takes to download pages.

"You're going to get performance based on a number of variables," says Surdan. "So how do we identify a problem that's within our ability to change? At the very least, we can make sure our customer-service organization is prepared to handle a problem somewhere in the country."

For some companies, a fast Web site is the key to success, and handing off control to someone else isn't an option. For those companies, it's essential to build a site strong enough to handle whatever spikes come across a network. That was the challenge for online brokerage CharlesSchwab.com, which had problems last year when its site slowed down as hundreds of thousands of customers tried to buy and sell stocks during significant fluctuations on the stock exchanges.

The company spent the past year upgrading its Web site. On April 4, when the Nasdaq moved 500 points in one day, Schwab's Web site handled the record traffic with only minor problems, says Fred Matteson, executive VP for technology services at the San Francisco company. On that day, Schwab's site handled 95,000 simultaneous logons and more than 600,000 orders. "We looked at our peak transactions from the previous month, and then we built a capacity reserve of 2-1/2 to three times that," says Matteson. "Our engineering design goal is to delight our most demanding customer at the busiest time of the busiest day."

Keeping those customers happy took considerable effort. Schwab created the 12th-largest supercomputing network in the world, Matteson says, by linking approximately 1,200 IBM SP2 and Sun servers running AIX and Solaris to operate its site. The firm also uses 10 IBM and Hitachi mainframes running MVS.

However, adding hardware is only the first step in developing a blazingly fast Web site, says Matteson. Web-site administrators and designers must plan for growth on their site from the beginning. "It's clear that it's not one thing that makes a Web site fast; it's a really complex set of services," Matteson says. "You can make one side of it blindingly fast and overwhelm the other aspect of it. You have to engineer it from an end-to-end basis for what the traffic is going to be. That's everything from the pure capacity of the links to the hardware and software capabilities."

Optimal Web-site performance is a moving target that depends a lot on what you put into your site--and what you want from it. But as the numbers keep dropping, it's clear that anyone who stands on the sidelines will get passed.

return to page 1, 2