There's been a great deal of improvement in the quality and reliability of voice-over-IP calls since the early days when a computer equipped with a microphone and speakers was connected to another over a dial-up connection to make calls. The result: a barely audible sound stream filled with gaps, delays, and distortion. Now, VoIP calls are made from traditional phones over high-speed Internet connections using services from hundreds of providers, including some of the nation's largest phone companies.

Still, VoIP calls have a long way to go before they can match the quality and reliability of conventional phone calls made over the U.S. public-switched telephone network, which for more than a century has set the standard for voice calls.

Those are the conclusions reached by a first-of-its-kind study by Keynote Systems Inc., which analyzed the quality and reliability of more than 154,000 VoIP calls and 9,000 public-network calls placed between May 21 and June 25. Keynote tested six VoIP service providers and seven network carriers, including residential cable modem and DSL providers, as well as business-class T1 carriers.

"VoIP performance across most providers is nowhere close to the dial-tone reliability and clarity of communications that consumers have become used to over the years," says Dharmesh Thakker, Keynote's senior project manager. The company plans to release some of the research results this week.

Among the highlights: Keynote was able to complete VoIP calls 96.9% of the time, compared with 99.9% for calls made over the public network. Voice quality for VoIP calls on average was rated at 3.5 out of 5, compared with 3.9 for public-network calls and 3.6 for cellular phone calls. And the amount of delay the audio signals experienced was 295 milliseconds for VoIP calls, compared with 139 milliseconds for public-network calls. A delay of 300 milliseconds can produce poor voice quality, according to Keynote.

"Generally speaking, the quality isn't there," Thakker says. "For the amount of money consumers pay each month, they expect pretty high voice quality. On average, VoIP isn't a whole lot better than wireless."

That isn't slowing the move to VoIP. More than 1.1 million residential customers used VoIP services at the end of 2004, a number that should grow to 28.5 million by 2009, according to the Yankee Group research firm. That compares with more than 110 million regular phone lines.

Keynote tested VoIP and public-network calls between New York and San Francisco using services from AT&T, Lingo, Packet8, Skype, Verizon, and Vonage. It tested those services over various carrier networks, including cable-TV providers Comcast and Time Warner Cable, local phone companies SBC Communications and Verizon Communications, and long-distance companies AT&T, Sprint, and UUnet. It picked those companies because they're among the leading providers and carriers in each city.

VoIP research is a new market for Keynote, which has specialized in Web-performance testing since 1996. To conduct the VoIP tests, the company rented an apartment in each coastal city and subscribed to VoIP services just like any consumer. In each apartment, Keynote obtained high-speed Internet service from the local phone company and cable-TV company in each area, as well as from three long-distance networks. Keynote set up automated dialing, transmitting, and recording equipment to conduct the test.

The goal of the study was to assess the market readiness of service providers in those two cities by comparing the quality of their service to that provided by the public network, comparing the service providers to each other, and analyzing the effect the underlying network carrier had on service quality, Thakker says.