Voice Over IP's Quality Surprise

A robot dialer made the calls, trying every combination of VoIP service provider and carrier network. Every 30 minutes, the dialer placed 35 calls, trying each service provider over each carrier network infrastructure. There were seven service providers because Vonage offers a hardware phone and a software-based phone and Keynote tested both. There were five carriers connected to each apartment, one cable company, one local phone company, and three long-distance companies. Once a call was placed, Keynote's equipment played a 30-second audio recording that contained 5 or 6 bits of talking with pauses in between to represent a typical phone conversation. Keynote recorded the results and compared them with a reference recording.

Keynote analyzed 160 elements for each call, including time to set up a call, network latency, jitter, and packet loss. It also measured the percentage of calls that went through, the average number of dial attempts, the number of dropped calls, and other reliability factors.

Rankings were established in two overall categories: reliability and audio clarity. Reliability was based on service availability, average number of call attempts, and dropped calls. Audio clarity was based on sound quality and audio delay. Keynote only released the top performers in each category; it wouldn't disclose the scores or ranking of the other companies.

Vonage was rated the most reliable VoIP provider overall, although the company could improve its dropped-call perform- ance factor, Keynote says. The two lowest-rated (unnamed) service providers trailed their competition by a substantial margin, showing a large difference between high-ranking and low-ranking providers in reliability.

When it comes to service availability, or the number of call attempts that get connected, the Vonage software-based phone was available 99.4% of the time. That compares with an industry average of 96.9%. The lowest-rated VoIP provider was available only 94.8% of the time.

AT&T CallVantage had the best audio clarity of VoIP service providers, although Keynote says the company could reduce the geographic variability of audio delay with its service. There's a noticeable gap between the audio clarity of AT&T CallVantage and the competition, Keynote says.

Time Warner Cable was rated the most reliable network carrier, although it could improve its dropped-call performance factor, according to Keynote. Time Warner's score was substantially higher than its competitors.

UUnet and Time Warner Cable tied for best audio clarity among the network carriers. All the network carriers tested provided consistent audio clarity, and the Mean Opinion Score--an International Telecommunications Union standard used to measure sound quality--didn't vary by more than 10% among network carriers.

Keynote isn't the only company testing VoIP services. E-business consulting firm Productivity Solutions conducted tests earlier this year after a client asked the firm to recommend a VoIP provider. Productivity Solutions tested 24 VoIP services, placing hundreds of calls and asking employees and recipients to assess the sound quality.

"We were surprised. We didn't expect to get reports that VoIP was as good or close to as good" as regular phone service, says David Moskowitz, CIO and chief technical officer at Productivity Solutions. "There were two standouts in terms of audio quality: AT&T and Vonage. We liked AT&T CallVantage so much that we decided to keep it and have been using it exclusively for the past three months" for the firm's 100 employees.

As for the client, Productivity Solutions recommended that it go with one of the top five VoIP providers. Says Moskowitz, "With more than 300 VoIP service providers out there, there's going to be a major shakeout at some point."