Review: Streaming Media Servers

To download or to stream? That's the question when it comes to distributing online media. Downloaded files guarantee users receive every frame and pixel of data, but they also quickly clog up storage media. Streaming media lets users store small bits of data on their hard disks and fast forward before the whole video downloads, saving bandwidth. And unlike progressive downloads, streamed files can be logged to show a user's bandwidth and viewing time.

Four vendors compete in the streaming server market: Apple Computer, Macromedia (now part of Adobe Systems), Microsoft and RealNetworks. A 2004 Frost & Sullivan survey had Apple, RealNetworks and Microsoft with a 99 percent combined market share (Microsoft, 38.2 percent; Apple, 36.8 percent; and RealNetworks, 24.9 percent). Separately, Macromedia claimed to have a 98 percent installed base before Adobe acquired the company in December 2005. We invited all four to submit their products for comparison, and all four accepted our challenge.

We tested only streaming video--not other features of each vendor's technology. Apple's QuickTime Player, for example, supports 3-D panoramic images, and RealNetworks' RealPlayer supports online music download stores. We didn't factor these nonstreaming capabilities into our evaluation.

When all was done, Apple's Darwin Streaming Server won our Editor's Choice award for its hassle-free operation and ease of use. However, if you're an all-Microsoft server shop and have a Windows 2003 Server licensed and available, Windows Media Server should work fine for you. And you can't beat the price (free) for Apple's and Microsoft's offerings. All four products are decent, separated by mere tenths of a point on our report card.

Apple squeaked into first place for its excellent stream stability, easy management user interface and price, but our testers ranked its video quality the lowest. Microsoft came in a close second. Macromedia offers excellent picture quality and a wide range of flexibility through Flash applets--as long as you're willing to do the Flash programming. RealNetworks' Helix Mobile Server has the best feature set but didn't score high on picture quality. Real can also stream QuickTime and Windows Media files in addition to those in its native format.

We focused our grading on four major areas--stream quality, management, features and price.

Stream quality, our most heavily weighted area, included image appearance and stability. The latter is particularly important, as frequent rebuffering and lost connections frustrate users and hinder streaming video's acceptance. Even a sharp picture looks terrible if the stream pauses to buffer every few seconds.

We didn't give extra credit for supporting Linux, Solaris or other Unix systems on the server or client side. Only Apple's product works on Mac OS X Server. Other platforms supported by Darwin aren't eligible for Apple technical support, but Apple hosts several discussion forums and mailing lists on other products. Apple had no objection to our testing Darwin under Windows. The server is available under the Apple Public Source License. Microsoft's streaming server works only on Windows 2003 Server. The products from Adobe and RealNetworks both support Linux servers. On the client side, all the products we tested support Windows and Mac. RealPlayer is available for Linux and Solaris. Portions of Real's Helix server are available under an open-source license. Flash 7 is available on Linux but Flash 8 isn't yet.

If there were an Oscar for most random images in a test video, we'd submit ours to the academy. Recorded on to a mini-DV camera, our high-motion movie featured fire, running water, detailed close-ups of local flora and fauna, a busy roadway, fast motion over the ground and text overlays, and a head shot of a miniature dachshund.

We digitally imported the raw footage into Apple's iMovie, edited the content and digitally exported it back to the DV camera. We used the exported content to encode all the videos, with separate Mac and Windows boxes for encoding the test videos. Per our request, each vendor specified codec and encoding settings. Darwin Streaming Server has a wide range of codecs and options, but Apple told us to use H.264 for all our tests. We encoded the video in three bit rates for each vendor: 1 Mbps, 768 Kbps and 256 Kbps. Each product had its own computer as a streaming server and a separate computer and monitor as the client. We used identical hardware for each product, and the monitors' contrast, brightness and refresh rates were the same. We used a Shunra Storm to simulate a typical LAN, as well as good and poor broadband connections between the client and server.

We created a quality survey to rate the video clarity, color, text rendering and overall playback smoothness. Nine Network Computing editors and contributors took the test. To make sure the test was fair, we set up the clients next to each other, so the participants could see all the media players at the same time. Each video was embedded into a blank Web page with a black background, and we hid the video controllers.

We ran all four players simultaneously and let the participants view any video as many times as they wanted. Apple's and RealNetworks' logos appeared for a few seconds during buffering, but we blocked the screen with a poster board during that period to keep the test blind. The first 14 seconds of the video was a stationary image, which gave us enough time to start up the streams, remove the poster board blockers and get out of the way. Participants graded the streams on a scale from 1 to 10 at each bit rate.

As we expected, the average score increased along with the video bit rate. Dropping from 1 Mbps to 768 Kbps didn't result in a big drop in quality, but going down to 256 Kbps did. The average score across all vendors at 1 Mbps was 7.4 out of 10 points. This dropped to 6.9 for the 768-Kbps test but plunged to 4.5 for 256 Kbps.

We were surprised by Darwin's poor quality showing at high bit rates. The frame rate was noticeably lower than the other competitors and produced a jerky video. This was less apparent in the 256-Kbps stream, and there was no jerky motion when viewing the video on the streaming server's console. Apple has released an update to QuickTime that touted improved H.264 playback, but it came out after our tests. Darwin's saving grace in our quality survey was its superior sharpness, vibrant color and excellent text rendering. Other vendors' players produced a slightly fuzzier, softly out-of-focus look, especially at lower bit rates. Our testers rated Flash's picture quality the best. The moving images were smooth and crisp, though small text didn't encode particularly well and was somewhat hard to read.

Our video streams occasionally dropped or stopped for 10 to 30 seconds during playback. There was no discernable or reproducible pattern to these performance glitches, and in some test runs, no videos dropped at all. Since the dropped streams were so sporadic, we instructed our testers to ignore them when answering the quality survey, and we considered stream stability as a separate factor in the overall quality grade. Darwin Streaming Server scored the best in stream stability; it never lost or rebuffered a stream during any of our test runs. All the other products did, even on a 10-Mbps network with no packet loss and 10-ms round-trip latency. This occurred rarely, however, at 10 Mbps, and showed up more predominately in the simulated broadband-connection test. Even at 2 percent packet loss, which is significant, Darwin was playable with minimal artifacts and no rebuffering. Although the servers offer some control over buffering time, the end user chiefly controls this setting. Flash is the exception, as the programmer dictates the applet's features.

We listened to audio quality with a 128-Kbps encoding from the audio track in our DV test video but found no discernable difference among vendors. Based on our experiences as media consumers, we'd recommend using at least 128-Kbps stereo audio in your streaming media, even if it means sacrificing some video quality or frame rate. Talking heads, with no music, can get away with 56-Kbps mono if necessary. Although all vendors support streaming audio only (Flash doesn't support MP3 streaming outside of the Web browser), the rise of podcasts has made this component obsolete for on-demand content.

Server and media management made up the next largest chunk of our grading. All the products' installation procedures are simple enough for an intern to handle, so we didn't make distinctions in the grading here. Rather, we focused on advertising insertion, playlist creation and management user interface. RealNetworks' Helix has the most complex server configuration. The interface isn't as straightforward as that of the other products, and the breadth of options made it a bit difficult to figure out what to do first. All the vendors offer limited Web administration. The Microsoft playlist wizard is available only in the console GUI, and Darwin Streaming Server's Web GUI doesn't allow for user and password administration.

We were sorely disappointed in all the products reporting features. Darwin Streaming Server offers historical data presentation through the user interface, but only under OS X Server. That server provides historical graphs up to the most recent seven days. Still, all the vendors supply extensive logging information in text-based, standard-format log files, if you're willing to write or find your own log-file interpreters.

All the video encoders have presets for high, medium and low bit rates but also let power users tweak the settings. Although most vendors let you specify multiple bit rates and batch encode, Apple required us to encode each bit rate individually. Mac users could whip out some AppleScripts, but the rest of us were annoyed. Microsoft and RealNetworks both allow multiple bit rate videos within the same file; Darwin Streaming Server and Flash require a separate file for each bit rate.

Apple's server offers the most options for delivering custom streams. We could send different streams depending on the player version, bandwidth, language or CPU speed. You could deliver a video for English speakers, for example, or a Spanish dubbed video, depending on the user's system preferences.

We tested custom content delivery. The other vendors are reliant on player version or bandwidth settings. RealNetworks is unique in that it can also deliver Windows Media or QuickTime content in addition to its own native format. None of the other products stream commercial competitors' formats.

However, we've also noticed that many organizations want to standardize on one format for simplicity. If you're set on offering multiple formats, RealNetworks offers the advantage of using one streaming server instead of three. If you want to offer multiformat delivery to users, paying for Helix Mobile Server may be better than setting up QuickTime and Windows Media servers.

Finally, Apple's and Microsoft's products are essentially free. We used QuickTime Pro for encoding the video, which costs just $30. Windows Media Server is included with Windows 2003 Server and isn't available as a standalone product. Some would argue this doesn't fit the definition of "no cost," which can be valid reasoning depending on your Microsoft volume license agreements. Flash and Helix both involve writing large checks, based on the number of simultaneous streams you want.

Streaming media offers benefits and frustrations. Users sometimes prefer downloaded data, as it allows for offline viewing and a higher-quality presentation. This comes at the expense of higher CPU requirements for the server, due to HTTP and TCP protocol overhead, potentially higher-bandwidth needs and more difficulty jumping ahead in long presentations. The rise of podcasts has overshadowed streaming audio, and video podcasts are just starting out.

Whether users will prefer downloadable video over streaming remains to be seen. Fortunately, streaming servers are simple to set up, reasonably inexpensive, and the data files can be offered for both mechanisms.

Mac shops can feel confident choosing Darwin Streaming Server, the no-cost, open-source version of QuickTime Streaming Server (QTSS). In fact, this choice is safe for almost any environment except for one dominated by low-end PCs. In our quality tests, QuickTime playback on Windows required more horsepower than the other products tested.

Apple earned its first-place score because of its excellent stream stability, strong but simple management and price (it's free without add-ons). Of all the products tested, only Darwin Streaming Server managed never to drop a session or rebuffer in midstream. We ran a test simulating a poor broadband connection where the network dropped 2 percent of all packets. Our stream played back with almost the same performance as our tests conducted with no packet loss.

Although Darwin's stability was unbeatable, our testers couldn't say the same for the image quality. Choppy playback, especially at the higher bit rates, gave QuickTime a low score on our quality survey. Unlike rebuffering, choppy playback does not pause for a few seconds but instead appears to have a low frame rate. At first we thought this was a deficiency in the streaming technology of QuickTime for Windows, as all the other products handled playback fine. So we tested the stream on a 1-GHz PowerBook, and it played at levels similar to the other participants. On a dual 3-GHz Xeon, the video played back with a decent frame rate. Apple recently released an update to QuickTime that touts improved H.264 playback, but it came out after our tests.

Apple's approach to server and media file management is simpler than that of any other vendor. The server has very few options to set. Playlist creation is elegant and intuitive, using HTML and JavaScript. Only Apple offers Web-based playlist creation. A list of our media files appeared in a left-hand column, and we just dragged the desired files into a right-hand column to create the playlist. We could also assign weights to make certain files play more often than others in random mode. Movies also can be played sequentially.

Streams can be user name/password protected, using a text file found in the media file's directory. We would have preferred integration with external directory systems, though. As with all the streaming servers, logging information is extensive but reporting is limited. Real-time information is available for throughput, total bytes and connections served, and stats on currently connected users. You can see connected users' IP addresses, bit rates, packet loss, time connected and what file they're watching.

Apple's solution doesn't offer a media-capturing component. Mac users can use iMovie, an application included with the Mac OS, to capture from Firewire cameras. It's also possible to convert any file QuickTime can read. We used iMovie on a Mac PowerBook to capture and save the DV stream from our camera. We then uploaded the DV stream from the PowerBook to the encoding PC and converted it to compare encoding times. Encoding required QuickTime Pro, a $30 add-on. Apple recommended the H.264 codec, which takes almost twice as long as RealNetworks' RealVideo 10 codec to encode. Several preset encoding options are available, but you can also configure your own. For streamed data, as opposed to progressive download, we added a "hints track" to the QuickTime file. The streaming server uses this information for better playback while streaming, but it does increase the file size. Our encoded 256-Kbps file went from 4.9 MBto 8.96 MB when adding server-optimized hints.

We used Apple's MakeRefMovie utility to create a reference movie and to link multiple encodings or versions of a movie, depending on the client's network and system settings. Unlike Real Networks' Helix Server or Microsoft's Windows Media Server 9, QuickTime's files are encoded at a specific bit rate. Every vendor offers player version and line speed, but QuickTime also let us transmit a different movie based on system language (such as a Spanish dub of a movie) or by CPU speed. The CPU speed feature is poorly documented but appears to be calculated based on QuickTime playback performance, not CPU gigahertz, based upon Apple's developer documentation.

Darwin Streaming Media Server 5.5.1. Apple Computer, (800) MY-APPLE, (408) 996-1010. www.apple.com

While Darwin Streaming Server is best for Mac and Unix environments, Microsoft's Windows Media Server 9 makes the most sense if you have a Win2003 Server. The software is free with Windows Server, is simple to deploy and provides good quality in its native environment.

To install Windows Media Server, we simply added it as a Windows component. A Web GUI is available in addition to the standard Windows MMC plug-in. The GUI is missing a few features, such as playlist creation, but most of the configuration options are there.

Microsoft also includes a free encoder with a wizard that walked us through the encoding process and helped us choose the appropriate quality levels. When capturing from our DV source, we could specify cut points on the timeline and perform two-pass encoding. Multiple-pass encoding took longer to perform, but produced higher-quality output. In multipass encoding, the video file is run through the compressor twice. Similar to text file compression, this lets the encoder work more effectively than it would with on-the-fly compression. RealNetworks' product could do two-pass encoding when working file to file but not during live capture. Apple and MacroMedia didn't have capturing products in our review, but the products we did test could perform multipass encoding. Windows Media Server let us store multiple bit rate encodings in the same file, whereas Darwin Streaming Server required separate files for each bit rate.

Windows Media Server had the second-best score on our quality survey. Color looked slightly washed out, and text was fuzzier than Flash or QuickTime, but the action was smooth at all bit rates. We found the stream to be less stable than the other competitors, frequently stopping during playback and not recovering. Oddly, dropped streams seemed to have happened less often in our 2 percent packet loss test than in the zero packet loss test.

We gave Windows Media Server high marks for its media and server management. A wizard made it easy to create a Web page with an embedded media player and to copy the HTML syntax to the clipboard. A playlist editor let us combine multiple media files, a directory of media, live encoders and other playlists into a stream. We used this tool to create interstitial ads and wrapper ads that appeared at the start and end of every stream.

Microsoft offers only IP access control, not user name/password, unless you want to use Windows Media DRM or tie user authentication into Active Directory. White- and blacklists of IP ranges may be set up. We successfully tested the access control mechanisms on a Network Computing subnet.

Real-time reporting is available through the standard Windows Performance Monitor application. We could look at connected players, bandwidth usage and error rates. Microsoft makes better use of graphics than Apple, but the data available from each is comparable. As with the other products, long-term reporting isn't available, but the log files contain extensive information, such as client type, playback statistics and access time.

Windows Media Server 9. Microsoft, (800) 642-7676. www.microsoft.com

Adobe's recently acquired Macromedia Flash Media Server is the newest player on the streaming server front. With its customizable, programmable media player for end users, this server may also be the most extensible product we tested. Now for the bad news. You can't just grab media and throw it on the Web with Macromedia Flash. But if you're looking to make advanced Web sites with tight integration between the content and streaming media, Flash is a good option.

Encoding video is simple. This server doesn't come with a video capture product; instead, you convert existing media files. We fed in a full-quality DV stream. There aren't many options in the encoder--the main choices are the video and audio bit rates and the codec choice. We used the On2 codec for our tests. Each bit rate encoding is saved in a separate file, though the encoder software does support batch encoding.

Video playback requires a Flash application, which must be programmed with Flash Professional authoring studio. This is the hardest part of setting up the server. With Macromedia's help, we created a basic video player with eight lines of code. There is also a video player library, which is built into the studio, but advanced functionality such as multi-bit-rate delivery requires additional programming.

Depending on the level of expertise and time you want to dedicate, this product can offer more options and features than the other products. If you want to require a user to enter a user name, password and phone number to access a video before playback, for example, that can be programmed into the Flash applet. However, unlike its competitors, the Flash server cannot stream MP3 playlists outside of the Web application.

Flash delivered the best video quality. Images were sharp, vibrant and smooth, but text rendering was a little fuzzy. We found Flash's stream was stable, with only occasional rebuffering. During our 2 percent packet loss test, the video paused three times during playback for 20 seconds, then 25, then 12, but we didn't lose any frames or notice any additional artifacts in the video.

The server is configured through a Web-embedded Flash application. You can centrally manage and get reports for multiple Flash media servers. Although this functionality isn't advanced, at least it's there. Reporting allows for a real-time graph of active connections, bandwidth utilization, CPU and memory use, but Flash doesn't include historical graphs.

Adobe Macromedia Flash Media Server 2. Adobe Systems, (800) 457-1774, (408) 536-6000. www.adobe.com

As with Macromedia Flash, Helix Mobile Server comes with a separate price for the Real media encoder and server license. The Helix server, however, is more versatile in one key respect: It can broadcast itself, QuickTime and Windows Media. We propped up the custom stream delivery score for this feature. If you're committed to delivering the same video in multiple video forms, this product is the way to go. This added to the stream delivery score, though Windows Media Server and Darwin are both free to install and use. For our tests, we looked at RealNetwork's native format.

The RealProducer Plus encoder, a $199 add-on, supports both live data capture and encoding of existing media files. Multiple bit rates may be encoded into the same file. RealNetworks includes a number of pre-defined connection speeds and suggested encoding settings, and you can easily modify or create your own. The product's live capture functions aren't as robust as those of Windows Encoder. To record DV content over the Firewire port, for instance, we had to start and stop the camera manually

The Helix Mobile Server's video quality was a bit disappointing, with lines that were less clearly defined than the competition's. Images appeared pale and slightly out of focus, especially at the 256-Kbps bit rate, and small text was hard to read. To its credit, the server produced a decent frame rate and smooth action. Although we didn't experience many dropped streams or rebuffering during our test, we saw poor results on our 2 percent packet loss test. Video frequently stopped, and we had to pause and restart the stream or rewind slightly.

The RealNetworks Helix Mobile Server is also a complex beast to administer, thanks to its wide range of configuration screens and settings. Playlists had to be created manually through a text file.

We were most pleased by the server's security features. The product supports authentication against a user name and password or by IP address. Two other pluses were that we could connect into external databases or NT domain authentication, and RealNetworks' support for multiple administrator accounts.

Helix Mobile Server, 11.0.0.1596. RealNetworks, (800) 444-8011, (206) 674-2700. www.realnetworks.com

Michael J. DeMaria is an associate technology editor based at Network Computing's Syracuse University's Real-World Labs®. Write to him at [email protected].

We used identical servers, clients and monitors for our test bed. The server was a dual 3.0-GHz Intel Xeon with Hyperthreading enabled, 3-GB of RAM and a 35-GB RAID hard disk. We ran Windows 2003 Server SP1 with IIS installed. Client machines were 1.2-GHz Pentium III systems with 512 MB of RAM and 18.6-GB hard disks running Windows XP Pro SP2.

Each client had an Intel 82815 graphics card built onto the motherboard. We connected a Samsung SyncMaster 171s, a 17-inch LCD monitor, running at 800x600 resolution, 24-bit color and 75 Hz. Clients and servers connected through a Shunra Storm STX-100 WAN emulator to a Hewlett-Packard ProCurve 2524. Our encoding machine was a 3.0-GHz Pentium 4-based computer with 1 GB of RAM and a 37.2-GB hard drive. An Osprey 300 EASE card was used for grabbing video from a Panasonic PV-DV952D camera. We also used a 1-GHz PowerBook and G4 X-serve for capturing DV footage. We encoded video on the Mac and PC, with no noticeable difference in quality.

We ran three quality test benchmarks. We encoded video at 1 Mbps, 768 Kbps and 256 Kbps, all at 30 frames per second.

In the first test, we used the Shunra Storm to simulate a 10-Mbps connection with 10-ms roundtrip latency. The second test was on a 3-Mbps link with 200-ms round-trip latency and 10-ms jitter. Our final test was on a 768-Kbps link with 200-ms round-trip latency and 50-ms jitter. We separately modified the third test to have 2 percent packet loss to further test stream stability.

Participants in the quality test were not told which system had which player on it. Systems were referenced as Alpha (Apple), Beta (Adobe), Gamma (Real) and Delta (Microsoft). Content was embedded into a black Web page maximized to fill the screen. The raw scores of all participants were averaged for each of the three tests. The scores for all three tests were then averaged and converted to our 0 to 5 scale.

All Network Computing product reviews are conducted by current or former IT professionals in our own Real-World Labs®, according to our own test criteria. Vendor involvement is limited to assistance in configuration and troubleshooting. Network Computing schedules reviews based solely on our editorial judgment of reader needs, and we conduct tests and publish results without vendor influence.

Three companies share the market for streaming media servers: Apple Computer, with various versions of QuickTime; Microsoft, with its Windows Media Server; and RealNetworks, with its Real platform. Adobe Systems has its own offering, too, now that it has acquired Macromedia Flash.

We invited each of these companies to send us the latest versions of their streaming servers, and all four agreed. Then we asked nine Network Computing editors and contributors to judge the quality of the videos they viewed as served by each product at three different bit rates. We also compared each server's management tools and general features.

The final tally was remarkably close, with Apple's Darwin Streaming Server earning our Editor's Choice award for its streaming stability and ease of use. Apple's product and Microsoft's Windows Media Server--a close second--have the added bonus of being free with their respective operating systems.

Curiously, Apple did not get the highest scores for picture quality; at high bit rates, the product's video images were jerky. Rather, our viewers rated Flash's video quality the best. Still, the combination of features we tested yielded the highest overall result for Apple, against tough competition.

R E V I E W

Streaming Media Servers