For The NewsMarket, A Web site that hosts hundreds of hours of video footage, most of it PR-related background video known as B-roll that's used by 10,000 TV news outlets in 144 countries, search is among its most important capabilities. Journalists need to be able to find relevant video snippets for their stories quickly and easily.

The NewsMarket started out seven years ago trying to develop its own search engine but soon found video search was no easy task. Unlike text search, video search can't rely on the simple parsing of characters and words or on link analysis. Instead, it needs a way to characterize the video content itself, and today that's typically accomplished by using word-based descriptions or some sort of audio-to-text translation. Compared with Google, it's an immature art.

As video explodes on the Web and grows in importance in business, so will the need for sophisticated video search. Many companies want to be able to locate appropriate video, or reuse video they already have, for communicating and training, says Stopher Eagen, U.S. CEO of knowledge management and search software company Autonomy. And they want to make sure the appropriate people can find the video they need when they need it.

VIDEO TAG--YOU'RE IT

The NewsMarket, which now uses Autonomy's video search application, relies mostly on meta tagging its videos, attaching identifying details and, in some cases, even descriptions of the various scenes in the video file itself. The NewsMarket tags content by general subject--or beat--area, as well as by the source and specific topics, like "auto show" or "Microsoft Windows."

(click image for larger view) Video search has to work at The NewsMarket

Tagging is the typical way of searching video. But Autonomy's and other vendors' search engines do more than just look through the tags. "Tagging sounds great, but you're not really finding the video, you're finding a piece of data about the video," says Suranga Chandratillake, CEO of Blinkx, a consumer video search engine.

Embedded audio is a powerful tool for video search engines, whether through transcription or phoneme search, which searches the actual sounds rather than relying on dictionaries. One common method is automatic audio-to-text transcription. However, transcription can be sloppy, often resulting in meaningless or inaccurate words. "We've tried some tools that were voice to text, and they just didn't work very well," says Rogulja Wolf, streaming content manager at Sandia National Laboratories. Sandia has used BBN and several other companies' products to search its video, which includes everything from meetings to highly technical process reviews. "We spent more time editing than what it was worth," Wolf says. Sandia is now testing Sonic Foundry's phoneme search tool and finds it works better.

Both Autonomy and Sonic Foundry, as well as other vendors such as Nexidia, rely more on phoneme-based search rather than transcription. With phoneme search, users type the subject into the search engine, and the software matches what was typed with a list of phonetically spelled words drawn from the audio track of the video being searched. Phonemes aren't matched to text until someone actually searches for them, and even then words don't need to be spelled correctly because the engine is searching for sounds rather than words.

Autonomy uses both phoneme search and transcription. The two act as multipliers, since Autonomy uses a probabilistic model that builds on what it's heard before to clean up phonemes and account for accents in the transcription process. Eagen says Autonomy also can do speaker recognition. For example, if a company has a lot of videos from its CEO, the engine can be taught to recognize any video in which the CEO is speaking.

Since Sonic Foundry was built for searching Webcasts, it reads text in PowerPoint presentations that run in a segregated window alongside the video. "You're improving the accuracy of these results through these multimodal processes," says Rimas Buinevicius, Sonic Foundry's chairman and CEO. Other applications, Autonomy included, can actually read text and subtitles embedded into the video itself.