The Search For Microsoft Researcher Jim Gray

It's not unheard of for recreational sailors to go missing off the California coast. The U.S. Coast Guard searches for them, executing precisely aligned grid patterns around their last known location. After that, if they're not found, they're considered gone.

Computer scientist Jim Gray disappeared Jan. 28 after sailing out of San Francisco Bay to scatter his mother's ashes at the Farallon Islands, 27 miles offshore. An extended, four-day search by the U.S. Coast Guard by air and sea turned up nothing, and that might have been that. But the search for the 63-year-old Gray--a distinguished engineer with Microsoft Research, database expert, and Turing Award winner for his work in transaction processing--didn't end there.

A worldwide network of friends, associates, and fellow researchers launched an all-out search of their own. For 16 days in February, they undertook one of the most wide-reaching volunteer search efforts ever organized. They used the latest technologies--and even developed their own --to scour the ocean surface, collaborate, and crunch data, while also resorting to the gumshoe work of posting missing-person fliers at marinas up and down the California coast.

There were promising moments, times when a speck on a satellite image caused the volunteers to strain their eyes for a closer look--sometimes jumping in planes to follow leads. In the end, however, the volunteers couldn't find Gray either, and on Feb. 16, the ad hoc search was disbanded. Those closest to Gray may still hold out hope that he's somehow alive. His personal Web page at www.microsoft.com/research appears just as it did on the day he set sail. Others, however, are almost sure he is dead.

To those who knew him, it defies reason that this hardy, experienced sailor vanished in the Pacific. "He was very comfortable sailing in the ocean off San Francisco Bay," says Michael Stonebraker, a friend for 35 years, former professor of computer science at the University of California at Berkeley, and founder of Ingres, Illustra, and Vertica Systems.

Paula Hawthorn, former VP of research and development at Informix (now part of IBM), first raised the question of manipulating satellite data to search for Gray's 40-foot sailboat, Tenacious. The idea was to apply large database techniques Gray himself pioneered in the effort to find him.

Gray was already known for "dragging astronomy into the 20th century" by showing astrophysicists how they could capture and share terabytes of data on the heavens, says James Bellingham, chief technologist at the Monterey Bay Aquarium Research Institute. More recently, Gray had developed an interest in oceanography, and he and Bellingham were planning to build a database system that could be used to model an ocean's currents and surface. In the face of possible climate change, "Jim was saying, 'We've got to solve this problem. We've got to get good information out of the ocean.'"

FLURRY OF HOPE

Many of Gray's contemporaries regularly work with large amounts of digital imagery, and they seized the opportunity to put that expertise to work in search of their peer. Gray, who worked for IBM, Digital Equipment, and Tandem Computers before Microsoft, never lost touch with his friends at Berkeley. When it was clear the Coast Guard search was ending, Joe Hellerstein, a computer science professor at Berkeley, established a Web site (openphi.net/tenacious) where searchers could collaborate. The group was told by the Coast Guard, "If we could turn up any evidence that he was out there to be saved, they would jump right back into the search," Hellerstein says.

Oracle VP Mike Olson met Gray as a Berkeley grad student and worked with Gray's wife, Donna, at Illustra. An experienced organizer of volunteers from his work with open source software, Olson began talking to Bellingham, Hellerstein, and other technologists about their expertise and how it might be used.

The effort extended well beyond the Bay Area. In Seattle, Werner Vogel, CTO of Amazon.com and a former student of Gray's, would play a key role. Dozens of Gray's co-workers at Microsoft's Redmond, Wash., headquarters were freed up to help. Associates at Carnegie Mellon University in Pittsburgh joined in, as did Istvan Szapudi, a physicist at Eotvos Lorand University in Budapest, with whom Gray had worked.

This story was modified April 2 to correct the date Gray set sail. A MOVING TARGET

There were many obstacles. Real-time satellite data isn't available without special government clearance, so the searchers were forced to use ocean images that were delayed by 24 hours. By the time they could work with it, the data would be two days old. Winds, tides, and currents would have moved Gray's boat even if they spotted it. "I decided I'd organize a group that would predict where an object would drift in that period," Bellingham says.

Antonio Baptista at Oregon Health and Science University and Yi Chao at NASA's Jet Propulsion Lab, both experienced ocean surface modelers, tried to predict where a boat located somewhere between the coast and the Farallons might have drifted over several days. Working with different data sets, they came to a similar conclusion: If the boat had come to harm between the coast and islands, it would have been carried back toward shore somewhere north of San Francisco near Drake's Bay.

The models also showed that if Gray had for some reason sailed past the Farallons out to sea, Tenacious would still be heading west--winds were offshore at the time, contrary to their usual pattern. Bellingham climbed aboard a Beechcraft Bonanza, a single-engine plane, and served as spotter as civilian pilot Scott Wedge flew up and down the coast. They found nothing. If Tenacious had washed up along the shore, Bellingham says, "at 1,300 feet, we would have seen it."

On Feb. 1, the first data available gave the group a look at patches of ocean south of San Francisco. It came from a NASA pilot flying a civilian version of the U2 spy plane, the ER2, which took off from Moffett Field and flew up and down the coast, its camera eye periodically collecting data. David Tennenhouse, a former manager of Amazon's A9 product search site, funneled the ER2 data into Amazon's storage service, where it was divvied into pieces by Mechanical Turk, experimental software that lets many people work on a task simultaneously.

Images that were 8,000-by-8,000 pixels in size were parsed into 400-by-400 "tiles" that were enlarged so volunteers could examine them for objects. Mechanical Turk automatically handled the distribution over the Web. Each tile was viewed and rated by three volunteers, and only those that got a high rating from all three would move to a second stage of review.

Even so, it was a mind-boggling task. There were 560,000 tiles, requiring 1.68 million viewings. A single tile consisted of 160,000 pixels of monochromatic gray; Gray's 40-foot, red-hulled boat would appear as a mere 6 to 8 pixels with a slightly darker gray tone. "There were more boats in the water than just Jim's," Vogel says. "It was hard to determine with any certainty which might have been Tenacious."

Nevertheless, word of the effort spread, and the task was completed in three days by 12,000 volunteers. Tennenhouse, in charge of coordinating the image data search, groaned at the prospect of thousands of people viewing the ER2 flight data via Mechanical Turk. But the volunteers had examples of what to look for, and some became skilled at distinguishing between a discoloration representing a wave pattern and one representing an object in the water. When a boat was detected, some tried to determine whether a wake could be seen behind it. (A wake indicated a boat under way as intended; they were looking for one adrift.)

At first, the volunteers wanted to identify every object down to the size of a dinghy, in case Gray had to take to the sea after Tenacious sank. But the resolution of the flight data wasn't good enough to pick out a dinghy without thousands of false hits. Tennenhouse insisted that the focus stay on a 40-foot boat.

At that point, Gray had been missing for nearly 10 days. If he was in a dinghy, his survival prospects were poor to nil, given the cold ocean temperatures and lack of fresh water. If he was still on the sailboat, his chances were much better. "We had to throw away some targets. A dinghy couldn't really be distinguished from noise," Tennenhouse says. "My team had to focus on positive outcomes."

As Mechanical Turk turned up targets, called "possibles," Gray's Microsoft Research colleague Tom Barclay was arranging flights by civilian pilots to go check them out. Sending pilots with a spotter 70 miles offshore was itself a risk, so decisions on what to pursue weren't made lightly. As 15 to 20 possibles accumulated, Navy image analysis experts helped determine what they might be. Locations were plotted using the drift trajectories developed by Bellingham's group.

"The Pacific Ocean is a very big place," says Barclay, who worked with Gray in the development of TerraServer, a database of satellite earth images. "Obvious, I know, but it is really hammered home when you try to 'mow it' with low-flying, fairly slow aircraft."

FALSE LEADS

When Alex Szalay, an astronomy professor at Johns Hopkins University, learned of Gray's disappearance, he "couldn't believe what had happened." The two had been collaborating on astronomy databases for 10 years. Upon hearing that a volunteer search was in the works, he stayed up all night in touch with his son at Cal Tech in Pasadena. Together, they wrote a Photoshop script that "stretched" satellite image data of the sort that would soon be available so that the human eye could discern a 40-foot boat. They then put 16 postdoctoral students and grad students with image-viewing experience to work examining satellite data available from DigitalGlobal, a com- pany that Gray had worked with in his own research. DigitalGlobal had reoriented satellite cameras to pick up images for the area of the Pacific that Szalay wanted to search.

As with the Amazon effort, Szalay developed software that sliced up im- ages that were 8,192 pixels on each side into smaller tiles. Unlike Mechanical Turk's reliance on slow-loading Web pages, reviewers at Johns Hopkins could move through these tiles quickly, using the university's broadband network.

They came up with two strong possibles, one found by Julian Bunn, a particle physicist at Cal Tech. He detected a pattern of a boat that no one else could see under a thick cloud cover, and it was verified through image-enhancement techniques. "That was a brilliant piece of work," recalls Szalay. But the sighting was eventually ruled out.

Another boat, spotted by the team about 70 miles offshore from California's Point Reyes, was confirmed by expert analysis as a strong target. Radar data from the Canadian Space Agency was cross-referenced and produced evidence of a boat. Its location corresponded with the drift trajectory analysis of where a boat would be had it veered off course from the Farallons. Barclay chartered a flight, and Tennenhouse climbed aboard. The boat was located, but it was yet another disappointment. It turned out to be a fishing vessel the same size as Tenacious.

Looking back, Tennenhouse gives the volunteer search effort a B for solid performance, but he regrets it was too slow in getting under way and too uncertain of its own results. No evidence of what happened to Gray turned up in the immense data search. "We had some hope of finding him, but it turned out to be like looking for a needle in a haystack," he says.

Despite the disappointing result, there is hope that the techniques applied in the search for the admired researcher--the use of satellite images, ocean-current modeling, Mechanical Turk, Web collaboration--can be used in future searches of vast open spaces. As someone who devoted his life's work to using technology in breakthrough ways, Gray would surely smile at that legacy.

The exhaustive effort that followed Gray's disappearance is a testament to how highly he is regarded by his peers in business and science. He brought new ideas to their work and sometimes assistance in the form of computing power. "All his projects had in common the goal of stretching what computing can do beyond its present capabilities," says John White, CEO of the Association for Computing Machinery.

Gray was expert at "stripping away mystery by making things simple," says Eric Allman, CTO of Sendmail and the author of the Sendmail messaging system. "It's an irony to me that he should end in a mystery."