Microsoft Describes How Virtual Earth Was Built

You want to know what the definition of "cool" is? It's sitting in the front row of a hotel meeting hall, watching a demo of Microsoft Virtual Earth on the 12-foot display in the front of the room, as the camera plunges from the sky to swoop and soar around detailed digital models of the Staples Center in Los Angeles and the streets of Philadelphia. Even cooler: Listening to John Curlander, general manager of Microsoft Virtual Earth, explain how it was built.

Virtual Earth started with a vision articulated by Bill Gates on his 50th birthday in 2005, Curlander said. Curlander quoted Gates: "You'll be walking around in downtown London and be able to see the shops, the stores, see what the traffic is like. Walk in a shop and navigate the merchandise. Not in the flat, 2-D interface that we have on the Web today, but in a virtual reality walk-through."

Curlander gave his presentation at the O'Reilly ETech Emerging Technology Conference in San Diego.

The Microsoft Virtual Earth team is working on building the database of images and data that will enable technologists to realize that vision. Virtual Earth APIs and data will be made available to game and application developers to add the third dimension to Web applications. The result will be, not a single world like Second Life, but many thousands of worlds, with common navigation.

To implement the project, Microsoft turned to, and later acquired, Vexcel, an imagery and remote sensing company. Vexcel built a prototype based on a single city block, hiring someone to walk around taking photos with a consumer digital camera, and building a three-dimensional model of the street based on the resulting image. The work was code-named Project Pyramid, because it would take as long to do an entire city as it took to build the pyramids, Curlander said.

Microsoft wanted to digitize and render in three dimensions 3,000 cities in less than five years. To do that, Vexcel needed to speed up and automate the process. The team built its own digital camera, the UltraCAM, which it used to take aerial photographs from hired planes flying at 1,500 feet above the subjects. The resulting images have 15 centimeter, or 6-inch resolution, fine enough to see individual people walking and casting shadows.

The sensors on the camera suck in data at greater than 400 Mbytes per second, 10 times higher than the throughput of commercial imaging satellites. The camera can capture a whole city in a few hours of flying.

The camera captures images in a broad spectrum, including full-color infrared, which allows software to distinguish what's being photographed. Different terrain types have different signature wavelengths: Solids, vegetation, water, roofs, soil, and swimming pools.

The result is what Curlander calls a "2-1/2 D" model. For every X and Y value, there's only one Z value. That means that the resulting model doesn't support layers: Tree branches, or, in buildings, inset windows, porticos, ledges, terraces, and other things that break the vertical line of an exterior wall. That results in an image that appears unnatural and (in the case of trees) even grotesque when viewed from the side. VirtuaL Earth uses software to correct the problem.

The software initially strips the buildings and trees from the images to get the shape of the land, and then adds back model trees. The team manually extracts the vertices of a roof, and uses the result to build the geometry of the building, and then adds photo textures with images from the UltraCAM.

Flying around the streets of a virtual city, one element is grossly unrealistic: Cars. Virtual Earth makes no effort to render the cars in 3-D. They are completely two-dimensional, looking like life-size photos of cars laying on the ground. "We extract the topography from the cars, smooth it out, and smash it to the ground," Curlander said.

Traffic reporting is one application for Virtual Earth. Curlander demonstrated an aerial view of Los Angeles, with highways color-coded based on traffic speed, with red highways indicating where traffic was moving at 25 mph or less. The map was real time, and there was a fair amount of red. "L.A. is always good for traffic because it's always jammed up," he said.

Microsoft's goal is to render 100 cities in 3-D by June 30, the end of the company's fiscal year. It is on its way to achieving that goal, although it may fall a few cities short. The decision on which areas to image comes from Microsoft headquarters, and is based on marketing and business considerations. Another 20 cities are due to come online within a couple of weeks, including New York.

The team is working on automating building rendering, which is now done manually. When that's done, it will be able to render 50,000 buildings in a city, instead of 5,000, which will make a city far more detailed and realistic.

The team is collecting streetside data, mounting the UltraCAM cameras externally on vans and driving them around cities and taking pictures. Curlander displayed an image of a vehicle, a large, white wagon or mini-SUV, festooned with external cameras, road dirt, and a Microsoft logo. It looked like a vehicle out of the old Max Headroom TV show. By taking drive-by pictures, Microsoft will be able to improve the details on the sides of buildings.

Curlander closed his presentation by showing a video of a fly-through of virtual New York, with a soundtrack of Frank Sinatra singing "New York, New York."

During the Q&A, one person asked about privacy issues. Curlander responded that the resolution on the aerial photos isn't good enough to make out individual faces. Microsoft has face-blurring software, and may use that when the renderings based on drive-by photography is made public. Microsoft currently blurs images of some buildings, for security reasons.

The Virtual Earth team is working with other groups at Microsoft to integrate into other products, including Flight Simulator and instant messaging, Curlander said.