Inside Second Life's Data Centers

Even while Second Life struggles with its explosive growth and new popularity, the developer, Linden Lab, is redesigning the service to handle hundreds of times as much traffic as it's dealing with today.

Second Life is seeing 20% monthly growth in the number of concurrent users, going from a then-record 18,000 users logged in at once in December to a record 36,000 on Sunday. The maximum capacity of the current network is 100,000 simultaneous users, says Joe Miller, VP of platform and technology development for Linden Lab.

Rapid growth has led to problems for Second Life users, who often complain about slow response times during peak periods. Users also see problems with screen graphics, moving around between regions of the virtual world, and discrepancies in transactions involving the in-world currency, called "Linden Dollars." Search results are becoming out of date quickly.

To help deal with current and future growth, Linden Lab is looking to redesign the network. Linden Lab is looking to increase capacity from 100,000 simultaneous sessions to tens of millions of simultaneous logins. Linden Lab also needs to manage that upgrade without interrupting service to existing users.

"We're swapping engines out at 40,000 feet while still flying," Miller says.

Second Life runs on 2,000 Intel and Advanced Micro Devices servers in two co-location facilities in San Francisco and Dallas. The company has a commitment to open source, with servers running Debian Linux and the MySQL database. Linden Lab chose Debian Linux because the software is suited to scaling massively with a small IT staff, say Linden Lab CTO Cory Ondrejka. MySQL allows the server farms to scale horizontally, by adding large numbers of low-power servers as needed, rather than vertically, which would have required Second Life to run on a few, powerful systems, Miller says.

The architecture of Second Life is based on the illusion of a virtual world, built out of virtual land. Each geographic area in Second Life runs on a single instantiation of server software, called a simulator or "sim." And each sim runs on a separate processor on a server. So, when an avatar walks or flies or swims or drives from one sim to an adjacent sim, it's actually moving from one processor or server to another.

Users in Second Life -- who are known as "residents" -- buy virtual land to house their buildings and virtual furniture, vehicles, and other stuff. In real-world terms, what the residents are doing is leasing software capacity on a server. Nearly everything in Second Life, all the buildings, clothes, vehicles, furniture, avatars, and other objects, was built by residents and operates on resident-owned land.

When residents buy a whole island, they get dedicated use of an entire Intel- or AMD-based server, with pricing based on the processor, memory, and storage of that server. Landowners aren't locked down to a particular server, though -- Linden Lab moves sims around for system maintenance or if there's a server crash. But landowners' fees guarantee them to access to particular classes of server, with particular processors, memory, and storage allocation.

For now, Linden Lab is the only company that runs sims. It derives the majority of its revenue from land sales. However, that will change, as the company is committed to allowing others to run their own sims on their own servers.

While many Second Life residents are convinced that Linden Lab will release the server source code as open source, in fact that hasn't been decided, says Miller. Linden Lab might simply decide to license the code to other companies, the way vendors now license any application. Linden Lab might publish all the APIs, and allow other companies to build clones of the Second Life sim software. Linden Lab might simply allow other companies to provide the hardware, which Linden Lab would run at its own co-location facility, running Linden Lab's software. And there are other options as well, which Miller declined to elaborate on.

Linden Lab opened the source code to its client software, which runs on Windows, Mac OS X, and Linux, in January, and the company is using that project as an opportunity to learn about governance of open source projects, says Ondrejka.

The servers support 34 terabytes of user-created content, and the traffic load for accessing that content is very different than a conventional Web site. "If you think of a Web site like Digg, there are relatively few creators of information, and lots and lots of consumers of information. In Second Life, the equation is different, in that two-thirds of the users are actively creating content over time."

To help manage growth, Linden Lab is looking for IT staff who have done it before. "We're looking for an operations-specific person who was involved in scaling a server farm from where we are today, which is 2,000 boxes, to 10,000 boxes. There are very few people in the industry who have done that," Miller said.

Linden Lab went live with its Dallas co-location facility in December. "We're consuming a lot of floor space. It makes sense to have it geographically distributed. It creates redundancy and backup. If we lose bandwidth at one facility, we can shift load and resources to another location," Miller says.

Since then, residents have expressed concerns that the split locations have contributed to performance slowdowns -- which the residents call "lagginess."

To maintain the illusion that Second Life is a virtual world, avatars and their associated attachments and processes need to be able to transition from one sim to another seamlessly. Avatars near the edge of one sim need to be able to look over and see activity in the next sim, just like people in the real world can see activity going on in the next lot of land. All of this needs to happen even if the sims are located on different servers -- even if the handoff between sims must bridge 1,700 miles between Dallas and San Francisco.

However, Linden Lab has designed the network so that the handoff between co-location facilities doesn't result in increased lag, Miller said. One method is to cluster the sims so that regions that are in proximity to each other in Second Life are running on servers in the same real-world location.

"If a Texas-based simulator needs data from a database in San Francisco, it gets it, and it doesn't block action on the part of the user," Miller says.

To help make the user experience more pleasant, Linden Lab is looking to improve performance of the scripting language used in Second Life. The language is called LSL, for Linden Scripting Language. The language is similar to C.

Objects in Second Life are built using a combination of primitive shapes, or "prims," which are sized, scaled, and stretched as needed, with images, called "textures," applied to the surface of the prim to alter its appearance. For example, on the exterior wall of a virtual skyscraper, textures would be use to provided the appearance of bricks, windows, and other features.

LSL determines how the objects behave, controlling the waving of virtual hair, driving a virtual boat or automobile, or behavior of a virtual gun. Everything in Second Life is composed of some combination of prims, textures, and LSL scripts.

In an August video, Ondrejka described how he wrote LSL. "It got written in a week, like all scripting languages," Ondrejka said. He decided he needed to roll his own scripting language for Second Life after he determined that he knew more about writing a scripting language than he did about modifying another language, like Java, to meet Second Life's requirements.

Performance is slow, at only 150 million instructions per second, with inconsistent syntax and limited support outside of Second Life. "Nobody's using LSL to do real-world things," he said.

To improve performance, Linden Lab is porting LSL to Mono, the open source implementation of Microsoft.Net. In tests, the Mono implementation of LSL runs 1,000 times faster than Linden Lab's current version.

Linden Lab plans to begin testing the Mono implementation in the second quarter. Later, Second Life's transition to Mono will enable the software to support scripting in other Mono-compliant languages, such as Visual Basic and C#.

As a stopgap measure to manage growth, Linden Lab said in mid-February that it might limit logins on the weekends. Linden Lab said it might lock out some free accounts during periods of peak traffic, limiting access to the network to premium accounts -- who pay $9.95 per month -- and free accounts that have bought or sold Linden Dollars through Linden Lab. The announcement proved controversial, with some residents saying it was a sign that Second Life is collapsing.

Another measure to manage growth is to move some of Second Life to the Web. To experience most of Second Life, users need to download and install a specialized client. But Second Life already allows users to manage parts of their Second Life experience from the Web. Users can create accounts, check whether their Second Life friends are online, and buy and sell land and Linden Dollars using a Web browser. Linden Lab is considering how it should expand the Web capabilities to reduce the burden on Second Life servers. This will involve creating open APIs to allow third-party developers to write their own applications.

Technology problems aren't the only problems created by the influx of users. New users need to learn how to get around in Second Life, where to go, what to do, and how to get along with their fellow users. The Second Life user community itself has always been a big part of that effort. After an initial orientation, users are thrust into the world, and more experienced users step in as volunteers to help the newcomers out. There are even formal programs in place to provide volunteer mentors and classes for new Second Life users.

Second Life often sees influxes of international residents. For example, one recent weekend saw a major influx of users from France, driven by a presentation on a popular news program there. Before that, Second Life saw an influx of users from Australia.

Co-branded Second Life areas can help ease the transition, Miller said. Only one of those exists now, a site for fans of the TV show The L Word, run by Showtime and built by Second Life marketing company Electric Sheep. Fans of the show can download the client from the The L Word Web site. When The L-Word fans log in for the first time, they materialize in an orientation area specially built for fans of the show, not the general-purpose orientation area where most residents first experience Second Life.

When fans of the show leave the orientation area, the first place they go is to the L-Word's _The L Word area in the world, although they are free to move throughout Second Life.

Linden Lab plans more co-branded deals in the future, Miller says, although he declined to provide specifics.

Search capabilities are also straining under the load of traffic. Responses to search can be several hours out of date. Linden Lab is looking to solve that problem by moving searches to separate partitions and databases.

As another means of improving performance, Linden Lab is looking to provide tools to users to allow users to see how much system resources they're using. Some of the attachments that users create to decorate their avatars can make intensive use of system resources, and Linden Lab hopes that, if users become more aware of the resources they're using, they'll use them more wisely.