Golden Moment

It was 11:15 on a cold Wednesday morning when I received the call. While athletes competed in alpine skiing, biathlon, hockey, and several other Olympic events at venues along Utah's Wasatch Front, an IT nightmare was unfolding. In recent days, we'd dealt with a host of problems: equipment failures, software bugs, timing-system glitches, radio-frequency conflicts, and network outages. Now, things were getting worse.

Early indications were that someone had hacked into our network and was taking control of our entire IT operation. Bob Cottam, chief integrator for SchlumbergerSema and acting director of the Olympic Information Technology Center, called to report that we had servers failing at every venue. I resisted the urge to dig in to how this happened and, instead, let our team of Unix administrators and network and security specialists pursue the problem following the procedures we had worked long and hard to develop. I sat in my office, imagining the headlines: "Hackers' Olympics Dream: Security Breach Delays Games."

Thank heavens this was just a drill.

The truth is, the XIX Olympic Winter Games were still a couple of months away. The events of that day felt real, but they were simulated--and they occurred in December 2001. We were conducting Technical Rehearsal II, a five-day, 800-person exercise designed to test our technology operations, processes, and procedures. We ran the first rehearsal two months earlier.

The comprehensive drills were just one of the ways that delivering Olympics technology is different from the typical business IT project. It's a five-year, $300 million effort that's guaranteed to get international attention if it doesn't work perfectly from the moment it goes live. Then it's shut down after just 17 days of operation.

It's also more costly than most companies would tolerate. It's possible to cut 10% or more from the IT budget without hurting the Games, something that needs to be considered amid the spiraling cost of the Olympics. Yet when it came to determining success, we relied on the same factors as any good project manager: Put together a strong team, control the project's scope, set up good processes, meet deadlines--and try to have some fun along the way.

The rehearsals weren't always fun. Engineers from the Games' timing-system supplier, Seiko, used simulators to send timing and scoring data to scoreboards and on-venue results applications. These systems updated scoreboards, TV graphics, and commentator systems with start lists, finish times, and event results, and simultaneously sent the same data across Qwest's Olympic Sonet network to central databases for distribution to our press, broadcast, print, and Web-site applications. The rehearsals' objectives were to make sure our backup and failover strategies worked and that the technology team was prepared to detect and respond to virtually any situation that might arise. Given that we had already performed tens of thousands of hours of software and infrastructure testing, the team was pretty confident that our systems and networks were ready. The question was, were we ready for Games operations?

Marco de Palma proved we weren't. I hired de Palma as a full-time consultant to plan and execute the two five-day exercises; he created a team whose job was to dream up diabolical, yet plausible, problems we might face during the Olympics. The first rehearsal in October was pretty grim. We found out that some key personnel were overloaded, that the physical layout of our IT command center needed a major redesign, and that people at our command centers and competition venues weren't always communicating as well as needed.

The December rehearsal went better. We had made sure all the server ports were locked down at the venues so an outsider couldn't plug into the network. But the rehearsal team had someone sneak in and plug a wireless device into an open port at headquarters, where the ports weren't all locked down, so the "hacker" could sabotage the network remotely from a nearby garage. We resolved that problem before the Games.

The technology project for the 2002 Olympics got started in 1997, when Sharon Kingman, managing director of telecommunications, recruited and signed our telecom sponsors: AT&T, Lucent, and Qwest (US West at the time). Together, these heavyweights would deliver local and long-distance telecommunications services, the Sonet network, CDMA and TDMA wireless networks, PBXs, cabling, a theaterwide cable-TV system, and a large, capable project team.

Alice Mahmood, managing director of information services, brought in Seiko for timing and scoring. Sponsors' contributions provide three-fourths of the Games budget, but the negotiations can be complicated at times. Mahmood enlisted Epson, Seiko's sister company, to build and integrate our on-venue results applications, but a marketing-rights conflict with another sponsor, one of many we would deal with over the years, scuttled the deal. Eventually, Mahmood and Frederic Wojciechowski, the Salt Lake Organizing Committee's results director, crafted an agreement with Wige Data, a German software-development company, to deliver the results applications for the venues.

I joined the Olympic committee in October 1998, just in time for the decision to replace long-time Olympic sponsor IBM. No other company in the world offered the breadth of products and services of IBM, but the committee and company couldn't come to terms. The Olympic committee was left with no choice but to assemble a consortium of companies to provide the myriad products and services previously provided by IBM. Driven more by economic need than desire, the 2002 technology team was also substantially smaller than teams assembled by IBM for previous Olympiads. In the end, the sponsors and suppliers in the 2002 technology consortium were the most effective, capable team ever to take on Olympic technology.

For the information services anchor position, the Salt Lake Organizing Committee and the International Olympic Committee signed SchlumbergerSema (then the Sema Group). A leading systems integrator in Europe, SchlumbergerSema had been around the Olympics since the 1992 Summer Games in Barcelona, when it began subcontracting for IBM. SchlumbergerSema's role in the 2002 Games included systems integration; development of all central systems; and management of data centers, staging areas, and the Information Technology Center.

From 1998 to 2001, a cadre of new Olympic sponsors joined the Salt Lake technology consortium, including Gateway, a great partner that provided all PCs, Windows NT servers and related support; Ikano, a small Utah technology-services company with a huge talent for supporting complex data networks; Sun Microsystems, which complemented bulletproof servers with top-notch tuning and security folks; and Quokka Sports and Logictier, which provided Internet content and infrastructure, respectively, for the official 2002 Web site. Adding to this were long-time IOC sponsors Xerox, Samsung, Panasonic, and Kodak to make a 13-sponsor technology consortium that collectively provided more than $225 million in goods and services to the Salt Lake Games.

Assembling and managing such a consortium wasn't without challenges. Consider the official 2002 Web site. Quokka Sports and Logictier, recruited and signed in late 2000, were complex partnerships from the beginning. Quokka was to produce an exhaustive amount of content, including live results, enhanced athlete biographies, and timely feature stories on the Games and events. Logictier, as our Internet infrastructure sponsor, was to support the 1 billion page views forecasted during the Games with an infrastructure that included seven data centers around the world, farms of Unix servers, and a robust network design. They seemed like smart choices, since both providers met all their early milestones. But in early 2001, the dot-com bubble burst. Quokka had never made a profit, and the venture funding it had relied on for years dried up. Logictier, despite a very talented crew, fell victim to an aggressive capitalization strategy and a collapsing Internet advertising market. Both went out of business.

To the rescue came MSNBC. Perkins Miller, who had signed up in late 2000 to manage the official Web site, worked with our U.S. broadcasting partner, NBC, to hastily put together a deal with MSNBC. Keeping with the long-standing mandate of Mitt Romney, CEO of the Salt Lake Organizing Committee, the Web site couldn't cost any cash. Like the Web deals that preceded it, the MSNBC agreement met this criterion and offered the added bonus of requiring us to work with just one company for content and hosting services. There was one caveat, not surprising in light of MSNBC's heritage. The official Web site had to run exclusively on Windows NT. While I'm a real fan of NT (we used it for all of our operational applications during the Games), I was leery of running the largest event Web site in history on an Intel-based platform. My concerns proved unfounded, though, when MSNBC not only produced the site in record time, including the development of complicated real-time results applications, but effectively hosted the site on Microsoft's server farms.

So, how did the 2002 Olympics technology fare? According to Jacques Rogge, newly installed president of the IOC, "The technology system was flawless." Well, not quite. You can't deploy the mountains of software and infrastructure we did without a few bumps. However, years of testing, our participation in a series of pre-Olympic test events, combined with two technical rehearsals, did their job. When a problem did occur during the Games, it rarely became visible to anyone except the venue technology managers or one of the 200 or so IT folks monitoring our systems and networks in the technology command centers.

What could we have done better? For starters, we began work too late on the detailed design of our overall architecture. This delayed the development of our security and disaster-recovery strategies, definition of our operating system release levels, and determination of our equipment and network needs.

We also failed to introduce good version-control tools early, and the regression-testing environment we employed lacked needed structure and documentation. Exceptionally talented configuration-management and-testing teams overcame these shortcomings, but introducing changes was even more anxiety-ridden than it needed to be. I also should have created a flatter IT-operations management structure, instead of having both a systems and a telecom manager at each venue. The structure we employed worked, but having a single technology manager at each venue would have been more efficient.

I also believe our IT project was too costly. Like Olympic organizing committees before us, we inherited a broad range of requirements, some of which were more the result of Olympic tradition than legitimate need. Under its new president, the IOC has launched an effort to reduce the expense of staging future Olympics, and technology costs should be examined closely.

The Games' organizers spend far too much on backup IT systems. We had two levels of backup systems to manage the results, and never once had to use more than the first backup. More than 200 of our 450 servers were some sort of backup device. Yet among our 150 Sun Unix servers, we experienced only one hard-drive problem and one bad adapter card in 17 days of the Games, and Gateway's 300-plus NT servers delivered a similar level of reliability. Given that IT represents about 20% of an Olympic committee's total costs, we're paying too much for peace of mind.

There are other savings opportunities: replacing the 20 million pages of printed reports and results with searchable CDs, or combining the separate systems that provide information to broadcast commentators and to accredited visitors online, since they provide essentially the same information. But to achieve its cost-savings goal, the IOC will have to win concessions such as less functionality and higher risk tolerance from some of its most powerful constituencies, most notably the press, broadcasters, and international sports federations--all groups used to getting their own way. Not an easy task.

On balance, we accomplished our goals of taking care of business and having some fun. Some folks had so much fun that they've accepted positions at the Summer Games in Athens in 2004 and the Winter Games in Torino, Italy, in 2006. For the rest of us, Salt Lake marks the end of our Olympic journey. By May 1, only a handful of Salt Lake Organizing Committee employees will remain. The majority of the team will return to the business world. Few expect to fully match their experience working on the 2002 Winter Games. But if you've got a tough business challenge, I'd suggest giving one of them a call.

Want to hire an Olympic IT staffer? Call Lisa Wardle at the SLOCs Job Transition Program Office: 801-212-3603