Obama's Developer Brain Trust: Inside The Big Battle

Social Studies: Obama vs. Romney

Five tech leaders of the Obama for America team recently spoke about their efforts to re-elect the President. They described a culture clash, not unlike that often found inside businesses, between a free-wheeling group of nerds and their goal-oriented and buttoned-down masters determined to win an election.

Both sides appeared satisfied with the results, but it took some jockeying and jostling with each other before they got to the win.

"We didn't do anything revolutionary," said Scott VanDenPlas, director of DevOps. "We just didn't screw up the things we would normally screw up," he said as part of a five-member panel that appeared at the Hyatt Regency Jan. 10 in San Francisco. The event was sponsored by the maker of an application performance monitoring tool used by the technology team, New Relic.

[ How did the Obama tech team use Amazon Web Service's EC2? See 6 Ways Amazon Cloud Helped Obama Win. ]

The five team leaders and their staff of 40 in Chicago and San Francisco produced what Alexis Madrigal, writer for The Atlantic, would describe after the election as "the new Chicago machine," 200 software applications meant to help volunteers collect donations, get out the vote, monitor the vote, track possible voter suppression, and apply the right resources, such as a local lawyer versed in election law, to resulting incidents.

Both sides invested in election technology, but from the start, the Obama tech team adopted the role of the underdog, regardless of whether that was actually the case. "We knew we would be outspent …We knew we had to make good use of what we had, " said Jason Kunesh, director of the user experience.

The team faced many conflicting demands from hardened politicos and organizers, who each felt they had a priority on the team's resources. "The political organizers would push, push, push. They were used to pushing third-party vendors," recalled Kunesh. But the team had been hired full time to be part of the campaign staff. Its members were committed to winning the election and didn't want to be treated like ignorant outsiders. At the same time, "None of us had a political background. We didn't understand much about how things work, as a campaign pops up every four years," said VanDenPlas. The team responded by capturing everyone's requirements and posting them on a wall at Chicago campaign headquarters, where the unsightly mess struck many as an impossible jumble. In fact, the visual aid, along with the approaching deadline of the election, proved a powerful winnower.

[ Read another view: Software Development in the Obama Campaign. ]

As the election drew nearer, the team applied two tests to all requests. "Will your feature help re-elect the President?" and "Can it get done before Nov. 6?" Even if a feature met the first test, it was discarded if it didn't meet the second. The team leaders became adept at explaining, "You're not getting your feature, but the good news is the application will get done in time," said Nick Leeper, the financial application technology lead.

One of the requirements he had to meet was being able to collect donations online, even if the database was down. The average donation was $55, which meant along with a few large donations there were thousands of $3 and $5 donations, he said. At one point, the database system was down for an hour, but none of the donations were lost. They were logged and added once it was running again. As an ecommerce specialist, he was used to building systems that initially collected $10-$20 an hour. It was a different experience to mount a system that would reliably collect $2 million an hour, he noted.

At first the political operatives doubted the technologists would produce anything of value in time for the election. Some of them were veterans of the 2008 campaign where a third-party software maker had produced the Houdini application to help with voter logistics; it had failed by 9:30 a.m. on Election Day.

The group started out with five servers running a handful of apps, but soon expanded by making use of Amazon Web Services servers in EC2. It produced 200 applications in all, but 48 proved the most useful. They were given whatever server power they needed during peaks of activity in the campaign. At one point, the Re-elect Obama effort occupied "thousands of servers in (Amazon's Northern Virginia data center) U.S. East," VanDenPlas said. The group worked at a breakneck pace through the summer, preparing applications for the election. VanDenPlas termed it, "our summer of messing up," learning what could be done in time and what wouldn't work. Kunesh said the sheer scale of what they were trying to do was daunting. When it came time to test how well an application would interact with voters who might respond to an email query, he was amazed to learn the sample would consist of 400,000 actual users. The whole list would have meant a mailing to 20 million.

As the team entered October, everyone agreed to a feature freeze and hardening of systems. On Oct. 21, Harper Reed, the technology chief who had hired the five, announced a dry run of key voter tracking and get-out-the vote election day applications and provided them with a script of how they would be tested. When the morning for the test arrived, Reed overrode the script and started firing unexpected challenges at them. At the same time, Amazon experienced a service outage in U.S. East, and instead of meeting Reed's challenge head on, the team had to divide in two, with one part dealing with the challenge posed by Amazon, the other dealing with Reed.

"Half responded to the real incidents (Amazon). Half responded to fake incidents," said Chris Gansen, technology lead for the Dashboard application. Reed posed dilemmas, such as the database server lost its cache memory function, then actually flushed out the database server cache. The team scrambled under his discerning eye to solve a "fake" problem that had suddenly become real.

At the end of the day, said Gansen, both teams were exhausted. They repeated the drills at the end of the month, then made additional preparations as Hurricane Sandy approached the East Coast in advance of the election. Believing Amazon's U.S. East data center complex was at risk, the team replicated all its systems to Amazon's U.S. West in northern California before Sandy struck.

The team had implemented a service-oriented architecture where many applications relied on services from a backend platform, called Narwhal. As word leaked out of the tech team's work, the technologists in the Mitt Romney camp announced they also had an Election Day system, called Orca. The Orca killer whale is the only known predator of narwhals.

At that point, the tech team was beginning to have faith in the reliability of the system they had built and it was not concerned by the reports. Its Narwhal platform could bring all the different activities that a volunteer or organizer had been involved in together to give a headquarters organizer a clearer idea of who they were dealing with and what they should do next.

As election day dawned, Narwhal worked as expected, while reports from the Romney camp indicated that Orca either failed to function or tied up communications lines by sending out 80-page PDFs to seniors who had volunteered to help turn out the vote. Nine days after the election, Ars Technica ran an insider piece on how the two respective systems performed, "Built to win: Deep inside Obama's campaign tech."

"Great relationships" developed across technology teams and in some cases, between the technologists and the politicos. "Towards the end, we got to where we needed to have people working cross-discipline on different functions," said Kunesh.

Reed "made us go for simplicity. The older users had iPads. The younger users had iPhones, and that was it," recalled Gansen. The technology team used HTML to straightforwardly gear its applications to mobile users needing simple communications and information fast.

"We've just established the new normal for what a political campaign will have to do," said Gansen. But he doubted the code created would survive as the backbone of a system for the next election. The next team should throw it out and start over, with conditions as it finds them at the time, he warned.

Several team members questioned why the Romney technology team didn't do better. "It's a little strange to me the party of business didn't do better… This isn't going to happen again," Gansen said.

Despite their reputation as the young guns of political technology, none of the five team members announced any clear plans of what he was going to do next.

"I have no idea," said the long-haired Kolak, perhaps the most noted member of the team as the mastermind of Narwhal. "We're in San Francisco. Is there any work out here?"

His audience of 300, heavily salted with young programmers from Google, Twitter, Zynga and Facebook, laughed appreciatively and broke into sustained applause.