Obama's Developer Brain Trust: Inside The Big Battle

Five tech leaders of the Obama for America team recently spoke about their efforts to re-elect the President. They described a culture clash, not unlike that often found inside businesses, between a free-wheeling group of nerds and their goal-oriented and buttoned-down masters determined to win an election.

Both sides appeared satisfied with the results, but it took some jockeying and jostling with each other before they got to the win.

"We didn't do anything revolutionary," said Scott VanDenPlas, director of DevOps. "We just didn't screw up the things we would normally screw up," he said as part of a five-member panel that appeared at the Hyatt Regency Jan. 10 in San Francisco. The event was sponsored by the maker of an application performance monitoring tool used by the technology team, New Relic.

[ How did the Obama tech team use Amazon Web Service's EC2? See 6 Ways Amazon Cloud Helped Obama Win. ]

The five team leaders and their staff of 40 in Chicago and San Francisco produced what Alexis Madrigal, writer for The Atlantic, would describe after the election as "the new Chicago machine," 200 software applications meant to help volunteers collect donations, get out the vote, monitor the vote, track possible voter suppression, and apply the right resources, such as a local lawyer versed in election law, to resulting incidents.

Both sides invested in election technology, but from the start, the Obama tech team adopted the role of the underdog, regardless of whether that was actually the case. "We knew we would be outspent …We knew we had to make good use of what we had, " said Jason Kunesh, director of the user experience.

The team faced many conflicting demands from hardened politicos and organizers, who each felt they had a priority on the team's resources. "The political organizers would push, push, push. They were used to pushing third-party vendors," recalled Kunesh. But the team had been hired full time to be part of the campaign staff. Its members were committed to winning the election and didn't want to be treated like ignorant outsiders. At the same time, "None of us had a political background. We didn't understand much about how things work, as a campaign pops up every four years," said VanDenPlas. The team responded by capturing everyone's requirements and posting them on a wall at Chicago campaign headquarters, where the unsightly mess struck many as an impossible jumble. In fact, the visual aid, along with the approaching deadline of the election, proved a powerful winnower.

[ Read another view: Software Development in the Obama Campaign. ]

As the election drew nearer, the team applied two tests to all requests. "Will your feature help re-elect the President?" and "Can it get done before Nov. 6?" Even if a feature met the first test, it was discarded if it didn't meet the second. The team leaders became adept at explaining, "You're not getting your feature, but the good news is the application will get done in time," said Nick Leeper, the financial application technology lead.

One of the requirements he had to meet was being able to collect donations online, even if the database was down. The average donation was $55, which meant along with a few large donations there were thousands of $3 and $5 donations, he said. At one point, the database system was down for an hour, but none of the donations were lost. They were logged and added once it was running again. As an ecommerce specialist, he was used to building systems that initially collected $10-$20 an hour. It was a different experience to mount a system that would reliably collect $2 million an hour, he noted.

At first the political operatives doubted the technologists would produce anything of value in time for the election. Some of them were veterans of the 2008 campaign where a third-party software maker had produced the Houdini application to help with voter logistics; it had failed by 9:30 a.m. on Election Day.

The group started out with five servers running a handful of apps, but soon expanded by making use of Amazon Web Services servers in EC2. It produced 200 applications in all, but 48 proved the most useful. They were given whatever server power they needed during peaks of activity in the campaign. At one point, the Re-elect Obama effort occupied "thousands of servers in (Amazon's Northern Virginia data center) U.S. East," VanDenPlas said. The group worked at a breakneck pace through the summer, preparing applications for the election. VanDenPlas termed it, "our summer of messing up," learning what could be done in time and what wouldn't work.