Obama's Developer Brain Trust: Inside The Big Battle
Tech leaders for the Obama for America team discuss their re-election project -- and a business-like culture clash between free-wheeling nerds and their goal-oriented bosses.
Kunesh said the sheer scale of what they were trying to do was daunting. When it came time to test how well an application would interact with voters who might respond to an email query, he was amazed to learn the sample would consist of 400,000 actual users. The whole list would have meant a mailing to 20 million.
As the team entered October, everyone agreed to a feature freeze and hardening of systems. On Oct. 21, Harper Reed, the technology chief who had hired the five, announced a dry run of key voter tracking and get-out-the vote election day applications and provided them with a script of how they would be tested. When the morning for the test arrived, Reed overrode the script and started firing unexpected challenges at them. At the same time, Amazon experienced a service outage in U.S. East, and instead of meeting Reed's challenge head on, the team had to divide in two, with one part dealing with the challenge posed by Amazon, the other dealing with Reed.
"Half responded to the real incidents (Amazon). Half responded to fake incidents," said Chris Gansen, technology lead for the Dashboard application. Reed posed dilemmas, such as the database server lost its cache memory function, then actually flushed out the database server cache. The team scrambled under his discerning eye to solve a "fake" problem that had suddenly become real.
At the end of the day, said Gansen, both teams were exhausted. They repeated the drills at the end of the month, then made additional preparations as Hurricane Sandy approached the East Coast in advance of the election. Believing Amazon's U.S. East data center complex was at risk, the team replicated all its systems to Amazon's U.S. West in northern California before Sandy struck.
The team had implemented a service-oriented architecture where many applications relied on services from a backend platform, called Narwhal. As word leaked out of the tech team's work, the technologists in the Mitt Romney camp announced they also had an Election Day system, called Orca. The Orca killer whale is the only known predator of narwhals.
At that point, the tech team was beginning to have faith in the reliability of the system they had built and it was not concerned by the reports. Its Narwhal platform could bring all the different activities that a volunteer or organizer had been involved in together to give a headquarters organizer a clearer idea of who they were dealing with and what they should do next.
As election day dawned, Narwhal worked as expected, while reports from the Romney camp indicated that Orca either failed to function or tied up communications lines by sending out 80-page PDFs to seniors who had volunteered to help turn out the vote. Nine days after the election, Ars Technica ran an insider piece on how the two respective systems performed, "Built to win: Deep inside Obama's campaign tech."
"Great relationships" developed across technology teams and in some cases, between the technologists and the politicos. "Towards the end, we got to where we needed to have people working cross-discipline on different functions," said Kunesh.
Reed "made us go for simplicity. The older users had iPads. The younger users had iPhones, and that was it," recalled Gansen. The technology team used HTML to straightforwardly gear its applications to mobile users needing simple communications and information fast.
"We've just established the new normal for what a political campaign will have to do," said Gansen. But he doubted the code created would survive as the backbone of a system for the next election. The next team should throw it out and start over, with conditions as it finds them at the time, he warned.
Several team members questioned why the Romney technology team didn't do better. "It's a little strange to me the party of business didn't do better… This isn't going to happen again," Gansen said.
Despite their reputation as the young guns of political technology, none of the five team members announced any clear plans of what he was going to do next.
"I have no idea," said the long-haired Kolak, perhaps the most noted member of the team as the mastermind of Narwhal. "We're in San Francisco. Is there any work out here?"
His audience of 300, heavily salted with young programmers from Google, Twitter, Zynga and Facebook, laughed appreciatively and broke into sustained applause.
Multicloud Infrastructure & Application ManagementEnterprise cloud adoption has evolved to the point where hybrid public/private cloud designs and use of multiple providers is common. Who among us has mastered provisioning resources in different clouds; allocating the right resources to each application; assigning applications to the "best" cloud provider based on performance or reliability requirements.