How Dropbox Moved 500PB Of Customer Files Off AWS

7 PaaS Startups To Watch

Over the last two-and-a-half years, Dropbox has been migrating its customers' files off the Amazon cloud. The files have moved from an Amazon data center to one of two storage centers run by Dropbox.

The San Francisco-based company had been running on Amazon Web Services Simple Storage System since it was founded in 2007. It had 500 million customers and 500 petabytes of data by the time it started the migration. Dropbox needed to pay particular attention to the way it was going to provide the storage networking that would serve all those customers.

"The [AWS] cloud is highly performant and reliable. Moving out of it isn't a decision that a company should make lightly," or without a commitment to planning where it's moving to, said Akhil Gupta, vice president of infrastructure at Dropbox, in a recent interview with InformationWeek.

A Changing Cloud Storage Landscape

Four years ago, Dropbox was watching the growth of other cloud services "and saw the writing on the wall," Gupta said. It needed to become a cloud service itself, with its own infrastructure. It couldn't piggyback forever on Amazon, he said.

"We could see how it would be valuable to create our own storage system," Gupta said.

Gupta also pointed out, however, that Dropbox as a company didn't move everything off Amazon. It may have set up a massive storage system of its own, but "a lot of our infrastructure is still on Amazon and will continue to be for the foreseeable future," he said. Business systems and a limited amount of customer storage still run there.

Getting Started

To begin the storage move, Dropbox in 2013 set up a small team to start architecting the move, build a prototype storage system, and find a way to scale it up to massive proportions, Gupta said.

One of the people called in to lead the task was James Cowling, who had gotten to know Dropbox cofounder Drew Houston when the two were in computer science at MIT. Cowling had been with the company for a year when he got the assignment to replace AWS's S3 with something that could operate at scale, but was geared to Dropbox's specific needs.

That meant a new storage software system, along with storage hardware based on shingled magnetic recording technology. SMR is a dense data recording technique in which one track on a hard drive is partially overwritten by the next, as shingles on a roof overlap. The approach can increase disk density by 25%, although it also necessarily slows writes. The storage software would use RAM and additional forms of storage to offset SMR's drawbacks.

"We're one of the first companies to be deploying SMR" as a core production storage system, said Cowling in an interview. Given the nature of the technology, he had to research how best to design a system based on SMR for Dropbox, because its performance lags behind magnetic disks using regular or non-overlapping writes. Shingled magnetic recording "is not designed for high-performance random writes," he said. Rather, it thrives on many sequential writes.

The software controlling the storage system was first written in the Go language, which originated at Google, but the team found it too hungry for memory. They switched to another modern language, Rust, which required less memory. Rust was created at Mozilla.

The team also designed a compact unit of storage that could hold a petabyte of data, which was dubbed Diskotech. The goal was to concentrate as much storage as possible in a unit that consumed as little space and power as possible, Cowling said. The Diskotech is a little longer than a standard rackmount server tray from front to back, but it takes up only 4U (7-inch) or pizza-box-server spaces in a rack.

After its design, Cowling said the team members aimed at getting what they would call their Deep Pocket system built and into production in six months. Since Gupta and Cowling are both skydivers, they dubbed the project BaseJump. In a sport that both men consider riskier than skydiving, BASE jumpers leap off a cliff or high point and free-fall or glide before opening their parachutes.

[Want to see how Zynga moved off Amazon? Read Zynga's Big Move to Private Cloud.]

Getting everything done meant that the physical storage data center needed to be constructed on a tight schedule, with logistics carefully managed, including the supply of backup parts. Backups were needed on two occasions, when trucks bringing the SMR drives to the facility were involved in accidents. The planning went down to the detail of "How many racks can you put on a loading dock?" Cowling said.

The Final Move

The data was not shipped to the facilities on hard drive, such as Snowball, a 50TB sealed, encrypted box offered by AWS that can only be opened when it gets to its proper destination. Rather, Dropbox used both private lines and the internet to move the data. That process isn't 100% complete, but he said it's over 90% done.

Cowling says, "We are a cloud service provider." He also refers to Dropbox as "a post-cloud company." That is, it's a company that began as heavily dependent on the public cloud, then migrated the production core of its business from that to set up its own service.

"We created a new company, a post-cloud company. Now we can leverage the cloud as much as we want to," he said.