How Dropbox Moved 500PB Of Customer Files Off AWS - InformationWeek
IoT
IoT
Cloud // Cloud Storage
News
6/1/2016
09:05 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

How Dropbox Moved 500PB Of Customer Files Off AWS

With 500 petabytes of customer files to manage, Dropbox decided to become a post-cloud company. That meant moving a core operation off AWS. Here's how it was done.

7 PaaS Startups To Watch
7 PaaS Startups To Watch
(Click image for larger view and slideshow.)

Over the last two-and-a-half years, Dropbox has been migrating its customers' files off the Amazon cloud. The files have moved from an Amazon data center to one of two storage centers run by Dropbox.

The San Francisco-based company had been running on Amazon Web Services Simple Storage System since it was founded in 2007. It had 500 million customers and 500 petabytes of data by the time it started the migration. Dropbox needed to pay particular attention to the way it was going to provide the storage networking that would serve all those customers.

"The [AWS] cloud is highly performant and reliable. Moving out of it isn't a decision that a company should make lightly," or without a commitment to planning where it's moving to, said Akhil Gupta, vice president of infrastructure at Dropbox, in a recent interview with InformationWeek.

A Changing Cloud Storage Landscape

Four years ago, Dropbox was watching the growth of other cloud services "and saw the writing on the wall," Gupta said. It needed to become a cloud service itself, with its own infrastructure. It couldn't piggyback forever on Amazon, he said.

Akhil Gupta (right) and James Cowling, together, at their Dropbox offices. (Image: Dropbox)

Akhil Gupta (right) and James Cowling, together, at their Dropbox offices. (Image: Dropbox)

"We could see how it would be valuable to create our own storage system," Gupta said.

Gupta also pointed out, however, that Dropbox as a company didn't move everything off Amazon. It may have set up a massive storage system of its own, but "a lot of our infrastructure is still on Amazon and will continue to be for the foreseeable future," he said. Business systems and a limited amount of customer storage still run there.

Getting Started

To begin the storage move, Dropbox in 2013 set up a small team to start architecting the move, build a prototype storage system, and find a way to scale it up to massive proportions, Gupta said.

One of the people called in to lead the task was James Cowling, who had gotten to know Dropbox cofounder Drew Houston when the two were in computer science at MIT. Cowling had been with the company for a year when he got the assignment to replace AWS's S3 with something that could operate at scale, but was geared to Dropbox's specific needs.

That meant a new storage software system, along with storage hardware based on shingled magnetic recording technology. SMR is a dense data recording technique in which one track on a hard drive is partially overwritten by the next, as shingles on a roof overlap. The approach can increase disk density by 25%, although it also necessarily slows writes. The storage software would use RAM and additional forms of storage to offset SMR's drawbacks.

"We're one of the first companies to be deploying SMR" as a core production storage system, said Cowling in an interview. Given the nature of the technology, he had to research how best to design a system based on SMR for Dropbox, because its performance lags behind magnetic disks using regular or non-overlapping writes. Shingled magnetic recording "is not designed for high-performance random writes," he said. Rather, it thrives on many sequential writes.

The software controlling the storage system was first written in the Go language, which originated at Google, but the team found it too hungry for memory. They switched to another modern language, Rust, which required less memory. Rust was created at Mozilla.

The team also designed a compact unit of storage that could hold a petabyte of data, which was dubbed Diskotech. The goal was to concentrate as much storage as possible in a unit that consumed as little space and power as possible, Cowling said. The Diskotech is a little longer than a standard rackmount server tray from front to back, but it takes up only 4U (7-inch) or pizza-box-server spaces in a rack.

After its design, Cowling said the team members aimed at getting what they would call their Deep Pocket system built and into production in six months. Since Gupta and Cowling are both skydivers, they dubbed the project BaseJump. In a sport that both men consider riskier than skydiving, BASE jumpers leap off a cliff or high point and free-fall or glide before opening their parachutes.

[Want to see how Zynga moved off Amazon? Read Zynga's Big Move to Private Cloud.]

Getting everything done meant that the physical storage data center needed to be constructed on a tight schedule, with logistics carefully managed, including the supply of backup parts. Backups were needed on two occasions, when trucks bringing the SMR drives to the facility were involved in accidents. The planning went down to the detail of "How many racks can you put on a loading dock?" Cowling said.

The Final Move

The data was not shipped to the facilities on hard drive, such as Snowball, a 50TB sealed, encrypted box offered by AWS that can only be opened when it gets to its proper destination. Rather, Dropbox used both private lines and the internet to move the data. That process isn't 100% complete, but he said it's over 90% done.

Cowling says, "We are a cloud service provider." He also refers to Dropbox as "a post-cloud company." That is, it's a company that began as heavily dependent on the public cloud, then migrated the production core of its business from that to set up its own service.

"We created a new company, a post-cloud company. Now we can leverage the cloud as much as we want to," he said.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
6/8/2016 | 1:47:12 PM
Shingled magnetic storage isn't for everybody
SaneIT, good question. The why: Dropbox looked around and saw how other companies were building their own cloud services geared to what they wished to do rather than depend on Amazon's general purpose cloud infrastructure. AWS has not adopted the shingled magnetic recording disks for S3 because they are less suited to general purpose storage than they are for Dropbox' storage. The how is also a big part of the why.
SaneIT
50%
50%
SaneIT,
User Rank: Ninja
6/2/2016 | 8:38:34 AM
The how but what about the why?
The how seems fairly straight forward and not all that different from moving much much smaller amounts of data.  I'm curious about the why, ""We could see how it would be valuable to create our own storage system," Gupta said."  That statement doesn't begin to get into why it was a good idea to migrate away.  At what point are you limited by AWS and can do it better yourself?  What limitations did they hit that couldn't be worked through with Amazon?  I would think that if you have 500PB on Amazon's servers they would be willing to work with you if you have needs they aren't meeting. 
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of the Cloud Report
As the use of public cloud becomes a given, IT leaders must navigate the transition and advocate for management tools or architectures that allow them to realize the benefits they seek. Download this report to explore the issues and how to best leverage the cloud moving forward.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll