Cloud // Cloud Storage
News
5/14/2014
10:26 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Social Science Site Using Azure Loses Data

Dedoose, a data analytics system, suffered a failure on Azure that may mean three weeks of lost data for customers.

8 Data Centers For Cloud's Toughest Jobs
8 Data Centers For Cloud's Toughest Jobs
(Click image for larger view and slideshow.)

Dedoose, a social science data company, has lost some of its customers' data while operating on the Microsoft Azure cloud. "This is a horrible moment for our company. We have never lost data or had a breach," said Dedoose president Eli Lieber in an interview.

At best, it will be able to restore data stored through April 11, Lieber said, and perhaps only up to March 30. For an uncertain minority of customers, data added to their accounts after April 11 has been lost, said Lieber.

"It's impossible to say" how many customers have lost data because the firm doesn't monitor how customers are using their accounts, he continued. "Only users can assess losses in their projects, not Dedoose," he said, but the company is doing everything it can to help them recover data.

Dedoose.com, founded in 2006 in Manhattan Beach, Calif., provides social scientists with an online analytics application, EthnoNotes, used by both commercial and academic researchers. Lieber said the application is used for marketers, pharmaceutical company researchers, academic social scientists, and others.

[Want to learn more about a documented failure in Azure operations? See Microsoft Pins Azure Slowdown On Cloud Software Fault.]

Dedoose offers its EthnoNotes application entirely on the Microsoft Azure cloud, and offers Gladinet.com, a free cloud storage service, as backup storage. Many customers put data into Word documents and Excel spreadsheets. Through its consulting services, Dedoose will help customers retrieve that data and rebuild their databases when other sources aren't available.

The Los Angeles Times reported that Dedoose officials sent an email Monday saying that both its operational and storage systems had failed. "The timing of this event was such that our entire data storage container was corrupted -- including the master database and all local backups," the company wrote in the email.

"Within minutes of discovering the problem, we contacted Microsoft Azure support. Unfortunately, Microsoft was unable to recover these data... from its servers," the Dedoose email to customers said. Microsoft officials couldn't be reached for comment.

"Since our initial communication about this issue, we have also learned that the backup files stored to an independent location were also corrupted. We are working with Gladinet services to have the non-corrupted data transferred to alternative storage locations and restore the integrity of these files," the message continued, according to the L.A. Times.

Asked who was responsible for the data loss, Azure or Dedoose, Lieber said: "It's a complicated situation." Dedoose produced its EthnoNotes system and installed it on Azure, but didn't expect that its data could be lost when it failed. He declined to estimate the respective shares of responsibility. Dedoose is still investigating why the system failed. Lieber is one of the authors of the EthnoNotes system.

"Things happened on the platform that we were blind to. Similar events have happened (to other Azure customers) and the data was not recoverable," Lieber said, without naming any other Azure users who suffered a similar failure.

Dedoose is revising its all-cloud approach to include a redundant system separate from the operational one on Azure. "We are putting measures in place so that if the platform is destroyed again, we will live on," said Lieber. "Within two weeks, our systems will be tremendously more robust."

But he added, "Ultimately, we are the responsible party. As we learn more about what happened, we are taking steps to protect our users." Lieber declined to say how many customers were affected or how many customers in total use EthnoNotes. But he said the number affected, while a minority, is a higher than the "dozens of customers" suggested by another description.

Are you better protected renting space for disaster recovery or owning a private cloud? Follow one company's decision-making process. Also in the Disaster Recovery issue of InformationWeek: Five lessons from Facebook on analytics success. (Free registration required.)

Charles Babcock is an editor-at-large for InformationWeek, having joined the publication in 2003. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive Week. He is a graduate of Syracuse ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
PaulS681
50%
50%
PaulS681,
User Rank: Ninja
5/17/2014 | 2:41:22 PM
Back ups?

So where are the backups? Isn't this data backed up? If it is...is it checked? Losing data is a very bad thing obviously, which is why you have a good backup plan in place. How does this happen?

Ok, I just read a few post down and see the answers to my questions. I commend them for explaining the situation. That being said, the most important piece of a backup plan is testing, testing and more testing. Test restores must be done. You need to know before that disaster happens if your backups are corrupt.

 

cust_faithful01
50%
50%
cust_faithful01,
User Rank: Apprentice
5/16/2014 | 12:33:44 PM
Proprietary data formats add insecurity
As a Dedoose user, I'm impressed with the attempt by the firm to provide a need that's out there.

They've added a bunch of features that rivals like nVivo have not offered, but users have cried out for. And of course, its cloud-based so any operating system can play.

BUT...and it's a big but, Dedoose creates and handles its data in some pretty unique ways. Its export functions are clumsy, and you can't even do basic things like search a project full of documents for a text string. (and don't even think about editing a document that's been loaded - a complete no-no apparently). That gives you some idea of how difficult it is to get data out of Dedoose.

The out-take of this for a user is that you worry about how you can maintain and sustain the life of your valuable information. And the first rule of data maintenance is fool-proof back up. Dedoose...and its users have learnt the hard way that niche software which has little universally standard backup or export is highly vulnerable to a catastrophic failure, in that local mirrors of projects made by users are not supported, and Dedooses' own backups clearly failed (failure across multiple backs in multiple locations is hard to fathom - I can't imagine how one could designa system like that, but hey, I'm just a user, what do I know /matter?)

I'm determined to stick with Dedoose becasue I believe they will learn from this and certainly will backup properky in future. But whether they will extend this to developing funcionality and format that enables users to access and store data in a more friendly and flexible way - that remains to be seen....

 

 

 

 

 

 
open_minded
50%
50%
open_minded,
User Rank: Apprentice
5/16/2014 | 8:36:08 AM
Re: Catastrophic Failure..
What do you expect from a Microsoft cloud? It's running Windows thus it will break.
Apparently Microsoft is unable to fix that with whatever stuff they use.
So the big tip to Dedoose: use a cloud provider that knows how to run a stable redundant infrastructure stack, like the largely backed OpenStack, or even AWS with HA options.

Yes you are to blame, for making the wrong choice and believing the marketing BS.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/15/2014 | 9:03:04 PM
Dedoose takes full responsibility
I think Dedoose is taking full responsibilty here and appropriate corrective actions. The lost data is likely to damage the projects of of some of its customers and that will be upsetting to them, as well it should be. I hope customers understand the company is going through a tough phase and responding appropriately to future threats to continuous operations.
Laurianne
100%
0%
Laurianne,
User Rank: Author
5/15/2014 | 5:31:19 PM
Thanks
Thanks to Dedoose for weighing in here with your side of the backup story.
Dedoose
50%
50%
Dedoose,
User Rank: Apprentice
5/15/2014 | 4:25:56 PM
To clear a few things up
 We are terribly sorry about this event. Dedoose is built and designed by well known academic researchers. We are not a giant corporation, we are a small team of researchers, and technology visionaries that built a collaborative tool for our own research needs, and are trying to share this tool with the world at large. This is a giant tragedy, and we accept responsibility for this event, and are offering our staff of methodologist to assist in recreating lost data, coding, analysis, etc. If you are affected by this data loss, please call us or shoot an email to support@dedoose.com. We did have a multiple backup strategy in place using proprietary software. It has been very challenging finding software to handle this job at the scales of data we are working with. Unfortunately the software we chose to use corrupted occasional backups. In our testing we did not encounter an issue, but during the event the full backups needed to restore were unrecoverable. This situation is complicated by the many layers of encryption our systems use, the database encryption, the backup encryption, the transfer encryption, etc. We are now running mirrors upon mirrors, off-site mirrors on a seperate cloud, off-site mirrors locally, and our team developed a tool that is automatically copying the encrypted database backups to amazons s3 cloud, glacier storage, as well as an onsite copy. We will be adding additional measure to ensure the app automatically downloads a local backup of the data needed to restore a project whenever logged in. When we developed Dedoose, we developed it initially for our research team. The biggest factor affecting our teams was the inability to collaborate on projects, thus we designed an online tool to do so. We continue to work on a non-collaborative offline version, but that was never our main focus. We recognize the seriousness of this event, but urge you to understand we have put the protections in place to ensure a data loss event is not possible in the future short of a global cataclysmic event. Ultimately it was our fault for not having backups of backups, backups of those backups, and more backups of those backups. We have resolved that issue and this will not happen again. We are deeply, deeply, sorry and more than willing to assist affected users with our team of Dedoosers.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/15/2014 | 2:47:30 PM
Dedoose implements better data protection on Azure -- and Amazon
In a blog May 9 following up the crash, CEO Eli Lieber said Dedoose was implementing the following for better data protection:
  1. Deploying a database mirror/slave in Azure
  2. Deploying a database mirror/slave into Amazon S3
  3. Keeping a mirror copy of the entire blob storage including all file data, backups, video data synchronized nightly to our private server in an encrypted volume
  4. Storing nightly database backups on the VHD, Azure Blob Storage, and Amazon S3 Storage
  5. Mirroring all Azure file data into an Amazon S3 bucket
  6. Carrying out a weekly restore exercise for the database backups to ensure integrity

 
moarsauce123
0%
100%
moarsauce123,
User Rank: Ninja
5/15/2014 | 12:26:24 PM
Welcome to the cloud!
All eggs into one basket without a backup and any means to control the infrastructre...I blame Dedoose squarely for the data loss. Anyone who has any experience with networked systems and server hardware should know that failure is always an option and it is unfortunately fairly high up on the list. Anyone who deploys cloud services without having a reliable plan B is just asking for well-deserved trouble.
HichemS973
50%
50%
HichemS973,
User Rank: Apprentice
5/14/2014 | 8:39:23 PM
Important: Not an Azure Failure based on Deddose Saying But it was an application issue on their side
http://blog.dedoose.com/2014/05/dedooses-black-eye-crash-and-recovery-efforts/

This devastating system 'collision' of Tuesday night resulted from an series of events leading to the failure of Dedoose services running on the Microsoft Azure platform.  To be clear, Dedoose services failed, not Azure.  In short, work done on one aspect of Dedoose led to the failure of another, cascading to pull down all of Dedoose.  The timing was particularly bad because it occurred in the midst of a full database encryption and backup.  This backup process, in turn, corrupted our entire storage system. Our immediate work with Microsoft support did not result in any substantial recovery.  Here's where we are and where we're going:

 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/14/2014 | 4:23:53 PM
It was all Dedoose's fault? Hmmm. Maybe....
Strong statements here from Joe Emison (we expect no less). But I'm not adopting the 0-100% split between Azure and Dedoose until I know a little bit more. For example, was any monitoring available to indicate the system was about to fail? I've asked Microsoft to respond. Still waiting. 
Page 1 / 2   >   >>
Google in the Enterprise Survey
Google in the Enterprise Survey
There's no doubt Google has made headway into businesses: Just 28 percent discourage or ban use of its productivity ­products, and 69 percent cite Google Apps' good or excellent ­mobility. But progress could still stall: 59 percent of nonusers ­distrust the security of Google's cloud. Its data privacy is an open question, and 37 percent worry about integration.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest Septermber 14, 2014
It doesn't matter whether your e-commerce D-Day is Black Friday, tax day, or some random Thursday when a post goes viral. Your websites need to be ready.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.