Social Science Site Using Azure Loses Data - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Cloud // Cloud Storage
News
5/14/2014
10:26 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Social Science Site Using Azure Loses Data

Dedoose, a data analytics system, suffered a failure on Azure that may mean three weeks of lost data for customers.

8 Data Centers For Cloud's Toughest Jobs
8 Data Centers For Cloud's Toughest Jobs
(Click image for larger view and slideshow.)

Dedoose, a social science data company, has lost some of its customers' data while operating on the Microsoft Azure cloud. "This is a horrible moment for our company. We have never lost data or had a breach," said Dedoose president Eli Lieber in an interview.

At best, it will be able to restore data stored through April 11, Lieber said, and perhaps only up to March 30. For an uncertain minority of customers, data added to their accounts after April 11 has been lost, said Lieber.

"It's impossible to say" how many customers have lost data because the firm doesn't monitor how customers are using their accounts, he continued. "Only users can assess losses in their projects, not Dedoose," he said, but the company is doing everything it can to help them recover data.

Dedoose.com, founded in 2006 in Manhattan Beach, Calif., provides social scientists with an online analytics application, EthnoNotes, used by both commercial and academic researchers. Lieber said the application is used for marketers, pharmaceutical company researchers, academic social scientists, and others.

[Want to learn more about a documented failure in Azure operations? See Microsoft Pins Azure Slowdown On Cloud Software Fault.]

Dedoose offers its EthnoNotes application entirely on the Microsoft Azure cloud, and offers Gladinet.com, a free cloud storage service, as backup storage. Many customers put data into Word documents and Excel spreadsheets. Through its consulting services, Dedoose will help customers retrieve that data and rebuild their databases when other sources aren't available.

The Los Angeles Times reported that Dedoose officials sent an email Monday saying that both its operational and storage systems had failed. "The timing of this event was such that our entire data storage container was corrupted -- including the master database and all local backups," the company wrote in the email.

"Within minutes of discovering the problem, we contacted Microsoft Azure support. Unfortunately, Microsoft was unable to recover these data... from its servers," the Dedoose email to customers said. Microsoft officials couldn't be reached for comment.

"Since our initial communication about this issue, we have also learned that the backup files stored to an independent location were also corrupted. We are working with Gladinet services to have the non-corrupted data transferred to alternative storage locations and restore the integrity of these files," the message continued, according to the L.A. Times.

Asked who was responsible for the data loss, Azure or Dedoose, Lieber said: "It's a complicated situation." Dedoose produced its EthnoNotes system and installed it on Azure, but didn't expect that its data could be lost when it failed. He declined to estimate the respective shares of responsibility. Dedoose is still investigating why the system failed. Lieber is one of the authors of the EthnoNotes system.

"Things happened on the platform that we were blind to. Similar events have happened (to other Azure customers) and the data was not recoverable," Lieber said, without naming any other Azure users who suffered a similar failure.

Dedoose is revising its all-cloud approach to include a redundant system separate from the operational one on Azure. "We are putting measures in place so that if the platform is destroyed again, we will live on," said Lieber. "Within two weeks, our systems will be tremendously more robust."

But he added, "Ultimately, we are the responsible party. As we learn more about what happened, we are taking steps to protect our users." Lieber declined to say how many customers were affected or how many customers in total use EthnoNotes. But he said the number affected, while a minority, is a higher than the "dozens of customers" suggested by another description.

Are you better protected renting space for disaster recovery or owning a private cloud? Follow one company's decision-making process. Also in the Disaster Recovery issue of InformationWeek: Five lessons from Facebook on analytics success. (Free registration required.)

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/15/2014 | 9:03:04 PM
Dedoose takes full responsibility
I think Dedoose is taking full responsibilty here and appropriate corrective actions. The lost data is likely to damage the projects of of some of its customers and that will be upsetting to them, as well it should be. I hope customers understand the company is going through a tough phase and responding appropriately to future threats to continuous operations.
Laurianne
100%
0%
Laurianne,
User Rank: Author
5/15/2014 | 5:31:19 PM
Thanks
Thanks to Dedoose for weighing in here with your side of the backup story.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/15/2014 | 2:47:30 PM
Dedoose implements better data protection on Azure -- and Amazon
In a blog May 9 following up the crash, CEO Eli Lieber said Dedoose was implementing the following for better data protection:
  1. Deploying a database mirror/slave in Azure
  2. Deploying a database mirror/slave into Amazon S3
  3. Keeping a mirror copy of the entire blob storage including all file data, backups, video data synchronized nightly to our private server in an encrypted volume
  4. Storing nightly database backups on the VHD, Azure Blob Storage, and Amazon S3 Storage
  5. Mirroring all Azure file data into an Amazon S3 bucket
  6. Carrying out a weekly restore exercise for the database backups to ensure integrity

 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/14/2014 | 4:23:53 PM
It was all Dedoose's fault? Hmmm. Maybe....
Strong statements here from Joe Emison (we expect no less). But I'm not adopting the 0-100% split between Azure and Dedoose until I know a little bit more. For example, was any monitoring available to indicate the system was about to fail? I've asked Microsoft to respond. Still waiting. 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/14/2014 | 1:35:27 PM
Rumors of its death may be highly exaggerated
I don't know how recoverable the data is by users themselves. This is social science project data, not necessarily business transactions with customer accounts screwed up and revenue lost. I suspect some of it is recoverable or able to be reconstructed by other than the usual backup and recovery means. But I don't know for sure. The company may survive this because of the value of its analytics application, EthnoNotes. 
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

News
How CIO Roles Will Change: The Future of Work
Jessica Davis, Senior Editor, Enterprise Apps,  7/1/2021
Commentary
A Strategy to Aid Underserved Communities and Fill Tech Jobs
Joao-Pierre S. Ruth, Senior Writer,  7/9/2021
Slideshows
10 Ways AI and ML Are Evolving
Lisa Morgan, Freelance Writer,  6/28/2021
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Slideshows
Flash Poll