How Did T-Mobile Suddenly Recover Unrecoverable Data?
Microsoft today said it had recovered most of affected Sidekick customers' lost data. But this past weekend, like a doctor issuing a terminal prognosis, T-Mobile told affected customers that their data "almost certainly has been lost." So, what changed between then and now?
Microsoft today said it had recovered most of affected Sidekick customers' lost data. But this past weekend, like a doctor issuing a terminal prognosis, T-Mobile told affected customers that their data "almost certainly has been lost." So, what changed between then and now?The statement at T-Mobile's site comes from Roz Ho, corporate VP of "Premium Mobile Experiences, Microsoft Corporation." (Amusingly, there is no mention in the statement of the Microsoft subsidiary, called "Danger," that runs the Sidekick data service under contract with T-Mobile. Wonder if Microsoft's damage control experts have murdered the Danger name and buried it forever?) Ho wrote: "We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data."
This is a complete reversal from the finality of T-Mobile's statement posted on Saturday, in which it told customers that if they were among those who had lost data, forget about getting it back.
In an earlier blog, readers joined in my astonishment that Microsoft wouldn't have back-up systems in place to make data recovery possible. But, now we know there were systems in place. So, why did they originally say they couldn't recover the data? Here are a few ideas:
#1 Microsoft/Danger-um, I mean-Microsoft knew it could recover the data all along, but didn't want to put the time and expense into doing that. Or, T-Mobile didn't want to pay its data services contractor, Microsoft, to put in the time and expense of doing that. But after the big backlash about the outage in the press, decided the initial decision was the wrong one.
#2 There was a communications error. When T-Mobile issued the statement on Saturday, it didn't really know what was going on, and thus, did not realize the data was recoverable. In that case, the use of such finality in the language-your data "almost certainly has been lost"-was the faux pas.
#3 Those in charge at the Microsoft (Danger!) data center on Saturday didn't know the data was recoverable. They didn't know, or didn't think about, rebuilding the system "component by component," and thus told T-Mobile to tell its customers to expect the worst.
Those are just a few ideas. Those of you who work in data centers and are more familiar with the details of backup and recovery may have better ones. Feel free to share.
Nonetheless, this situation has highlighted for both businesses and consumers, that if you are going to have your data managed "in the cloud" by another party, don't take anything for granted. Cloud vendors should present you with a specific and solid data recovery plan, and reliable, consistent communications-no matter how many vendors are involved--if customers are affected by system failures.
How Enterprises Are Attacking the IT Security EnterpriseTo learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
2017 State of IT ReportIn today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.