"Unfortunately, it's a huge file and it's taking a long time," says a Verizon spokesman in this report about a database gone astray. Unfortunately, the file problem left 750,000 landline customers here in Southern California without any voice mail service for two days.Most backup problems aren't quite this public or monumental; lots of storage pros looked at Verizon's problem and thought "There but for the grace of god goes my data center." Your own internal end users are never too pleasant to deal with when things like voice mail aren't available. Imagine having to face down your boss's boss, and potentially, regulators and shareholders, too, thanks to a "customer-facing" incident.

The telco is tapping its last backup before the database glitch to restore all saved or pending messages (which weren't available while the service was on the blink Wednesday and Thursday). That's presumably the big-ass file that Verizon couldn't get to run fast enough.

With all the alleged mirroring, snapshotting, and continuous protection at work, it's a little surprising to see survey results that backups take on average nine hours, according to this survey. Then again, that's for tape backup, and the survey was done by a vendor of IT support services, giving the findings that Nostradamus, end-of-days tone.

I doubt tape was the problem in Verizon's case. I'm also a bit dubious over how much incidents like these will hasten the rush to hosted backup services. Seems like the bigger driver for EMC's Mozy service (for which Verizon will be a reseller) is the convenience from offloading responsibility for a major function of the data center.

Reliability's great -- but it's even better when it comes with speed.