Verizon Cloud Outage Draws Overblown Angst

Cloud Storage: 8 Ways You're Wasting Money

Verizon Cloud customers were notified that the service was slated to undergo maintenance that would take the service offline starting 1:00 a.m. this past Saturday (Jan. 10) and last for up to 48 hours. In reality, the Verizon Cloud was out of service for about 40 hours over the weekend. After it came back online, Verizon explained that the update's purpose was to enable "seamless upgrades" -- i.e., to prevent the need for downtime when future updates are made.

Verizon's announcement of the planned service outage drew intense flak from analysts last week, who said being down for up to 48 hours was "insanely stupid" for a cloud vendor, according to Rob Enderle, principal of the Enderle Group. IDC's Frank Gens said, "Forty-eight hours seems very excessive." Dan Olds at the Gabriel Group said, "A two-day outage, for any reason, is a very big deal to enterprise cloud customers. I can't recall any cloud outage that is this lengthy."

Well, Amazon's Easter outage in 2011 was closer to three days. I'm just saying... it's not that shrouded in the mists of time. And that was an unplanned, unannounced, and unprepared-for outage, which is different from what Verizon did.

[Does your SLA cover downtime? See Amazon SLAs Didn't Cover Major Outage.]

In the interest of having more than the three cloud suppliers that Microsoft seems to think is the maximum allowed, I harrumphed loudly at the nature of Verizon's customer notice and was glad it did what needed to be done and got it over with.

In the group of big suppliers, Verizon Cloud is the youngest of the Amazon, Google, Microsoft, Rackspace set. It became generally available only in October 2014 after 12 months of beta operation. It represents a fraction of the enterprise cloud customers that Verizon acquired when it purchased Terremark in January 2011. Terremark's customer base represents the bulk of Verizon's Enterprise Cloud customers, who would resent a two-day outage, but they were not affected by the upgrade to Verizon Cloud. They represent the users of a legacy cloud -- as much managed hosting system as self-provisioning cloud -- that Verizon is trying to evolve away from, but wisely has left intact for now.

I disagreed with the redoubtable David Linthicum when he said, "Verizon's enterprise cloud will go offline for as long as two days starting Saturday, Jan. 10" (emphasis added). If anything, the majority of parties affected are consumers, small businesses, and developers, not enterprises. If Verizon actually were cavalierly shutting down enterprise customers, I guess I'd be indignant, too. But the Verizon Enterprise Cloud (primarily the legacy Terremark customer base) was not affected. Nor were Enterprise Cloud Managed Edition and Enterprise Cloud Federal Edition, again part of the Terremark legacy.

In fact, there are some businesses among those customers being shut down, and they were justifiably concerned about being offline. But I suspect they, too, want Verizon to upgrade its cloud operations software, if only to make this sort of maintenance outage unnecessary in the future. That's a primary goal of this upgrade. As I said, by all means, Verizon, get on with it.

Not everyone is as irritated as the foregoing pundits. In a comment on a GigaOm story, "Verizon Cloud warns customers of a 2 day (!) maintenance closure," Carl Brooks of the 451 Group said, "This is a whole bunch of nothing, IMO. I think it's lousy PR on their part but it's not some grand failure of design. I seriously doubt anyone at Verizon is going to mash the big red button on Verizon Cloud until Monday morning. They're telling a small subset of their customers to [get out] while they do some planned maintenance, which is miles better than the usual practices from cloud providers."

In a check with Verizon, spokesman said "less than 10%" of its cloud customers were affected. That probably included managed hosting customers as well -- strictly speaking, not cloud users.

Brooks, in his comment to the GigaOm story, added: "You know what [Amazon Web Services] says to users before it upgrades and reboots a bunch of its platform? Nothing."

Well, it's possible AWS migrates its customers' virtual machines to an availability zone or data center that won't be affected by the maintenance, rather than forcing them to shut down. I haven't heard of AWS staging 48 hours of downtime for a software changeover. So I'd be cautious on that point. The fact that Verizon has had to do so is scarcely a triumph of cloud services marketing.

The commentator that I most appreciated on this issue is the one who sounded the most well-informed on it. Forrester's James Staten, in an email exchange in advance of the outage, told me: "Verizon has been in the IaaS cloud business for several years and with different underlying architectures. Most of these solutions up to this point have been more traditional hosting in design and architecture than IaaS (meaning less self-service, elastic, API controlled, etc.)...

"Their latest, Verizon Cloud, is more the latter but a very new offering. As such there are some elements of the architecture that were not optimized at first release. One such issue is not having the ability to apply firmware changes non-disruptively. This outage will add this capability."

For those of us who want to see viable service providers in the marketplace -- in addition to Amazon, Microsoft, and Google -- this upgrade was the best way for Verizon to get there. The consequences of staging it in this manner are between Verizon and its customers. Post-outage, I'd be surprised to hear from a large group of outraged customers who say they're mad as hell and not going to take it any more (but don't hold your breath).

If Verizon gets to a more competitive position, and customers get Verizon's promised continuous service in the future, then I'm glad they got on with it, and I'll try to restrain my temptation to snipe.

Verizon Cloud offers three types of service.

Staten noted: "Of these three resource types, only Reserved is affected by this outage. Granted it is the primary resource type, but most customers have not committed to these resources yet. As you would expect, companies tend to test things out on Research before deploying to Reserved, just as you would in-house by building the app in test & dev, then moving it later to production."

Speaking of SLAs, until recently the best SLA you could get from cloud providers, including Amazon, was a guarantee of 99.9% availability (which was improved over the last two years to 99.95%). Under that earlier standard, up to 21.5 hours of unplanned, unannounced downtime was allowed in any 12-month period, even if all 21.5 hours occurred in one month. The planned Verizon Cloud outage essentially doubled what has been the accepted SLA for unplanned outages

As Staten said in his message: "Given the early state of Verizon Cloud and that the purpose of the upgrade is to prevent these [maintenance downtimes] going forward, I think the concerns being raised are a bit overblown." 

Apply now for the 2015 InformationWeek Elite 100, which recognizes the most innovative users of technology to advance a company's business goals. Winners will be recognized at the InformationWeek Conference, April 27-28, 2015, at the Mandalay Bay in Las Vegas. Application period ends Jan. 16, 2015.