Verizon Cloud Outage Draws Overblown Angst - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Cloud // Infrastructure as a Service
Commentary
1/12/2015
10:45 AM
Charles Babcock
Charles Babcock
Commentary
Connect Directly
Twitter
RSS
100%
0%

Verizon Cloud Outage Draws Overblown Angst

Verizon Cloud's planned 40-hour outage set off outraged critics, but long-term implications will make the cloud service more stable and more competitive.

Cloud Storage: 8 Ways You're Wasting Money
Cloud Storage: 8 Ways You're Wasting Money
(Click image for larger view and slideshow.)

Verizon Cloud customers were notified that the service was slated to undergo maintenance that would take the service offline starting 1:00 a.m. this past Saturday (Jan. 10) and last for up to 48 hours. In reality, the Verizon Cloud was out of service for about 40 hours over the weekend. After it came back online, Verizon explained that the update's purpose was to enable "seamless upgrades" -- i.e., to prevent the need for downtime when future updates are made.

Verizon's announcement of the planned service outage drew intense flak from analysts last week, who said being down for up to 48 hours was "insanely stupid" for a cloud vendor, according to Rob Enderle, principal of the Enderle Group. IDC's Frank Gens said, "Forty-eight hours seems very excessive." Dan Olds at the Gabriel Group said, "A two-day outage, for any reason, is a very big deal to enterprise cloud customers. I can't recall any cloud outage that is this lengthy."

Well, Amazon's Easter outage in 2011 was closer to three days. I'm just saying... it's not that shrouded in the mists of time. And that was an unplanned, unannounced, and unprepared-for outage, which is different from what Verizon did.

[Does your SLA cover downtime? See Amazon SLAs Didn't Cover Major Outage.]

In the interest of having more than the three cloud suppliers that Microsoft seems to think is the maximum allowed, I harrumphed loudly at the nature of Verizon's customer notice and was glad it did what needed to be done and got it over with.

In the group of big suppliers, Verizon Cloud is the youngest of the Amazon, Google, Microsoft, Rackspace set. It became generally available only in October 2014 after 12 months of beta operation. It represents a fraction of the enterprise cloud customers that Verizon acquired when it purchased Terremark in January 2011. Terremark's customer base represents the bulk of Verizon's Enterprise Cloud customers, who would resent a two-day outage, but they were not affected by the upgrade to Verizon Cloud. They represent the users of a legacy cloud -- as much managed hosting system as self-provisioning cloud -- that Verizon is trying to evolve away from, but wisely has left intact for now.

I disagreed with the redoubtable David Linthicum when he said, "Verizon's enterprise cloud will go offline for as long as two days starting Saturday, Jan. 10" (emphasis added). If anything, the majority of parties affected are consumers, small businesses, and developers, not enterprises. If Verizon actually were cavalierly shutting down enterprise customers, I guess I'd be indignant, too. But the Verizon Enterprise Cloud (primarily the legacy Terremark customer base) was not affected. Nor were Enterprise Cloud Managed Edition and Enterprise Cloud Federal Edition, again part of the Terremark legacy.

In fact, there are some businesses among those customers being shut down, and they were justifiably concerned about being offline. But I suspect they, too, want Verizon to upgrade its cloud operations software, if only to make this sort of maintenance outage unnecessary in the future. That's a primary goal of this upgrade. As I said, by all means, Verizon, get on with it.

Not everyone is as irritated as the foregoing pundits. In a comment on a GigaOm story, "Verizon Cloud warns customers of a 2 day (!) maintenance closure," Carl Brooks of the 451 Group said, "This is a whole bunch of nothing, IMO. I think it's lousy PR on their part but it's not some grand failure of design. I seriously doubt anyone at Verizon is going to mash the big red button on Verizon Cloud until Monday morning. They're telling a small subset of their customers to [get out] while they do some planned maintenance, which is miles better than the usual practices from cloud providers."

In a check with Verizon, spokesman said "less than 10%" of its cloud customers were affected. That probably included managed hosting customers as well -- strictly speaking, not cloud users.

Brooks, in his comment to the GigaOm story, added: "You know what [Amazon Web Services] says to users before it upgrades and reboots a bunch of its platform? Nothing."

Well, it's possible AWS migrates its customers' virtual machines to an availability zone or data center that won't be affected by the maintenance, rather than forcing them to shut down. I haven't heard of AWS staging 48 hours of downtime for a software changeover. So I'd be cautious on that point. The fact that Verizon has had to do so is scarcely a triumph of cloud services marketing.

The commentator that I most appreciated on this issue is the one who sounded the most well-informed on it. Forrester's James Staten, in an email exchange in advance of the outage, told me: "Verizon has been in the IaaS cloud business for several years and with different underlying architectures. Most of these solutions up to this point have been more traditional hosting in design and architecture than IaaS (meaning less self-service, elastic, API controlled, etc.)...

"Their latest, Verizon Cloud, is more the latter but a very new offering. As such there are some elements of the architecture that were not optimized at first release. One such issue is not having the ability to apply firmware changes non-disruptively. This outage will add this capability."

For those of us who want to see viable service providers in the marketplace -- in addition to Amazon, Microsoft, and Google -- this upgrade was the best way for Verizon to get there. The consequences of staging it in this manner are between Verizon and its customers. Post-outage, I'd be surprised to hear from a large group of outraged customers who say they're mad as hell and not going to take it any more (but don't hold your breath).

If Verizon gets to a more competitive position, and customers get Verizon's promised continuous service in the future, then I'm glad they got on with it, and I'll try to restrain my temptation to snipe.

Verizon Cloud offers three types of service.

  • Reserved service, which comes with service level agreements for individuals and companies deploying their workloads to it
  • Elastic services, managed by Verizon and allocated to existing (Reserved) workloads when they need to scale -- and there are resources available for them to do so
  • Research pools, which are basically test-and-dev servers run to a scale as the customer sees fit, as resources allow; no SLAs available with Research Pools

Staten noted: "Of these three resource types, only Reserved is affected by this outage. Granted it is the primary resource type, but most customers have not committed to these resources yet. As you would expect, companies tend to test things out on Research before deploying to Reserved, just as you would in-house by building the app in test & dev, then moving it later to production."

Speaking of SLAs, until recently the best SLA you could get from cloud providers, including Amazon, was a guarantee of 99.9% availability (which was improved over the last two years to 99.95%). Under that earlier standard, up to 21.5 hours of unplanned, unannounced downtime was allowed in any 12-month period, even if all 21.5 hours occurred in one month. The planned Verizon Cloud outage essentially doubled what has been the accepted SLA for unplanned outages

As Staten said in his message: "Given the early state of Verizon Cloud and that the purpose of the upgrade is to prevent these [maintenance downtimes] going forward, I think the concerns being raised are a bit overblown." 

Apply now for the 2015 InformationWeek Elite 100, which recognizes the most innovative users of technology to advance a company's business goals. Winners will be recognized at the InformationWeek Conference, April 27-28, 2015, at the Mandalay Bay in Las Vegas. Application period ends Jan. 16, 2015.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
1/13/2015 | 1:22:57 PM
Verizon's entry too tentative
Does anyone remember the ins and outs of what Amazon did three months into general availability of EC2? No, because it was a new service granted a little leeway for shakedown. Part of Verizon's problem is that the standard for cloud computing has risen as it somewhat tentatively stuck its toe in the market. Now it's trying to play catch-up. Future upgrades will occur in background as customers' systems continue running, it says.
danielcawrey
50%
50%
danielcawrey,
User Rank: Ninja
1/13/2015 | 12:59:06 PM
Re: Verizon completes upgrade in 40 hours
It's really not a good idea for a cloud service to be down for any amount of time. 

At least Verizon is making it clear the point here is to limit future outages. Nevertheless, I don't think this was a very smart move for a service that really should be up at all times. That's the point of offering the concept of the cloud to customers. 
Somedude8
50%
50%
Somedude8,
User Rank: Ninja
1/13/2015 | 10:55:24 AM
Its just not right
What this thing comes down to for me are these:

1) I don't trust the phone or cable companies. Far too many bad experiences, and far too long of a track record of taking the evil option when given the choice.

2) Storing enterprise data at the phone company seems like a very strange fit, about like asking Pizza Hut to deliver some bandwidth along with that large pepperoni.

With these two considerations dominating my thought process, the idea of a 2 day scheduled downtime makes total sense. Its exactly the kind of 'just does not get it' action I would expect. There will be more in the future.
eekygeeky
50%
50%
eekygeeky,
User Rank: Apprentice
1/12/2015 | 8:11:20 PM
Clarification
My comment about AWS was not meant to convey the impression that AWS pulls the rug out from customers without notice; rather it was meant to convey that most of AWS' maintenance is totally invisible to the end users and it can and does make major changes to the platform without informing users. I still maintain that this is a tempest in a teapot. 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
1/12/2015 | 7:03:33 PM
Did you hear that, Verizon?
And I fervently hope they don't do it again or I will suffer angst myself. My name will be mud. 
David Wagner
50%
50%
David Wagner,
User Rank: Strategist
1/12/2015 | 6:58:46 PM
Re: Verizon completes upgrade in 40 hours
@Tom- Right. and you have to take them at their word that this update means all other updates won't require this. one assumes as technology moves, it might be required again. That said, planned outages are what they are-- standard practice in IT for decades.
Thomas Claburn
50%
50%
Thomas Claburn,
User Rank: Author
1/12/2015 | 6:34:16 PM
Re: Verizon completes upgrade in 40 hours
I'm inclined to agree that the outage was overblown, but at the same time, it makes me wonder about the viability of cloud services in industries or situations where delays are unacceptable. You couldn't have hospital systems, for example, just shut down for 40 hours without a contingency plan. 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
1/12/2015 | 2:33:24 PM
Verizon completes upgrade in 40 hours
The upgrade took longer than the most optimistic predictions, about 40 hours, but less than the 48 allotted. When it was completed, Verizon acknowledged it was staged to eliminate the need for such blackout upgrades in the future:  "All maintenance and upgrades to Verizon Cloud will now happen in the background with no impact to customers," spokesman Kevin King wrote in a blog on the Verizon site this morning. http://news.verizonenterprise.com/2015/01/cloud-upgrade-seamless-verizon-platform/
Slideshows
9 Steps Toward Ethical AI
Cynthia Harvey, Freelance Journalist, InformationWeek,  5/15/2019
Commentary
How to Assess Digital Transformation Efforts
Lisa Morgan, Freelance Writer,  5/14/2019
Commentary
Is AutoML the Answer to the Data Science Skills Shortage?
Guest Commentary, Guest Commentary,  5/10/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
A New World of IT Management in 2019
This IT Trend Report highlights how several years of developments in technology and business strategies have led to a subsequent wave of changes in the role of an IT organization, how CIOs and other IT leaders approach management, in addition to the jobs of many IT professionals up and down the org chart.
Slideshows
Flash Poll