Commentary

Jonathan Feldman
 

RIM's Death By Speculation

After a botched explanation for the BlackBerry outage, RIM customers are imagining the worst. RIM must take two steps right away to regain IT's trust.

RIM's choice to stay vague about the details behind the almost four-day outage during Thursday's conference call update has generated lots of speculation about what, exactly, went wrong. Precise communication would have made things more clear, but RIM kind of dropped the ball on that one. Customers are now left to speculate about things that might be far worse than the reality.

It's easy to be sympathetic to RIM, especially if you've worked for any length of time in enterprise IT. When it comes to outages, we've all been there. The system is down, users are wondering, and it's hard to know what to do. But this type of situation is IT 101. Support handles communication, infrastructure folks handle the problem. Is that so hard?


More Global CIO Insights

White Papers

More >>

Reports

More >>

Webcasts

More >>

RIM's major mishandling of this incident is that they focused on having top management in the trenches working the problem instead of communicating. RIM positioned this as heroics. During the conference call, listeners questioned why it took so long to start communicating, why co-CEO Mike Lazaridis took until Thursday morning to explain, given that the outage started on Monday. Jim Balsillie, co-CEO, jumped on defense. "I'll speak for Mike: He was directly commanding the team, nobody's gone home since Monday."

Dude. You are talking to the most sympathetic audience in the world, don't mess it up. Again, there's not a CIO on the planet that doesn't understand how it feels to have a major outage going on. So much is vague. So much is unknown. So much pressure. You try everything. So, all you had to do was have someone who isn't working on the problem make the apology or announcements. You do have a co-CEO, don't you? And for organizations that don't, surely there's someone who is not in the trenches working on the problem.

What about those marketing folks? If you're in the trenches, someone else needs to communicate. You haven't slept. You're in no shape to communicate. And, you're so focused on solving the problem that you would NEVER waste time taking your eye off the ball to send out a press release or go on video. Of course someone else needs to do that. That's what your business partners are for, and it would be dysfunctional to expect infrastructure folks to do that. That's how IT has sensibly evolved over the last 20 years.

It's a simple mechanism. But RIM screwed it up. And that's why InformationWeek readers and others have started to speculate. After all, they had no good information.

As InformationWeek's Laurianne McLaughlin pointed out earlier today, the Twittersphere is all abuzz with complaints and cracks about RIM. (New definition of Crackberry?) But the folks on the private mailing lists that I'm on are even more abuzz.

On one private list, a CIO summed up the group's feelings well, saying that RIM's architecture "was just what was needed when there was low bandwidth and no mobile apps." Despite RIM's statements yesterday to the contrary, he went on to say, "this problem highlights how bad a single point of failure architecture is and why it should be avoided."

One CTO for an ISP observed, in the information vacuum, "it appears that their services are very regionalized with little or no redundancy in other regions. They tried to imply they have redundancy by stating they have multiple data centers around the globe, but it's obvious that each region runs through one or a few data centers in that region so they've only split the problem into smaller pieces, but haven't solved it."

Global CIO
Global CIOs: A Site Just For You
Visit InformationWeek's Global CIO -- our online community and information resource for CIOs operating in the global economy.

And a reader comment on my story yesterday speculated that "they've laid off a majority of their IT staffs that built and maintain their networks for cheaper India laborers who are mostly right out of school. When things go wrong they continue to cast blame and fire the seasoned IT guys on the ground. If you speak up and tell the emperor he has no clothes then you get fired too." RIM brushed this type of concern aside yesterday, but they stopped short at specifics. Earth to RIM: specifics are VERY needed right now.

If RIM is going to survive this debacle, they need to do two things. First, take a course in IT 101 communications. This speculation is killing them. Second, RIM needs to be very specific about how they will be testing in the future.

Netflix regularly throws a "chaos monkey" into their software infrastructure, randomly breaking some of the infrastructure. That's the ONLY way that you really know whether something is going to fail over or not.

Yesterday's bonked up backup switch was inexcusable, and it would have been caught had RIM tested it. RIM needs to adopt the open communication practices of companies like NetFlix. It may be operationally and culturally difficult, but RIM must do so to regain user trust.

Jonathan Feldman is a contributing editor for InformationWeek and director of IT services for a rapidly growing city in North Carolina. Write to him at jf@feldman.org or at @_jfeldman.

Virtualization support, memory, and bandwidth are in, our annual State of Server Technology Survey finds. Download the issue now. (Free registration required.)


Related Reading




Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

InformationWeek encourages readers to engage in spirited, healthy debate, including taking us to task. However, InformationWeek moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing/SPAM. InformationWeek further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
T-Shirt Giveaway T-Shirt Giveaway: Each week we're selecting one great comment from our readers. The author of the comment will receive an InformaitonWeek Community t-shirt. So get posting!
Subscribe to RSS

Resource Links