Commentary

Bob Evans
Senior VP, Global CIO  

Global CIO: IBM's Bank Outage: Anatomy Of A Disaster

IBM personnel inadvertently triggered a 7-hour outage at Singapore's largest banking network last month by using unapproved procedures. Here's a detailed look at what went wrong.

At 2:58 a.m. on the morning of July 5, disaster struck for Singapore's largest banking network and a small IBM support team troubleshooting a communications problem between one of the bank's storage devices and its mainframe. Here's how the bank, DBS Group, and its IT services provider, IBM, described that harrowing moment, which occurred after 40 hours of on-and-off attempts to fix the problem:

"[The on-site IBM engineer] replaced the cable using the same procedures as before. This caused errors that threatened data integrity. As a result, the storage system automatically stopped communicating with the mainframe computer, to protect the data. At this point, DBS banking services were disrupted."


More Global CIO Insights

White Papers

More >>

Reports

More >>

Webcasts

More >>

Disrupted, indeed: for the following seven hours—until 10:00 a.m.—DBS's customers were unable to access banking services via branches, ATMs, online, or mobile. One media report said DBS has about 1,000 ATMs and, early in 2008, had almost 1,000,000 online-banking customers.

And while bank CEO Piyush Gupta 8 days later issued a long and deeply apologetic letter to DBS Group's customers in which he took full responsibility for the outage and resulting customer inconvenience and loss of trust, IBM has also been embroiled in the controversy over what happened, why it happened, and how it can be prevented in the future.

What we do know at this point is that the outage will cost DBS Group a great deal more than the negative impact on customers as the government agency that oversees the banking industry, the Monetary Authority of Singapore, has ordered DBS to place $230 million in regulatory capital as a result of the outage.

What is not known—at least publicly—is whether IBM will have to compensate DBS for the cost of the outage and/or related costs. In two separate media reports from a joint press conference involving both DBS and IBM, both articles said the companies would not answer questions about that possibility.

Global CIO
Global CIOs: A Site Just For You
Visit InformationWeek's Global CIO -- our new online community and information resource for CIOs operating in the global economy.

According to ChannelNewsAsia.com's coverage of the press conference, IBM regional general manager Cordelia Chung said that "the personnel directly involved with this incident have been removed from direct customer support activity and disciplined" and that "IBM has taken steps to enhance the training of all related personnel on the most current procedures."

And the BusinessTimes.com.sg article quoted Chung as saying, "We have also taken steps to review installations of the same storage system at other financial institutions in Singapore for whom we provide maintenance services."

(If I may inject an opinion here: if I were DBS CEO Gupta, I'd have very mixed feelings about that preventive-maintenance approach taken by IBM not only on behalf of DBS's competitors but also on the shoulders of a very troublesome incident for DBS. That's why I would guess that while neither company would comment on whether IBM would be compensating DBS for its role in the crash, IBM's going to be paying DBS very generously in either cash or extended and comprehensive additional services. On top of that, the removal from the involved employees from future dealings with customers, plus disciplinary action against them, plus Chung's very public apologies to both DBS and its customers all seem to add up to a very uncomplicated admission by IBM of some level of culpability in the outage.)

The article goes on to quote Chung as describing IBM's top priority once they realized a crash had occurred:


Page 2:  Integrity Of Customer Data
 1 | 2 |Next Page » 

Related Reading




Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

InformationWeek encourages readers to engage in spirited, healthy debate, including taking us to task. However, InformationWeek moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing/SPAM. InformationWeek further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
T-Shirt Giveaway T-Shirt Giveaway: Each week we're selecting one great comment from our readers. The author of the comment will receive an InformaitonWeek Community t-shirt. So get posting!
Subscribe to RSS

Resource Links