New York Times Recoups After Network Failure

The New York Times was back to its regular schedule Thursday after the internationally known newspaper experienced a network failure of more than three hours Wednesday afternoon.

In a detailed internal e-mail to staffers, New York Times general manager and president, Scott Heekin-Canedy, said a network and phone outage crippled the paper's critical network-dependent systems, including ones used to "produce the news and advertising content."

"We have redundant server capabilities to protect us from the difficulties we experienced," the memo said. "They have been working effectively to manage many disruptions that have been invisible to you. But they failed the test yesterday."

According to the memo seen by InformationWeek, the backup server management software failed, leaving redundant systems waiting for word to step in and save the network.

The Times' IT staff and executives from Nortel, the media company's equipment provider, managed to restore the newspaper's systems in New York by 7:30 p.m. in time to publish its Thursday morning edition.

"The New York Times is a valued customer; however, we cannot comment on its network," a Nortel spokesperson said in an e-mail.

The New York Times did not respond to a direct request for comment.

Heekin-Canedy's memo noted that The New York Times is going over its backup plan to make it better. "We are continuing to identify, isolate, and correct the root cause software issues," the memo said. "We are already reviewing and revising our contingency plans, and we are undertaking a reconsideration of the infrastructure design."

A similar backup system failure this summer at 365 Main, a San Francisco ISP, caused similar outages at media companies including AdBrite.com, CurrentTV.com, Craigslist.org, RedEnvelope.com, SecondLife.com, Six Apart's blog sites (LiveJournal.com, TypePad.com, Vox.com), Technorati.com, and Yelp.com.