Software Bug Linked To Blackout

NEW YORK (AP) -- A programming error has been identified as the cause of alarm failures that might have contributed to the scope of last summer's Northeast blackout, industry officials said Thursday.

Ralph DiNicola, spokesman for FirstEnergy Corp., said the utility has since applied fixes developed by the system's vendor, General Electric Co., and has accelerated plans to replace GE's system with a system from French nuclear engineers Areva SA.

A U.S.-Canadian task force investigating the blackout said in November that FirstEnergy employees failed to take steps that could have isolated utility failures because its data-monitoring and alarm computers weren't working.

Without a functioning emergency management system or the knowledge that it had failed, the company's system operators "remained unaware that their electrical system condition was beginning to degrade," the report said.

At the time, task force members said it remained unclear whether the software malfunctioned or if FirstEnergy's computers had difficulty running it that day.

DiNicola said Thursday that the company, working with GE and energy consultants at Kema Inc., had pinned the trouble on a software glitch by late October and completed its fix by November 19, coincidentally the same day the task force issued its report.

GE Energy spokesman Dennis Murphy said the company distributed a warning and a fix to its more than 100 other customers the following day.

DiNicola said FirstEnergy had informed the task force at the time; the company went public with it this week in a report on the Web site SecurityFocus.

The discovery of the programming error took "weeks of going through millions of lines of data," DiNicola said.

The failures occurred when multiple systems trying to access the same information at once got the equivalent of busy signals, he said. The software should have given one system precedence.

With the software not functioning properly at that point, data that should have been deleted were instead retained, slowing performance, he said. Similar troubles affected the backup systems.

Joseph Bucciero, senior vice president for transmission services at Kema, said the public should not lose confidence in utilities.

"There are a lot of systems out there and they are running for many years already, and this is the first time a problem like this has arisen," he said.

Bucciero said the software bug surfaced because of the number of unusual events occurring simultaneously -- by that time, three FirstEnergy power lines had already short-circuited.

The GE system at FirstEnergy was a 1996 model, DiNicola said. The newer system from Areva will be installed at two locations, Akron, Ohio, and Reading, Pa., for redundancy, he said.