Prosecution Witness: UBS PaineWebber Network Still Suffering Four Years After Attack
The logic bomb had a "catastrophic impact," bringing operations to a standstill and wiping out servers around the country, according to testimony from an IT manager for the company.
She and her team of 13 IT professionals worked full-time on the incident until June 2 of that year.
Part of the problem, she said, was that about 20% of the downed servers didn't have backup tapes. That multiplied the trouble they had bringing the machines back to life and at times made it impossible to restore all the information that had been wiped out when the logic bomb was triggered at 9:30 that morning--just as trading started on the stock market.
"There were a lot of problems," said Rodriguez. "Some branches didn't have backup. There were no tapes to go to. We continued to encounter problems for the next year at least."
Rajeev Khanna, manager for UBS's Unix Systems Group at the time of the attack, also didn't get any sleep that night or for the next two nights. Khanna, who oversaw the recovery process, testified Wednesday that 400 to 500 UBS workers--application developers, project managers, systems administrators, and database administrators--were pulled off their normal jobs to work on the restoration.
"The most important thing was for users to be able to log in to their desktops," he said. "They couldn't log in. They couldn't do the work they do on a daily basis, in terms of pulling data on their clients, making trades, and checking market data."
UBS hasn't reported how much money was lost in business because of the server and broker downtime.
To avoid a repeat of the incident, Rodriguez said, for the next two or three years she prepared to fend off a similar attack before every March 4. She took critical servers offline, so if there was any malicious code still lurking on the network, at least those servers wouldn't be affected. "We had to make sure there was no more business impact," she said.
On his cross-examination of Rodriguez, Adams read down through a help desk log from the day the malicious code was triggered. While the logic bomb went off at 9:30 that morning, the log showed there were reports of much smaller incidents before that. For instance, a Sybase server was having trouble at 7:14 that morning. A user was having trouble logging in to a branch server at 7:39. And there was more trouble with the Sybase server at 8:19 a.m.
Rodriguez called the problems "routine support."
Later in his testimony, Khanna said he was generally only notified of a problem if the systems administrators were unable to handle it on their own. And he added that before 9:30 on March 4, 2002, he hadn't received calls about any trouble on the network or with the servers.
But Adams pressed Rodriguez about the company's computer security.
The defense attorney noted that in a January 2002 group internal audit report on the UBS PaineWebber IT department, it said there were issues with the company's Unix and Sybase security, specifically involving passwords.
And during Rodriguez's testimony, she said that immediately after the attack began, she stepped out of the office and used the open "root" access on another systems administrator's computer to monitor what was happening on the network.
When asked if it was company policy for an administrator to walk away and leave root access up on a computer, she said it wasn't policy, but she wasn't surprised it happened.
"I found an open session, so obviously that time [the policy] was not followed," she said, adding that she "would not be surprised" if it happened on another occasion.
And Adams asserted that a March 2000 review of the UBS virtual private network showed that another session could open under a username and password that was already in use. Rodriguez said she wasn't sure if that could be done at the time, but it can't be done now.
5 Top Federal Initiatives For 2015As InformationWeek Government readers were busy firming up their fiscal year 2015 budgets, we asked them to rate more than 30 IT initiatives in terms of importance and current leadership focus. No surprise, among more than 30 options, security is No. 1. After that, things get less predictable.
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.