Prosecution Witness: UBS PaineWebber Network Still Suffering Four Years After Attack
The logic bomb had a "catastrophic impact," bringing operations to a standstill and wiping out servers around the country, according to testimony from an IT manager for the company.
She and her team of 13 IT professionals worked full-time on the incident until June 2 of that year.
Part of the problem, she said, was that about 20% of the downed servers didn't have backup tapes. That multiplied the trouble they had bringing the machines back to life and at times made it impossible to restore all the information that had been wiped out when the logic bomb was triggered at 9:30 that morning--just as trading started on the stock market.
"There were a lot of problems," said Rodriguez. "Some branches didn't have backup. There were no tapes to go to. We continued to encounter problems for the next year at least."
Rajeev Khanna, manager for UBS's Unix Systems Group at the time of the attack, also didn't get any sleep that night or for the next two nights. Khanna, who oversaw the recovery process, testified Wednesday that 400 to 500 UBS workers--application developers, project managers, systems administrators, and database administrators--were pulled off their normal jobs to work on the restoration.
"The most important thing was for users to be able to log in to their desktops," he said. "They couldn't log in. They couldn't do the work they do on a daily basis, in terms of pulling data on their clients, making trades, and checking market data."
UBS hasn't reported how much money was lost in business because of the server and broker downtime.
To avoid a repeat of the incident, Rodriguez said, for the next two or three years she prepared to fend off a similar attack before every March 4. She took critical servers offline, so if there was any malicious code still lurking on the network, at least those servers wouldn't be affected. "We had to make sure there was no more business impact," she said.
On his cross-examination of Rodriguez, Adams read down through a help desk log from the day the malicious code was triggered. While the logic bomb went off at 9:30 that morning, the log showed there were reports of much smaller incidents before that. For instance, a Sybase server was having trouble at 7:14 that morning. A user was having trouble logging in to a branch server at 7:39. And there was more trouble with the Sybase server at 8:19 a.m.
Rodriguez called the problems "routine support."
Later in his testimony, Khanna said he was generally only notified of a problem if the systems administrators were unable to handle it on their own. And he added that before 9:30 on March 4, 2002, he hadn't received calls about any trouble on the network or with the servers.
But Adams pressed Rodriguez about the company's computer security.
The defense attorney noted that in a January 2002 group internal audit report on the UBS PaineWebber IT department, it said there were issues with the company's Unix and Sybase security, specifically involving passwords.
And during Rodriguez's testimony, she said that immediately after the attack began, she stepped out of the office and used the open "root" access on another systems administrator's computer to monitor what was happening on the network.
When asked if it was company policy for an administrator to walk away and leave root access up on a computer, she said it wasn't policy, but she wasn't surprised it happened.
"I found an open session, so obviously that time [the policy] was not followed," she said, adding that she "would not be surprised" if it happened on another occasion.
And Adams asserted that a March 2000 review of the UBS virtual private network showed that another session could open under a username and password that was already in use. Rodriguez said she wasn't sure if that could be done at the time, but it can't be done now.
The Business of Going DigitalDigital business isn't about changing code; it's about changing what legacy sales, distribution, customer service, and product groups do in the new digital age. It's about bringing big data analytics, mobile, social, marketing automation, cloud computing, and the app economy together to launch new products and services. We're seeing new titles in this digital revolution, new responsibilities, new business models, and major shifts in technology spending.