Exclusive: Anatomy Of A Brokerage IT Meltdown

The network slowdown was one of the first clues that something was amiss at GunnAllen Financial, a now defunct broker-dealer whose IT problems were only a symptom of widespread mismanagement and deeper misconduct at the firm.

It was the spring of 2005. Over a period of roughly seven business days, traffic had slowed to a crawl at the Tampa, Fla.-based firm, which had outsourced its IT department to The Revere Group. GunnAllen's acting CIO, a Revere Group partner, asked a member of the IT team to investigate.

Dan Saccavino, a former Revere Group employee who at the time served at GunnAllen as the IT manager in charge of the help desk, laptops, and desktops, says he and another network engineer eventually pinpointed the cause of the slowdown: A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem. As a result, none of the company's trades, emails, or phone calls were being archived, in violation of Securities and Exchange Commission regulations.

Despite the fact that at least five people at The Revere Group knew about the engineer's action, it's unclear whether it was reported at the time to GunnAllen or regulators. The SEC didn't reference the incident in a subsequent announcement about a settlement with GunnAllen for unrelated privacy and data security violations, and interviews with former Revere Group employees reveal that regulators may have known about only a fraction of the data security failures at the firm.

What follows is a chronicle of one firm's myriad IT and other missteps over a period of at least four years, as related by former employees and various official documents. It's a cautionary tale of what happens when a company tosses all IT responsibility over a wall and rarely peeks back. It also reveals what happens when an IT outsourcing vendor gets in over its head, and it points to the failures of regulators to identify and clean up a corporate mess on a grand scale.

While these missteps go back as far as seven years, they have continuing relevance today in the context of how businesses oversee outsourcing, information security, regulatory, and employee matters.

Rogue Home Router

Why would a network engineer route all of his employer's traffic through his home RoadRunner cable modem? "You can direct where your traffic is going, and we found out that he'd sent the traffic home to ensure that his routing patterns at work were correct," Saccavino told InformationWeek in a recent interview. But after a week, Saccavino said, he'd forgotten to turn it off.

During the week or so in 2005 that all brokerage traffic was being piped through the home router, the data being sent by GunnAllen's 200 or so employees included bank routing information, account balances, account and social security numbers, and customers' home addresses and driver's license numbers, says Roger Sago, a former Revere Group SQL Server database administrator who was working at the GunnAllen offices at the time. Sago was in charge of defining the data stream to and from Pershing (a unit of Bank of New York Mellon that provides prime brokerage and other services to financial services organizations), which involved thousands of transactions per day. "They transmitted it over the system, online, to the clearinghouse, and if anyone had access to that data ... the ramifications would be huge," Sago said. "There's enough data there that a person could run off and live forever off of what they found."

Sago contacted InformationWeek, saying that the SEC's 2011 settlement announcement relating to prior information security and privacy failures at GunnAllen had failed to mention additional security breaches at the firm. By way of background, Sago filed a civil action--since settled--against The Revere Group and GunnAllen in December 2008, alleging that he'd been unfairly laid off. During the course of that lawsuit, Sago says he learned about the undisclosed breaches from other former employees. Because such security breaches must be reported to the relevant authorities, Sago says he brought them to the attention of The Revere Group and GunnAllen lawyers involved in his case and asked them to respond within 30 days--and preferably, to report the incidents to the relevant authorities.

When neither responded, according to Sago, he says he then alerted the Federal Trade Commission, the Financial Industry Regulatory Authority (FINRA), the SEC, and attorneys general in the 42 states where GunnAllen had conducted business.

Negligence, Incompetence, or Sabotage?

Other former IT staffers, in interviews with InformationWeek, confirmed Sago's assertions, saying the home router incident was indicative of a pattern of either security negligence or incompetence--or possibly sabotage--at GunnAllen, much of which could be traced to the previously mentioned senior network engineer. "The network would get screwy over the weekend ... then [he] would show up, and five minutes in on a Monday, he'd fix the problem," Saccavino said.

It's the opinion of Thomas Lynott, a former senior systems engineer at Revere Group who worked at GunnAllen at the time, that the network engineer's actions suggested a pattern of sabotage. "He'd purposefully break things, then come in in the morning and be the hero," Lynott claimed. "I ended up key-logging all the servers, and I logged him logging in from home at 2:30 in the morning, logging on to BlackBerry servers and breaking them."

After the router incident was brought to the attention of the acting CIO, the offending network engineer received a "written warning and corrective action plan" from his manager, Jerome DiMarzio, the Revere Group IT operations manager assigned to GunnAllen. DiMarzio reported to the acting CIO.

DiMarzio's "confidential memorandum" to the network engineer--dated August 24, 2005, and copied to the acting GunnAllen CIO as well as a Revere Group HR official--outlines episodes involving "insubordination and/or indifference" as well as "dereliction of duty," including failure to obtain formal change control permission for undertaking BlackBerry server maintenance, rebooting the Cisco Call Manager, rebooting the domain controller "without ensuring it had fully recovered," and "changing the default gateway for Exchange."

The memorandum, which DiMarzio confirmed as legitimate, also accused the network engineer of "purposely pulling a cable out of a production environment in order that you would not have to travel to Jacksonville to attend an HP event at the request of the CIO." It also accused him of failing to identify the root cause of problems, including a Microsoft Exchange "data store corruption" and "BlackBerry server MAPI profile loss," and failing to note that logging had been disabled on the company's WatchGuard firewalls.

Officials from The Revere Group, including president and COO Todd Miller and CEO Michael Parks, didn't respond to multiple email requests from InformationWeek to comment on the episodes detailed by the former employees. Our multiple calls to The Revere Group seeking comment also weren't returned.

Keep Quiet

Lynott said he'd been brought in to clean up one case in which the engineer pulled tables from a server to crash it so he could skip an offsite meeting. "But if you pull tables out, you can corrupt data, transactions, all sorts of stuff," Lynott said. "I don't know whether or not it was intentionally turned off or due to incompetence, but the end result was when I brought it to their attention, I was told to turn it on and not tell anybody. We were told on so many occasions not to tell anyone anything."

Another alleged incident at GunnAllen involved a database that had been set to disable email logging, though SEC regulations require broker-dealers to retain copies of all emails for seven years. "For email, they did all the transaction logging, where they'd send all mail incoming and outgoing and they'd log it all offsite," Lynott said. "There was a point in time for probably two months where no one's email was logged. I brought it up in a meeting once and was told to shut up [by the acting CIO]," he said.

"The protocol from the CIO was to shut your mouth, don't say anything, and just brush it under the rug," Lynott said.

The former acting CIO of GunnAllen didn't respond to our requests for comment, sent via LinkedIn. Revere Group officials didn't respond to our request for the former CIO's current contact information.

Microsoft Threatens Shutdown

Not all of GunnAllen's alleged IT missteps had SEC implications. One incident detailed by two former employees involved unpaid Microsoft SQL Server licenses. The Revere Group had been receiving from Microsoft license-renewal bills for GunnAllen, which the acting CIO had ignored, according to the two former employees. Ultimately, Microsoft issued a final warning with a bill for about $20,000, saying it would disable the license at a specified date and time. "It was like an hour or two before the deadline--before Microsoft shuts the SQL servers down, which would bring GunnAllen to its knees," Saccavino recalled.

That ultimatum led one of DiMarzio's employees to contact Microsoft and share GunnAllen's licensing details. But two former employees say the acting CIO at the time, when given the licensing news and bill by the employee, threatened to fire the employee if he spoke of the matter again.

In another incident, DiMarzio relates that an internal network project plan he first delivered in the spring of 2005 to help GunnAllen comply with the Sarbanes-Oxley Act was dismissed by the acting CIO. When, in January 2006, DiMarzio heard that he was to be replaced, he reached out to a GunnAllen executive for perspective on the matter and was allegedly told that the acting CIO's incentive plan included a bonus tied to the longevity of the SOX project. DiMarzio says he also heard at the time that multiple GunnAllen executives, fed up with the pace of the SOX work, were calling for The Revere Group's contract to be canceled.

DiMarzio said he immediately held a conference call with a Revere Group HR official, as well as the acting CIO's boss at The Revere Group, to bring the concerns to their attention. He also requested that the meeting participants not identify him to the acting CIO as the source of the information. But the next week, DiMarzio said, he was called into the acting CIO's office and told to sign a resignation letter in exchange for receiving severance benefits. DiMarzio said he signed the letter and never looked back.

In September 2008, Sago said, he too was dismissed by The Revere Group. Revere ascribed the layoff to declining market conditions, though Sago said that after strong work reviews during his 11 years with the company, he was offered only two weeks of severance pay instead of the standard two weeks per year of employment. Sago rejected the severance offer, filed a civil action for harassment, retaliation, and unfair treatment, and entered into arbitration with his former employer in October 2009, at which time he says he learned about the home router incident, among other IT incidents. Ultimately, Sago and The Revere Group settled out of court.