USPS Played Cat And Mouse With Cyber Attacker

H-1B Visa Program: 13 Notable Statistics

When US Postal Service (USPS) officials received word about a major network intrusion earlier this year, one of its first instructions was to take no immediate action.

In an effort to prevent the intruders from knowing they had been discovered, the postal service's Office of the Inspector General advised the USPS's corporate information security officer Charles McGann not to initiate any mitigation measures. That included such actions as network scanning, reimaging systems, resetting passwords, taking systems offline, or searching for IP addresses.

Instead, for several weeks investigators from the postal service, the US Computer Emergency Response Team (US-CERT), and the FBI Service worked quietly to determine the scope and nature of the intrusion before finally shutting it down almost two months later.

[What should you keep to yourself about a hack? Read NOAA Blames China In Hack, Breaks Disclosure Rules.]

Randy Miskanic, VP of the secure digital solutions group at the postal service, outlined details of the high-stakes cat-and-mouse game to a subcommittee of the House Committee on Oversight and Government Reform this week.

"From the technical perspective, experts within the Postal Service and from supporting agencies provided prudent warnings that short-term remediation efforts would be seriously compromised if the threat actor became aware that the intrusion had been discovered," Miskanic said in written testimony.

"If provided advance warning of network actions intended to expel and block the intruder from the Postal Service network, the adversary could take bolder steps to further infiltrate or sabotage systems," he added. The potential of greater damage or sabotage heavily influenced the postal service's decision to delay notification and public disclosure of the breach.

It's unclear if Miskanic's explanation will help assuage criticism that has been directed at the USPS over its handling of a breach that exposed data on some 800,000 employees and 2.9 million customers. But his testimony provides a glimpse into the struggles that organizations face dealing with an intrusion by a sophisticated adversary.

According to Miskanic, the US Postal Service first learned of a potential intrusion on Sept. 11, after being alerted to it by the Inspector General's office.

Over the next several days, members of the investigative team quietly installed monitoring devices and performed forensic imaging on the four servers that were initially believed to be the only affected systems. They later configured and installed what Miskanic described as the "technical architecture and tools" necessary to understand the full scope of the breach.

That effort revealed another 29 servers and three Postal Service user accounts that had been compromised. Because of the broadening scope of the incident, the Postal Service then decided to seek the help of the US Department of Defense's Cyber Crime Center.

It wasn't until October 7, nearly a month after being first alerted to the intrusion, that investigators found signs that a large encrypted data file had been copied from one of the compromised systems and transferred to an external system.

It took another several days for investigators to determine that the file potentially contained personally identifiable information on all postal service employees, as well as recent retirees. Around this time, the postal service finally decided to bring in private sector experts in intrusion detection and remediation to assist in the effort to shut down the breach.

Around mid-October, postal services CIO James Cochrane decided to invoke the Mass Data Compromise Response Plan and set up a formal incident response center for coordinating investigation, mitigation, and incident communication activities. Also in mid-October, the FBI's cyber unit provided a Top Secret briefing to command center leadership, again emphasizing the sophisticated nature of the adversary and the need for operational secrecy, Miskanic said.

The FBI also warned that "implementing mitigation activities or communicating the threat to employees or the public at that point could result in the threat being further embedded into the Postal Service network," he said.

On November 7, Cochrane's organization finally activated a remediation plan, developed in conjunction with US-CERT and private firms, to remove the threat.

The operation required a "network brownout" that limited the US Postal Service' Internet connectivity, virtual private network (VPN) connections, and remote network access, Miskanic said. All email from non-postal accounts was blocked and workstation administrator rights were revoked during the brownout. To mitigate the risk of spear-phishing attacks, all access to personal email accounts such as Gmail and Yahoo was also blocked, and continues to be blocked, according to Miskanic.

"Direct database access is now enabled only to technology support staff, and a number of business applications have been retired," he noted, adding that the safeguards will be periodically reviewed and updated if needed.

Without knowing the exact causes, it is difficult to speculate on why the USPS's initial response was to allow the attack to continue, said John Pescatore, director of emerging security trends at the SANS Institute. "In order to be prepared to respond rapidly and effectively to an incident, you need to have some processes and controls in place," he said in an email to InformationWeek.

Pescatore also recommended that organizations need to have a baseline, or a known good state that they can revert back to quickly in an emergency. "[It] sounds like some or all of that was missing with USPS, or they were depending on contractor services that couldn't start right away."

Considering how prevalent third-party attacks are, we need to ask hard questions about how partners and suppliers are safeguarding systems and data. In the Partners' Role In Perimeter Security report, we'll discuss concrete strategies such as setting standards that third-party providers must meet to keep getting your business, conducting in-depth risk assessments -- and ensuring that your network has controls in place to protect data in case these defenses fail. (Free registration required.)