Cyber Risk Dashboards: False Sense Of Control?

Dashboards are used throughout business and industry to provide a measure of success. A correctly designed and implemented dashboard can provide critical information to an organization about performance and risk measures in near-real time. The dashboard information should drive the organization to excel in meeting goals while minimizing risk and provide early warnings of possible problems. Dashboards are a good thing when used correctly, but how do we know if we are measuring the correct indicators?

As part of the Department of Homeland Security's Continuous Diagnostics and Mitigation (CDM) program, organizations must develop a dashboard to allow DHS to assess the level of risk by agency. DHS states it will do this by collecting input from sensors placed in 144 agencies. These sensors will allow an agency to "quickly identify which network problems to fix first, and empower technical managers to prioritize and mitigate risks on their respective networks." While this goal is admirable, we need to examine the methodology proposed to ensure the dashboard will function as intended.

[Make sure you can bounce back fast. Read Cyber Attacks Happen: Build Resilient Systems.]

At a very basic level, risk is a function of the likely impact a threat could impart by exploiting a vulnerability. If a dashboard is to accurately reflect the true risk posture of an organization, it must take into account all aspects of the risk equation including threat, vulnerability, likelihood, and impact. The following diagram from NIST illustrates the relationship between the components of risk and also helps explain how we should arrive at organizational risk:

So how well does CDM capture organizational risk? Presently, CDM focuses on four main control areas: hardware assets, software assets, configuration settings, and vulnerabilities. The second phase of CDM promises access control management, security-related behavior management, credentials and authentication management, privileges, and boundary protection. The final stage will provide event planning and response, generic audit/monitoring, document requirements, quality management, and risk management.

Given the structure of risk, agencies must determine if DHS CDM feeds provide a collectively exhaustive picture of threats, vulnerabilities, impacts, and likelihoods for their risk dashboards. If these feeds are inaccurate, not timely, or incomplete, the dashboard risks will be inaccurate or worse -- divert scarce resources to the wrong risk areas. Agencies must map sensor information and capabilities to ensure they are not only getting accurate, thorough, and timely risk information, but also ensure they are leveraging the considerable investment in deploying and operating the sensors.

For the most part, the first four areas -- hardware, software, configuration and vulnerability -- are focused on vulnerability detection. Vulnerability is a major component of the risk formula, but we would be wrong to equate vulnerability alone with risk because we have not considered threats, likelihood, or impact. Therefore, a vulnerability dashboard treated like a risk dashboard might result in spending resources on systems that might not need the most urgent patches or repairs.

For example, two identical systems might be running the same operating system on the same hardware with the same configuration. One system is used to store classified information, and the other simply processes publicly available information. If a vulnerability scan is executed, both machines will be subject to the same vulnerabilities, as they have the same configuration, hardware, and software. Should a vulnerability exist, determining which machine should be patched first is a challenge, as they both show the same level of vulnerability criticality. Worse, let's assume the machine with classified information has a moderate vulnerability, while the public machine contains a high vulnerability due to a configuration difference. In this situation, if vulnerability is equated with risk, the machine with public information will be prioritized higher than the system with classified information.

Although this is a very simple example, it shows how distorting one area of the risk equation for another can cause disastrous results. When we don't accurately measure and combine the elements of risk, we might gain a false sense of control in our environments and be caught off guard when a breach, outage, or compromise occurs. Agencies must use existing information such as their FIPS-199 categorizations for impact and threat information from their security operations centers or intelligence programs to help bolster their real-time risk scoring dashboards. Combining disparate sources from existing C&A work in addition to often organic and nebulous threat information will prove to be the ultimate challenge for an accurate risk dashboard.

In the meantime, agencies should use the dashboards with a full understanding of what they provide -- and, more importantly, what they do not.

NIST’s cyber-security framework gives critical-infrastructure operators a new tool to assess readiness. But will operators put this voluntary framework to work? Read the Protecting Critical Infrastructure issue of InformationWeek Government today.