How to Measure Anything in Cybersecurity Risk

by Douglas W. Hubbard

“The measure of success is not whether you have a tough problem to deal with, but whether it is the same problem you had last year.”

This global attack surface is a macro-level phenomenon driven by at least four macro-level causes of growth: increasing users worldwide, variety of users worldwide, growth in discovered and exploited vulnerabilities per person per use, and organizations more networked with each other resulting in “cascade failure” risks.

What risks are acceptable is often not documented, and when they are, they are stated in soft, unquantified terms that cannot be used clearly in a calculation to determine if a given expenditure is justified or not.

For those who believe something to be immeasurable, the concept of measurement—or rather the misconception of it—is probably the most important obstacle to overcome.

For all practical decision-making purposes, we need to treat measurement as observations that quantitatively reduce uncertainty.

Definition of Measurement Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations.

A nominal scale expresses a state without saying that one state is twice as much as the other or even, for that matter, more or less than the other—each state scale is just a different state, not a higher or lower state.

Ordinal scales, on the other hand, denote an order but not by how much.

And since this uncertainty can change as a result of observations, we treat uncertainty as a feature of the observer, not necessarily the thing being observed.

When we conduct a penetration test on a system, we are not changing the state of the application with this inspection; rather, we are changing our uncertainty about the state of the application.

What we call a “clarification chain” is just a short series of connections that should bring us from thinking of something as an intangible to thinking of it as a tangible. First, we recognize that if X is something that we care about, then X, by definition, must be detectable in some way.

Clarification Chain If it matters at all, it is detectable/observable. If it is detectable, it can be detected as an amount (or range of possible amounts). If it can be detected as a range of possible amounts, it can be measured.

If the clarification chain doesn’t work, I might try what scientists would call a “thought experiment.” Imagine you are an alien scientist who can clone not just sheep or even people but entire organizations. You create a pair of the same organization, calling one the “test” group and one the “control” group. Now imagine that you give the test group a little bit more “damage to reputation” while holding the amount in the control group constant. What do you imagine you would actually observe—in any way, directly or indirectly—that would change for the first organization?

To most people, an increase in security should ultimately mean more than just, for example, who has attended security training or how many desktop computers have new security software installed. If security is better, then some risks should decrease.

Definitions for Uncertainty and Risk, and Their Measurements Uncertainty: The lack of complete certainty, that is, the existence of more than one possibility. The “true” outcome/state/ result/value is not known. Measurement of Uncertainty: A set of probabilities assigned to a set of possibilities. For example: “There is a 20% chance we will have a data breach sometime in the next five years.” Risk: A state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. Measurement of Risk: A set of possibilities, each with quantified probabilities and quantified losses. For example: “We believe there is a 10% chance that a data breach will result in a legal liability exceeding $10 million.”

There is no single, universal sample size required to be “statistically significant.” To compute it correctly, statistical significance is a function of not only sample size, but also the variance within a sample and the hypothesis being tested. These would be used to compute something called a “P-value.” This result is then compared to a stated “significance level.” Lacking those steps, the declaration of what is statistically significant cannot be trusted. Once you know not only how to compute statistical significance but also how to understand what it means, then you will find out that it isn’t even what you wanted to know in the first place. Statistical significance does not mean you learned something and the lack of statistical significance does not mean you learned nothing.

For now, it is probably better if you drop the phrase “statistically significant” from your vocabulary. What you want to know is whether you have less uncertainty after considering some source of data and whether that reduction in uncertainty warrants some change in actions.

Rule of Five There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population.

No matter how complex or “unique” your measurement problem seems, assume it has been measured before. If you are resourceful, you can probably find more sources of data than you first thought. You probably need less data than your intuition tells you—this is actually even more the case when you have a lot of uncertainty now.