How to Measure Anything in Cybersecurity Risk

by Douglas W. Hubbard

For all practical decision-making purposes, we need to treat measurement as observations that quantitatively reduce uncertainty.

We use probability because we lack perfect information, not in spite of it.

There is no greater impediment to the advancement of knowledge than the ambiguity of words.

Clarification Chain If it matters at all, it is detectable/observable. If it is detectable, it can be detected as an amount (or range of possible amounts). If it can be detected as a range of possible amounts, it can be measured.

Uncertainty: The lack of complete certainty, that is, the existence of more than one possibility. The “true” outcome/state/ result/value is not known. Measurement of Uncertainty: A set of probabilities assigned to a set of possibilities. For example: “There is a 20% chance we will have a data breach sometime in the next five years.” Risk: A state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. Measurement of Risk: A set of possibilities, each with quantified probabilities and quantified losses. For example: “We believe there is a 10% chance that a data breach will result in a legal liability exceeding $10 million.”

What you want to know is whether you have less uncertainty after considering some source of data and whether that reduction in uncertainty warrants some change in actions.

Rule of Five There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population.

Define a list of risks.

Define a specific period of time over which that risk event could materialize (e.g., “A data breach will occur for application X in the next 12 months, a loss of availability for system X long enough to incur a productivity loss will occur in the next 5 years, etc.”).

For each risk, subjectively assign a probability (0% to 100%)

For each risk, subjectively assign a range for a monetary loss if such an event occurs as a “90% confidence interval” (CI).

Get the estimates from multiple experts if possible, but don’t have a joint meeting and attempt to reach consensus.

some ways that we actually have more data

health risk even though you never made

When

Clear: Everybody knows what you mean. You know what you mean. Observable: What do you see when you see more of it? This doesn’t mean you will necessarily have already observed it but it is at least possible to observe and you will know it when you see it. Useful: It has to matter to some decision. What would you do differently if you knew this? Many things we choose to measure in security seem to have no bearing on the decision we actually need to make.

Decomposition Rule #1: Decompositions should leverage what you are better at estimating or data you can obtain (i.e., don’t decompose into quantities that are even more speculative than the first). Decomposition Rule #2: Check your decomposition against a directly estimated range with a simulation, as we just did in the outage example. You might decide to toss the decomposition if it produces results you think are absurd, or you might decide your original range is the one that needs updating.

Overconfidence: When an individual routinely overstates knowledge and is correct less often than he or she expects. For example, when asked to make estimates with a 90% confidence interval, many fewer than 90% of the true answers fall within the estimated ranges. Underconfidence: When an individual routinely understates knowledge and is correct much more often than he or she expects. For example, when asked to make estimates with a 90% confidence interval, many more than 90% of the true answers fall within the estimated ranges.

Repetition and feedback. Take several tests in succession, assessing how well you did after each one and attempting to improve your performance on the next one. Equivalent bets. For each estimate, set up the equivalent bet to test if that range or probability really reflects your uncertainty. Consider two pros and two cons. Think of at least two reasons why you should be confident in your assessment and two reasons you could be wrong. Avoid anchoring. Think of range questions as two separate binary questions of the form “Are you 95% certain that the true value is over/under (pick one) the lower/upper (pick one) bound?” Reverse the anchoring effect. Start with extremely wide ranges and narrow them with the “absurdity test” as you eliminate highly unlikely values.

Knowing the outcome of the penetration test was informative since P(MDB | PPT) > P(MDB) > P(MDB | ~PPT). Think of informative conditions like a teeter-totter with the original prior in the middle. If a condition increases the probability, the opposite of that condition must decrease it, and vice versa.