Upstream: The Quest to Solve Problems Before They Happen

by Dan Heath

short-term measures

He's talking aobut short-term measures, but I think he should be talking about leade measures, the activities you do today that drive important lag measures in the future.

Think about this: An NYPD official is held accountable for rape statistics. There are two ways to make those numbers look better. The first way is to actually prevent rape—to project the police’s presence into dangerous areas and thereby deter the violent acts. (That’s what would have happened if Ritchie and his partner had arrived at the scene just a few minutes earlier.) The second way to reduce the rape count is to reclassify actual rapes as lesser crimes—in this case, Ritchie’s boss tries to reframe the incident with the prostitute as a “theft of service.” The first way constitutes a victory; the second way is an abomination. But, tragically, both would look the same in the data.

This isn't "gaming the system," this is lying. It's fraud.

Here’s what makes this whole subject even trickier: Crime really did go down—way down—in New York City. But

And to make it even tricker, CompStat wasn't the only policing change made in New York at the time and compustat was used for more than just the fraud described.

CPS’s success is no ghost victory. Their measures matched the mission. And the way the district’s leaders accomplished that is instructive. They used what Andy Grove, the former CEO of Intel, called “paired measures.” Grove pointed out that if you use a quantity-based measure, quality will often suffer. So if you pay your janitorial crew by the number of square feet cleaned, and you assess your data entry team based on documents processed, you’ve given them an incentive to clean poorly and ignore errors, respectively. Grove made sure to balance quantity measures with quality measures.

Any upstream effort that makes use of short-term measures—which, presumably, is most of them—should devote time to “pre-gaming,” meaning the careful consideration of how the measures might be misused. Anticipating these abuses before the fact can be productive and even fun, in sharp contrast to reacting to them after the fact. Here are four questions to include in your pre-gaming: The “rising tides” test: Imagine that we succeed on our short-term measures. What else might explain that success, other than our own efforts, and are we tracking those factors? The misalignment test: Imagine that we’ll eventually learn that our short-term measures do not reliably predict success on our ultimate mission. What would allow us to sniff out that misalignment as early as possible, and what alternate short-term measures might provide potential replacements? The lazy bureaucrat test: If someone wanted to succeed on these measures with the least effort possible, what would they do? The defiling-the-mission test: Imagine that years from now, we have succeeded brilliantly according to our short-term measures, yet we have actually undermined our long-term mission. What happened?

Chicago’s leaders tried an experiment by banning lightweight plastic bags; it failed at first, but they knew why it failed, so they tried a different experiment, which worked better, and hopefully no city on earth has to repeat the dumb version of the ban again.

Also, they repealed their original regulation, something that's devilishly hard to do

Donella Meadows deserves the last word: “Systems can’t be controlled, but they can be designed and redesigned. We can’t surge forward with certainty into a world of no surprises, but we can expect surprises and learn from them and even profit from them.… We can’t control systems or figure them out. But we can dance with them!”

But sometimes averages don’t just blur an underlying reality, they obliterate it. I’m amazed, for instance, by how many perfectly smart people seem to believe that the 47.3-year life expectancy in 1900 is synonymous with “most people lived significantly shorter lives back then.” I suppose they picture our ancestors, in their mid-forties, tottering around with canes and false teeth and trying frantically to get their affairs in order. From this perspective, the passage of the Social Security Act in 1935 would have been a cruel joke indeed—Yes, you can start collecting retirement

This is a great explanation, but it begs the question of why we keep using "average life span"

That’s an example of what’s called the “wrong pocket problem”: a situation where the entity that bears the cost of the intervention does not receive the primary benefit. One pocket pays, but the returns are scattered across many pockets.

trust me when I say there is an endless wormhole of complexity that lies beyond):

This is part of the problem, not something to dismiss in a parenthetical statement. We write laws and the laws spawn agencies wo write regulartions and no one is responsible for seeing that the regulations don't conflict. It might not even be possible, but this complexity makes common sense solutions almost impossible to achieve without herculean effort.

hacking. In other words, if a hacker was dead set on breaking into West Aurora School District No. 129—or any other specific institution, for that matter—then the difference between 29% and 5% is immaterial. For many hacking purposes, you just need one open door. Just that one gullible person who will click on anything.

Bostrom notes that we haven’t drawn a black ball so far, but “The reason is not that we have been particularly careful or wise in our technology policy. We have just been lucky.… Our civilization has a considerable ability to pick up balls, but no ability to put them back into the urn. We can invent but we cannot un-invent. Our strategy is to hope that there is no black ball.”

The idea of a "black ball" implies that you can predict that such an event will occur. Sometimes what you have is a "black swan" event that you can't predict.