Black Box Thinking: Why Some People Never Learn from Their Mistakes - But Some Do

by Matthew Syed

sales would have gone up even more if you had not cha...

But how often do we actually test our policies and strategies? How often do we probe our assumptions, in life or at work?

How often do we evaluate the design of our experiments and learn to improve them?

“It is about marginal gains,” he said. “The approach comes from the idea that if you break down a big goal into small parts, and then improve on each of them, you will deliver a huge increase when you put them all together.”

20 mile March, see Collins good to great in uncertainty

The best way to adjudicate between these stances would be to conduct a randomized control trial.

But there is a rather obvious problem. There is only one Africa.

When it comes to really big issues, it is very difficult to conduct controlled experiments.

But now suppose that instead of looking at the big picture, you examine an individual program.

the economists started to think about the problem in a fresh way. They tried something completely new: a de-worming medication.

This time the results were excellent.

This was a marginal gain. It was just one program in one small region.

by looking at education at this level of magnification, it

was possible to see what really works, an...

Small bet to test hypotheses in a complex system. Lean startup and design approach. Trial group assumes representative of larger group, but nonetheless empirical. Catalog conditions of group and continue to test representative groups to invalidate hypothesis

They could now roll it out in other areas, while continuing to test, and iterate, and crea...

Each test provides a small gain of one kind or another (remember that failure is not inherently bad: it sets the stage for new ideas).

By breaking a big problem into smaller parts, it is easier to cut through narrative fallacies. You fail more, but you learn more.

Now we can see a clear answer. Marginal gains is not about making small changes and hoping they fly. Rather, it is about breaking down a big problem into small parts in order to rigorously establish what works and what doesn’t.

to find out if something is working, you must isolate its effect. Controlled experimentation is inherently “marginal” in character.

Marginal gains, as an approach, is about having the intellectual honesty to see where you are going wrong, and delivering improvements as a result.”

Once you have gone through a practice cycle with the initial strategy, you immediately realize that there are miscellaneous items that you are not measuring.

So the second stage of the cycle is about improving your measurement statistics,

We have talked about the concept of an open loop. This is where a strategy is put in action, then tested to see if it is working. By seeing what

going wrong, you can then improve the strategy.

use the first test not to improve the strategy, but to create richer feedback.

Designing better experiments by getting started with the experiments. Arm chair thinking misses real world unknowns, especially in complex systems

it is about hundreds of thousands of small items, optimized to the nth degree.

You start with a sensible design, but it is the iterative process that guides you to the best solution. Success is about creating the most effective optimization loop.

You need judgment and creativity to determine how to find solutions to what the data is telling you, but those judgments, in turn, are tested as part of the next optimization loop.

Success is a complex interplay between creativity and measurement,

The fact that Divine’s shade lost out in this trial didn’t mean he was a poor designer. Rather, it showed that his considerable knowledge was insufficient to predict how a tiny alteration in shade would impact consumer behavior.

Google, came up with a more systematic trial. She divided the relevant part of the color spectrum into forty constituent shades and then ran another test.

As of 2010, the company was carrying out 12,000 RCTs every year.

Every year since it was founded Capital One has run thousands of similar tests. They have turned the company into a “scientific laboratory where every decision about product design, marketing, channels of communication, credit lines, customer selection, collection policies, and cross-selling decisions could be subjected to systematic testing and using thousands of experiments.”

RCTs, whether in business or beyond, are often very dependent on context. A trial that improves, say, educational outcomes in Kenya has no claim to improve outcomes in London.*

We need to run lots of trials, lots of replications, to tease out how far conclusions can be extended from one trial to other contexts.

we need to create the capacity for running experiments at scale and at a lower unit cost.

Marginal gains

a willingness to test assumptions is ultimately about a mindset.

is about intellectual honesty and a readiness to learn when one fails. Seen in this way, it is relevant to any business; in fact to almost any problem.

This is the potency of marginal gains. By dividing a big challenge into small parts, you are able to create rigorous tests, and thus deliver incremental improvements.

this visualization also reveals the inherent limitations of marginal gains.

creative leaps. It is about acts of imagination that can transform the entire landscape of a problem.

the problem is also obvious: the business model was eventually superseded by Netflix and the like, rendering videos and DVDs, to a large extent, obsolete.* The entire landscape fundamentally changed.

In the diagram, the new landscape is represented by the taller hill.

Marginal gains is a strategy of local optimization: it takes you to the summit of the first hill.

focus on the bold leaps that lead to new conceptual terrain, or on the marginal gains that help to optimize one’s existing fundamental assumptions.

Industry dependent, but some level of both is required

success is about developing the capacity to think big and small, to be both imaginative and disciplined, to immerse oneself in the minutiae of a problem and to stand beyond it in order to glimpse the wider vista.

Zoom out to check environment for change, zoom in for local optimization and discipline

the aversion to failure is the single largest obstacle to creative change,

Creativity is something that has to be worked at, and it has specific characteristics.

He was curious, inquisitive, and willing to engage with a difficulty rather than just accepting

The first is that the creative process started with a problem, what you might even call a failure, in the existing technology.

It was the very nature of the engineering problem that sparked a possible solution (a bagless vacuum cleaner).