Why Greatness Cannot Be Planned: The Myth of the Objective

by Kenneth O. Stanley

We assume that any worthy social accomplishment is best achieved by first setting it as an objective and then pursuing it together with conviction. It makes you wonder, is there such a thing as accomplishment without objectives?

Maybe all this objective mania makes sense. At some level, we must believe it does to have allowed it to dominate our lives so completely. Or maybe it’s something else—maybe we’ve become so used to objectives defining everything we do that we’ve forgotten that their value can even be questioned. Either way, there’s a certain appeal to this kind of routine. The idea that all our pursuits can be distilled into neatly-defined objectives and then almost mechanically pursued offers a kind of comfort against the harsh unpredictability of life. There’s something reassuring about the clockwork dependability of a world driven by tidy milestones laid out reliably from the starting line. Though often unspoken, a common assumption is that the very act of setting an objective creates possibility. The very fact that you put your mind to it is what makes it possible. And once you create the possibility, it’s only a matter of dedication and perseverance before you succeed. This can-do philosophy reflects how deeply optimistic we are about objectives in our culture. All of us are taught that hard work and dedication pay off—if you have a clear objective.

Objectives might sometimes provide meaning or direction, but they also limit our freedom and become straitjackets around our desire to explore. After all, when everything we do is measured against its contribution to achieving one objective or another, it robs us of the chance for playful discovery. So objectives do come with a cost. Considering that this cost is rarely discussed in any detail, maybe it’s a good idea to look a little harder at what we’re really giving up in exchange for such objective optimism.

There’s a lot our culture has sacrificed in the name of objectives, and we’re going to take it back. They’ve stolen our freedom to explore creatively and blocked us from serendipitous discovery. They ignore the value of following a path for its own uniqueness, rather than for where it may lead.

Obviously we’re going to raise a lot of questions about the benefits of objectives, but one important point is that we’re mainly focused on ambitious objectives—those whose achievement is anything but certain. One of the reasons that objectives aren’t often questioned is that they work perfectly well for more modest pursuits. If a manufacturer decided to increase efficiency by 5 %, no one would be shocked if it succeeded. A software company upgrading its product from version 2.0 to version 3.0 is similarly likely to succeed, as happens all the time. Everyday successes like these mislead us into believing that setting objectives works well for almost everything. But as objectives become more ambitious, reaching them becomes less promising—and that’s where the argument becomes most interesting.

Objectives are well and good when they are sufficiently modest, but things get a lot more complicated when they’re more ambitious. In fact, objectives actually become obstacles towards more exciting achievements, like those involving discovery, creativity, invention, or innovation—or even achieving true happiness. In other words (and here is the paradox), the greatest achievements become less likely when they are made objectives. Not only that, but this paradox leads to a very strange conclusion—if the paradox is really true then the best way to achieve greatness, the truest path to “blue sky” discovery or to fulfill boundless ambition, is to have no objective at all.

Even after vacuum tubes were first discovered, no one would realize their application to computation for over 100 years. The problem is that the stepping stone does not resemble the final product.

It often turns out that the measure of success—which tells us whether we are moving in the right direction—is deceptive because it’s blind to the true stepping stones that must be crossed. So it makes sense to question many of our efforts on this basis. But actually the implications are even more grave than just questioning particular pursuits and their objectives. At a deeper level, we might ask why we think ambitious pursuits should be driven by objectives at all.

We want to show you that it’s possible to explore a search space intelligently even without an objective. In other words, there is a third way—just because you don’t have an objective doesn’t mean you have to be wandering. We can align ourselves towards discovery and away from the trap of preconceived results. Over the remainder of this book an important principle will begin to emerge: Sometimes the best way to achieve something great is to stop trying to achieve a particular great thing. In other words, greatness is possible if you are willing to stop demanding what that greatness should be.

The idea that an improving score guarantees that you’re approaching the objective is wrong. It’s perfectly possible that moving closer to the goal actually does not increase the value of the objective function, even if the move brings us closer to the objective.

This situation, when the objective function is a false compass, is called deception, which is a fundamental problem in search. Because stepping stones that lead to the objective may not increase the score of the objective function, objectives can be deceptive.

The deception of the Chinese finger trap is that the path to freedom is to push inward, away from freedom. In other words, the stepping stone to freedom is to become less free. This situation illustrates deception well because it shows how wrong it can be to measure progress towards your goal: If your objective is freedom from the Chinese finger trap, then measuring progress by how close you are to freedom is exactly the wrong approach. Maybe you think the Chinese finger trap is unfair. After all, it’s designed to trick you. But the Chinese finger trap is actually a lot simpler than the kinds of problems we’d like to solve or the kinds of discoveries we’d like to make. The Chinese finger trap has only one deceptive stepping stone. There’s almost no question that ambitious problems like designing artificial intelligence or curing cancer will require well more than a single deceptive stepping stone. In fact, the search space of any complex problem is sure to be littered with deceptive stepping stones. Consider for example the game of chess: There are many moves that seem promising (such as capturing a piece) that later actually lead to trouble because of subtle unforeseen implications. As tricky as that makes chess, the world we live in is far more complicated than even that, so deception is going to be everywhere. The moral is that we can’t expect to achieve anything great without overcoming some level of deception. And any problem without deception is trivial, because the ...

Collecting stepping stones isn’t like pursuing an objective because the stepping stones in the Picbreeder collection don’t lead to somewhere in particular. Rather, they are the road to everywhere. To arrive somewhere remarkable we must be willing to hold many paths open without knowing where they might lead.

The problem is that the stepping stones to intelligence do not resemble intelligence at all. Put another way, human-level intelligence is a deceptive objective for evolution. Once again, deception rears its confusing and misdirecting head. Rather than increasing intelligence, the stepping stones that lead from single-celled organisms to humans include such unrelated innovations as multicellularity and bilateral symmetry. Millions of years ago our ancestor was a flatworm. It would not score any accolades for its intellect, but its one great achievement was bilateral symmetry. Who would ever think that bilateral symmetry is essential to writing poetry? But it was an essential stepping stone on the road to Shakespeare. The problem with the intelligence test approach is that it entirely fails to detect such monumentally important discoveries. Instead it wastes precious effort measuring a property that will not come into play in any important way until eons in the future. As philosopher Marshall McLuhan said, “I don’t know who discovered water, but it wasn’t a fish.” Or in the words of the scientist Chuck Thacker, “You can’t build railroads before it is railroad time.” Neither should you administer intelligence tests to single-celled organisms if you want to evolve intelligence. That may seem obvious because subjecting a cell to an IQ test (or any intelligence test) is unquestionably ridiculous. But the fact that it is ridiculous is exactly the point, and is why this experiment...

Because most interesting inventions are simply the most recent fruits of chains of ideas spanning centuries, they too will necessarily depend upon a prior invention created for an entirely different purpose. In fact, stretching this line of thinking to its logical conclusion leads to a provocative hypothesis about invention in general: Almost no prerequisite to any major invention was invented with that invention in mind. While this idea sounds strange, if it’s even partially true then its implications for objective-driven innovation are sobering. After all, what hope is there of achieving ambitious objectives if their prerequisites are almost sure to come from someone with an entirely different objective? But there is good reason to believe that the world works exactly in this way. Electricity was not discovered with computation in mind, or even with vacuum tubes in mind, and neither were vacuum tubes invented to foster building computers. We simply lack the foresight to comprehend what one discovery will later make possible.

Great invention is defined by the realization that the prerequisites are in place, laid before us by predecessors with entirely unrelated ambitions, just waiting to be combined and enhanced. The flash of insight is seeing the bridge to the next stepping stone by building from the old ones. And the story of those stepping stones is not a story of intentional objective-driven building, one piece at a time towards some distant über-invention as conceived by an overarching plan.

The reason that searching for novelty leads from simplicity to complexity is itself surprisingly straightforward: When all the simple ways to behave are exhausted, the only new behaviors that remain to discover are more complex.

As Stephen Jay Gould has pointed out in evolution, once all the simple ways to live are exhausted, the only way to create a new species or niche is to become more complex [48]. In other words, there are only so many ways of being a bacteria. That’s why increasing complexity is almost inevitable if evolution is to continue. But these increases in complexity are not arbitrary. Rather, they reflect the properties of the world in which evolution takes place: Eyes represent the presence of light in the universe. Ears signify mechanical vibration. Legs are reflections of gravity, and lungs of oxygen.

Take for instance the robot searching for novel behaviors in the maze in the last chapter (Fig. 5.​2). While it might be a stretch to call the objective of running through the maze ambitious, it turns out that it’s still deeply deceptive anyway—when the robot tries to solve the maze as its objective, it almost always fails. Why is the objective even in a relatively simple maze so deceptive? The answer has to do with walls—when walls block the robot from running straight to the target, the objective-driven robot will crash into the closest wall in the direction of the goal. This closest wall attracts the robot because by approaching it the robot edges closer to the goal—which is its objective—causing its reward to go up. But to come any closer to the goal than the wall allows would require first driving much farther away. But that won’t work out well because moving away from the wall looks worse to the objective-driven search. So any further exploration of the idea of going around the wall is likely to be cut off prematurely. The problem is that the only time the objective actually pushes the search in the correct direction is at the very end, when the robot has learned to navigate to a point within a straight shot of the goal. But at that point the problem becomes trivial anyway: Learning to run towards a goal that is directly in front of you is not what one would call a challenge. Outside that small area where the goal is within a straight shot, the objective remains a fa...

Campbell’s law, which is well known in the social sciences [58]: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

When the quest for progress is packaged into a measure, the result is an objective-driven approach. If the objective is ambitious, then a drive to increase objective performance is likely to produce deception, preventing the best possible result from being discovered.

the main effect of the system is to reward consensus. In other words, the more the reviewers agree that the proposal is excellent, the more likely it is that the agency will fund it. The problem is that consensus often works directly against exploring stepping stones.

Seeking consensus prevents traveling down interesting stepping stones because people don’t agree on what the most interesting stepping stones are. And resolving this kind of disagreement often leads to a compromise between opposing stepping stones. Like a faded gray that results from mixing the sharp contrasts of black and white together, the product of such compromise often dilutes the two original ideals. For the scientist writing a proposal, the result is that the best way to win a grant is to propose the perfect compromise, the best faded gray—good enough to satisfy everyone, but unlikely to lead anywhere highly novel or interesting. So when consensus is sought in exploration, the result is a generic washout effect. Instead of allowing each person to discover their own chains of stepping stones, the system squashes a diversity of opinions into a generic average.

One of the main reasons we tend to stick to objectives in general is a fear of risk. While some level of risk is necessary for progress, at the same time those paying the bills generally don’t want so much risk that resources are simply being wasted on crackpot projects. But our fears can’t change the fact that risk is essential in science. The whole point is to cross many stepping stones over the long run. Because the hope is to travel far, risk-averse objective thinking limits and constrains progress. On the other hand, in areas likes business investing, the hope is for more short-term payoff. When considering innovative businesses, most investors recognize that the only good objectives are on nearby stepping stones. The point is, in the business world, it’s often also a good idea to follow the interesting, but when the idea is presented to investors, it needs to be clearly reachable from the current stepping stone. So the innovative solution must be discovered before approaching the investor.

Realistic objectives, which tend to be the province of investing, tend to be exactly those that are one stepping stone away. This fact is reflected in how most people invest—a solid business plan guides us only to the next stepping stone. But that doesn’t mean that business can’t be innovative—an innovative business idea reveals a nearby stepping stone that we didn’t previously realize was there. The innovator in business also searches for the interesting, but waits to present anything to investors until the unexpected stepping stone is fully understood.

what we hope you gained is an appreciation for just how omnipresent objectives can be. Their tentacles entangle every facet of our lives. From the most critical societal initiatives to the more mundane rituals of daily life and even the milestones of youth, objectives run almost everything. They’re not always wrong, and even when they are wrong replacing them is no simple matter. But by seeing how they shape our view of the world and its potential, at least we can highlight that there is sometimes another way to think or to approach life. It is possible sometimes that the entrenchment of stagnant ways can be escaped by abandoning their false security. In some cases, the power of non-objective discovery and divergent exploration could indeed create a better future. And though non-objective search itself is not a universal solution, we’re still better off seeing with sober eyes that slavish faith in objective exploration and assessment is often a formula for mediocrity and conformist stagnation.

The quest for cures, elegant theories, beautiful structures, brilliant machines, soul-stirring melodies, epic tales, unfettered creativity, travel beyond the stars, national healing, passionate release, true happiness—these are places where objectives lose their power. When we seek out there, beyond the horizon, where stepping stones fall silently behind shadows, that is where the objective compass begins to quaver. And the farther out we go, the more the objective becomes a deceiver, holding us back from our deepest potential—which is exactly where it starts making sense to change our approach.

you can still search by following principles, even if you aren’t following any particular objective. The only concession we have to make is that we can’t be sure what we’ll end up finding. In other words, our destination becomes unknown. We have to concede control of the final destination.

When people do actually achieve amazing feats, they almost always build upon eons of accumulated innovations during which the final outcome was not the objective.

That’s the stepping stone principle—one good idea leads to another. Treasures lead to more treasures, chaining and branching out across the infinite stepping stones of possible discoveries. So what you need to do is become a skilled treasure hunter. For that to happen, you have to learn to search for clues. But instead of clues that the objective is nearby, they are clues that something is nearby that may be worth finding, like a mysterious but sweet aroma hanging in the air.

When you’re standing on the edge of possibility looking out over the unknown, objectives become false beacons, but interestingness is different. Interestingness forms a network of roads leading from one treasure to another. The needles in the haystack pop into view only when we pause to appreciate the promise of the moment—the current stepping stone and the nearby stones to which it leads. Contrary to popular belief, great inventors don’t peer into the distant future. A false visionary might try to look past the horizon, but a true innovator looks nearby for the next stepping stone. The successful inventor asks where we can get from here rather than how we can get there. It’s a subtle yet profound difference.

In the past, and even into the time of the Wright brothers, those who pursued flight were primarily motivated by the inspiring vision of claiming the skies. Interestingly, Samuel Langly, their primary rival, attracted significant government funding to match the ambition of his effort, compared to the meager self-financing of the Wright brothers. But the Wright brothers had a different kind of motivation. In fact, they were originally bicycle manufacturers. It turns out that for them bicycles were stepping stones to flying machines. You could say that they saw in the present an echo of the future, rather than trying to drag a preconceived vision of the future into the present. So while flight was an objective for many failed visionaries over hundreds of years, it was when the bicycle-makers realized that planes are bicycles in the sky that it actually happened [105].

It’s in your interest that some do not follow the path you think is right, because one day they will build the stepping stones that lead to your greatest discovery.

In fact, evolution is just one of many major scientific discoveries that took humans off our pedestal as intrinsically special. Copernicus demonstrated that Earth wasn’t the center of the universe [107], Wöhler showed that the chemistry of living things wasn’t separate from the chemistry of the non-living [108], and similarly Darwin discovered that humans were just one of the many leaves on the tree of life [109].

You’d think that experts on writing world-class search algorithms would also be skilled at guiding a search themselves. So an interesting question is whether the search experts in the AI community avoid chasing the false compass of ambitious objectives—or whether even the experts are drawn to the myth of the objective like everyone else.

Recall that AI is a search for algorithms that search—a meta-search. So with this meta-search in mind it’s probably not a good idea always to reject new ideas just because their performance is unimpressive. Recall that performance is exactly the heuristic against which novelty search compared favorably in Chap. 5. There’s no reason to suspect that the same heuristic, but used one level up (to guide the AI community as a whole instead of a search algorithm), will avoid the problems plaguing objective-driven search. When performance is the rule of thumb that filters which algorithms are shown to the larger community, all other kinds of stepping stones are rejected and ignored. It would be one thing if criticisms about performance were just criticisms—and not a standard reason for rejection. But because performance is so widely accepted as a filter for new ideas, it becomes a classic deceptive objective function.

the field of AI is drawn into a classic objective-driven search—which is fueled by the assumption that objective-driven searches work well. But there’s an interesting irony here: The experimentalist heuristic is driven by such a simple objective that few AI researchers would actually employ an algorithm based on such a naïve heuristic today. While algorithms that search have become more sophisticated to deal with some level of deception, these insights aren’t being applied to how AI researchers themselves behave in their search as a community. So even as the simple experimentalist heuristic hides in plain sight, the field is driven faithfully by it.

The problem is that the theorem isn’t a theorem about what the community should do next. The theorem is about one algorithm—one point in the huge search space of all possible algorithms. For all we know, a tiny change to an algorithm that isn’t proven to perform well could create an algorithm that performs even better than algorithm OldReliable does. A proof for one algorithm doesn’t say anything about other algorithms that may come up in the future. So to believe that algorithm OldReliable is a promising stepping stone because of its theorem only makes sense if there’s something about this particular theorem that suggests the algorithm may lead to other promising algorithms. Whenever you choose which rule of thumb to rely on—whether it’s the experimentalist or the theoretical heuristic—don’t forget that you’re still just relying on your intuition. Whether you like performance or guarantees, they’re both intuitions about what clues hint that an algorithm is a promising stepping stone.