The Worlds I See: Curiosity, Exploration, and Discovery at the Dawn of AI

by Fei-Fei Li

AlexNet, it appeared, was no mere contest entry. It was a moment of vindication a quarter century in the making.

AlexNet could process images about ten times larger than those fed into LeNet, scanning their contents with a convolution kernel—the “focal point” of the network, so to speak—of about twice the size.

A defining flaw of neural networks—long considered fatal—was the difficulty of training them.

In yet another twist of fate, the style of number-crunching favored by neural networks is functionally similar to the kind used in rendering the graphics for video games—a multibillion-dollar industry that had been driving the advancement and commercialization of custom hardware since the 1990s, fueling the growth of megabrands like Nvidia, the company at the forefront of the field.

So, for a stretch of seven days in early 2012, while millions of GPUs all over the world were running hot to render jittering machine guns, charging hordes of zombies, and shrapnel-laced explosions, two of them, somewhere in Toronto, were bringing a new kind of neural network to life.

If one were to point to something truly different about the world of 2012—something categorically absent in the days of LeNet—it had to be the abundance of data used to train the network.

So it’s unsurprising that the one application for which a training set could be found at the time stood for more than twenty years as the algorithm’s sole achievement.

In 2011, believing he was closer than ever to a turning point, he began to reach out to his colleagues in a style that was both confrontational and collaborative, soliciting advice on what he should do next in ways that sounded more like a challenge than a question. One of those calls was to Jitendra, a longtime friend who was skeptical of his project.

It’s just too small. There aren’t enough examples, so the network doesn’t generalize very well when we show it something new.”

Whether Jitendra had truly had a change of heart regarding the project or was simply trying to get under an old friend’s skin—and both seemed plausible—Hinton took the advice seriously.

By the time I arrived, the workshop was so crowded that LeCun himself had to stand against the back wall, having arrived minutes too late to find a seat.

Hinton couldn’t attend because of a chronic back problem that made international travel almost impossible for him, so he’d sent Alex Krizhevsky in his place.

But as with many brilliant people, his personal presentation didn’t quite live up to the gravity of his work—something I’m not sure even he fully appreciated.

After Rosenblatt’s perceptron, Fukushima’s neocognitron, and LeCun’s LeNet, it was a long-overdue next step, decades in the making, finally realized at a scale befitting its potential.

Like all neural networks, AlexNet’s initial state is shapeless and inert, like a tapestry in a void. Then the onslaught begins: one after another, a photograph is chosen at random from the ImageNet library and the network is tasked with correctly assigning it one of a thousand labels.

Mistakes trigger corrective signals, rippling across the network’s tens of millions of constituent parts, each assessed for its contribution to the result and pushed, proportionately, to behave differently next time.

Its 650,000 individual neurons are networked together by way of 630 million connections in total, with 60 million tiny, nearly imperceptible weights influencing the strength of those connections, making some stronger and others weaker, as signals flow from one end of the network to the other.

The weights change from one round to the next, some growing stronger, some weaker, and some merely vacillating, making for a pliable fabric that responds to its training with organic grace. Bearing the weight of these gargantuan quantities are two Nvidia GPUs, highly specialized silicon running in parallel, conducting round after round at maximum speed.

Like something out of geology, these imprints coalesce into a single terrain that reaches from one end of AlexNet to the other. Pencil sharpeners, mosques, starfish, hockey pucks—all embedded somewhere in the landscape. The algorithm hasn’t merely “seen” these things; it’s become them.

It’s an apparently magical combination of hardware, software, and data, and it’s come closer than anything our field has ever built to capturing the spirit of the evolution that shaped the minds of mammals like us.

This neural network, the largest our field has ever seen, trained by more data than any in history, can generalize.

My lab had bet everything on a yearslong pursuit of data at an unprecedented scale, while Hinton’s had staked their reputations on a commitment to a family of algorithms the field had all but abandoned.

I spent more time traveling to and from Florence than I did on the ground.

With garbage bags and a dolly in hand, on an unassuming afternoon in the early months of 2013, we were standing in the former hub of the world-renowned SAIL—the Stanford AI Lab. Over the course of decades, the field that had once boldly called itself “artificial intelligence” had fractured into a multitude of narrower disciplines, many of which downplayed their cognitive roots in favor of more mechanistic terms like “pattern recognition” and “natural language processing.”

Our field appeared to be unifying, albeit under the banner of a slightly different moniker—and an increasingly popular buzz phrase—“machine learning.”

The most substantial sign of change, however, came as more and more of us were gripped by a fixation on the tech industry, with a number of them departing academia altogether for Silicon Valley careers.

When Sebastian Thrun left Stanford to help kick-start Google’s burgeoning self-driving efforts, Silvio’s reputation made him a front-runner for the position.

Next, Andrew Ng, who’d long balanced his role as an educator with leadership positions across Silicon Valley, stepped down as the director of SAIL.

So, with a call to an electronics recycling specialist and the offer of free lunch to lure my fellow professors to a new schedule of meetings, I set about reestablishing SAIL—not just as a channel for funding, but also as the social, interpersonal, and even cultural center of our community.

It was a place I loved to wander whether or not I had anything on the calendar. Every room seemed to house a new clutch of students, at least one of whom was always free for a few minutes of chatting about their research or some stray idea.

For the first time, the highest ambition of a computer vision student wasn’t one of a handful of coveted faculty positions scattered across the country, but a path into the technology industry, whether a job with a start-up or one of the giants.

ImageNet wouldn’t have been possible without WordNet, after all—it provided the framework that gave each category not just its label, but its place within a tree of connected ideas.

Although a project at Cornell University dwarfed that number with a data set of photographs covering hundreds, it’s estimated there are more than ten thousand species across the world, leaving even the state-of-the-art orders of magnitude behind reality. I grinned, reminded of the breathless tone taking hold in the tech press, with article after article heralding the age of machine learning and declaring image classification a suddenly “solved problem.”

Among science’s greatest virtues, however, is its ability to recast a lesson in humility as a moment of possibility.

Years of effort and a global competition between some of the brightest minds on earth, all for a baby step toward true visual intelligence. And yet, when I looked around the room, I didn’t see intimidation or despair on my students’ faces. I saw the gears behind their eyes beginning to turn.

The journey isn’t over yet. We have so much more to explore.

As is so often the case in life, the demands of a career, a marriage, and motherhood had seemed to explode overnight. But I still made time, on occasion, at least, to tag along with my father when he was pursuing his favorite activity. They were rare moments of stillness and nostalgia in a life that felt perpetually accelerated, and they helped preserve the bond that had seen us through from our earliest days in a strange new country.

What new insights—societal, cultural, even political—might be revealed by simply looking more closely at the world that surrounds us every day?

Darlings of the academic community that had enthralled researchers only a year or two earlier—algorithms like support vector machines and Bayesian networks—all but vanished from conference lectures, published articles, and even conversation around the lab.

For instance, humans might be better than computers at explaining why they believe the bird on a nearby branch is a coastal blue jay, drawing on all sorts of general knowledge, visual cues, and intuitions, but we can only stretch that ability so far. Even experienced bird-watchers can rarely identify more than a couple hundred species, which leaves the vast majority of the avian universe inaccessible to any one observer. As AI struggled to overcome the last few percentage points that divided human-level performance in general object classification from its own, it seemed tantalizingly close to outclassing us in other dimensions—and quite dramatically. We just can’t hold anywhere near as much knowledge in our heads as a computer can.

Because we intended to use cars as a proxy for exploring larger socioeconomic questions—correlating them with aspects of their owners’ identities like income, education, and occupation—we had to face the fact that dramatic gaps in monetary value often translate to only subtle differences in outward appearance.

We found the concept of trim levels especially vexing, as option packages totaling thousands of dollars, and sometimes more, often entailed only minor modifications to the car’s body style and badging.

But she made an immediate impression on me—not just because she was the only Black woman pursuing an engineering PhD, but because her willingness to ask questions demonstrated a hunger to learn that professors immediately notice. When she asked to join the lab, I said yes without hesitating, dispensing with even basic formalities like reference letters.

Life in a three-generation, two-culture family quickly taught Silvio the art of cohabitating with my mother, who took kitchen cleanliness to an almost pathological extreme—following the maxim of cleaning while one cooks so slavishly it might be more accurate to say she cooks while she cleans.

right. I remember thinking ‘gist’ was a funny word to see in an article like that.”

“The ideas were so big, but the language was totally straightforward.”

Although Pietro, Christof, and I considered such nuanced awareness a distant dream for computer vision, we were convinced that the journey could only begin with a better understanding of what humans do, and devised a way to explore it.

Photographs may be still, but we excel at extracting the motion frozen within them, from the grand and sweeping to the nearly imperceptible, and all with impressive acumen.

We imagine the circumstances leading to the moment the picture captures and the outcome that may result, as in the fraction of a second following a photograph of a skateboarder leaping off a curb, or the lifetime that follows an image of a young couple exchanging wedding vows.

I rarely had any conscious memory of enumerating the individual objects surrounding me—the roomful of furniture, my mother and father, the clothes they wore, kitchen utensils, an unopened package or envelope, Silvio’s espresso machine, the family cat, and so on.