More on this book
Community
Kindle Notes & Highlights
For one thing, although the algorithm bore a conceptual elegance when drawn on a whiteboard, the computation required to bring even a simple implementation to life was staggering, and still well beyond the capabilities of even most corporations and governments. Equally damning was the state of digital data—a comparatively rare commodity at the time, especially when it came to perceptual data like images, video, audio, and the like.
And with the dot-com boom in full swing, it was a genuine dilemma: the financial world was eager to recruit anyone with a mind for numbers and a fancy degree from the right school, making even physics geeks like me the target of aggressive recruiting efforts from a revolving door of Wall Street players.
In return, all they asked was that I give up science.
It was my first trip to Southern California, and the weather lived up to its sunny reputation, with a dry heat that felt like instant refuge from the humidity of New Jersey.
And although it may sound trivial, I couldn’t pretend that the opportunity to escape years of shivering through Northeastern snowstorms wasn’t a selling point all by itself.
I had made time for a handful of art classes at Princeton and was excited to recognize it as a Mondrian.
Christof in particular often seemed so captivated by his own thoughts that he was more interested in exploring them in a monologue than in talking to me, even in one-on-one conversations.
My career as a grad student began with the purchase of an especially large textbook.
As far as my advisors were concerned, an essential first step toward the promise of machine intelligence was a better understanding of the human kind.
An early step toward that understanding came from the pages of my textbook Vision Science, with the introduction of the Princeton psychologist Anne Treisman.
“I’ve got something for your reading list, Fei-Fei,” Pietro said, dropping a copy of an article on the desk in front of me.
was an example of the greatest tradition in all of science—the disruption of established ideas, intuitive and familiar as they may be, by a more intricate reality.
They read computer science journals as regularly as anyone else in the department, but they pored over publications like Psychological Review, Proceedings of the National Academy of Sciences, and the especially prestigious Nature just as intently.
Braun was researching a similar hypothesis—that our brains process extensive visual details without our conscious awareness—using what he called the “dual-test method,” in which he engaged a subject’s attention with a central task that required deliberate focus while presenting a peripheral task requiring only passive observation, the high level of attention demanded by the first task ensuring the second won’t be processed consciously.
Because willing subjects were never quite as abundant as we’d have liked, we were at the mercy of their schedules.
But I loved even that. In its own way, it, too, was a part of science.
As important as our experiment was, Pietro and Christof made it clear that a good scientist needs to keep up with the literature as well.
Whereas EEG measures electrical impulses across the brain, which are exceedingly fast but spread diffusely over its surface area, fMRI measures blood oxygen level changes when specific patches of neurons are engaged.
Evolution is parsimonious to a fanatical extent, responding only to environmental pressures so extreme that the alternative to adaptation is extinction.
For all the attention Pietro paid to his ritual, it was easier to talk with him over coffee than lunch, as he’d developed a habit of arranging our trays into colorful tableaus he likened to the work of pop artist David Hockney. Although amusing at first, his “Hockney collages,” as he called them, were mostly an opportunity for him to entertain himself while I grew hungrier and hungrier, remembering how sophisticated his love for art history had once seemed.
Vision works at a higher, more meaningful level, arming us with knowledge—the awareness of leaves we can imagine swaying in the breeze or holding between our fingers, or a branch with a texture and weight that we can instantly estimate, both of which differ dramatically from the untouchable atmosphere and colored light that hangs miles above.
It felt like we were trying to reverse-engineer clockwork crafted with meticulous patience by some unknowable colossus.
There’s no limit on what artificial intelligence might ultimately become, but that began to feel like a secondary point; I was increasingly convinced that this particular challenge—making sense of the visual world by understanding the myriad categories of objects that fill it—was an ideal first step toward unlocking it. It appeared to have worked for our species, after all. I now believed it could work for our machines as well.
I’d found a North Star of my own.
Again and again we were asked about the mathematical core of the machine learning algorithm we chose—a probabilistic technique called a “Bayesian network”—but not a single question about the data we trained it on. While that wasn’t unusual—data was not so subtly dismissed as an inert commodity that only mattered to the extent that algorithms required it—I began to realize that we’d underestimated something important.
They were looking to fill an intern-level analyst position that promised extensive on-the-job experience, which meant that researchers from Ivy League schools with even tenuous connections to math and computer science were ideal candidates.
As exciting as it was to publish our own research, feeling as if we were contributing to the ideas of others—and playing even a small role in their success moving the field forward—was an even greater thrill.
Most of all, I was passionate about my studies, to the point of an almost daily sense of exhaustion.
I met with each once a week, attended journal clubs reviewing the latest literature in both neuroscience and computer science, and, because both labs offered free food, was eating better than I might have expected.
Most of the algorithms we explored were so complex—so “computationally intractable,” to put it more technically—that they couldn’t be configured manually. The range of permutations for their countless parameters was simply too great, like a control panel of knobs and switches stretching beyond the horizon. Instead, automated techniques allowed them to approximate the ideal balance of those parameters through a long, iterative sequence of trial and error.
I returned to the literature again, this time with a vengeance.
It took some digging, but I eventually found something.
I was somehow married, alone, and living with my parents, all at once. Still, my research into the categorizing nature of vision remained the center of my world, and I’d been invited back to Princeton to present my latest work to the computer science department. I’d grown accustomed to delivering lectures by this point, but I’d picked up hints that this invitation might be something more—the first step in a recruitment process, and potentially a faculty position.
She hadn’t attended herself, but a colleague of hers had wound up in the audience and had a feeling she’d appreciate my work, connecting us immediately after.
Barely a year after I’d become an assistant professor at Urbana-Champaign, Princeton offered me a job.
Science may be an incremental pursuit, but its progress is punctuated by sudden moments of seismic inflection—not because of the ambitions of some lone genius, but because of the contributions of many, all brought together by sheer fortune.
And hey, your husband is six hundred miles away, so you’ll definitely have the time.”
There was no escaping the fact that algorithms were the center of our universe in 2006, and data just wasn’t a particularly interesting topic. If machine intelligence was analogous to the biological kind, then algorithms were something like the synapses, or the intricate wiring woven throughout the brain.
As a leading thinker in microprocessor architecture—the art of arranging millions upon millions of nanometer-scale transistors into some of the world’s most sophisticated devices—Professor Kai Li understood the power of exponential thinking better than most.
Kai was the only other Chinese immigrant among Princeton’s computer science faculty. Born in the 1950s, he was part of a generation that put him among the first class of students to attend college in the aftermath of the Cultural Revolution, ultimately coming to America to attend grad school in the 1980s—a period during which such immigration was rare and relatable peers were scarce.
He’d built a reputation as a pioneer in efficiently connecting microprocessors with huge stores of memory, cofounding a company to commercialize his research that eventually sold for more than $2 billion.
His unusual background not only endowed him with engineering skills of a caliber the average computer vision student would be unlikely to have, but spared him the burden of expectations. This was an unorthodox project, if not an outright risky one, and far out of step with the fashions of the field at the time. Jia didn’t know that.
Only nouns referring to physical objects—generally speaking, things tangible enough to be counted: one something, two somethings, a hundred somethings—would be included. Everything else was stripped away. All told, we cut the majority of WordNet’s 140,000 entries, leaving a visual, countable subset just north of 22,000.
“No amount of images will be enough. So whatever we think the number is, we should probably think bigger. Then think even bigger than that. We’re guessing either way, so let’s guess big.” We settled on a goal of one thousand different photographs of every single object category.
“Yeah: nineteen years, give or take. Fei-Fei, I believe in this project— I really do—but I can’t wait that long for my PhD.”
Jia and I met to discuss the issue in the campus’s Mathey Dining Hall, a place I’d grown reliant on as ImageNet’s mental hold on me made the idea of taking time out for cooking all but unbearable.
We talked through every step our labelers followed to identify, categorize, and label each image, streamlining them wherever we could with shortcuts and bespoke tools.
Our lab computers stay put, but a dynamic IP connects us to middlemen that continually change, so Google thinks they’re coming from different users.”
When we found the images in a given category looked too similar, thus diluting the variety we sought, we used international translations of WordNet to submit the query in different languages in the hopes that images from around the world would vary more widely. When we couldn’t find enough images at all, we’d add related terms to the search query, turning “corgi” into “corgi puppy” or “corgi dog park.”
For a guy who’d been designing microprocessor architectures only a year before, these were awfully prosaic engineering challenges. Still, we both knew our efforts were in service of something worthwhile.
