More on this book
Community
Kindle Notes & Highlights
the point of predictive analytics is not the relative size or unruliness of your data, but what you do with it. I have found that “big data often equals small math,” and many big data practitioners are content just to use their data to create some appealing visual analytics. That’s not nearly as valuable as creating a predictive model.
In short, we live in a predictive society. The best way to prosper in it is to understand the objectives, techniques, and limits of predictive models. And the best way to do that is simply to keep reading this book.
Yesterday is history, tomorrow is a mystery, but today is a gift. That’s why we call it the present. —Attributed to A. A. Milne, Bill Keane, and Oogway, the wise turtle in Kung Fu Panda
predictive analytics combats financial risk, fortifies healthcare, conquers spam, toughens crime fighting, and boosts sales.
You cannot see the future because it isn’t here yet. We find a work-around by building machines that learn from experience. It’s the regimented discipline of using what we do know—in the form of data—to place increasingly accurate odds on what’s coming next.
The only source of knowledge is experience. —Albert Einstein
data embodies a priceless collection of experience from which to learn.
As data piles up, we have ourselves a genuine gold rush. But data isn’t the gold. I repeat, data in its raw form is boring crud. The gold is what’s discovered therein.
How come you never see a headline like “Psychic Wins Lottery”? —Jay Leno
Predictive analytics (PA)—Technology that learns from experience (data) to predict the future behavior of individuals in order to drive better decisions.
forecasting. Forecasting makes aggregate predictions on a macroscopic level. How will the economy fare?
PA leads within the growing trend to make decisions more “data driven,” relying less on one’s “gut” and more on hard, empirical evidence.
PA is the process by which an organization learns from the experience it has collectively gained across its team members and computer systems. In fact, an organization that doesn’t leverage its data in this way is like a person with a photographic memory who never bothers to think.
The leading free, open-source software tool for PA, called R (a one-letter, geeky name), has a rapidly expanding base of users as well as enthusiastic volunteer developers who add to and support its functionalities.
Machine learning’s task is to find patterns that appear not only in the data at hand, but in general, so that what is learned will hold true in new situations never yet encountered. At the core, this ability to generalize is the magic bullet of
What often happens to you that cannot be witnessed, and that you can’t even be sure has happened afterward—but that can be predicted in advance?
Want to come up with your own new innovative use for PA? You need only two ingredients. Each application of PA is defined by: 1. What’s predicted: The kind of behavior (i.e., action, event, or happening) to predict for each individual, stock, or other kind of element. 2. What’s done about it: The decisions driven by prediction; the action taken by the organization in response to or informed by each prediction.
What is the great revolution of science in the last 10, 15 years? It is the movement from the search for universals to the understanding of variability. Now in medical science we don’t want to know . . . just how cancer works; we want to know how your cancer is different from my cancer.”
Predictive model—A mechanism that predicts a behavior of an individual, such as click, buy, lie, or die. It takes characteristics of the individual as input, and provides a predictive score as output. The higher the score, the more likely it is that the individual will exhibit the predicted behavior.
But all predictive models share the same objective: They consider the various factors of an individual in order to derive a single predictive score for that individual. This score is then used to drive an organizational decision, guiding which action to take.
For this reason, machine learning is also called predictive modeling, which is a more common term in the commercial world. If deferring to the older metaphorical term data mining, the predictive model is the unearthed gem.
Predictive modeling generates the entire model from scratch. All the model’s math or weights or rules are created automatically by the computer. The machine learning process is designed to accomplish this task, to mechanically develop new capabilities from data. This automation is the means by which PA builds its predictive power.
Knowing is not enough; we must act. —Johann Wolfgang von Goethe
The predictive score for each individual directly informs the decision of what action to take with that individual.
Predictive scores issue imperatives to mail, call, offer a discount, recommend a product, show an ad, expend sales resources, audit, investigate, inspect for flaws, approve a loan, or buy a stock. By acting on the predictions produced by machine learning, the organization is now applying what’s been learned, modifying its everyday operations for the better.
we have mangled the English language. Proponents like to say that predictive analytics is actionable. Its output directly informs actions, commanding the organization about what to do next. But with this use of vocabulary, industry insiders have stolen the word actionable, which originally has meant worthy of legal action (i.e., “sue-able”), and morphed it. This verbal assault comes about because people are so tired of seeing sharp-looking reports that provide only a vague, unsure sense of direction.
Thomas Davenport and Jeanne Harris put it in Competing on Analytics: The New Science of Winning, “At a time when companies in many industries offer similar products and use comparable technology, high-performance business processes are among the last remaining points of differentiation.” Enter predictive analytics. Survey results have in fact shown that “a tougher competitive environment” is by far the strongest reason why organizations adopt this technology.
Before each launch, organizations establish confidence in PA by “predicting the past” (aka backtesting). The predictive model must prove itself on historical data before its deployment. Conducting a kind of simulated prediction, the model evaluates across data from last week, last month, or last year. Feeding on input that could only have been known at a given time, the model spits out its prediction, which then matches against what we now already know took place thereafter.
predictive modeling is a kind of reverse engineering to begin with. Machine learning starts with the data, an encoding of things that have happened, and attempts to uncover patterns that generated or explained the data in the first place. John was attempting to deduce what the other team had deduced. His guide? Informal hunches and ill-informed inferences, each of which could be pursued only by way of trial and error, testing each hypothetical mess-up he could dream up by programming it by hand and comparing it to the retrospective predictions he had been given.
Every new beginning comes from some other beginning’s end. —From the song “Closing Time” by Semisonic
the predictive modeling process is a form of automated data crunching that learns from training examples, which must include both positive and negative examples. An organization needs to have positively identified in the past some cases of what it would like to predict in the future. To predict something like “Will buy a stereo,” you can bet a retailer has plenty of positive cases. But how can you locate Target customers known to be pregnant?
Information about transactions, at some point in time, will become more important than the transaction themselves. —Walter Wriston, former chairman and CEO of Citicorp
A joint study by Columbia University and Ben Gurion University (Israel) showed that hungry judges rule negatively. Judicial parole decisions immediately after a food break are about 65 percent favorable, but then drop gradually to almost zero percent before the next break. If your parole board judges are hungry, you’re more likely to stay in prison.
The emotions aren’t always immediately subject to reason, but they are always immediately subject to action. —William James
Predictive modeling derived one model, for example, that identifies anxious blog posts by the presence of words such as nervous, scared, interview, and hospital, and, conversely, by the absence of words that are common within blog posts that are not anxious, such as yay, awesome, and love.
Free public data is also busting out, so a wealth of knowledge sits at your fingertips. Following the open data movement, often embracing a not-for-profit philosophy, many data sets are available online from fields like biodiversity, business, cartography, chemistry, genomics, and medicine. Look at one central index, www.kdnuggets.com/datasets, and you’ll see what amounts to lists of lists of data resources. The Federal Chief Information Officer of the United States launched Data.gov “to increase public access to high value, machine readable datasets generated by . . . the Government.”
...more
Data always speaks. It always has a story to tell, and there’s always something to learn from it. Data scientists see this over and over again across PA projects. Pull some data together and, although you can never be certain what you’ll find, you can be sure you’ll discover valuable connections by decoding the language it speaks and listening. That’s The Data Effect in a nutshell.
Prediction starts small. PA’s building block is the predictor variable, a single value measured for each individual. For example, recency, the number of weeks since the last time an individual made a purchase, committed a crime, or exhibited a medical symptom, often reveals the chances he or she will do it again in the near term. In many arenas, it makes sense to begin with the most recently active people first, whether for marketing contact, criminal investigation, or clinical assessment.
Similarly, frequency—the number of times the individual has exhibited the behavior—is also a common, fruitful measure. People who have done something a lot are more likely to do it again.
In fact, it is usually what individuals have done that predicts what they will do. And so PA feeds on data that extends past dry yet essential demographics like location and gender to include behavioral predictors such as recency, frequency, purchases, financial activity, and product usage such as calls and web surfing. These behaviors are often the most valuable—it’s always a behavior that we seek to predict, and indeed behavior predicts behavior. As Jean-Paul Sartre put it, “[A man’s] true self is dictated by his actions.
When applying PA, we usually don’t know about causation, and we often don’t necessarily care. For many PA projects, the objective is more to predict than it is to understand the world and figure out what makes it tick.
PA a kind of “metascience” that transcends the taxonomy of natural and social sciences, abstracting across them by learning from any and all data sources that would typically serve biology, criminology, economics, education, epidemiology, medicine, political science, psychology, or sociology. PA’s mission is to engineer solutions. As for the data employed and the insights gained, the tactic in play is: “Whatever works.
The heart has its reasons which reason knows nothing of . . . We know the truth not only by the reason, but by the heart. —Blaise Pascal
adjustment in the formation of these diamonds. In particular, emotional intensity is relative. It’s the change in intensity that tells us something. Rather than tracking the absolute level of anxiety, the Anxiety Index depicted in the figure above measures how quickly overall anxiety is changing from one day to the next.
The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” but rather “Hmm . . . that’s funny . . . ” —Isaac Asimov
PA fosters serendipity. Predictive modeling conducts a broad, exploratory analysis, testing many predictors, and in so doing uncovers surprising findings,
Dutch firm SNTMNT (“sentiment”) provides an API to enable any party to trade based on public sentiment as expressed on Twitter. “There are a lot of smart people (quietly) trading on textual sentiment of the news and tweets,” financial trading and predictive analytics expert Ben Gimpert told me in an e-mail.
Most discussions of decision making assume that only senior executives make decisions or that only senior executives’ decisions matter. This is a dangerous mistake. — Peter Drucker, an American educator and writer born in 1909
Eric Webster, a vice president at State Farm Insurance, put it brilliantly: “Insurance is nothing but management of information. It is pooling of risk, and whoever can manipulate information the best has a significant competitive advantage.” Simply put, these companies are in the business of prediction.
we can define PA in these very terms.3 Here’s the original definition: Predictive analytics (PA)—Technology that learns from experience (data) to predict the future behavior of individuals in order to drive better decisions.
