Why economists should learn machine learning

Why economists should learn machine learning

Economists analyze data. Machine learning (ML) offers a powerful set of tools for doing just that. But while econometrics and ML share a foundation in statistics, their aims and philosophies often diverge. The questions they ask and the tools they prioritize can differ dramatically. To clarify these differences—and the reasons economists might ultimately use ML—it helps to begin by deliberately sharpening the contrast between the two.

Quantifying vs. predicting

At its core, econometrics is about explanation. The typical economist is interested in quantifying the effect of a specific variable, often within a framework of causal inference. For example: What is the effect of raising the minimum wage on employment? Do peers influence students’ academic achievements? What is the average wage gap between men and women? These questions focus on estimating one or a few key parameters, with great attention to the rigor of the identification strategy. The emphasis is on the assumptions under which we can identify the parameters and on inference—constructing confidence intervals, testing hypotheses, and, above all, establishing causality.

This approach gained prominence in recent decades, culminating in Nobel Prizes for economists like Joshua Angrist, Guido Imbens, David Card, and Esther Duflo, whose work emphasizes empirical strategies to identify causal effects in natural, field, or experimental settings.

Machine learning, by contrast, is largely concerned with prediction. The primary goal is to develop models—or more precisely, algorithms—that deliver accurate predictions for new data points. Whether it’s recommending movies, classifying elements of an image, translating text, or matching job-seekers with firms, ML prioritizes predictive performance under computational constraints. Rather than focusing on a particular parameter, the goal is to learn complex patterns from data, often using highly flexible (sometimes opaque) models.

That said, forecasting is one area where econometrics and ML converge. Econometric forecasting often imposes structure on messy data to reduce noise, while ML emphasizes complexity and flexibility. Nevertheless, many traditional econometric tasks can be reframed as prediction (sub)problems or built upon them. Estimating a treatment effect, for example, involves building a counterfactual and is inherently a predictive exercise: being able to credibly predict what would have happened to this individual had they not been treated?

Models and assumptions

Econometric models tend to be simple, theory-driven, and interpretable. They often rest on strong assumptions—like linearity or exogeneity—that are difficult to verify but motivated by behavioral or economic theory. These models aim to isolate the effect of a particular variable, not to simulate the entire system.

In ML, simplicity is often sacrificed for performance. Black-box models, such as deep neural networks, are acceptable (and even preferred) if they generate more accurate predictions. A battery of performance metrics—like precision and recall—guide model selection, depending on the stakes. For instance, in fraud detection, a model with high precision ensures that flagged cases are likely real; in cancer screening, high recall ensures few real cases are missed.

Nevertheless, within a particular defined problem, ML offers algorithms whose predictive performance often surpass the standard (non)parametric toolkit in data-rich environments. For example, when selecting a model, they allow modeling complex interactions between variables or being robust to possibly high-dimensional nuisance parameters. The issue is that the theoretical behavior of these tools is often intractable, making them difficult to use within the classic econometric framework. Fortunately, over the past fifteen years, econometric theory has advanced to incorporate ML techniques in a way that enables statistical inference—allowing researchers to understand the working assumptions and their limits, construct tests, and build confidence intervals using ML-powered estimators.

Data and deployment

Another important divergence lies in how data is used. Econometric models are typically built on a single dataset, intended for a specific study. Replication is possible, but each new dataset generally leads to a different model. The focus is on understanding a particular phenomenon using the data at hand.

In ML, models are developed to be deployed in production, where they will continuously generate predictions as new data becomes available. This makes it crucial to guard against overfitting—when a model performs well on training data but poorly on unseen data. This risk is mitigated by techniques like cross-validation, and by splitting data into training and test sets. Modern ML even grapples with new phenomena like “double descent” where larger models trained on more data can paradoxically generalize better.

Complex data, new frontiers

ML’s rise is partly fueled by its success in handling complex, unstructured data—images, text, audio—that traditional statistical approaches struggle to process. These data types don’t fit neatly into rows and columns, and extracting meaningful features from them requires sophisticated techniques from computer science. ML excels in these domains, often matching human-level performance on tasks like facial recognition or language translation. As such, ML is the key ingredient to compress or extract information from such unstructured datasets, unlocking new possibilities.

Think about it:

classifying the sentiment of an internet review on a numerical scale to enter a regression model,compressing a product image into a fixed-size vector (an embedding) to analyze consumer behavior,measuring the tone of a central banker’s speech.

Text data is undoubtedly one of the richest sources of economic information that largely remains out-of-reach for traditional econometric approaches.

A two-way street

The distinctions above are real, but they are not absolute. Economists have long used prediction tools, and ML researchers are increasingly concerned with issues that economists know well: fairness, bias, and explainability. Recent public controversies—from racial bias in criminal risk algorithms (e.g., the COMPAS tool) to gender stereotypes in language models—have underscored the social consequences of automated decision-making.

Likewise, econometrics is not immune to methodological pitfalls. The replication crisis, “p-hacking,” and specification searching can be seen as forms of overfitting problems that ML addresses through careful validation practices. Techniques like pre-analysis plans (committing to a set of statistical tests before receiving the data in order to reduce false positives) have been adopted by economists to mitigate these risks. However, possible solutions can draw inspiration from ML’s train/test split approach.

Bridging the divide

So, should economists learn machine learning? Absolutely. ML extends the standard econometric toolkit with methods that improve predictive performance, extract insights from text and images, and enhance robustness in estimation. For economists looking to stay at the frontier of empirical research—especially in a data-rich world—ML is not just useful. It’s essential.

Feature image credit: Photo by NOAA sur Unsplash.

OUPblog - Academic insights for the thinking world.