Nate Silver's Blog, page 86

Politics Podcast: Trump Meets Kim

FiveThirtyEight

More: Apple Podcasts |
ESPN App |
RSS
| Embed

Embed Code

President Trump is hours away from sitting down with North Korean leader Kim Jong Un. The FiveThirtyEight Politics podcast uses some game theory to discuss what got us to this point and what could happen next. The crew also asks what on earth happened at the G7 summit in Canada over the weekend and debates the merits of election reforms in Maine.

You can listen to the episode by clicking the “play” button above or by downloading it in iTunes , the ESPN App or your favorite podcast platform. If you are new to podcasts, learn how to listen .

The FiveThirtyEight Politics podcast publishes Monday evenings, with occasional special episodes throughout the week. Help new listeners discover the show by leaving us a rating and review on iTunes . Have a comment, question or suggestion for “good polling vs. bad polling”? Get in touch by email, on Twitter or in the comments.

View more on Nate Silver's website »

Like • 0 comments • flag

Published on June 11, 2018 14:40

Sometimes The Parties Do Decide, After All

In the wake of the 2016 presidential nomination process, it became fashionable to talk about the declining influence of the two major U.S. political parties on their voters. That’s for pretty good reason. The Republican nomination of Donald Trump, despite his tepid support from Republican elected officials and his frequent breaks from Republican orthodoxy, defied the scholarly consensus 1 about how “party elites” are supposed to have sway over their voters. Whether Bernie Sanders’s vigorous run against Hillary Clinton also fits the pattern is a more difficult question2 — but there was certainly plenty of conflict on display between Democratic party officials and a sizable contingent of Democratic voters.

It’s important to remember, however, that even if the parties’ power over their voters has declined to some extent, they’re still extremely powerful institutions that get their way most of the time. And in Tuesday’s primaries in California, New Jersey and six other states, the parties had a really good night.

That’s not to say that the parties pitched a perfect game. In the U.S. House primary in California’s 50th Congressional District, for example, the Democratic Congressional Campaign Committee’s preferred candidate, Josh Butner, projects to finish only in third or fourth place, behind Republican incumbent Duncan Hunter and another Democrat, Ammar Campa-Najjar. But by luck or design — undoubtedly a bit of both since there were a number of close calls — it looks as though Democrats may avoid having any districts in California where there are no Democrats on the November U.S. House ballot. (In California, the top two finishers advance to the general election regardless of political party, potentially leading to one of the parties being shut out of the November ballot.) That’s a real boon to the Democrats’ prospects in a state where at least nine Republican-held U.S. House seats are potentially in play.

Meanwhile, the Democratic candidates who had the lion’s share of endorsements from party officials finished first in the primaries for U.S. senator and governor in California. In the Senate race, incumbent Sen. Dianne Feinstein easily outdistanced another Democrat, state Sen. Kevin de León, although it looks as though de León will hold on for second place. And in the gubernatorial race, we’ll have a battle between fairly traditional, establishment-backed nominees in Democrat Gavin Newsom, the lieutenant governor, and Republican John Cox.

Sometimes, the parties get their way even if it isn’t necessarily in their electoral best interest. In New Jersey, incumbent senator Bob Menendez was re-nominated, but an obscure Democratic candidate who raised almost no money and who had never held elected office, Lisa McCormick, received 38 percent of the vote. Menendez probably won’t put the seat at risk for the Democrats in November — one recent poll shows a close general election race there, but two others don’t — but with another Democratic nominee who didn’t have Menendez’s baggage, the seat would be almost certain to remain blue. That leaves open the question of whether a better-funded, more recognizable primary challenger could have beaten Menendez, whose popularity waned amid a corruption trial last year. (The trial resulted in a hung jury, and the corruption charges were dismissed earlier this year.) The answer is: possibly yes. Such a candidate would at least have made for a heck of race. It wasn’t a coincidence, however, that no such candidate existed, since the New Jersey Democratic Party has a long history of clearing the field for its preferred nominee.

The Republican establishment also had a good night, including the strong second-place finish for Cox in California. Cox got more of the vote than polls projected after receiving the endorsement of President Trump. Thus, California Republicans will avoid the potential turnout-depressing nightmare of having neither a gubernatorial nor U.S. Senate candidate on the ballot in November.

In Montana, the establishment-backed Matt Rosendale, the state auditor, won the Republican Senate nomination; he’ll face a slightly uphill but by no means unwinnable race against Democratic incumbent Jon Tester. And in South Dakota, the moderate Republican Dusty Johnson, a former chief of staff to Gov. Dennis Daugaard, won the Republican nomination for the at-large U.S. House seat; he’ll probably avoid putting the seat at risk against Democrat Tim Bjorkman, whereas a further-right nominee might have made things more interesting. These results come on the heels of last month’s primaries in West Virginia, which were another fairly successful night for the Republican establishment. Insurgent Don Blankenship finished in a distant third place behind state Attorney General Patrick Morrisey in the U.S. Senate primary.

Again, this is not to suggest that the political parties bat 1.000. Keep in mind, for instance, that there is currently a Democratic senator from Alabama because the state’s Republican Party could not figure out how to prevent its voters from backing an accused child molester, Roy Moore. But these cases are still more the exceptions than the rule, even if the exceptions have become more frequent.

CORRECTION (June 6, 2018, 11:02 a.m.): An earlier version of this article incorrectly described a Republican candidate for the U.S. House seat in South Dakota as the chief of staff to the governor. Dusty Johnson left that position in 2014.

View more on Nate Silver's website »

Like • 0 comments • flag

Published on June 06, 2018 03:53

Politics Podcast: A Conversation About Our Pollster Ratings

FiveThirtyEight

More: Apple Podcasts |
ESPN App |
RSS
| Embed

Embed Code

FiveThirtyEight has updated its pollster ratings for the first time since before the 2016 presidential election. In this episode of the FiveThirtyEight Politics podcast, Nate Silver discusses the state of polling in the U.S. and what to expect in 2018.

You can listen to the episode by clicking the “play” button above or by downloading it in iTunes , the ESPN App or your favorite podcast platform. If you are new to podcasts, learn how to listen .

The FiveThirtyEight Politics podcast publishes Monday evenings, with occasional special episodes throughout the week. Help new listeners discover the show by leaving us a rating and review on iTunes . Have a comment, question or suggestion for “good polling vs. bad polling”? Get in touch by email, on Twitter or in the comments.

View more on Nate Silver's website »

Like • 0 comments • flag

Published on May 31, 2018 06:44

Which Pollsters To Trust In 2018

As FiveThirtyEight has evolved over the past 10 years, we’ve taken an increasingly “macro” view of polling. By that, I mean: We’re more interested in how the polls are doing overall — and in broad trends within the polling industry — and less in how individual polls or pollsters are performing. As we described in an article earlier this week, overall the polls are doing … all right. Contrary to the narrative about the polls, polling accuracy has been fairly constant over the past couple of decades in the U.S. and other democratic countries.

Still, in election coverage, the “micro” matters too, and our newly updated pollster ratings — in which we evaluate the performance of individual polling firms based on their methodology and past accuracy — are still a foundational part of FiveThirtyEight. They figure into the algorithms that we design to measure President Trump’s approval ratings and to forecast elections (higher-rated pollsters get more weight in the projections). And sometimes those pollster ratings can reveal broad trends too: For example, after a reasonably strong 2012, online polls were fairly inaccurate in 2016.

The ratings also allow us to measure pollster performance over a large sample of elections — rather than placing a disproportionate amount of emphasis on one or two high-profile races. For instance, Rasmussen Reports deserves a lot of credit for its final, national poll of the 2016 presidential election, which had Hillary Clinton ahead by 2 percentage points, almost her exact margin of victory in the popular vote. But Rasmussen Reports polls are conducted by a Rasmussen spinoff called Pulse Opinion Research LLC, and state polls conducted by Rasmussen and Pulse Opinion Research over the past year or two have generally been mediocre.

So which pollsters have been most accurate in recent elections? Because some races are easier to poll than others, we created a statistic called Advanced Plus-Minus to evaluate pollster performance. It compares a poll’s accuracy to other polls of the same races and the same types of election. Advanced Plus-Minus also adjusts for a poll’s sample size and when the poll was conducted. (For a complete description, see here; we haven’t made any changes to our methodology this year.) Negative plus-minus scores are good and indicate that the pollster has had less error than other pollsters in similar types of races.

The table below contains Advanced Plus-Minus scores for the most prolific pollsters — those for whom we have at least 10 polls in our database for elections from Nov. 8, 2016 onward. These polls cover the 2016 general election along with any polling in special elections or gubernatorial elections since 2016.

How prolific pollsters have fared in recent elections

Advanced Plus-Minus scores for pollsters’ surveys conducted for elections on Nov. 8, 2016, and later

pollster
Methodology
No. of Polls
Avg. Error
Advanced Plus-Minus
Bias

Monmouth University
Live
24
4.8
-1.5
D+3.9

Emerson College
IVR
51
4.1
-1.0
D+1.2

Siena College
Live
18
4.0
-0.9
D+1.5

Landmark Communications
IVR/online
14
4.4
-0.6
D+4.3

Marist College
Live
17
3.7
-0.6
D+1.5

Lucid
Online
14
2.6
-0.4
D+2.4

SurveyUSA
IVR/online/live
18
4.5
-0.2
D+1.0

Trafalgar Group
IVR/online/live
15
4.0
-0.1
R+0.8

YouGov
Online
33
4.3
+0.0
D+2.8

Opinion Savvy
IVR/online
11
4.3
+0.1
D+2.8

Quinnipiac University
Live
26
4.4
+0.1
D+4.2

Rasmussen Reports/Pulse Opinion Research
IVR/online
55
5.1
+0.4
D+3.6

CNN/Opinion Research Corp.
Live
10
4.3
+0.6
D+1.4

Gravis Marketing
IVR/online
53
4.6
+0.7
D+2.5

Remington Research Group
IVR/live
32
4.9
+0.8
D+2.1

Public Policy Polling
IVR/online
28
5.2
+1.0
D+5.2

SurveyMonkey
Online
195
7.3
+2.3
D+5.6

University of New Hampshire
Live
19
8.9
+3.4
D+8.9

Google Surveys
Online
12
8.4
+5.0
D+1.8

Negative plus-minus scores are good and indicate that the pollster has had less error than other pollsters in similar types of races.

The “average error” is the difference between the polled result and the actual result for the margin separating the top two finishers in the race.

“Bias” is a pollster’s average statistical bias toward Democratic or Republican candidates.

The best of these pollsters over this period has been Monmouth University, which has an Advanced Plus-Minus score of -1.5. That’s not a huge surprise — Monmouth was already one of our highest-rated pollsters. After that, the list is somewhat eclectic, including traditional, live-caller pollsters such as Siena College and Marist College, as well as automated pollsters such as Emerson College and Landmark Communications. Polling institutes run by colleges and universities are somewhat overrepresented among the high performers on the list and have generally become a crucial source of polling as other high-quality pollsters have fallen by the wayside.

The lowest-performing pollsters in this group are the University of New Hampshire’s Survey Center, Google Surveys and SurveyMonkey. UNH uses traditional telephone interviewing, but its polls were simply way off the mark in 2016, overestimating Democrats’ performance by an average of almost 9 percentage points in the polls it conducted of New Hampshire and Maine.

Google Surveys and SurveyMonkey are newer and more experimental online-based pollsters. Google Surveys has an unusual methodology in which it shows people a poll in lieu of an advertisement and then infers respondents’ demographics based on their web browsing habits. While national polls that used the Google Surveys platform got fairly good results both in 2012 and 2016, state polls that used this technology have generally been highly inaccurate. Some Google Surveys polls also have a highly do-it-yourself feel to them, in that members of the public can use the Google Surveys platform to create and run their own surveys. We at FiveThirtyEight are going to have to do some thinking about whether to include these types of do-it-yourself polls in our averages and forecasts.

SurveyMonkey, which sometimes partners with FiveThirtyEight on non-election-related polling projects, conducted polling in all 50 states in 2016, asking about both the presidential election and races for governor and the U.S. Senate. Unlike some other attempts to poll all 50 states,1 SurveyMonkey took steps to ensure that each state was weighed individually and that respondents to the poll were located within the correct state. Thus, FiveThirtyEight treated these polls as we did any other state poll. Unfortunately, the results just weren’t good, with an average error2 of 7.3 percentage points and an Advanced Plus-Minus score of +2.3.

It wasn’t just Google Consumer Surveys or SurveyMonkey, however — overall, online polls (with some exceptions such as YouGov and Lucid) have been fairly unreliable in recent elections. So have the increasing number of polls that use hybrid or mixed methodologies, such as those that mostly poll using automated calls (also sometimes called IVR or interactive voice response) but supplement these results using an online panel.

In the chart below, I’ve calculated Advanced Plus-Minus scores and other statistics based on the technologies the polls used. An increasing number of polling firms no longer fall cleanly into one category and instead routinely use more than one mode of data collection within the same survey or switch back and forth from one methodology to the next from poll to poll. Therefore, I’ve distinguished polls that use one methodology exclusively from those that employ mixed methods.

Online polls have been less accurate in recent elections

Advanced Plus-Minus scores for pollsters’ surveys conducted for elections on Nov. 8, 2016, and later

Poll type
No. of Polls
Average Error
Adv. Plus-Minus
Bias

Live caller
77
4.9
+0.1
D+2.2

Live caller only
62
4.8
-0.1
D+2.5

Live caller hybrid
15
5.2
+0.7
D+1.2

IVR
35
4.6
-0.0
D+2.0

IVR only
13
4.5
-0.7
D+0.8

IVR hybrid
17
4.7
+0.4
D+2.6

Online
32
5.3
+1.1
D+3.0

Online only
15
5.4
+1.6
D+3.3

Online hybrid
17
5.1
+0.7
D+2.8

All pollsters
119
4.9
+0.3
D+2.4

Negative plus-minus scores are good and indicate that the pollster has had less error than other pollsters in similar types of races. Averages are weighted based on the square root of the number of polls that each firm conducted. Pollsters that are banned by FiveThirtyEight because we know or suspect that they faked their data are not included in the averages.

The clearest trends are that telephone polls — including both live caller and IVR polls — have outperformed online polls in recent elections and that polls using mixed or hybrid methods haven’t performed that well.

The relatively strong performance of IVR polls is surprising, considering that automated polls are not supposed to call cellphones and that more than half of U.S. households are now cellphone-only. It ought to be difficult to conduct a representative survey given that constraint.

We’ve sometimes seen the claim that IVR polls are more accurate because people are more honest about expressing support for “politically incorrect” candidates such as Trump when there isn’t another human being on the other end of the phone. This feeling of greater anonymity would presumably also apply to online polls, however, and online polls have not been very accurate lately (and they tended to underestimate Trump in 2016).

Another answer may be that the IVR polls were more lucky than good in 2016. In general, online polls tend to show more Democratic-leaning results, IVR polls tend to show more Republican-leaning results, and live-caller polls are somewhere in between. Thus, in years such as 2012 when Democratic candidates beat the polling averages, online polls tend to look good, and in years when Republicans outperform their polls, IVR polls look good. If undecided voters largely broke to Trump in 2016, polls that initially had too many Republicans in their samples would wind up performing well.

Over the long run, the highest-performing pollsters have been those that:

Exclusively use live-caller interviews, including calls placed to cellphones, and
Participate in professional initiatives that encourage transparency and disclosure.3

FiveThirtyEight’s pollster ratings will continue to award a modest bonus to pollsters that meet one or both of these standards and apply a modest penalty to those that don’t. Thus, the letter grades you see associated with polling firms are based on a combination of their historical accuracy and their methodological standards. Polling firms with non-standard methodologies can sometimes have individual races or even entire election cycles in which they perform quite well. But they don’t always sustain their performance over the long run.

As for online polls, we don’t want to discourage experimentation or to draw too many conclusions from just one cycle’s worth of polling. But we at FiveThirtyEight are becoming skeptical of what you might call bulk or “big data” approaches to polling using online platforms. The polling firms that get the best results tend to be those that poll no more than about six to eight states and put a lot of thought and effort into every poll. Online firms may want to do less national polling and fewer 50-state experiments and concentrate more on polling in electorally important states and congressional districts. Results in these contests will go a long way toward determining whether online polling is an adequate substitute for telephone polling.

View more on Nate Silver's website »

Like • 0 comments • flag

Published on May 31, 2018 04:00

The Polls Are All Right

With the 2018 midterm elections approaching, we’ve updated FiveThirtyEight’s pollster ratings for the first time since the 2016 presidential primaries. Based on how the media portrayed the polls after President Trump’s victory over Hillary Clinton later that year, you might expect pollsters to get a pretty disastrous report card.

But here’s a stubborn and surprising fact — and one to keep in mind as midterm polls really start rolling in: Over the past two years — meaning in the 2016 general election and then in the various gubernatorial elections and special elections that have taken place in 2017 and 2018 — the accuracy of polls has been pretty much average by historical standards.

You read that right. Polls of the November 2016 presidential election were about as accurate as polls of presidential elections have been on average since 1972. And polls of gubernatorial and congressional elections in 2016 were about as accurate, on average, as polls of those races since 1998. Furthermore, polls of elections since 2016 — meaning, the 2017 gubernatorial elections and the various special elections to Congress this year and last year — have been slightly more accurate than average. This isn’t just a U.S. phenomenon: Despite often inaccurate and innumerate criticism over how polling fared in events like Brexit, a recent, comprehensive study of polling accuracy by Professor Will Jennings of the University of Southampton and Professor Christopher Wlezien of the University of Texas at Austin found polling accuracy has been fairly consistent over the past several decades in a variety of democratic countries in Europe, Asia and the Americas.

The media narrative that polling accuracy has taken a nosedive is mostly bullshit, in other words. Polls were never as good as the media assumed they were before 2016 — and they aren’t nearly as bad as the media seems to assume they are now. In reality, not that much has changed.

That’s not to say there aren’t reasons for concern. National polls were pretty good in the 2016 presidential election, but state-level polling was fairly poor (although still within the “normal” range of accuracy). Polls of the 2016 presidential primaries were sometimes way off the mark. And in many recent elections, the polls were statistically biased in one direction or another — there was a statistical bias toward Democrats in 2016, for instance.

There’s also reason to worry about what’s going into the polls as response rates to polls decline and as newsrooms cut their budgets for traditional, high-quality surveys. Internet-based polling may eventually be a part of the solution, but for the most part,1 it was quite inaccurate in 2016 (we’ll go into more detail on this point in another article later this week). Let’s dig further into the evidence:

FiveThirtyEight’s pollster ratings database, which you can download here, includes all polls in the final 21 days2 of gubernatorial and congressional3 elections since 1998 and presidential primaries and general elections since 2000. It also includes polling of special elections for these offices, such as the race in Pennsylvania’s 18th Congressional District in March. We’ve also done a bit of cleanup on the pre-2016 polls in our database (see the footnotes for details4). In total, the database contains more than 8,500 polls.

Our preferred way to evaluate poll accuracy is simply to compare the margin in the poll against the actual result.5 For instance, if a poll showed the Democrat winning by 2 percentage points in a race that the Republican ended up winning by 3 points, that would be a 5-point error. It would also be a 5-point error if the Democrat won by 7 points. We consider these errors to be equally bad even though the pollster “called” the winner correctly in one case and failed to do so in the other.

In the table below is the average error in different types of races and different election cycles since 1998. A few methodological notes as you browse through it: Special elections and off-year elections are grouped with the next even-numbered year; for instance, the 2009 Virginia gubernatorial race is included as part of the 2009-10 political cycle. Pollsters that are banned by FiveThirtyEight because we know or suspect that they faked their data or we are otherwise not confident in the legitimacy of their polling operation are not included in the averages. And some polling firms are considerably more prolific than others — for instance, Gravis Marketing conducted 33 polls of the 2016 presidential general election that count in our database, while the Iowa-based Selzer & Co. conducted just three. To partly counteract this, the averages are weighted in such a way6 that the highly prolific firms don’t dominate too much.

How accurate have U.S. polls been?

Weighted-average error in polls in final 21 days of the campaign

Presidential
State-Level

Cycle
Primary
General
Governor
U.S. Senate
U.S. House
Combined

2017-18
—
—
5.2
6.0
4.1
5.1

2015-16
10.1
4.8
5.4
5.0
5.5
6.8

2013-14
—
—
4.4
5.4
6.7
5.4

2011-12
8.9
3.6
4.8
4.7
4.7
5.1

2009-10
—
—
4.9
4.8
6.9
5.7

2007-08
7.4
3.6
4.1
4.7
5.7
5.4

2005-06
—
—
5.0
4.2
6.5
5.3

2003-04
7.1
3.2
6.1
5.6
5.4
4.8

2001-02
—
—
5.2
4.9
5.4
5.2

1999-2000
7.6
4.4
4.9
6.1
4.4
5.5

1998
—
—
8.1
7.4
6.8
7.5

All years
8.7
4.0
5.4
5.4
6.2
5.9

Pollsters that are banned by FiveThirtyEight because we know or suspect that they faked their data are not included in the averages. Averages are weighted based on the number of polls a particular firm conducted. Specifically, the weights are based on the square root of the number of polls in a particular category that each firm conducted. Each poll receives a weight of sqrt(n)/n, where “n” is that firm’s number of polls in that category.

There’s quite a lot of info to digest in that table. But it’s worth starting with the number in the bottom right corner: The average error in all polls conducted in the late stage of campaigns since 1998 is about 6 percentage points. If the average error is 6 points, that means the true, empirically derived margin of error (or 95 percent confidence interval) is closer to 14 or 15 percentage points! That’s much more than you’d infer from the margins of error that pollsters traditionally list, which consider only sampling error and not other potential sources of error and which pertain only to one candidate’s vote share and not the margin between the candidates.

This means that you shouldn’t be surprised when a candidate who had been trailing in the polls by only a few points wins a race. And in some cases, even a poll showing a 10- or 12- or 14- point lead isn’t enough to make a candidate’s lead “safe.”

In other cases, you can expect a bit more accuracy. For instance, you may have more than one poll, and polling averages are more accurate than individual polls. The average isn’t foolproof — it doesn’t help when all the polls miss in the same direction — but you’re usually better off taking your chances with it than with individual surveys. And polls are slightly more accurate in the final few days of the campaign than in the final few weeks, although the accuracy gains are more modest than you might assume.7

Polling error also varies based on the type of election. The general rules of thumb are that polling error in the primaries is much greater than in the general election and that polling error increases the further down the ballot you go. Thus, polls for U.S. House races are more error-prone than gubernatorial or U.S. Senate polls, which in turn are more error-prone than presidential election polls. This is important to remember once you start seeing polls of House races later this year. While some will be spot-on, many others will be off by 5 or 10 points or even more — and this will be perfectly “normal.”

This is a lot of words to spend without addressing the question of how the polls performed in 2016. In the case of House, Senate and gubernatorial polls — as the table shows — the answer is a pretty straightforward “about average” (and in the case of House polls, maybe even slightly better than average).

It’s also relatively easy to address the case of presidential primary polls: They were pretty darn bad in 2016, with an average error of 10.1 percentage points. Polling the primaries is hard — the average polling error in all presidential primaries since 2000 is 8.7 percentage points. But primary polls aren’t usually as bad as they were in 2016. Because voting in general elections operates along increasingly predictable demographic lines, pollsters can use demographic weighting to make up for other problems in their samples. They don’t always have that luxury in the primaries, where demographic coalitions are more fluid and turnout is more difficult to model. Polling in the 2020 primaries could be a pretty wild ride.

Polling of the 2016 presidential general election is the trickiest case to evaluate. The average error was 4.8 percentage points — slightly higher than in 2000 (4.4 points) and considerably higher than in 2004 (3.2 points), 2008 (3.6 points) or 2012 (3.6 points).

However, the error was about average as compared to the long-term accuracy of presidential polls. Our 2016 presidential election model gave Trump a much better chance than other forecasts did, in part because it derived its probabilities based on polls from elections dating back to 1972 (not just since 2000). Our data from those earlier election cycles isn’t quite as comprehensive or well-curated as the stuff in our official pollster ratings database. But it’s certainly good enough for a comparison on an aggregate basis. Below are the error calculations for polls of presidential elections dating back to 1972. I’ve listed the error for state polls and national polls separately and combined.

2016′s presidential polls were about as accurate as average

Weighted-average error in polls in final 21 days of the campaign

Cycle
National
State
Combined

2016
3.1
5.2
4.8

2012
3.3
3.7
3.6

2008
2.3
3.9
3.6

2004
2.2
3.5
3.2

2000
3.9
4.6
4.4

1996
6.4
4.8
5.3

1992
4.6
5.2
5.1

1988
3.5
5.0
4.6

1984
5.4
4.5
4.7

1980
8.9
8.6
8.6

1976
2.5
3.8
3.4

1972
2.6
4.6
4.3

Average
4.1
4.8
4.6

Pollsters that are banned by FiveThirtyEight because we know or suspect that they faked their data are not included in the averages. Averages are weighted based on the number of polls a particular firm conducted.

On average since 1972, polls in the final 21 days of presidential elections have missed the actual margins in those races by 4.6 percentage points, almost exactly matching the 4.8-point error we saw in 2016. As we tried to emphasize before the election, it didn’t take any sort of extraordinary, unprecedented polling error for Trump to defeat Clinton. An ordinary, average polling error would do — one where Trump beat his polls by just a few points in just a couple of states — and that’s the polling error we got.

That error was concentrated much more in state polls, which missed by an average of 5.2 percentage points, than in national polls, which missed by just 3.1 percentage points.8 This is somewhat typical, as national polls have been more accurate than state polls over the long run. The gap in 2016 was larger than usual, however. Polls underestimated Trump’s margin in states with large numbers of white voters without college degrees, but they also underestimated Clinton in states with large non-white or college-educated populations such as California. At the national level, these errors somewhat canceled each other out, but not so much at the state level.

But even the state polling errors were well within the normal range. Their 5.2-point average error isn’t far from the 4.8-point error that state polls have had on average since 1972. It wasn’t a year like 1980, when both state polls and national polls were off by almost 9 points, incorrectly showing a near dead heat between Jimmy Carter and Ronald Reagan (Reagan won the Electoral College 489-49).

Two other factors undoubtedly contributed to the widespread criticism about how polls performed in 2016.

One is that people had gotten spoiled by recent presidential elections. When looked at in historical context, what stands out isn’t that polling in 2016 was unusually poor, but that polling of the 2004, 2008 and 2012 presidential races was uncannily good — in a way that may have given people false expectations about how accurate polling has been all along.

The other factor is that the error was more consequential in 2016 than it was in past years, since Trump narrowly won a lot of states where Clinton was narrowly ahead in the polls. By contrast, in 2012, the polls somewhat underestimated Barack Obama’s numbers in several swing states as well as in the national popular vote. (National polls were actually a bit more accurate in 2016 than in 2012.) But it didn’t usually change the winner in these contests — Obama just won them by a clearer margin instead of a narrower one.

In the table below, you can see what percentage of races were “called” correctly in different election years. (A “call” is correct if the candidate who is leading in the poll wins the race.9) Over the long run, polls get the winner right about 80 percent of the time. That accuracy rate was just 71 percent in the 2016 presidential election, however. Down-ballot polls — House, Senate and gubernatorial — also had a bad year.

How often do polls “call” races correctly?

Weighted-average share of polls that correctly identify the winner in final 21 days of the campaign

Presidential
State-Level

Cycle
Primary
General
Governor
U.S. Senate
U.S. House
Combined

2017-18
—
—
87%
29%
68%
66%

2015-16
86%
71%
68
77
59
77

2013-14
—
—
79
74
75
76

2011-12
63
81
90
88
73
78

2009-10
—
—
85
86
75
82

2007-08
80
94
95
96
83
88

2005-06
—
—
89
92
71
83

2003-04
94
78
76
82
69
79

2001-02
—
—
87
82
77
82

1999-2000
100
68
80
84
56
76

1998
—
—
86
86
57
78

All years
83
79
83
83
71
79

Pollsters that are banned by FiveThirtyEight because we know or suspect that they faked their data are not included in the averages. Averages are weighted based on the number of polls a particular firm conducted.

But this is definitely not how FiveThirtyEight recommends evaluating pollster accuracy. In a true toss-up race, where public opinion is split evenly, a poll is going to be “wrong” approximately 50 percent of the time no matter what it says. There have been a lot of close elections recently — and polls get too much criticism when they correctly point toward a close race but the outcome goes against the media’s expectations. Polls often get too little criticism, however, when they “call” the winner correctly but are way off on the margin in landslide elections, such as in the French presidential election last year.

A more serious concern is that polls are sometimes statistically biased in one direction or the another.10 We measure statistical bias by accounting for the direction of the polling error — for instance, if a poll shows the Democrat winning by 9 percentage points and she actually wins by 4 points, that poll is biased in the Democrat’s favor by 5 percentage points.

In the 2016 general election, polls had a pro-Democratic bias of about 3 percentage points. This was fairly consistent across presidential, gubernatorial and congressional races; Trump outperformed his polls, but Republican candidates for Congress and governor did so by just as much.11 Polls also had a pro-Democratic bias in 2014.

Polling bias shifts from election to election

Weighted-average statistical bias in polls in final 21 days of the campaign

Pres.
State-Level

Cycle
General
Governor
U.S. Senate
U.S. House
Combined

2017-18
—
R 4.2
R 3.7
D 0.7
R 2.3

2015-16
D 3.2
D 3.3
D 2.8
D 3.7
D 3.0

2013-14
—
D 2.4
D 2.3
D 3.8
D 2.6

2011-12
R 2.5
R 1.6
R 3.3
R 2.1
R 2.5

2009-10
—
0.0
R 0.7
D 1.7
D 0.6

2007-08
D 1.1
D 0.5
D 0.8
D 1.0
D 1.0

2005-06
—
D 0.3
R 1.3
D 0.2
R 0.1

2003-04
D 1.1
R 4.3
D 1.7
D 2.5
D 0.9

2001-02
—
D 3.0
D 1.4
D 1.3
D 2.2

1999-2000
R 2.6
D 0.6
R 2.9
D 0.9
R 1.8

1998
—
R 5.8
R 4.8
R 1.5
R 4.3

All years
D 0.2
D 0.3
0.0
D 1.1
D 0.3

Bias is calculated only for races in which the top two finishers are a Democrat and a Republican. Therefore, it is not calculated for presidential primaries. Pollsters that are banned by FiveThirtyEight because we know or suspect that they faked their data are not included in the averages. Averages are weighted based on the number of polls a particular firm conducted.

But the bias tends to shift unpredictably from election to election. Polls had a pro-Republican bias in 2012, for example. They’ve also had a pro-Republican bias in elections so far in 2017 and 2018. We strongly encourage readers to remember that polling error can occur in both directions and that it’s almost impossible to predict which direction in advance.12 There have been cases, such as in last year’s U.K. general election, in which pollsters overcompensated for past errors and introduced new ones that caused polls to miss in the opposite direction. Over the long run, U.S. election polls have had very little overall bias toward either Democrats or Republicans.

I recognize that to some readers, this will have seemed like an overly sanguine view of the state of the polling industry. But it’s not that we don’t have concerns; in fact, we’ve been concerned about problems like declining response rates for a long time.

Nonetheless, those concerns are not particularly larger or smaller than they were a few years ago because polling performance has been about average for the past few years. While there have been some genuine trouble spots, like polling in the 2016 presidential primaries, overall there simply hasn’t been a clear trend toward polls becoming either more accurate or less accurate over time. Polling continues to present new challenges, but pollsters also continue to learn from their mistakes and make improvements to their methods.

Media organizations need to do a better job of informing their readers about the uncertainties associated with polling and of distinguishing cases in which polls were within a few percentage points of the correct result, like in the Brexit vote, from true polling failures, like the 2016 Michigan Democratic primary. But they also need to recognize that polls pick the winner correctly most of the time — about 80 percent of the time — and that media’s attempts to outguess the polls have a really bad track record. Since 2016, we’ve already seen examples in which the media overcompensated for its past failures by mischaracterizing polls or refusing to draw any conclusions from them at all, badly misinforming their readers.13

Finally, it’s worth re-emphasizing that while the House will probably be “in play” this November for the first time since 2010, polling of individual House races is historically some of the least accurate polling. House polls only identify the right winner about 70 percent of the time, as compared to 80 percent of the time in the other types of elections we track. Given a long list of potential Democratic pickups, it’s likely that the outcomes of dozens of House races will be at least somewhat uncertain heading into Election Day. Be wary of news accounts and of statistical models that claim to be able to forecast the number of Democratic pickups within just a few seats; that sort of precision isn’t realistic. Instead, this will be another election in which it’s important to think probabilistically about a fairly broad range of outcomes.

View more on Nate Silver's website »

Like • 0 comments • flag

Published on May 30, 2018 08:00

Politics Podcast: Things Are Looking Worse For The Democrats

FiveThirtyEight

More: Apple Podcasts |
ESPN App |
RSS
| Embed

Embed Code

The Democrats’ advantage in generic ballot polls has declined in recent months, as President Trump’s approval rating has risen. The FiveThirtyEight Politics podcast team debates what that means for Democrats’ prospects this fall and how significant a change it is. The crew also tracks the recent developments in possible talks between Trump and North Korean leader Kim Jong Un.

You can listen to the episode by clicking the “play” button above or by downloading it in iTunes , the ESPN App or your favorite podcast platform. If you are new to podcasts, learn how to listen .

The FiveThirtyEight Politics podcast publishes Monday evenings, with occasional special episodes throughout the week. Help new listeners discover the show by leaving us a rating and review on iTunes . Have a comment, question or suggestion for “good polling vs. bad polling”? Get in touch by email, on Twitter or in the comments.

View more on Nate Silver's website »

Like • 0 comments • flag

Published on May 29, 2018 14:28

Should The GOP Force Paul Ryan Out Early?

Welcome to FiveThirtyEight’s weekly politics chat. The transcript below has been lightly edited.

micah (Micah Cohen, politics editor): Welcome, everyone. It’s soooooooooo good to be with you all again.

For us to debate today: How much of a lame duck is House Speaker Paul Ryan?

There have been a few news developments that sorta raise this question — White House budget director Mick Mulvaney reportedly talking to supposed-next-in-line House Majority Leader Kevin McCarthy about replacing Ryan early, the discharge petition (which would force a vote on DACA) gaining steam in the House despite leadership’s opposition, etc.

So, let’s start with how much of a lame duck Ryan is. But I’m also kinda curious how and why that matters.

perry (Perry Bacon Jr., senior writer): He’s a lame duck in the most obvious sense that he’s not running for another term, so he will be gone after December. Usually, the speaker of House leads, in part, through the perception that he or she has outsized power (he or she controls committee chairmanships, has access to donors you need, etc.), and I wonder if you lose power if everyone knows you are leaving soon.

I guess the only question is whether he was a lame duck already — before announcing his pending retirement — because of the perception that Democrats will win the House anyway.

clare.malone (Clare Malone, senior political writer): I’ve been tuned out of the House shenanigans for a bit, so I’m not as filled in on this discharge petition stuff that Micah referred to, but the basic gist is that Ryan was against bringing an immigration vote to head and told his caucus as much, but then a lot of them ignored him?

Sounds like a loss of power, though can we directly tie it to his lame duck-age? Isn’t the House Republican caucus just generally disobedient?

natesilver (Nate Silver, editor in chief): I think he’s a lame duck figuratively — but not literally. Paul Ryan is a person and not a bird.

But more seriously — he hasn’t been an especially effective speaker. He doesn’t really have that much of a hold of his caucus. And there’s no reason for Kevin McCarthy or other people who want to be speaker not to try to oust him early.

clare.malone: Propriety, I guess, is what the counterargument would be

micah: Is there an affirmative reason for them to oust him early? What’s the benefit?

perry: The context here is that conservatives and moderates joined together to kill a farm and food stamps bill that Ryan was pushing on Friday. That increased the buzz that Ryan has no power with his caucus. The conservatives were mad that he wasn’t moving forward on their immigration bill, which is along the lines of what Trump wants. The moderates didn’t like the food stamps cuts. Separately, the moderates are pushing to get a vote on a bill that would basically grant legal status to the “dreamers,” who were illegally brought to the U.S. as children, and trying to force it to the floor over the objections of Ryan.

clare.malone: I’m interested in someone explaining the logic of this Mulvaney quote to me:

“Wouldn’t it be great to force a Democrat running in a tight race to have to put up or shut up about voting for Nancy Pelosi eight weeks before an election? That’s a really, really good vote for us to force if we can figure out how to do it.”

natesilver: Yeah, that quote is obvious bullshit.

micah: I don’t think Mulvaney understands how the midterms are shaping up, no?

clare.malone: If the Republicans decide to force a leadership vote on their end, Democrats also get forced into a vote, right?

perry: If Ryan resigns now, there is a vote by the full House for who is speaker.

micah: Yeah, it’s one vote, right?

perry: Democrats would have to vote for either Pelosi or vote present.

It’s one vote of the whole chamber, yeah.

micah: But, Nate, explain why you think it’s BS.

natesilver: First of all, there are very few vulnerable Democrats running in the House. The GOP won almost all the competitive seats in 2014 and 2016.

Secondly, the Democrats have already voted on Pelosi as speaker, so it’s hard to see another vote having any marginal impact.

Thirdly, some Democrats in tough races (and again, there are almost none of them) might appreciate an opportunity to throw Pelosi under the bus when there’s essentially no consequence to doing so.

And fourthly, this is exactly the sort of self-serving excuse that one should be disdainful of. It’s a transparent excuse from McCarthy’s allies to do something they have lots of other reasons to do.

perry: To take Mulvaney’s side, if you think Ryan is suboptimal as leader because he is a lame duck and want to dump him, that’s a reasonable position. And if you are at a Weekly Standard event (that is where Mulvaney made his comments), it might be easier to say, “Let’s use the speaker vote to beat up on Pelosi,” than, “Ryan has no power and couldn’t manage the firing of a chaplain, let’s get him out already.”

clare.malone: Right, Nate, isn’t politics all about saying something that is actually a transparent excuse to get to actually DO something else? Or Congressional politics, at the very least.

natesilver: Of course it is, Clare, but it’s the job of reporters to call out that bullshit.

clare.malone: You wanted more context to the news story, correct?

Or you wanted reporters to thinly editorialize within the piece that this was BS?

perry: Let me also try defending Mulvaney’s view of the politics: Wouldn’t a vote in September on House speaker be a huge media story and basically require every Democratic candidate for the House — and to some extent the Senate — to give their views on Pelosi? Isn’t this a net good for Republicans? Isn’t any day/week that Pelosi’s unpopularity is in the news a good day for Republicans?

micah: That last point seems pretty persuasive to me.

perry: I’m not saying it’s going to win the GOP any seats. But the Republicans don’t have a lot to run on.

natesilver: I think it would be a half-day story.

slackbot: Micah used to taunt people leaving even a few minutes early with, “Half day?” as they walked out the door. He thought he was very funny. Many, many others disagreed.

clare.malone: lol

micah: Haha — someone made that automated reply when anyone says “half day” apparently.

clare.malone: Can we leave slackbot in there?

natesilver: Please leave that in the chat.

perry: Republicans are already investing heavily in the anti-Pelosi strategy in ads and so on.

natesilver: Whereas … replacing the speaker of the House two months before an election would be a bigger story? What if the vote doesn’t go smoothly?

perry: Good point. Any process that relies on Freedom Caucus cooperation will not be smooth.

Imagine what they would ask McCarthy for! “Obamacare repeal votes every day if you are speaker.”

natesilver: Yeah, for me it’s like — if you can do it quietly, sure, go ahead and do it. But if it becomes a big news story, there’s more downside than upside risk for Republicans.

perry: The Ryan question, in part, gets at something broader: What would you do to save the majority if you were Trump/Mike Pence/McCarthy/Ryan? And is Ryan bad at running the House or is the GOP conference ungovernable? Maybe the second question has the more obvious answer: yes and yes.

micah: OK, yeah, so let’s take that one part at a time.

Step 1: Is Ryan’s continued occupation of the speakership hurting the GOP as it heads into the midterms?

Hurting electorally, that is.

clare.malone: Eh, is it?

micah: I’m asking you!

clare.malone: I’m not sure that it is.

micah: I do the asking around here!!!

clare.malone: OK, that’s my answer: I’m not sure that it is!

I’d call it neutral.

micah: Yeah, that’s my view too.

natesilver: Ryan isn’t a popular guy. He and Senate Majority Leader Mitch McConnell are really just as unpopular as Pelosi, depending on which polls you look at.

perry: Hard to know. It’s not helping. He says he is great at raising money. I think GOP donors would give to whoever was in that job, because they are focused on the majority.

micah: Ryan is unpopular in the way all Congressional leaders are unpopular.

natesilver: Also, the GOP agenda is quite unpopular. People forget that Trump’s approval ratings hit some of their lowest points in the midst of the health care debate, and then later in the midst of tax debate, when the GOP Congress was dominating the news.

micah: But again, that suggests this isn’t about Ryan.

Whoever occupies that job will represent 1. Congress, and 2. the GOP agenda.

Both of which, as you say, are unpopular.

natesilver: And Kevin McCarthy is not exactly a guy who screams, “Here’s a break from the status quo.”

micah: Very true.

clare.malone: He’s from exotic CALIFORNIA!!!!

micah: lol

OK, Step 2 …

Would a different speaker (McCarthy or someone else) or the process of getting a different speaker, in any way improve GOP’s 2018 fortunes?

natesilver: “In any way” is a pretty big qualifier.

micah: I’m trying to encourage outside-the-box thinking.

natesilver: I think there are probably some consequences to the GOP caucus being in disarray before the midterms. I’m not sure if getting rid of Ryan will lead to more or less disarray, however.

But I don’t think this is really an electoral politics story. It’s more a future-of-the-Republican-Party story.

clare.malone: Love those.

Ryan is publicly “with Trump,” but you can read between the lines and see that he also probably doesn’t like the guy. But he’s gone along with him. I’d be interested to see what a Freedom Caucus speaker would look like — i.e., a super-duper Trump buy-in person.

micah: I mean, isn’t McCarthy pretty super-duper Trumpy?

clare.malone: Sure. But he’s not a radical conservative.

That’s what I mean … like, of the stylistically radically conservative set.

natesilver: I don’t think McCarthy is particularly Trumpy.

perry: Ryan is, “Let Mueller finish.” McCarthy is basically, “Whatever Trump is for.” And a Freedom Caucus speaker would be, “Fire Rosenstein, we need a second special counsel, go Nunes!”

clare.malone: Yes.

Chaos agents.

micah: That’s a good way to break it down.

lol

Go Nunes!

perry: And I think that is what we are really debating ahead of 2018. Ryan wants to do conservative policy (reforming the food stamp program, for example). The moderates don’t, because conservative policy is dangerous politically if you are in a swing district. The Freedom Caucus wants to do conservative policy and anti-Mueller/Deputy Attorney General Rod Rosenstein stuff. McCarthy wants to please Trump and become speaker (those goals are related but not necessarily perfectly aligned with one another). He is not particularly policy-oriented, and I don’t mean that as a insult, because being in leadership in Congress is not really a policy role.

And as Nate hinted, these play into visions of the future too.

natesilver: I mean, McCarthy has voted with Trump a lot since he’s been in leadership. But I think he’s sort of a generic conservative Republican, frankly. He’s neither as ideologically-driven nor as wonky as Ryan, I don’t think. He’s just a partisan who votes the way most of the GOP caucus does.

micah: Doesn’t that describe Ryan too?

clare.malone: Zing!

micah: I’m serious!

clare.malone: I know.

natesilver: Ryan has at least the patina of being an intellectual.

micah: *had

clare.malone: The Ryan-McCarthy swap out would be more of the same.

perry: Ryan really wants to do a bill reforming the food program because he believes in that policy. McCarthy, I don’t think, would really push that — unless he was told that was what other people wanted and it would guarantee him speaker votes.

micah: So if that’s true, then maybe getting McCarthy in would marginally help the GOP.

natesilver: Ryan has called out Trump on various occasions. Maybe not when it mattered and not in a meaningful way. But more than McCarthy has.

perry: That is true as well.

I think McCarthy, if you can believe it, might make the House slightly more Trump-aligned than it is now.

clare.malone: Do you think voters actually care about a new speaker pre-midterms?

micah: Noooooooooooooooo

clare.malone: That’s the original q …

I’m not sure they do!

Micah and I are on the “nothing really matters” bandwagon

micah: Team Nihilism!!!

perry: There might be ways to change House policy that would matter in the midterms.

Like if I were them, I would stop doing food stamps and go full culture war — defend the police, build the wall, a bill encouraging NFL players to stand during the pledge, etc.

There are probably ways to run the House that are more Trump-like, in other words,

and I think that might have marginal electoral effects.

clare.malone: OK, that’s fair.

And that would be a definite departure from the current course of action.

But that sounds like a full Freedom Caucus speakership, not a McCarthy one. So it would have to be an upheaval speakership election.

natesilver: It seems like you guys are ignoring some important ways that the House could matter.

In Room, The Elephant

micah: Nate, what you talking about?

natesilver: Mueller.

What if Rudy Giuliani is right about something for the first time in many years, and Mueller actually does come back with findings before Sept. 1?

What if Trump fires Mueller? What if he fires Rosenstein?

What if Trump pardons Jared Kushner after Mueller indicts him?

All of these are very real possibilities.

micah: So?

Nate, what does this have to do with anything!!!!

clare.malone: Nate literally just changed the entire convo.

perry: Would Ryan react differently in any of those situations than McCarthy would?

Would Ryan react differently to that than Freedom Caucus member Jim Jordan?

micah: In Room, A Non Sequitur

natesilver: I’m just saying we’re debating all these minutia of what the House’s agenda will be for the rest of the year, and that’ll all be outweighed by an order of magnitude if any of the aforementioned Mueller-related things happen.

So, yes, how would Ryan react (as compared to McCarthy or Jordan) to an obstruction of justice finding, for example?

micah: The same.

natesilver: [citation needed]

micah: The same, according to my gut feeling which is based on not much at all.

clare.malone: Jordan might act differently.

micah: Oh, yeah …

Sorry.

Jordan might.

natesilver: One could maaaayyyyyyyyyyyyyyyyybeee argue that having Ryan still in place as speaker could help the GOP in that instance.

He could try to play like he’s the reasonable man in the room, a framing that the media tends to eat up with Ryan, just so the GOP can buy time and see how the midterms go.

clare.malone: I’ve lost track. Does Nate think Ryan should resign?

natesilver: I think Republicans should do whatever the hell they want. I don’t think the overall electoral effects are liable to be profound either way unless there’s a messy transition or speakership battle, and even then they’ll be like the seventh most important issue.

If somehow that messy transition battle coincided with a big development in the Mueller probe where Congress was compelled to weigh in — I guess that’s the worst-case scenario for the GOP, insofar as this goes.

clare.malone: So you’re lightweight on Team Nihilist.

micah: OK, actually, if we use Trump score as a proxy for “will do what benefits Trump,” then here’s the ranking from most pro-Trump to least:

McCarthy
Ryan
Jordan

perry: The Trump score is broken in this case then. On a Mueller probe scale, it should be, in terms of loyalty to Trump: 1. Jordan 2. McCarthy 3. Ryan. Or maybe: 1. Jordan/McCarthy 2. Ryan. Or: 1. Jordan/McCarthy/Ryan.

micah: lol

OK, closing thoughts?

natesilver: I’d just keep in mind that the next GOP leader will likely face either a Democratic House or a very narrow GOP majority, neither of which is much fun.

clare.malone: Do you think if Netflix offered Ryan a development deal he’d leave early?

I’m half serious. If it’s such a shit job and his leaving will have no real effect, why wouldn’t he leave early?

I guess that’s my ultimate last thought: I don’t think the speakership matters to voters.

micah: Team Nihilism!

natesilver: It’s a bit humiliating, I guess? Ryan tries to brand himself as someone who’s principled rather than transactional.

micah: The Trump presidency has been really bad for the Ryan brand.

However his tenure ends.

natesilver: Arguably Ryan has been bad for the Trump presidency too.

View more on Nate Silver's website »

Like • 0 comments • flag

Published on May 22, 2018 13:02

Paul Ryan Is A Lame Duck. Should The GOP Force Him Out Early?