Trusting opinion polls?

How do we know some partisans aren't being properly reflected in polls?

A frequent question about public opinion polls in politics asks how we know that pollsters aren’t being avoided by partisans of one stripe or another, so their opinions just don’t show up in poll results. The most frequent version of this suggests that Republicans avoid pollsters altogether, or give false answers, or give answers they think the pollsters want to hear. The theory often suggested is that such behavior skews poll results in favor of Democratic candidates… so “true” Republican sentiment goes unrevealed in opinion polls until actual election results are reported.

Pew Research recently applied some smart testing and analysis to that concept, as it applies to opinion polling on President Trump’s approval rating at the end of his term. As he left office, Mr. Trump’s approval rating, 29%, was lower than any time in his four years in office. Down from its previous low of 38% around the Republican National Convention last August.

The most recent Pew poll was taken just after his January 6 rally outside the White House and the subsequent deadly siege on the U.S. Capitol, which forced Vice President Pence and members of Congress and the Senate to evacuate their chambers.

Most of the 9-point decline in Trump’s approval occurred among Republicans, according to the Pew poll. But Pew senior survey adviser Scott Keeter asked this question:

“How can we know that the change reflected a real shift in public opinion and was not an artifact of the poll itself, such as the possibility that some Republicans were less willing to be surveyed because of the events of Jan. 6?”

Here’s the key: Pew was comparing its August 2020 poll with its January 2021 poll. But Pew interviewed many of the same exact people in both surveys (not a “sample” of different people in each survey). “Most of those who participated in the January survey also had participated in the August survey,” Keeter says. So… Pew compared their answers in the two different interviews. And these were not trivial survey sizes. More than 4-thousand people were interviewed both in August 2020 and in January 2021.

25% of Americans who had been interviewed in both surveys changed their answers from “approval” to “disapproval” between August and January.

Pew knows that very few Democrats registered any approval of Trump back in August, so it concludes that very few Democrats were counted in that dramatic shift of public opinion in January. The bottom line: It was Republicans who turned against Trump after the siege in January:

Among Republicans and Republican-leaning independents who approved of Trump’s performance in August, 23% disapproved in January.

Pew knows that “virtually equal shares” of Republicans and Democrats took part in the August and January polls, “so the overall decline in approval was not a result of a disproportionate decline in cooperation among the most enthusiastic Trump supporters,” says Keeter.

It’s normal to be highly skeptical of a huge, 9-point change in public opinion between two surveys, especially when one of them took place after the very dramatic siege events on January 6. But in this case, because the two surveys interviewed the same people, Pew researchers believe its fully accurate to say that it was Republicans who lost faith in Mr. Trump.

You can read the full Pew Research report, including viewing the survey questions themselves and other methodology details, here.