Twitter polls aren’t real life, part two

Soon after he took control of Twitter, the social media platform’s new owner Elon Musk instituted an unexpected management mechanism: key decisions were suddenly being presented to the site’s users for final arbitration.

Should former president Donald Trump be returned to the platform? Users said yes, so — “vox populi, vox Dei,” as Musk liked to tweet! — he was back. Should previously banned accounts be granted amnesty? You bet! The populi had spoken!

Having seen prior examples of Twitter polls being taken seriously, I wrote at the time that the unscientific results of Musk’s surveys were not necessarily useful reflections of sentiment. Just because 15 million people weigh in, as happened with the Trump survey, that doesn’t make the results statistically significant.

But this was just rhetoric, an argument about such polls in general. It’s not like an actual pollster was replicating the results and allowing us to compare scientific polling with Musk’s surveys, right.

Wrong. YouGov did exactly that.

The results are fascinating. The pollsters took nine Musk tweets that included polls and recreated them within their polling system precisely. They even surveyed a statistically significant number of Twitter users, allowing us to compare the unscientific surveying of Twitter users on Twitter with the scientific assessment of the views of Twitter users off of Twitter. YouGov even asked whether respondents followed Musk, so we can compare the views of Musk’s follower base with the actual results of the surveys.

Before we walk through the results, there are some caveats. For example, the YouGov polls necessarily followed the Musk ones, at times by several weeks. In other words, public sentiment could possibly have shifted. There are also measurable margins of sampling error in the YouGov results, including relatively high ones for the Musk-follower subset.

But, barring an agreement between YouGov and Musk to run polls simultaneously (stay tuned for part three, maybe?), this is as good as we’re going to get.

First, the overview. On average, the results from Musk’s polls on Twitter varied by 16 points from the responses of everyone in YouGov’s surveys. Just among Twitter users responding to YouGov, the average was smaller, but still 11 percentage points. Among Musk followers, the average difference was 18 points. In other words, if the results of a tweet showed a 50-50 split, a YouGov result that matched the average deviation would show something like a 66-34 margin overall, a 61-39 margin among Twitter users and a 68-32 margin among Musk’s followers.

Nor did the number of responses make a difference. That 15-million response Twitter poll (the Trump one) was only 1 point off of the YouGov overall results — but the same difference occurred in a 3-million response Twitter poll. Among Twitter users, the Trump poll results on Twitter were each 6 points off of the YouGov results. Musk’s poll had it 52-48. YouGov had Twitter users at 58-42.

Precise, Twitter polls are not.

Here are the nine Twitter polls and the YouGov results, in chronological order. On our graphs, “on Twitter” refers to the results of Musk’s own poll. The other three are categories from YouGov: all respondents (“overall”), the subset that are Twitter users and the subset of that group that follows Musk.

Difference in margin with all YouGov respondents: 40 points. Among Twitter users in YouGov’s poll: 60 points.

Difference in margin with all YouGov respondents: 28 points. Among Twitter users in YouGov’s poll: 38 points.

Here I’ll note that the results may have been skewed by the likelihood that many non-Twitter users may not have known what Vine is. (It is a defunct video tool.) Difference in margin with all YouGov respondents: 62 points. Among Twitter users in YouGov’s poll: 24 points.

And here I’ll note that the verbiage of this poll is deeply leading. But YouGov recreated it faithfully nonetheless. Difference in margin with all YouGov respondents: 2 points. Among Twitter users in YouGov’s poll: 4 points.

There are no overall responses here since non-Twitter users wouldn’t have insight into the frequency of bots on Twitter. Difference in margin among Twitter users in YouGov’s poll: 2 points.

Difference in margin with all YouGov respondents: 2 points. Among Twitter users in YouGov’s poll: 16 points.

Difference in margin with all YouGov respondents: 28 points. Among Twitter users in YouGov’s poll: 6 points.

Difference in margin with all YouGov respondents: 26 points. Among Twitter users in YouGov’s poll: 8 points.

Difference in margin with all YouGov respondents: 77 points. Among Twitter users in YouGov’s poll: 45 points.

This last one is a useful example. There were 3.3 million responses to Musk’s Twitter poll, but there’s little doubt that the result was powered to some degree by users being motivated to participate. The results mirror Musk’s follower count closely — but that’s nowhere near public opinion in general or even Twitter overall.

Yes, some results from YouGov are close to the ones Musk got. Many aren’t. Many are way off the mark — as are the averages. Twitter polling is simply not a reliable measure of opinion, even of Twitter users.

On Thursday night, Musk tried another poll, asking when banned accounts that Musk claimed had revealed his location in real-time should be allowed back on the platform. “Now” won. So Musk declared there had been too many options presented in the poll and it should be re-run — certainly his prerogative but a move that reflects the malleability of Twitter polling.

As of writing, with six hours left, “now” is winning the new poll by an even wider margin than the first time.