How pollsters screw up

An old, sad story

In its Newfoundland and Labrador election poll, Mainstreet played up their poll result from less than a week before voting day that showed Liberals ahead of the Pea Sees by 30 points, 52 to 32. Supposedly.

In the Newfoundland and Labrador election, the polling firms, the pundits connected to them, and the news media who relied on their junk product got the election wrong.

Totally wrong.

Preddicted a Liberal victory.

Got a Pea Sea Victory.

Four out of four polls.

Dead wrong.

Go back through 20 years of Bond Papers and you will see a raft of posts and columns about the poverty of public polls about elections and politics. You’ll also find some writing about the richness of some polls that really are about more than just self-promotion. We’ll get to the problems with the polls in this election in a sec but understand two things up front.

First, what happened just now in Newfoundland and Labrador is on par with the complete polling cock-ups in Alberta and British Columbia a few years ago.

Second, while the BC and Alberta pooch-shagging turned into lots of hand-wringing and media stories about how badly the polls were wildly wrong, no one will spend any time honestly dissecting this failure of four national polling firms.

Okay, well, one sort of did. Alex Kohut’s the North Poll told folks that the lessons to be learned from Newfoundland and Labrador is that there are no lessons. After all, “Newfoundland and Labrador is a difficult place to conduct large surveys, and I credit the pollsters who decided to give it a try despite the risk of looking inaccurate.” That’s followed by a half-assed apologia. We’ll come back to this tomorrow (Friday) but for now, let’s go through the basic problems with any of these marketing polls.

1. They are about marketing not politics. All these polls exist solely to get the polling firm some free media coverage. On the surface they are about politics but the results don’t give you anything useful. Forum polled me in this election. They asked one or two very basic questions and then a half dozen on demographics but nothing meant anything about the election or what would happen. Whatever was wrong with their poll, they got it wildly wrong, as did the others. In 2019, Foreum forecast a Pea Sea win when the Liberals actually won.

2. They don’t poll voters. We cannot be sure what they actually are looking for but it ain’t voters. This was a big issue in BC where the poll-weighting - matching the sample to population - missed the simple and well-known fact that older people vote, younger people not so much. As a result, parties who did well with young age groups lost the election to folks who did well with actual voters. Local example: I called the 2018 Windsor Lake by-election using the raw data from the polling firm whose public numbers got it wrong. Their raw data showed clearly who had the support of likely voters.

3. So what do they survey? We don’t really know. The polls usually claim the results for party choice are the share of people who turn out to vote. But given that they poll all eligible voters - versus people who actually vote or are most likely to vote - the raw scores should tell us how many eligible voters will stay home. That is, if we look at the original table of responses that includes people who said they wouldn’t vote and so on we should be able to line that up with the actual vote in each district, including eligible voters who didn;t show up. Voter lists are inaccuracte some will say but that’s actually something we can figure out and allow for in the analysis. We’ll come back to this on Friday because one of the polls had some interesting raw numbers that they released along with the ludicrous 20 point lead for the Liberals. That firm was the Liberal polling firm, by the way.

4. Elections are not a horse race. People otuside political parties want to have some idea of what is going on. They want to know what the issues are, as well as what parties are leading where. These point-in-time polls mean aboslutely nothing. Literally. As the old science law goes, you cannot tell anything about a single point. Is someone trending up or down? Why do people support one party or not the other? You can get those details over time from some companies but most opt for the cheap and dirty point-in-time or horserace polls.

   Horserace polls and horse race reporting of election results give a completely and knowingly false perspective on how elections work. Elections actually happen on a seat by seat count of winners and losers. In 1989, the Liberals trounced the Pea Seas even though they lost the total vote, often mistakenly called the “popular” vote. They won more votes in enough seats to form a comfortable majority. And one firm, a predecessor of MQO was polling for the incubent Pea Seas *and* released its polling data to influence the campaign without declaring its obvious conflict of interest. Well, in some senses, the same thing happened on Tuesday. And like the Liberals in 1989, the Pea Sea strategy focused on winning the most seats not an imaginary horserace.

5. Margin of error is wildly useless. Even if the rest of that weren’t true, polling firms typically do no better than a five point margin of error on their public polls. That means that their result for each party in the so-called top line numbers could be within a range of 10 points ( five points plus or minus) 95 times if you repeated the same survey on the same population 100 times. Well, given the way a handful of votes can swing seats dramatically, that 10 point spread gives you a hand grenades and horse shoes idea of accuracy when it comes to trying to figure out what those vote totals might mean, if they mean anything at all. After all, as we said, people vote and parties win or lose in districts, not in totals.

6. Samples are often badly skewed. MainStreet’s database, for example, remains skewed Liberal thanks to early work for the party. Either that or, like lots of other firms, they tend to oversample people in the east end of Newfoundland and Labrador because they are accessible. That will always give you a skew. Don Mills’ CRA often skewed Pea Sea during the Danny Williams years for the same reason. There may be something off in the sample. What isn’t badly skewed might tend to treat all voters in the province as if they were equal without realising the opinion landscape can be subtly different even in neighbouring communities. Maybe that’s why this is a hard place to poll, well at least if you just want to do a quick and dirty self-promotion.

7. No poll result ever matches the actual election result. Go back to the point about who are we actually polling anyway. This is not so much a specific flaw as just a general observation. Whatever you see from polling firms is a mathmatical illusion with lots of rationalizations and voodoo incantations to pretend that numbers line up with other numbers in election results. The worst of it is in talking about decideds only or decideds and leanings. Whatever you call the people who say they will not vote, the undecideds, or the people who refuse to say, their numbers *never* line up with voting and turnout. Ever.

8. Undecided is a valid choice. One of my chronic irritations comes from firms that discount undecided for no good reason. One of the polls reported last week had an undecided of just 10%. Another showed undecideds of 37.5 percent but never discussed what that meant even though it was more than apparently supported the supposedly leading party, the Liberals. That 10% is absolutely unheard of in Newfoundland and Labrador political polling, meaning the firm did something (disclosed or not) to reduce that number to an extreme degree.

   Thing is, often in Newfoundland and Labrador, undecided is a code for something else like people with a vote intention they perceive as unpopular or running against the norm. Sometimes, “undecided” is the answer you give when you like the governing party but dislike something they are doing. Most firms never really probe UND or try to fit them into pre-determined categories based on potentially invalid assumptions by the polling firm, so what you get is often not what is actually going on.

9. Seat projections are junk science of the worse kind. Take a dodgy poll and match it up with previous election results. Guess what the next seat count might look like. That’s the game. Call it a con or a grift or just a shared delusion and you’d still be accurate. That’s why you sometimes see bizarro seat projections that show NDP seats in Newfoundland and Labrador where the candidate is a dud just because once, by accident or because of a decent candidate, the Dippers picked up a seat outside their usual pattern.

   Best example of that during this election: Eric Grenier’s suggestion mirrored by the television backroom crews that the PCs might win as few as eight seats and the Liberals 26 or more. First of all that’s a ludicrously unrelaible range. It’s the equivalent of saying the Liberals might win or they won’t. Second, don’t be confused by the fact some of these projections of who might win comes with a percentage chance, like we were told that there was only a 44% chance of the Pea Seas forming a majority. Those numbers were based on trends and bad polls and not surprisingly changed radically as the actual results poured in.

10. What they lack is ground-truthing. That’s information you cannot digitize, like the fact Christopher Mitchelmore won a seat in 2015 based on his own effort for the NDP. Ditto Jordan Brown. Once they were gone, the Dippers had no core vote in the district or even a campaign infrastructure. Sometimes, as in Lab West this time, they ran a candidate with no connection to the district other than a postal code. Any seat projection that showed an orange colour in Lab West or anywhere except Quidi Vidi or Centre was based on a bunch of bad infomration. Ground truthing requires knowldge of the electorate and the situation on the ground, hence the term. National polling firms lack that knowledge so their ability to reliably understand what is going on is all kindsa dodgy. The interpretations are based on nothing beyond looking at numbers that we know - now - can be nothing close to right. Always remember Allan Gregg in the 1980s who dismissed an election result in Newfoundland and Labrador he didn’t see coming with the sneering comment “small sample size.” That was code for Newfies don’t count.

Liberal “strategists” were shocked on Tuesday night because, aside from their unwarranted arrogance in their own abilities, they relied on polling information that was, from the outset, very badly skewed. You could see the shock on their faces in interviews as the night wore on just as you could see it on the faces of reporters and Liberal talking heads like Dwight Ball. People got fooled by poor information.

Hubris, that is, excessive pride or confidence that leads to a fall?

Maybe.

Nasty little fellows who always get their comeuppance, as Evie would say?

Meh.

Think of that ashen look of the Liberal losers on Tuesday as proof that, as Animal Mother told Private Joker and the Lusthog squad, payback is a mother.

---