I’m a great admirer of Nate Silver’s work. I’m old enough to remember the first incarnation of 538.com, and still remember excitedly following his reports from the field in 538’s first US election. In those days I was perfectly happy to have a very low profile. His reports from the field brightened otherwise monotonous days spent turning algorithmic ploughshares into swords.
However, I’m sorry to report that his latest piece on the UK elections hasn’t quite brightened my day. I don’t expect this response piece to be widely read. It certainly won’t reach the stratospheric reach of a piece from Silver, but it’s worth putting down my objections anyway. I think I owe an analyst of the calibre of Nate Silver a bit more than “it’s garbage so there!”. This is the first part. I’ve just pitched up at a cafe and wouldn’t mind doing this in parts, so here’s part one!
I advise you to read Nate’s piece first. It’s here: https://fivethirtyeight.com/features/are-the-u-k-polls-skewed/
It’s an ok piece. Nate runs through the records of the pollsters in the UK and then makes some contorted philosophical points about “First Rule’s” and conventional wisdom. Or something like that. I’m not entirely clear; it all seems to lack precision, which is a shame given it is written by someone who has built a career from such precision. But I’ll come to that in later pieces perhaps. For now, I have a very singular and precise issue with the piece.
Nate uses a simple average of the last ten polls to say the Conservatives are 7.1 per cent ahead. In fact he uses the simple average throughout the piece, citing for instance that the Conservatives started the campaign with “an average lead of 17 percentage points”. At the end of the piece he implores us to “focus on the polling average rather than the polls you like”. This is flawed and Nate has himself written extensively on why this is flawed. He knows better. In the US, he weights the pollsters. Not all pollsters are given equal weight. He does “like” certain pollsters over others and he doesn’t focus on the simple average! Here is his own [brilliant] description of that process: https://fivethirtyeight.com/features/how-fivethirtyeight-calculates-pollster-ratings/
Yet in the piece yesterday he simply added together the vote shares from each party in the most recent ten polls and divided them by n. Why does he apply weighting criteria to pollsters in the US but not in the UK? He may not have the time to do so. That’s fine. After all, the piece was aimed at the US market in all likelihood. “There’s an election on, people want something, I’ll write something quick”, is a perfectly reasonable editorial decision.
But if you’re doing a piece on the accuracy or otherwise of the polling houses then I’d like the same academic rigour brought to bear on it that Nate expects of himself and others in the US. Simply averaging the pollsters numbers, as Nate knows full well, is the entirely wrong approach to take. He would never do that in an American context. In fact, he’s written extensively on the criteria he applies to arrive at pollster weightings, and he’s very firm in deciding what is and isn’t a good poll, choosing in some instances to ignore polls from firms which fail to meet these criteria. None of that in this piece on our election: “Just add together the numbers from the [equally reliable] pollsters and divide by n”.
In America, Nate provides clear ratings for every poll and pollster, based on how recent the poll is carried out, the sample size, the track record of the pollster, etc, etc. He hasn’t done anything of the kind here. By using a simple average, he gives the same weighting to each of the pollsters. What he should have done is apply the same criteria he applies to American pollsters, weight them accordingly and then average them.
Except even then Nate doesn’t just average them in a US context. Because, as he admits, that is clumsy and statistically imprecise. Here is his long, and very good piece, on the approach he takes in the US: https://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/ . His piece on the UK elections doesn’t appear to be anywhere near as rigorous. Which is a shame, because it runs counter to his usual approach.
On Nate’s own terms, this singular lack of precision undermines much of the piece. Look, I’m not at all suggesting that he should spend an inordinate amount of time on our election. I suspect he spent all too much time on our election in 2015 and found the experience somewhat bruising. But if he is going to do comment on our polling, I only ask that he applies the same rigour to the exercise that he would apply to elections in the United States.
Anyway, back to my coffee and croissants.