April 28 –
May 4, 2014

Transcript: How Drew Linzer used R to Correctly Predict the US Presidential Election


Drew Linzer


Vote on HN

(Original post with audio and slides is here.)

­­Hey there. You’re listening to G33k Talk. Got a very exciting piece for you today. Drew Linzer from the Votamatic Project is going to talk to us about the dynamic Bayesian forecasting model he used to call the outcome of all fifty states as early as June. Check it out.

Drew Linzer: I’m Drew Linzer and I had a little fun last Fall with the presidential election. Actually very happy with Barack Obama for winning the election. I said he was going to. So I don’t know — if you’re anything like me, every four years, I get very carried away with the presidential election coverage. It’s high entertainment for me.

And in one of the more promising I’ve seen, and I’m sure you saw too over the last few presidential campaigns is that news reporters, and pundits have started to use more quantitative information. Primarily, that means polls. But at the same time, if you’re anything like me, I’ve felt a lot of frustration with how these polls are being used, misused, misunderstood, misinterpreted. And I thought that something better could be done.

And so that was the start of this project. This is a project that I started working on in 2010, so hopefully I’m ready for 2012. It sat under review for a long time. And while I was waiting for the peer review process to happen, I thought, might as well setup this website and get the forecasts out.

And so my website is votamatic.org. I don’t know if any of you have seen it. It’s not a specially complicated website but it gets the job, I think. And — so I want to talk about today is the modeling that went into the forecasts I was making. I’ll show you a little bit about, sort of from a political science standpoint. What we know about elections and campaigns that went into the model itself. How it worked. And also a few things about the 2012 presidential election itself. Some ways that the election was sort of predictable in historical context. And some ways that the election was unique. And then, I also have some R code to show of how I actually implemented it.

So the entire analysis and the website was all done in R, with a little help from a program called Winbooks, which I don’t know if any of you have used or not. But it’s a terrific piece of software for estimating Bayesian models using MCMC. And that’s what my model was estimated in. Okay. And at any point, if you want to interrupt me with questions, I’m happy to take questions. Just wave at me or whatever.

All right. So, I hope this isn’t a spoiler. Obama won the election whether you like it or not. And he got 332 electoral votes. So that was, you know, great for him. There was another score that was interesting to me, as the BBC put it in one of their reports — nerds — one, pundits — zero. And I was a nerd not a pundit. So — I imagine so are we all. So congratulations to all of us.

All right, so why did they say this? And the reason they said this was because there are really two camps of people — two types of people working on trying to predict the election outcome. On the one hand, there are people like me — people like my colleague here at Stanford, Simon Jackman who is working — who works in a — who is at Stanford and who was working with the Huffington Post.

Another political scientist named Josh Putnam at Davidson College. This guy you may have heard of called Nick Silver with the New York Times. And Sam Wang, who’s a part of the Princeton Election Consortium. And we all had it right. We all said on the eve of the election that Obama’s going to win. The only error there — Sam Wang didn’t get Florida, but Florida was a coin flip according to the data. So I’ll cut him some slack on that. All right?

On the other hand, you know, you have people like Carl Rove, Newt Gengrich. In some cases, you know, George Will — George Will is considered a very highly respected political analyst in politics. And they were predicting that Mitt Romney was going to win. And not only that Mitt Romney was going to win, but they expect possibly he was going to win big — 321 electoral votes.

And how are they doing this? You know, they had mental models, they were talking to people, maybe it was wishful thinking. I don’t know. I’ve had people say to me, well, they’re not trying to be accurate like you guys — like me. They’re entertainers. They’re trying to rally their side. But the fact remains is that they were on television telling people things that were false. I don’t mind — I think that’s fair to say.

So I’m happy that, you know, that those guys got shown up. And hopefully this will — we won’t have to deal with so much of this in the future. But what I want to show you is, the ideas how people like me got it right. And in particular how I got it right.

All right. So what we want when we’re talking about the forecasting problem for presidential elections, is accurate forecasts as early as possible. We want to know as early as possible. I mean, it was great that we could all do this on the eve of the election by looking at the polls. But really, if you trust the polls it’s not that hard.

So we want to get this right as early as possible. And the problem is that the data that are available early for this job aren’t accurate. And the data that are accurate aren’t available early. So what do I mean by that? Early on in the election there are variables having to — we call these the fundamentals. So things like how’s the economy doing? Is the incumbent president popular? These things correlate with election outcomes, as I’ll show you in a second. But they’re noisy. All right?

They’re — we could wait until the end of the campaign until the polls come out, and just rely on those polls. But by then it’s sort of too late. The forecasting problem isn’t as interesting anymore. It’s not as useful to us. So we need — we want to somehow combine this. Or, you know, bringing these two sources of information together.

The polls are great. The other problem is, though, that they’re full of error too — sampling error, house effects. House effects are when certain polling companies are systematically biased in favor of one candidate or the other. And then, of course, these polls that come out — most states don’t have very many. Even the states that have more polls — the states — the competitive states — they don’t come out on those states. So there’s all sorts of challenges involved, actually taking advantage of these different types of information.

All right. So the solution is, we need some statistical model — and this is what I came up with — that uses what we know about presidential campaigns, to take these long term forecasts, based on fundamental factors like the economy. And update them from the polls in real time. So what do we know about presidential elections? I’ll tell you five things real quick. These are the — we know more than these things that I’m going to show you, but these are the key things.

All right, so the first one is that if you know what the fundamentals are, they’re going to be able to predict election outcomes pretty well, but noisily. So here’s the scatter plot showing for all the presidential elections going back to 1948. On the vertical axis it’s the incumbent party candidate’s share of the major party vote. And on the horizontal axis is the GP growth, so the economic growth from quarter one to quarter two of the election year.

And so what this scatter plot is showing you is that in election years where the economy is growing, incumbent party candidates do better. In election years where the economy is flat or even in recession, like in 1980 over there, the incumbent party candidate does worse. So there’s a correlation there. Okay, it’s still noisy. There’s still error around there. But it’s not nothing.

Even better, you can go take a measure of the president’s approval rating. You get a much stronger relationship. So in June — this is in June of the election year. If you go and take a poll — and these are from Gallup. Gallup’s been doing this all the way back, post-World War II.

Go take a poll. Ask people, do you approve or disapprove of the job that the president’s doing? Just calculate the net approval — approval minus disapproval. Well, no big surprise, presidents that are more popular in June get more votes in November. And so we can use this information as well, but again, it’s noisy.

That’s about the national level. What about the state level? At the state level, that’s where — I mean, so if you know presidential elections in the United States, the trick is that there’s actually — it’s not one national election. It’s fifty state election. Win the state election — pfft. You get all the electoral votes from that state, then you add those up.

All right? So here’s something that we know of that can help us. These state election outcomes swing in tandem from election to election. So I stole this from the New York Times. I don’t usually steal things from the New York Times, but this one was so pretty.

All right, each one of those bars is one state. And what you can see is from 2000 to 2004, ’08 and — to 2012, the vote shares in the states — they go up by a little bit, they go down by a little bit. From ’04 to ’08, Obama did better than Kerry. And then Romney gained back some of that. It’s not a perfect pattern, but for the most part, these states moved together.

All right, so what else do we know? As I mentioned before, the polls are accurate on election day on average, but maybe not before. So this is data from 2008. And this is just for Florida in 2008. Each one of those blue dots represents one poll. And what it is, is actually it’s the share of the major party preference that poll respondents in Florida gave to Obama.

So the poll would say, do you — who do you want, Obama or McCain? What’s the percent that’s going for Obama, and they ran a smoother through it. So you can see that by the end of the campaign in November here, the polls are accurate on average, even though they’re noisy. But prior to that, in September, if you would try to forecast the outcome based on where the polls stood at that point, you would have been wrong. So Obama actually won Florida in ’08, but he was trailing in the polls two months before the election at that — in that race.

All right, one more thing we know. As these campaign trends move over time in each state, the state trends are very similar. And the reason for this is that even though elections in the United States are at the state level, the campaigns are largely national. And so when public opinion changes, or when preferences evolve during the campaign, they tend to evolve in more or less the same way across multiple states. This is going to be very useful in a moment.

All right, the last thing is maybe more something to be cautious of when modeling and forecasting than something to take advantage of. And that is, that we know that during the campaign there are going to be times when public opinion and reaction in the short term to big events like campaigns or, like, you know, maybe vice presidential candidate announcements. And these are just short term and temporary. And they dissipate after a while.

And so you can see these are the convention bumps. Someone actually measured before and after the party’s convention, going back to 1964, how support for the Democratic or Republican candidates changed. And usually, there’s a positive effect. Maybe two to five points, maybe more, maybe dissipating these days for some reason. But the point is that you don’t want to be misled by these short term bumps. These are factors that will go away and maybe won’t have a larger effect on the election outcome.

All right, so if we know all of this, how can we use it? And so that’s what I’ll show you. Putting this all together, what we were able to do is make a forecasting model that learns from the polls. So the amazing thing that we saw starting in 2008 and repeating in 2012 was an enormous — actually, enormous and shocking and terrific outpouring of public opinion research that was being done by news organizations, by pollsters trying to get their name out, or distinguish their brand.

People just doing polls and publishing them for free. The — people like me who do projects like this. It’s great. And I used to be a public opinion researcher. And they’re expensive. I don’t know why they do this, but I’m not really complaining.

All right, so in 2008, by the end of the campaign — and this is just state level polls. There were — there are over 1,700 of these done — a million interviews. This year there weren’t as many. But still about 1,200 of these state polls were done — a quarter of a million interviews. I’ll take that. And so what we want to do, is early on in the campaign, use those forecasts based on the fundamentals, like those scatter plots I showed you. As we get closer and closer to election day, use the polls to update and refine the forecast that we would have gotten from the historical fundamental factors.

All right. Everyone with me so far?

Drew: Okay. Great! So here’s how I did it in 2012. My colleague at Emory, a guy named Alan Abramowitz, came up with this forecast model, that he calls his Time for Change Regression. And it’s a very nice parsimonious model. And here’s what it is. It says, over the summer, if you go and you measure a quarter two GP growth, just like I showed you. And the June net approval like I showed you.

And in one more term — with one more variable in here, which is whether or not the incumbent party has been in office for more than two terms. You get penalized by that, actually. All it’s actually saying is that it’s very, very hard to be a first term incumbent, which you may know.

But that’s the equation. And it fit — that’s in fifteen or sixteen past elections. We can plug in the numbers for Obama, which is a very modest amount of GDP growth. And about an even net approval rating. And right off the top, we get a national forecast of Obama’s 2012 — this is major party vote share, of 52.2%. All right, so does anybody know what Obama actually got?

Female Speaker: Fifty-one percent

Drew: It’s actually fifty two of the major party vote. So this forecast was extremely accurate this year. And obviously, that’s just partially attributable to luck. I would have been just as happy as if this had been very wrong. Because then my model would have been able to do more, as I’ll show you. But it turned out to be very accurate.

So anyone — by the way, anyone who told you early on in the campaign that Obama was in a really bad position based on the lackluster economy and lukewarm approval ratings. And that it would take Mitt Romney to run a terrible campaign and say mean-spirited things on hidden video cameras. All right, that’s just not true. Obama was actually in a pretty good position to start off with.

All right, that’s only a national forecast obviously. We want to get it down to the state level. So we use that uniform swing assumption I showed you, where all the states move together. And what that means is that, well, we know that Obama got 53.7% of the vote in 2008. So that’s a difference of 1 ½%. Just take all of Obama’s 2008 vote outcomes, state by state, and subtract a point and a half from all those. And that gives us a starting point. So any actions of any polling information or anything — that’s going to be the starting point for my forecasts. Uh-huh?

Male Speaker: Yes, I have one question to — about your model.

Drew: Yeah.

Male Speaker: I — it looks as if your model was designed for congressmen or something. And I’m wondering how you know it’s valid for presidential elections when presidents can’t be in office for more than two terms.

Drew: Oh, I’m sorry. No, that’s a great question. So this is designed for presidential candidates. It’s — it refers to the party being in office for more than two terms.

Male Speaker: Oh, oh, I see.

Drew: Yeah, no, I appreciate that. So when George Bush Senior ran, it was the Republicans’ third term after Reagan. And then, when he ran again against Clinton, that was the fourth.

Male Speaker: Okay.

Drew: So at that point — this is why Alan Abramowitz calls it Time for Change. At this point, the voters are like, we’ve had enough of your — it’s someone else’s time or turn.

All right. So what I do with this is — these state estimates — I make them a Bayesian prior over my forecast. They can — they get updated from the polls. So what’s the trick of the model here? Well, we need to do a couple of things to make a forecast.

We have — we now have an estimate of where we think the election is going to end up. We don’t necessarily know how public opinion is going to trend to get us there. All right, so at some point in the campaign, this is data from 2012. And again, each dot is the percentage of people in the polls saying they support Obama.

And I’ve cut it off artificially right at the beginning of October. You know why I did that? So that all you Democrats out there could relive that feeling that you got after Obama mailed it in in that first debate. All right, so we’re — here we are, we’re in early October. Obama just sleptwalked through the Denver debate. And his polls crashed by a massive point and a half. All right.

It’s not hard — so it’s not hard to track the trend in a state like Florida where pollsters are very active. We can run — this isn’t as smooth as my model estimate. But we have lots of information there. We can get a trend line to go through it.

But the trick is that we also want to make forecasts about states like Oregon, where there were only three polls fielded — public polls, anyway — between May and October. Right? So how do we do that?

Well, the fact that we have states like Florida where there’s lots of polls, and Ohio where there’s lots of polls. And Virginia where there’s lots of polls. And Pennsylvania and Michigan — and that all these trends are going to be moving together. What we get is a pretty good sense of what the shape of that trend should be in any state, even if there aren’t any polls being conducted there.

So my model’s designed to borrow strength hierarchically across states and sort of impute or reconstruct what we think the trends are going to be in other states, even if there aren’t a lot of polls being done there. So that’s one of the neat tricks.

Now, obviously, here we are at the beginning of October. But we don’t necessarily just care about — oh, yeah.

Male Speaker: Do you see any spatial information? I mean, since Washington and Oregon are very similar and that — pretty much foresee a lattice model will be pretty effective?

Drew: Yeah. So — okay, so let me just repeat the question because I feel like I’m talking louder. All right, so the question is, do I use any spatial information to do this borrowing? I could but I don’t. And the trick would be — okay, either you estimate which states are more similar, or you make a decision a priori.

Drew: And I wasn’t comfortable doing either of those because of how little data there would be to base that on. And so I would get — if I could do it well, I would be happy to do it. I wasn’t confident I could do it well. And it worked — when I validated it, it worked well enough without that —

Drew: that I decided not to push it any further. But if you go and look at the Huffington Post, where Simon Jackman was working, his model did have a regional component in there. So you’ll see a bit more variation in his trend lines than in my estimated trend lines. But, you know, it’s hard to say which one of us is right or wrong without more data. You start to lose reliability anyway. But it’s a good point. Was there someone else?

Male Speaker: Yeah, actually —

Drew: Oh, yeah.

Male Speaker: I don’t know if this is the right time to ask, but — maybe not. But I’ll ask anyway.

Drew: Okay.

Male Speaker: You know, truly, you and a bunch of others were not surprised by the outcome. Was the Romney campaign surprised?

Drew: Apparently they were. Maybe we should talk about that at the end. But as I — I was saying my background is in public opinion. It’s also in campaigns.

Male Speaker: Oh.

Drew: I was very surprised that they were surprised. But, I mean, let’s talk about that if you want, because I could go on at length about that. There’s no R involved in that. Maybe that’s the problem.

Okay, so here we are at the beginning of October. And — okay, right. So we don’t necessarily care where we are right now, unless you’re trying to, like, calibrate how worried you should be. But what we want to do is project ahead. So there’s now a month left to go.

And what may be an intuitive way you might think about projecting ahead is what’s called a random walk model. Which is simply that my best estimate of what’s going to happen a month from now on election day is just today, plus whatever random noise, whatever random path public opinion takes between now and then. And that’s what that fan is meant to represent.

And it’s actually not an optimal model. I mean, it’s not a bad model. But we can do better than that. And this is where that historical information — those fundamentals come back in. And it’s actually preferable to use what’s called a — what we call a mean reversion model.

And what the mean reversion model is doing is saying, well, look — okay, it’s fine that Obama just dipped down a little bit in October. But that was sort of idiosyncratic. That was due to this random, weird debate performance that he mailed in. The fundamentals say that by election day, he should be back up ahead in Florida.

And so, what we want our model to be able to do is sort of correct for the idiosyncratic little blips — ups and downs, convention bumps, and those sorts of things. And bring it back, and as I said here, compromise — have forecasts compromise between where we think the election’s going to end up and where public opinion is right now, today.

This is my model, actually. I thought we might have some statisticians in the audience who didn’t want me to just say it’s magic. Okay, so this is actually the model that I use. The way it works — if I can run through this very quickly for you — is that we take these historical predictions that came out of the Abramowitz model and uniform swing. Those go into a informative prior over the — over each state’s election outcome.

And so these datas are filtered state trends. They’re filtered in that they’re the state trend of public opinion preferences minus the joint national effect that tied all those states together. All right. That national effect is defined to be equal to zero on election day. Though I’ve been noting election day with a capital J, as the Jth day — the last day of the election. And all that means is that on average, the polls are accurate on the last day of the campaign. Which is not actually true, but it’s true enough to make my model work. All right?

So that’s the prior. Where are we? To get the trends, I assume that the filtered state trends and that the national joint trends are connected by a random walk process — a random walk prior process. Except actually it’s a reverse random walk prior. And so what I mean by that is that since we have a good idea of where the model is ending up, with this prior process at the bottom here lets us do is work backwards from where we end up, where we are today.

Meanwhile, if you go up top there. This is in my paper so you don’t have to take a picture of it. You just download my paper it’s —

Female Speaker: Oh. I missed that part. Where’s your paper?

Drew: It’s on my website. Actually, it’s going to be in the Journal of the American Statistical Association if you subscribe to that.

Female Speaker: Uh-huh, okay.

Male Speaker: Not on the archive?

Drew: It’s coming out in the next issue. You’d think they could have published it before the election. I love the peer review process! What? It’s great! All right. They could have rejected it after two years.

Okay, so we’ve got these trends moving back in time to the current day. Meanwhile, what do we observe? What are the data? The data are these Y’s and these N’s. And so if we do a poll of 500 people, then the proportion of people who say that they’re going to — or, sorry, I should say the number of people who say they’re going to support the Democrat going to be binomially distributed.

And the pie perimeter there is representing the true proportion of people if we had an infinitely large poll of people in that state who would say that they were going to support the Democrat. And then I decomposed that here into the state and national trends. All right?

So we’ve got all these trends moving together. The thing we really care about is these pies. Because the pies are the altogether trend of public opinion for each state as we go forward. And so every time I get a new batch of polls, that’s my data, I’ve okayed these polls. And I run this whole big model, and I get these three trends. The pies, again, being the sum trends. And those giving my election forecasts. I’ll show you this in a second. Anyway, that’s the model. That’s it. Isn’t that nice?

All right. So what happens when you run this model? Here are some results. This is a record of every one of my forecasts that I made for Florida for the entire campaign. So anchoring the forecast to the fundamentals, rather than just going up and down with the polls, like some of my peers slash competitors did, produces much more stable forecasts.

So for example, in Florida, I never ever thought that Obama was going to lose Florida. All fall. Now, there’s uncertainty around that. And those shaded areas represent ninety five percent uncertainty bounds. So, you know — but that’s sixty to seventy percent chance that Obama would win Florida. But the point is I had him winning in a relatively stable way.

When you take all these state forecasts. Again, I have one of those for every state. And you add up the winners, you get a count of electoral votes. So, then — again, this is a record of every one of my electoral vote forecasts going back to when I started doing this in June. And you can see it’s very, very flat. And the reason is because the polls never really contradicted the basic story that the fundamental factors were pointing out to us. Which is that Obama was in a really good position to win re-election.

The only really systematic blip here is this little thing right there. And that’s right around the time of Obama’s successful convention, followed by Mitt Romney telling people that forty seven percent are never going to vote for him. Which was true. Okay.

All right, so one of the interesting things about 2012 is, if I actually try to evaluate this model, that there are really almost no surprises. So if you take the — if you take my baseline forecasts that I made with no polls at all back in June. And you calculate the difference, or the error between each one of those state forecasts and the actual election outcome in each state. And you get the average of those state by state errors, there was only two percent. And that’s already pretty good.

And you use my model. You start to put all the polls in and update these forecasts and in theory improve them, what happened to this average error? Well, it fell all the way to 1.7. That’s how little room there was for improvement this year, given the number of polls that we had. When I did this in 2008, there was much more improvement. And it was because it was a much more volatile election.

All right, so why didn’t the model improve forecasts by more in 2012? Well, it’s because these modeling assumptions worked really well this year. There really weren’t that many surprises. Here I plotted the 2012 Obama election vote outcome in each state against the 2008 Obama election outcome in each state. And what you see — Obama did slightly worse in 2012 state by state than in twenty — in 2008. And by about one and a half to two points.

He did a little bit better in Alaska than in 2008. I’ll let you ponder on why that might have been. And much worse in Utah. Okay. But really, these aggregate preferences state by state were extremely stable. So people in the news were very excited every time a poll would come out. This shows Obama surging. This shows Romney crashing.

No. Actually, if you average all the polls together and accept that polls are full of sampling error, what you get is that opinion really, over the entire campaign, state by state, never varied by more than about two points, maybe three points maximum. It was a very, very, very stable race. And you contrast that with 2008, for example, where state by state, the changes were anywhere between five and ten percent. So, much less volatility. I don’t know why that’s necessarily the case. Maybe we just — you know, we know Obama, and people stuck with their early opinions.

Okay. Could the model have done better? Yeah, absolutely. And so here are my final forecasts on election day, compared to the actual outcomes. And so positive values there for each state indicate states where Obama performed better than what my model predicted from the polls on election day.

What you can see is that for these competitive states here, it’s where there were more polls fielded. Obama was actually being underestimated by the polls. There was this whole controversy, you’ll probably remember, about the polls being skewed. And how they were all favoring Obama. And we had to un-skew them. And Romney was going to surprise us all with his victory. And actually, the polls were underestimating Obama the entire time.

All right. The thing that bugs me, I got to tell you, is this stuff down here. So, I couldn’t get really good forecasts — well, nobody could — of Hawaii and West Virginia and South Dakota. And you know why? It’s because people weren’t conducting free polls and giving me their data. All I need is, like — just give me — if you can give me twenty free polls — what does that cost nowadays — $100,000? I can get you to within two percent on election day. So, talk to me after.

All right. Before I get to the R code, I just want to say that there’s really a lot of other things you could do with this model other than just forecasting. And maybe, you know, you see this too in your work. You build a model for one thing and then it occurs to you, you can do other things with it as well.

But it’s not just about predicting who’s going to win. But also knowing early on which states are going to be the competitive states. Where is current opinion going? How is it changing? Why is it changing? How can we make better forecasts? And how can we make them better early? And then, again, there was this question that came up during the campaign. So I want to show you a graph that I made. Were some survey firms biased in one direction or the other?

And you might say, well, that’s really hard to figure out because how do you know — in the campaign, because how do you know if a firm is biased when you don’t know what the election outcome is yet? So what I thought to do — and I did this back in October. Is to notice that, well there — when these firms are doing these polls, there are big firms and there are small firms.

So the small firms are doing one or two polls at a time. There are a few hundred of these. The big firms like YouGov, and PPP and Rasmussen were doing fifty polls, eighty polls, a hundred polls. So what we could do is say that, well, these hundreds of small firms are probably not all ideologically in sync. They’re probably all, all over the map. And so they’re probably not biased on average.

So we can calculate the difference between — on any given day in any given state — we can calculate the difference between what they say the level of support for Obama is, and what my model says it is — the average. And plot the distribution of those errors. That’s that gray distribution there. And then kind of compare the distribution of the survey errors for all the other big firms to the distribution of the errors for the small firms, which we think is unbiased.

And so, sure enough, when I did that, I found that these supposedly left leaning firms like PPP actually had an error distribution that matched the error distribution of the, you know, arguably unbiased smaller firms. Whereas these Republican affiliated firms like ARG, Gravis Marketing and Rasmussen were clearly set apart on the pro Romney side over there.

And if you compare those modes, that’s about a two percent difference, which indicated to me that there was a pretty good chance that these three firms were systematically underestimating Obama by about two percent throughout the campaign, which turned out to be true. Yeah.

Male Speaker: Where’s Gallup in this?

Drew: Okay, so I don’t use — okay, the question is, where’s Gallup in this? Thankfully there’s no Gallup in this. Gallup has brought me great feelings of schadenfreude. They — as you may know, Gallup did a terrible job in the last election. But they were only polling nationally. And I did not look at any national polls, because the problem is, there are so few national polling firms. And they dominate so much of the work that, if there is a strong bias from one firm like Gallup, it totally throws the results off. So I don’t even consider the national polls in what I’m doing here.

Now, theoretically, I could link the national trends to the national polls. But I just chose not to because of that threat. I’m very pleased that Gallup has recently been dumped by USA Today and is now getting sued by the Justice Department. Sorry if any of you have friends who work there. All right. That’s it for the Powerpoint. There’s lots more on my website. Feel free to email me or Tweeter — Twitter, you know. There is the website. Lots of graphs and forecast related things.

So I thought what I would do is run through some R code and show you where this stuff came from. Show you how I did it. All right, so the website is a — the website is just a WordPress template that sucks pictures out of my public drop box. Let’s see if I can do this. So these are all R graphics files saved as PNG format that I stuck in a drop box folder and marked as public. That is the extent of my technical sophistication. No, I’m serious, that’s literally it. And it worked. It’s fine.

So, you know, I bet that there are some of you in this room who know D3 and JavaScript. All right. So I — that would be great. So, maybe next time, you know. But these are just R — these are R graphics files that get pulled into a WordPress template. And here are all your states, your public opinion trends.

I have — if these are too flat for you, by the way — a friend of mine asked me to make this other page here, called the trend detail. As he said, it doesn’t look that exciting because all the dots are in a line. So he said, why don’t you zoom in on those? So I did that. I put the swing states at the top. So then you can immediately, like, feel that something is happening, instead of running the most boring website in the world. Tracked a trend that didn’t change for four months.

All right, so isn’t that more exciting? It’s like —

Male Speaker: Yeah.

Drew: But all I did was I just stretched out the vertical axis. So, like, if you’re someone who likes to feel nervous, you can — Obama’s totally running away with Florida! No, he sucked in, eh, well. No, then, there’s the — whatever. Okay, so that is fifty one and a half percent. That’s forty nine and a half percent. Better luck next time.

Okay. And we — I mean, what’s nice about this, though, is that you can see how the model is producing a dominant national trend that then varies by the unique situation in each state. So it’s not like it’s fitting the same trend to every state. In New Hampshire, there is much more of a decline over the summer. In Wisconsin, you see a much n — bigger Paul Ryan effect or not.

But then if you go down here, the strong states where there’s really no polling being done — you see the models just filling in the same trend more or less over and over again. Yeah.

Male Speaker: In — did you go back, and, I mean, I’m sure you did and validated on this 2008 election.

Drew: Yeah.

Male Speaker: And did you see here shrinkage is big thing.

Drew: Yes.

Male Speaker: Was the shrinkage giving you more error in the 2008 polls?

Drew: Okay, so the question is, there’s obviously a ton of shrinkage happening in this model. Like, enormous amounts in this model. And was it also a good thing in 2008 where there was more volatility.

So one of the things I didn’t show you is — obviously, you want to look at not only the accuracy of the forecast but also the accuracy of uncertainty in the forecasts. And all the properties related to both the forecast accuracy and the uncertainty were exactly the same in 2008 as they were in 2012.

And I in part calibrated my 2012 forecasts to 2008. And when you do that, you know, it’s very risky, obviously. Like, calibrating to one case. So I was — I felt very anxious all summer. And it turned out to be exactly the same, which was a huge relief. But, yeah. The model is actually slightly under-confident early in the race, and slightly over-confident by about ten percent later in the race.

And that’s just — that’s how it works. And the reason is because I assume no net firm error, which is, like I said, wrong. But it leads to — the polls are over-dispersed, but my model does not assume that. So it leads to a bit of over-confidence. Yeah.

Male Speaker: This is really great and I’m really impressed by the fatness of the candidate who could do such a thing. I actually see start in June. But what I would like to point out, is there was another story there. Apparently, team Obama had this set of data scientists doing data mining —

Drew: Yeah, yeah.

Male Speaker: social media and everything. And they got kind of harassed about that. So if you could predict so accurately what was the benefit from that — did they change anything?

Drew: Yeah, well, okay, so — maybe we should save that with the other program. You want to talk about politics. I’d love to talk about politics. But just what I’ll say to that, briefly though, is that all these models — all of them — assume that the campaigns are working as hard as they can. Because if Obama just decided to go sit on a beach somewhere and not campaign, he would have lost. It doesn’t matter what the economy is doing. So all of these models assume that they’re working.

And — now, apparently, the Obama campaign ran historically unprecedented level of sophistication and — but did that net them one percent of the vote, two percent of the vote? We don’t know. But it wouldn’t have radically changed these things. And if it was reflected — as long as it was reflected in the polls — excuse me. I would have picked it up.

Okay. Incidentally, if you don’t want to read my paper, I wrote a more accessible version of how this all works here. And you can read this and I also made a blog. This is what you do when you’re on sabbatical. You write a blog and — there’s that graph. Okay. So anyway, you’re welcome to look at this.

Let me show you just a little bit how I did it. So this was all, as I said, done in R and WinBUGS. WinBUGS is a piece of stand-alone software that can be called from R using a package called R to WinBUGS. What it does is it takes model files that are interpretations of Bayesian models like the one I — like my model that I put on the slide there.

And it performs all of the Markov chain Monte Carlo for you behind the scenes. It’s really a pretty remarkable piece of software. And for people who are largely applied like me, it opens up whole worlds of model development that just wouldn’t be possible otherwise. It makes it much easier to do more creative modeling for people who are prim — more primarily substantively minded. Although I wouldn’t necessarily call myself that.

All right, so here’s what I did. Let me just do a little setup stuff here. As I said, all of the data are public. So here’s one of the sites where I would get data from. The Huffington Post. The Huffington Post very helpfully aggregates all of these polls for us. And I think they must have figured out that people like me were writing web scrapers, because at one point, they just said, oh, screw it. That’s fine! That’s fine.

Okay. So what I would do is every night, open up my laptop. And read in all these data. All this is doing is calling their website here and making sure that there is a file there. So if there were no polls, there’s no file to download. And as long as that all checks out — it’s just creating a data frame here that is — that contains the firm, when the poll was fielded, the percentage for Obama and Romney. I calculate Obama’s head to head percentage, so I throw out all the undecideds and the third party people. The state, the day — all this other stuff.

Well, what I get — let’s see here. Is, it’s going through all the states and this is just telling me that it’s working. How many polls are in each state? Alabama had one poll. Florida had 112. But this just ticks away for a moment.

And then when we’re done with that, we get this data file. Now, that’s 1,237 polls. But a lot of these polls are from really, really early in the campaign when the polls contained basically no information. So I limit myself to the polls only conducted over the final six months of the campaign.

If you tabulate that, you see that — well, 972 are worth keeping. Let’s go ahead and keep those and redo our factors here so that everything makes sense. And what we get is a data set that looks like this. Each line is one poll in one state with an outcome — head to head outcome, sample size and a state. And that’s that 750,000 interviews number that I was throwing out before.

To do the rest of the analysis, we need to setup our prior. So here’s my data set of historical information. For every election going back to 1948, the approval and disapproval ratings in June is measured by Gallup — this two terms dummy, growth rates, approval ratings, net. And the only thing that we don’t know is what’s the incumbent party candidate’s vote share going to be in 2012?

So, here is the Alan Abramowitz regression. I know you are all comforted that despite there being seventeen observations, everything has got lots of stars on it. And I’m a Bayesian so that’s a joke. I don’t know why R defaults to that. That’s the first question I want to ask the R people when I —

Anyway. Okay. So we can do a predicted value for 2012 — oops. And there — it’s just rounding differences. I showed fifty two point two in the slide. It’s actually fifty two point one four. We’ll do uniform swing. Actually put in a couple of house — home state effects. And then subtract that from 2008 and that’s our prior. So get that all loaded in. So this thing called pred.all, those are loges of the vote share predictions.

Then, we need to call BUGS. So, BUGS wants all of the inputs as vectors. So here we go, we setup the data. And what I’m going to do since this is just a demonstration, I’m not going to run the whole thing to convergence on every day. I’m going to separate the state — the filtered state trends up to seven day blocks. When I would do this for my website I’d run it overnight for eight hours. You all seem like very nice people. I’m sure you have families that you want to get home to. So, we’ll just do this more briefly.

This is a Markov Chain Monte Carlo procedure. Sometimes it helps to have start values. I give it random start values to make sure that it explores the posterior space fully. That’s that part of it. And then here’s the call to WinBUGS. It’s going to send it all my data that’s necessary for estimating the model. Our initializations function. I’m going to say, run the whole model but only send me back the P — the pies from that other slide. Just the estimates of the trend. And this is for all the data through election day.

I’ve got my model file here as prestracker. And what we’ll do is we’ll run three chains of — actually let’s run it for — this isn’t very long. We’ll run three chains — three parallel chains of five hundred iterations, burn in the first half, it should take about thirty seconds. And then we can play with the results.

All right. So it goes and calls BUGS. It runs the three chains for two hundred and fifty to burn those in. And then it will show you the result. There we go. Now it’s saving the remaining two hundred and fifty. And what you’ll see, by the way, when this is done — it’s going to show you the trace plots for these three chains. And the states where there aren’t that many polls, they’re not going to be converged. But the states where there’s lots of polls, they’re going to have converged already. So you’ll see that in a second.

But like I said, it’s good enough for this purpose. So it’s doing this, it’s saving it. It’s actually a lot of parameters if you think about it. It’s estimating one value of the proportion of people supporting Obama for a hundred and eighty nine days of the campaign for fifty states. So it’s a huge number of parameters. There’s actually more parameters than that, I’m just not saving them. But it’s actually kind of amazing that we can even do this at all.

Male Speaker: Did you run any of the diagnostics for convergence? Or —

Drew: Oh, yeah. Yeah. The question is do I do the diagnostics for convergence. I do lots of dia — I’m a big worrier, so I do — I like diagnostics. And they’re all pretty good.

Male Speaker: Do you have a favorite one?

Drew: Yeah, well, I mean maybe this is sort of low tech, but I like looking at the chains. And if they — well, okay, if they look like this, that’s my — I mean, that’s my favorite. But that’s not my only one.

Male Speaker: Right, right.

Drew: But my favorite one — if they looked like this, I’d feel good. If they look like that, I feel bad. So that’s my favorite one. So what you see here is — oops. Did I just lose it? Here we go. So what we get is for each state, state fifty on day a hundred and eighty seven — this is the mixing of the three chains for the estimate of the proportion of the people supporting Obama.

So let’s close this out. And it will now take a moment to get all of those posterior draws and send them back into R. This BUGS model object is actually going to be a list containing lots of different features of the estimated model. If I need to use the CODA Package, which allows a lot of nice visualization, you can ask BUGS to return the estimated stuff in a format that CODA can interpret. I haven’t done that but it’s often a nice thing to do.

All right. So, once that comes back, we can start doing some other interesting things with it. There we go. So one thing, for example, is, just pull out one of the vote forecasts. And what’s a ninety five percent credible interval around those forecasts. I’m going to do that right here. And so what we get is a vector of fifty state forecasts. We can put those next to their names, I guess.

All right, and so you see things like Florida just barely going to Obama. I won fifty bucks on that. Actually I — this guy on Twitter — I never even met him before. But I wrote on Twitter, I’ll bet Florida — anyone on Twitter. And this guy said, fifty bucks and another fifty on Virginia. And I said, now I’m just taking your money. And he actually — I never met this guy. He sent me a hundred dollars on Paypal!

That was the only gambling I did, by the way, on the polls. I know that Intrade had Obama, mysteriously at seventy percent chance of winning, which is crazy to me. But I never wired money to the UK. Took advantage of that. Although, many of my friends did.

Okay, so there’s the result. And we also want the probability that Obama’s going to win. And here’s what you see. I mean, very extreme probabilities. Like so extreme that you wonder what Karl Rove was thinking or not thinking.

Female Speaker: Yeah, that’s what we want to hear you talk about.

Drew: Oh, you — Karl Rove?

Female Speaker: Do you think Karl Rove —

Drew: Karl Rove is still employed. But Dick Morris is not, so —. Okay, but, the only one that was even a tossup was Florida. Other than that, very high probabilities that Obama would either win up to one or lose down to zero.

All right. How about some pictures? Here’s what my final — pretty much final forecastage probability of electoral vote looked like. I didn’t say that very clearly. On the day of the election, if you plotted a distribution of what the elect — a vector of what the electoral outcomes would be, that’s what it would look like. Three thirty two is Obama winning Florida. That’s what actually happened. What were the chance — you know, that he loses Florida — you get that outcome. The rest of it is just — the data didn’t indicate that any other outcome was really highly probable at all.

I didn’t put that on my website, because people would freak out. I’m serious. I put it on my website over the summer, and they freaked out. And I said, I don’t need that. So I took it off.

Female Speaker: In what way did they —

Drew: What’s that?

Female Speaker: In what —

Drew: Because it seems improbable. It doesn’t see intuitive to us, that I can say that there are this number of discreet outcomes. And any other outcome basically has a zero probability of happening. But that’s — it was right. I mean, that was the outcome right there. That big – . And especially if you do that in July. Then they say, well, now you’re crazy! I’m not crazy.

Female Speaker: So is this on the website now?

Drew: No! No, it’s not on — this is not on the website. In fact, if you go to my website, there is an unattractive blank space here. You see that? No. It’s — in July, my wife and I sat down and we said, I think it has to go! That’s what she wanted.

So that’s that. And, where are we? Here we go. The map I kept, I like that map. The map — there’s just a map. There are built in maps in R. And you can color them in. And the only thing that’s interesting about this to me is the color ramp palate here. But what I wanted was a continuous gradient of color that went from red for Romney to blue for Obama, and white for anything in the middle. And these just correspond to the probabilities that we thought each candidate would win. And so put a little legend on that. And there you go. Beautiful!

Female Speaker: It is beautiful.

Drew: Thank you. It’s beautiful because it’s right, not because of the colors.

Female Speaker: It’s also highly communicative.

Drew: Yeah. I mean, so the funny thing is — I think is, this and this is what’s funny. So I have South Carolina as a swing state. I don’t — you know, people have asked me, do you actually thought — do I actually think that Obama had a chance of losing South Carolina? No, but to be honest, there were no polls in South Carolina.

And given that I had no data specific to South Carolina, except for what South Carolina did in ’08. And given this large amount of error, to be honest — in the historical forecasts — to be honest about it, I have to say that no, Obama had maybe a twelve percent chance of winning South Carolina. Given that I had no polls and that’s why it’s a little pink there.

Female Speaker: But if you had had the richer models of how the states were related to each other —

Drew: Yeah.

Female Speaker: then you would have gotten that.

Drew: Yeah, that’s exactly right. So she said, if I had had a richer model of how the states are related. So if I had built in that South Carolina is really like Georgia and not like North Carolina —

Female Speaker: Yes.

Drew: then I would have solved that problem. Absolutely. Yes. Although, as a Georgia resident, I can tell you that the Democrats are eyeing Georgia very strongly, and South Carolina as well. So, look out.

Male Speaker: What – I mean, that’s a spatial out of correlation, right? I mean, how do — you across elections re-induce the spatial out of correlation?

Drew: Well what you could do — actually, okay, so I’ll tell you what Simon Jackman did, who’s the Stanford guy. He looked at past elections and he just looked at which states were similar to one another historically. And then he just blocked those off together. And said that the co-variances were — among those states were the same. But different than along the upper Midwest states or the western states. So he just assumed those blocks.

Male Speaker: So he’d rather use the past historic — you have — it basically imposed the past on the story.

Drew: That’s right. That’s right. Well, he — I think he estimated them. But he — once he blocked off the states, he allowed those parameters to be estimated. You don’t have to do that. You could assume the block and assume the value of the association. If you really want to get fancy, I don’t see why you couldn’t estimate the strength of the relationships as well. If you have a fully structural model that goes all the way back to 1948. But I think that’s — I don’t know, how complex do you want to get? I mean, that’s the question. Yeah.

Drew: Okay, so that’s that. The poll tracker — again, I put a lot of effort into just making the stuff what I felt like looked good. This is all of the R code to make one of these blocks. Whoops. So each one of these — it’s got transparency, it’s got lines, it has a little dashed thing at fifty. That’s this whole chunk here. And I’ll run it for you. And it’ll just run through them one at a time.

And that’s basically what it’s showing. I love being able to draw trends through no polls. I really do. That’s the most exci — actually, the most exciting thing for me would be when there would be no polls in a state. And I’d have this trend going through no polls. And then a poll would be released and it would land on the trend line.

And one of the nice things about this is that you can really where the bad pollsters were operating. Like, that’s one poll. That’s a poll where even though the trend in Arizona was for Romney to be winning and cruising along, these geniuses had a big error there. So —.

Other states like California were very regular. The transparency really comes in handy, I think. I don’t even know — well, I don’t know if you think this is good or bad. It’s hard to say. I mean, if you just look at those data, you don’t need a complicated model to tell you we have no idea what’s going to happen in Florida.

I mean, my model — and it’s so funny because I really wanted to be better than Nate Silver. I really wanted to. And — just for vanity. And I was really hoping that he would pick Romney to win Florida. And he didn’t. He — his model on the final day gave — I think gave a fifty point two percent chance of winning Florida. So, this is just — there’s luck, you know. Statistics, luck.

Female Speaker: –––

Drew: Yeah, mostly Excel, actually. Let’s see. Any other good ones in here? Yeah, I mean, some bad polling in the other direction as well. Okay, so that’s all of those.

Male Speaker: So when you say bad polling means that they have high sampling there in that particular poll or that particular polling firm to have really bad strata. ––  and they setup their sampling.

Drew: Okay.

Male Speaker: sampling for —

Drew: Yeah, so the question is, when I say bad poll, is it that they just got unlucky? Which is possible. Or, do I actually think they’re not doing a very good job. Let’s actually skip all this, because I’ve been talking a long time.

Male Speaker: Do you think they’ll be spoofing you at the next election?

Drew: Do I think they’ll be spoofing me? I don’t know. Let’s — let me — I have something to show you on this point. It’s the last thing I’ve got. This right here — what it’s going to do is compare the observed head to head numbers for Obama to the estimated values from my model — the mean trends. So this is every poll’s error, compared to what I think the truth was.

So one thing we want to do is see — well, is that error expected? Are ninety percent of the polls within the theoretical ninety percent margin of error? Are ninety five percent within the ninety nine — ninety five percent margin of error? So let’s — I’ll show you, actually. I have a plot to do that.

So here’s every poll. And what we have on the horizontal is the sample size. These dashed lines here represent the theoretical ninety five percent margins of error at every sample size. And the gray dashed lines represent the theoretical ninety nine percent — I should s — I guess confidence intervals at every sample size.

And so —

Female Speaker: But how many polls are there?

Drew: This is about 1,000 polls.

Female Speaker: So —

Drew: Yeah, I’ll actually — I’ll do the table for you. Don’t panic. But you can see, generally, they’re well behaved. I mean, this one is very problematic. If you do a poll of more than 2,000 people and you’re that far outside the ninety nine percent confidence interval, then I think you’re not a very good pollster.

But if you do 500 and you’re out by that much, you know, sometimes you lose some. But the proportions — the properties of the polls — these theoretical proportions were actually right on the money. So ninety six percent were within the theoretical ninety five percent margin of error. And ninety eight point six percent were within the ninety nine percent margin of error. So we expect over-dispersion due to house effects. In aggregate, we don’t see that much at all. All right, but there were house effects. It’s just that it wasn’t a lot.

And here’s another thing. We can actually look at this by firm. And so I’ll go through and break out that same plot. And show you firm by firm, who was doing a good job and who wasn’t. And then you tell me if you think that they were getting unlucky or if they were doing a bad job.

So, the horizontal lines that are colored correspond to the means of all of the polls done by each firm. So PPP is a Democratically affiliated firm in North Carolina. And positive values indicate polls that are more favorable to President Obama. So PPP showed a very slight house effect. And sometimes they missed. And I’ll give them that. Because they did a hundred and something polls and gave them to me for free. And sometimes you get unlucky and nothing happens.

Survey USA looks fine. Marist looks fine. YouGov — if anything, YouGov’s polls were under-dispersed, which is interesting, I think. That just shows you how smart they are. They have learned how to tame sampling error. Stop. Okay. Where are we? YouGov, Quinnipiac, We Ask America — these firms are all looking good.

All right, here’s where it gets fun. All right.

Male Speaker: What’s up with the vertical clustering on Rasmussen —

Drew: Yeah.

Male Speaker: and Purple and ARG?

Drew: So the reason is interesting. They — a firm like Purple Strategies only does polls on about 500 people. That’s it. And it’s actually smart, because think about it. If you go and you do a poll of 2,000 people and you mess up, you’ve got no sampling error to blame your error on. It’s just fine. Okay? But if you’re only doing polls of 500 and you mess up, you say, oh, I got unlucky.

Except if you’re Rasmussen and you do 100 polls. And they only have sample sizes of 750, but every single — I mean, every single one of their polls was more favorable to Romney than it should have been. I mean, that’s a red flag. Yeah.

Male Speaker: Is this likely voter models or registered voter models?

Drew: These — it varies. It’s whatever Huffington Post put on their website as the best at the current time. So some of these switched from being registered voters early to likely voters later on. But they still represent what the polling firms themselves considered to be their best estimates at the time.

Male Speaker: Okay.

Drew: Yeah, but it’s both.

Male Speaker: A question —

Drew: Yeah.

Male Speaker: Maybe I’m missing something, but since all the polls are presumably using different methods, how do you know what their confidence level is or what their standard errors usually are?

Drew: Yeah, I mean — you’re a statistician. Obviously, you’re right. But I’m just using the basic — the same — like, the basic, basic rule of calculating margins of errors. I don’t know what their weighting schemes are. I don’t know what their — I know what their sampling methods are. They say that they do it over the internet or auto dialing or what. But, you know, we don’t know — and how they stratify the sample. We don’t know any of that. They don’t publish any of that.

Male Speaker: Right.

Drew: So all I have to do is sort of take them at their word. Their sample size is this. And it’s hopefully at least random. And go from there. But, yeah, obviously, that’s your —. Gratis Marketing was my favorite. They were terrible. ARG was also quite bad. So all these things are discoverable. And then, do I have one more thing to show, or is that it?

Oh yeah, right! This is just because I’m — I can. What are the worst polls? Which are the most unlucky? I’m sorry. So what I did here is that I calculated a P value for each poll, assuming that my model’s estimates on each day were correct.

So what’s the probability of seeing a poll this extreme just due to bad luck? And so, let’s say that a P value of .01 even is okay. You just got one in 100 unlucky with your poll. And the reason I say one in 100 is because PPP and Rasmussen ran 100 polls. So you would expect to see PPP and Rasmussen get a P value of one percent, just because that’s what happens.

But I’ll show you what some of these really bad P values are. Okay. P values of less than .01 coupled by Quinnipia, Gravis Marketing — shows up on this list twice. This group in Atlanta got a P value of less than one in 1,000. And then this group here in Florida just needs to go out of business.

So, I like this. I mean, I have fun with this. Here are the absolute errors. You can see some of these polls are off by as much as five, six, seven, eight percentage points. But, you know, there’s so much data here. And lets us say, things like this, shame people. Yeah.

Male Speaker: And so will you count in the errors of your estimate that is the actual state’s percentage after the election date?

Drew: Oh. Yeah, so, no, that’s a good point. I’m using the truth as my model’s estimate, not the actual election outcome. That’s because a lot of these were done over the summer when I — the election outcome wouldn’t have been a proper benchmark. But I could have recalculated all of these. And it probably would have been more fair to do this first polls just done within say, the last two weeks. And compare them to the actual election outcome.

Male Speaker: The actual outcome also is — the underlying preference may be changing.

Drew: Yeah.

Male Speaker: Like the — Romney made a statement that the underlying and observed – maybe shift.

Drew: Yeah, no, that’s true. Yeah. No — I’m being a little obnoxious with this. I hope you don’t mind. All right. So, that’s it. So that’s the whole project. And I don’t have anything to do for the next three and a half years except wait for another election to come along. But I plan on running the website again and I welcome you all to check it out at that point.

Female Speaker: Congressional elections.

Drew: All right, well, you — I need a model. Yeah.

Female Speaker: So it’s one thing to be informed by all these statistics. But having worked closely with these numbers, I know that you know things about them that the computed statistics don’t reveal. And that that was why you bet on Florida.

Drew: Yeah.

Female Speaker: Why did you bet on Florida?

Drew: Okay, so I bet on Florida because even though my model was giving Romney — was giving Obama sixty percent chance, twenty percent of the polls in Florida were fielded by Rasmussen, ARG and Gravis Marketing.

Female Speaker: And so, your — here — your model assumption that the polls were unbiased, you knew was a little bit wrong. Okay.

Drew: Well, I knew it’s wrong. I mean, it’s — it is.

Female Speaker: Yeah.

Drew: It’s a little bit wrong. And the problem is usually you don’t have a way of telling that.

Female Speaker: Yes.

Drew: And so I didn’t want to put my thumb on the scale in the model in any way that could be wrong. That I didn’t have a way of systematically testing ahead of time. So I didn’t — that’s why it was only a bet. And it was done offline. It wasn’t on my website. But I had a very strong hunch that those three firms were pulling the numbers down and that turned out to be correct. But I was willing to lose fifty dollars on that. I wouldn’t have bet a hundred dollars on that.

If we have time for questions, I’m happy to —

Female Speaker: Okay, the Republicans.

Drew: Oh.

Female Speaker: Tell us why you think that —

Drew: Oh.

Female Speaker: Like, why were the Republicans so surprised?

Drew: Yeah. Okay. Okay, so most of you don’t know, Mitt Romney thought he was going to win. No, I don’t really mean that as a joke. He really —

Male Speaker: But it is a joke!

Drew: I mean, it’s amusing to half of us in the room, maybe more, I don’t know. I should do a poll before I start saying in appropriate things. No, so the Romney campaign, like all campaigns, hire — had pollsters working for them. And they were doing, just like Obama was doing, picking polls in all the competitive states. And trying to figure out where they were going to win.

And it came out after the campaign, because this is what losing campaigns do — they backstab and knife each other. The polls were really wrong inside the Romney campaign. They didn’t have a system of redundant polls like Obama did. They really did believe that the public polls were skewed, like a lot of the right wing commentators were saying, even though that was not true.

And they — how do I put this? No one ever denied it, I guess, is what —. Like, usually when these stories come out and they start criticizing each other behind the scenes and saying these things about how bad the campaigns were. Someone else comes along and says, no, that person’s just got an axe to grind. And they’re disgruntled. And they’re looking out for their — you know, their future employing prospects.

But no one ever denied the stories about Romney believing these bad polls. To the point where he had no concession speech prepared. And in fact apparently had fireworks lined up to go off in Boston harbor. And had all his supporters flying their private jets into Logan airport and had to send them home.

Like, he really believed he was going to win. And so reporters went into the Romney campaign and asked them, can we see your polls? It’s like, we don’t believe this. And one of these reporters got leaked the final Romney internal polls. And he was wrong in a couple of states. He wasn’t wrong terribly across the board. But he was wrong in a couple of states.

But the interesting thing about those polls was, that even in his own polls, they — he never had 270 electoral votes. And he believed he was going to win despite that. Even in his own polls he was losing. Because he believed that he had momentum.

And the reason he believed he had momentum was because they weren’t doing enough polls. And he was connecting the dots to form trends off sampling error. And they were projecting ahead based on polls that just happened to blip up from the last two weeks, the last week of the campaign. And they thought that would continue.

So, I — to me, I’ve worked in campaigns. I’ve worked in polling. That’s just absolutely mind boggling. That’s cognitive — what’s the — when you persuade yourself of something that doesn’t –

Male Speaker: The power of confirmation bias.

Drew: Yeah, it’s confirmation bias. Thank you. Yeah. I mean, it’s con — it’s the — I’ve never seen confirmation bias like that, ever, especially at the presidential campaign. And especially among a candidate whose whole claim was that he was in a — a business man who could run organizations and — I just — it was just a – like, I’m shocked. I just — I’m shocked by it.

Female Speaker: ––

Drew: I don’t —

Drew: Well, I don’t say — I don’t think it’s a personal failing. I don’t criti — I mean, I just — it’s just very surprising that the campaign fell prey to that. You just don’t see it at that level. Yeah.

Male Speaker: Yeah. I think the two he put in is a burning desire and a ream of data does not constitute evidence.

Drew: Well, there’s no — I mean, there’s really no excuse. There’s no excuse. I — you know, Nate Silver at the end of the campaign had Obama winning ninety percent probability. All right? It should have been higher than that. And he took a lot of heat because people were saying, well, he was in the tank for Obama.

He was actually adding uncertainty to his model to make it look like Romney had a higher chance of winning than he ever actually did just justified by the data. So, the — I mean, the Obama campaign knew all of this. They were on top of all of this. And their projections came out largely correct. And why the Romney campaign couldn’t have done this, like I said, it’s just absolutely mystifying to me. Yeah.

Male Speaker: Not only was Romney surprised, but the press seemed — had been surprised as well. And that’s — there’s no confirmation bias there presumably.

Drew: But the — there —

Male Speaker: I wonder why?

Drew: Well, the incentives were a little bit different. I mean, if you’re the press, you don’t want this story. I’m happy with this story because I’m — that — I’m still — I’m — the reason I’m here in front of you today is because of that graph. So, I’m happy about that. But the press wants conflict and change and close elections. And so that’s what they sell us. But I wish they wouldn’t.

Male Speaker: No, but I think they really believed it.

Drew: I don’t know.

Male Speaker: They — the last — like, two days before the election, you know, they were interviewing David Axelrod. And he said, well, we’re going to go with 300 electoral votes. And they were aghast. I mean, they really were genuinely surprised.

Drew: You know, Dick Morris, who has recently been relieved of his duties, actually admitted in an interview that his projections of Romney winning were made in order to rally the base. He came out and said that. So, it’s not clear to me what he really believes. That’s an understatement.

But, anyway, it’s hard to say. Are we done? Okay.

Male Speaker: How late do we have the room for?

Female Speaker: Another couple minutes. Just nine o’clock.

Drew: Is what — you want to do one more question real quick —

Male Speaker: Sure. Sure.

Drew: and then I’ll — I — I’m —

Male Speaker: Can you refresh my recollection? When was the Republican convention?

Drew: It was right before the Democratic convention, so —

Male Speaker: No, I’ll just start over here at the top left.

Drew: Yeah. Like, it would have been right there. It had no — the Republican convention —

Male Speaker: So at the Republican convention, you basically could predict the winner —

Drew: Yeah.

Male Speaker: clearly.

Drew: Yeah, so here.

Male Speaker: Now what if you ran this with the other — so the other Republican candidate? In other words, if you — during that time frame —

Drew: Yeah.

Male Speaker: up until the Republican convention.

Drew: Oh, oh, oh.

Male Speaker: Before they would pick their nominee. Do you have any charts that correspond to that?

Drew: No, because by this point it was clear, even though it wasn’t officially Romney, that it was going to be Romney. So.

Male Speaker: By that time.

Drew: Yeah. By this time, we pretty much knew that Romney was going to be the guy.

Male Speaker: All right, so you didn’t try anything to see.

Drew: No. Well, there was no polling on it anyway. You’re asking like, what if we put Obama head to head with Newt Gingrich. I don’t —

Male Speaker: There’s not enough polling data to give you a There’s not enough polling data to give you a

Drew: Right. Okay.

Female Speaker: Thank you.

Drew: Thank you. Thank you very much.

 

 

 

 

 

R Big Data Lang forecasting