Margin of Error in Polling

I have one more political post today, and something tells me there will be at least one more in the next week.

Lately, I have been seeing news channels occasionally report polling results with a particular oversight which is careless, if not incorrect. You maybe have heard something like this:

The Rasmussen Poll shows Obama leading Nevada 50-48, a statistical dead heat since it is within the 4.5% margin of error. However, the Las Vegas Review-Journal Poll shows Obama ahead 50-46, with only a 2.9% margin of error. (source: realclearpolitics.com)

The mistake is that both of these polls are within the margin of error (MoE) because the MoE refers to each individual statistic, not the difference between Obama and Romney.

What this means is that if the Rasmussen poll came from a simple, random
sample (which it likely didn’t because of several other inherent biases – this is one reason why people like Karl Rove disagree with the polls), we can say with 95% confidence that Obama is in the 45.5% to 54.5% range. With similar confidence we can say Romney has support within the 43.5% to 52.5% range. Though there is a higher probability that Obama is closer to 50 than either extreme, and Romney is closer to 48 than either extreme, we do not know what the true number is. And it is certainly not true that the poll shows Obama winning Nevada – it is within the MoE. This is sampling error – it could reasonably be the case that Romney is right now winning 52.5% to 45.5%, and you can not say that that is incorrect or unrealistic. Neither candidate is winning in this poll.

And if you compare this Rasmussen result to last week’s poll, and you notice Obama’s lead increased, it didn’t. Well, it might have – but the poll doesn’t tell us. If the results were within the MoE last week and are within the MoE this week, that means the leader couldn’t be determined last week, and it still can’t.  Remember, the MoE creates a range centered at 50-46. There is a slim probability that the real opinion is exactly 50-46, the MoE gives an idea of how close to that it probably is.

Now here is the bigger mistake that is being made: The Las Vegas Review-Journal (LVRJ) poll shows Obama ahead 50-46 with a MoE of 2.9%.  As stated earlier, the MoE applies to each value, not the difference between them, so this is also a dead heat and the poll does not show Obama winningWith 95% confidence, Obama has somewhere from 47.1% – 52.9%. He almost surely does not have exactly 50%. With 95% confidence, Romney has somewhere between 43.1% – 48.9%, and he almost surely doesn’t have exactly 46%, though it is probably very close. To summarize, the LVRJ poll does not show Obama winning 50-46, it shows Obama having as low as 47.1% and Romney as high as 48.9%; this poll shows that Romney could be in the lead by 1.8% at this moment. The MoE helps us to quantify how the limitations of not actually polling everyone, but only a small sample, turns the poll results into more of an estimate.

Now, If you have taken some time to think about MoE, you may have realized that the difference between Obama’s and Romney’s statistics must be more than twice the MoE to be outside of it (the MoE can be subtracted from the higher value and also added to the lower value). For example, The Rasmussen Indiana poll shows Romney up 52-43 (9 point difference) with a 4% MoE. That means Romney could actually have as low as 48% and Obama could have as high as 47% (1 point difference). That doesn’t change the result, therefore Indiana is outside the MoE.

Realistically, if we aren’t taking into account any other polling biases, Obama is looking good in the swing states. Though almost all of the swing states are dead heats (within the MoE), the values reported, e.g.; Obama up 50-48 in Nevada, primarily show Obama ahead, and the probability of several states actually lying toward the pro-Romney extremes seems pretty unlikely.

As a side note, no one is talking about Oregon as a swing state, but all three Oregon polls are well within the MoE. Just an observation.

Technical note: In a simple poll where the sampling is completely random, the margin of error is calculated as the standard error multiplied by the Z-score based on the confidence needed.   The standard error is the square root of [p(1-p)/n]. In this formula, “p” represents the percentage of the sample size compared to its population, “n” represents the total population of the respondents’ pool.

So, If 800 people were surveyed and 53% chose Obama, the standard error would be the square root of  [(.53)(.47)] / 800 , which is √0.000311375, or 0.017645…

For a 95% confidence interval, that number must be multiplied by about 1.96, and that gives 0.034585, or 3.4585%.

Advertisements

One thought on “Margin of Error in Polling

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s