A Poll at the Forum

Often I see polls saying that Rob Ford has a political base that is either made of concrete or completely at odds with the reality of what it means to be a responsible leader that represents the people. The most recent example is a tweet in #TOpoli today citing an article in the Toronto Star showing Rob Ford and Olivia Chow in a near dead heat.

One of the issues that has been bothering me recently is how many of these stories simply quote the numbers without question and appeal to the forums that generated them as being accurate. In fact the latest poll referred above states:

The Forum Research automated voice response telephone poll of 1310 residents is considered accurate within 3 percentage points, 19 times out of 20.

This statement on its own should not cause much concern but at the end of the article is the following statement.

Ford has fared worse in polls conducted by Ipsos Reid than polls conducted by Forum. A mid-November Ipsos Reid poll had Chow at 36 per cent, Tory 28 per cent, Ford 20 per cent, Stintz 13 per cent, Soknacki 3 per cent. In a mid-December Ipsos Reid poll, 61 per cent said they would not consider voting for Ford.

At this point I became curious and wanted to see just how reliable the numbers being quoted are and how it is possible that a poll that claims to be so accurate (within 3 percent) could be so at odds with other polling firms. Let’s find out!

Question #1: Where does this 3% come from?
For any poll, the population is asked a question. For example: “Will you vote for Rob Ford in the next Toronto mayoral election?” Say that there were 1310 people that you asked this question of and 406 of them said “Yes, yes I will vote for Mr Ford”. If this was the case then in the particular poll that you conducted, the proportion of responses that said “Yes” is 406/1310 or 0.3099. Let’s just call that 31 percent (percent is just a fancy way to say “divide by 100”). This is not really the true proportion since there is no possible way that every voter can be asked this question and so we must estimate the true proportion with a smaller sample and hope that this sample is large enough to give a reasonable estimate of the true proportion. Now suppose you do the poll three more times and get 32 percent, then 30 percent and finally 25 percent. Each time you poll, the randomness in selecting who is asked the question is reflected in the variability of observed proportion for that particular poll. However, all is not lost, and in fact doing the poll over and over will reveal a structure in the observed percentages. They will be centred at the true proportion but have some spread about that centre value. As it turns out, most of the variability in the observed 406 is between 406 – 36 = 370 and 406 + 36 = 442 and this number 36 is the square root of the sample size. So the proportion can be anything from 370/1310 = 0.2824 to 442/1310 = 0.3374 or about plus and minus 3% = 0.03 from the reported value of 31%.

For a poll with \(n\) people being sampled, the percentage error is simply \(100/\sqrt{n}\) so for a required accuracy of 1% one would need \(n = 10000\).

Question #2: How can polls differ by more than this “accuracy”
There are a number of ways that variability in polling results can occur and primarily these are due to sampling from a group that is not a faithful representation of the population as a whole. This is becoming a increasing concern with telephone polls that only sample from homes with landlines. In fact there is an alarming trend in individuals replacing their landline in favour of cell phone. A bit of digging reveals that 2000 about 97% of all households in Canada had a landline and this has been steadily decreasing (2006: 91%, 2010: 67%. Houses that only had a cell phone have correspondingly been on the increase (2008: 8%, 2010 13%). An ever more exacerbated situation has been observed in the US and there have been some effort to account for this effect. What is of concern is that this opting out of having a landline is falling along demographic lines with data from 2010 indicating that only 7% of households with a landline were in the age group 18-29. For those who consider numbers amongst their close and personal friends, the full report of Secondary Research into Cell Phones and Telephone Surveys is available.

Consider a toy model first with just six people that are polled: \(Y_1, Y_2, Y_3\) are young people, only one of them having a landline (say \(Y_1\)) and \(O_1, O_2, O_3\) are older, all of them having a landline. Consider a candidate for an election that is preferential to the older voters with support from \(Y_1, O_1\) and \(O_2\). Some other candidate is supported by \(Y_2, Y_3\) and \(O_3\). The table summarizes this situation.

Individual	Has a landline	Supports candidate
\(Y_1\)	Yes	Yes
\(Y_2\)	No	No
\(Y_3\)	No	No
\(O_1\)	Yes	Yes
\(O_2\)	Yes	Yes
\(O_3\)	Yes	No

The question is, how much support does that candidate have? If you were able to ask each of the six then three of them support the candidate and this gives 3/6 = 50% support. What would a landline based telephone survey reveal? Only four of the six candidates can be polled. And of these four, ‘poll-able’ individuals, three of them would support the candidate giving 3/4 = 75% support. The difference here lies in the coverage of the sampling and the fact that those not being sampled may have significantly different views than those being sampled.

What is really being measured is not the support of the candidate by the whole population, but rather the support of the candidate by that part of the population that has a landline. We could correct for this effect if two extra pieces of data are known. First, the proportion of people that do not have a landline and second, among these non-landline individuals, what proportion would support the candidate.

The probability of the candidate being supported comes from two sources, those that support them and have a landline and those that support them without a landline, each weighted by the probability of having a landline and not having a landline respectively. Suppose we use the 2010 estimate of 67% of households having a landline (it is most likely less than this in 2014) and let \(\alpha\) denote the probability of a candidate being supported by the individuals without a landline. In the most recent survey mentioned at the start of the blog, Rob Ford was said to command 31% of the support of the people polled. How does all this combine to find the true support of the candidate? By conditioning on having a landline,
\[
P(\textrm{candidate}) = P(\textrm{candidate}|\textrm{has landline})P(\textrm{landline})+P(\textrm{candidate}|\textrm{no landline})P(\textrm{no landline})
\] or in English: “The probability of support for a candidate is the probability they are supported by an individual with a landline weighted by the probability of having a landline together with the support by an individual without a landline weighted by the probability of not having a landline.”

Each of the ingredients to this recipe on the right hand side of this expression are:

• \(P(\textrm{candidate}|\textrm{has landline}) = 0.31\) in this case and represents the proportion observed in its unfiltered form;
• \(P(\textrm{candidate}|\textrm{no landline}) = \alpha\) is most likely unknown but could be significantly different than the 0.31 in the previous item;
• \(P(\textrm{landline}) = 0.67\) is taken as 67% and is mostly likely less than this in 2014;
• \(P(\textrm{no landline}) = 1-0.67 = 0.33\) is rising as individuals opt-out of having a landline.

Putting all of this information together gives the expression
\[
P(\textrm{candidate}) = 0.2077 + 0.33\alpha.
\]

There are three extreme cases that allow one to see just how much the reported numbers could vary.

1. If everyone without a landline does not support the candidate then \(\alpha = 0\). In this case, \(P(\textrm{candidate}) = 0.2077\) or about 21% support.
2. If everyone without a landline supports the candidate then \(\alpha = 1\). For this case, \(P(\textrm{candidate}) = 0.2077 + 0.33\) or about 54% support.
3. If those without a landline just as likely to support the candidate then \(\alpha = 0.31\) and \(P(\textrm{candidate}) = 0.2077 + 0.1023 = 0.31\) reproducing the reported 31%.

What are the take aways here? If you believe that those individuals without a landline have the same average viewpoint as those that have landline then the raw telephone poll surveys will suffice. If you think that perhaps there may be a difference, then this effect will swamp the results. The simple example here gives a range of 21% to 54% depending on the unobserved probability \(\alpha\). With probability it is not only what you can measure that is important, but recognizing what you are not measuring that yields accurate results.

If you got this far then I have a surprise. Since we know what was reported in the Ipsos Reid poll we can compare it to the Research Forum values to extract the most likely voting demographic of this unobserved group. That in itself will prove to be quite interesting and moreover, it can be used to help compensate future polls from the Research Forum. Stay tuned, and hey, isn’t math fun?