From bc58103445e3000c035ba789f59e6dccdfb56349 Mon Sep 17 00:00:00 2001 From: andrewflowers Date: Mon, 6 Oct 2014 12:27:53 -0400 Subject: [PATCH] poll of pollsters data --- poll-of-pollsters/README.md | 17 +++++++++++++++++ .../poll-of-pollsters-anonymous-answers.tsv | 1 + poll-of-pollsters/poll-of-pollsters.tsv | 1 + 3 files changed, 19 insertions(+) create mode 100644 poll-of-pollsters/README.md create mode 100644 poll-of-pollsters/poll-of-pollsters-anonymous-answers.tsv create mode 100644 poll-of-pollsters/poll-of-pollsters.tsv diff --git a/poll-of-pollsters/README.md b/poll-of-pollsters/README.md new file mode 100644 index 0000000..5dd8522 --- /dev/null +++ b/poll-of-pollsters/README.md @@ -0,0 +1,17 @@ +### Poll of Pollsters data + +This repo contains the responses from 26 pollsters to FiveThirtyEight's [poll](https://docs.google.com/forms/d/15vciUokilOUBF9JDZIQQfuZh-2i_fxkmRVIX3v2xXcc/viewform?usp=send_form) of professional political pollsters, as described in this article: + +[Pollsters Predict Greater Polling Error In Midterm Elections](http://fivethirtyeight.com/features/pollsters-predict-greater-polling-error-in-midterm-elections/) + +Some answers have been edited, primarily for spelling, grammar, style and to protect anonymity, when requested. + +The responses are broken up into two files: + +`poll-of-pollsters.tsv`: + +The tab-separated text file contains the names of 25 respondents, their polling organizations, and those responses that they have agreed can be attributed to them. (One respondent wished to remain anonymous.) The heading of each column contains a question, and below it, in each row the answer given by the pollster listed in that row. Empty fields either mean the pollster didn't answer that question, or didn't wish to have the answer attributed. + + `poll-of-pollsters-anonymous-answers.tsv`: + +The tab-separated text file contains those responses that pollsters didn't want attributed to them. The heading of each column contains a question, and below it, the responses to that question, in alphabetical order. That means that each row doesn’t correspond to any one respondent. For example, the answer in the fourth row, in the third column, wasn't necessarily given by the same respondent as the rest of the answers in the fourth row. This sorting step was taken to better protect anonymity, by making it harder to figure out who gave which answer. diff --git a/poll-of-pollsters/poll-of-pollsters-anonymous-answers.tsv b/poll-of-pollsters/poll-of-pollsters-anonymous-answers.tsv new file mode 100644 index 0000000..89676e0 --- /dev/null +++ b/poll-of-pollsters/poll-of-pollsters-anonymous-answers.tsv @@ -0,0 +1 @@ +How would you describe your polling organization? Do you release raw data to the Roper Archives? Why do you or don't you release raw data to the Roper Archives? Are you a member of the National Council on Public Polls? Why are you or aren't you a member of the National Council on Public Polls? Have you signed onto the American Association for Public Opinion Research transparency initiative? Why have you or haven't you signed onto the AAPOR transparency initiative? How many seats do you expect Republicans will hold in the Senate after the election? Why do you expect Republicans to hold that many seats? How do you expect the average poll's error in 2014 Senate races to compare to the average error in 2012 Senate polls? Why do you expect that of average poll error? What was your average response rate across election polls in 2012? What is your average response rate across election polls in 2014? How do you define response rate? What percentage of your election polls include cellphones? By how many percentage points does including cellphones increase the cost of a poll for your organization? In what year do you expect Internet-based polling (using a probability or non-probability sample) to overtake phone polling as the primary method used across the industry in election polling? Why do you expect that timeframe for Internet-based polling to overtake phone polling? What percentage of your revenue comes from news-media clients? What percentage of your revenue came from news-media clients in 2010? What do you think is the biggest reason for the decline in political polling volume in recent national elections? Why do you think that is the biggest reason? Which states are the hardest to poll in? Why are those states the hardest to poll in? "What percentage of your respondents are black, on a typical poll?" By what factor do you typically have to overweight black respondents in your polls? "What percentage of your respondents are Hispanic, on a typical poll?" By what factor do you typically have to overweight Hispanic respondents in your polls? Do you oversample blacks or Hispanics on polls that focus on racial issues? How do you oversample blacks or Hispanics? Why don't you oversample blacks or Hispanics on polls covering racial issues? "What question or questions would you want to ask, or want us to ask, your fellow pollsters in future rounds of this poll?" "Academic, Commercial, Nonpartisan" No I was never been contacted by them. We would consider releasing the information. No "Again, that is probably more important for polling firms that conduct national surveys." No Because we are limited by client restrictions. 49 Back of envelope Higher in 2014 "Define ""average poll error"" in a standardized way that we all agree upon and I'll answer your question." 5% 4% AAPOR R3 33% 20% 2016 "Because I think telephone or mixed-mode polling will still be reasonably accurate in 2016, and in the 2018 midterms (because of the demographics of midterm-election turnout). 2020 is likely to be very challenging for telephone polls, so that by 2024, the industry standard will have to change." 0% 0% Disagree with premise of question "I'm not sure I agree that the volume is declining. Perhaps at the state level, or for phone polls, which may be dictated by media companies' budgets. But the actual number of interviews done each election cycle appears to be increasing." "California, New York and Florida" Fewer people = fewer people online. 11% 1.1 8% 1.4 No "Assuming an RDD sample for the general population, we purchase additional RDD sample in zip codes that are disproportionately [black] [hispanic]. If we have (say) promised the client a 100% oversample of a particular minority group, we purchase enough additional ""targeted"" sample to make sure we achieve a 100% oversample of that minority group." "Do you consider yourself a functionary or a visionary? Have you spoken out against ... remained silent in the face of ... or do you wholly support ... attempts by those in the polling community to ostracize and marginalize the contributions of research innovators? Have you personally participated in the systematic hazing, or bullying, of those research innovators who develop new ways of gathering opinions?" "Academic, Nonpartisan" No "It is an obligation of polling firms to release data to the maximum extent possible. Firms must honor clients' wishes, but we urge all clients to archive at Roper. And most do." No "As a small firm, we simply do not have time to take advantage of the programs and offerings." No I am a member of AAPOR. I believe when you sign up you agree to this. I have not signed up for anything other then the membership site. 49 "I think they will end up winning Alaska, Arkansas, and Louisiana, and 1 out of Colorado, Iowa, and North Carolina, although I see all 3 as tossups at this point and they could certainly break totally in one direction or another. I think Pat Roberts will eventually end up surviving, particularly if Republicans can effectively frame his re-election as the only way to boot Harry Reid." Higher in 2014 Midterm elections are harder to predict accurately because of the lower turnout. 10% 9% AAPOR RR3 35% 20% 2016 Cheaper overall and cellphones will simply dominate the calling universe by then. 0% 0% Media companies' shrinking budgets "In my case, both of the papers I have regularly polled for have all but discontinued polling. Ten years ago, we were doing 4-6 polls per cycle for them. At one time we were doing monthly polls." Hawaii "New York -- horrible cooperation rates, particularly in NYC. California & Florida -- large non-English speaking populations." 12% 1.7 10% 1.5 Not Applicable "We purchase additional, proportionately-sized random samples from our voter-list provider." "Do you offer Spanish versions of your polls? If so, how much, if at all, do Hispanic respondents differ depending on their language choice?" "Commercial, Nonpartisan" No It is proprietary. No Geared more to commercial firms. No I will look into it. 50 "I'm really not in the business of predicting this kind of thing. Were I to answer, I'd simply be parroting something someone else has said." Higher in 2014 More difficult to prdict who will turn out in midterm. 24% 19% AAPOR2 100% 35% 2018 "I think it already has, if you calculate ""overtake"" by the number of online interviews conducted vs. the number of phone interviews conducted. We do 130k+ online interviews annually with Reuters. There are enough online pollsters doing larger samples (e.g. YouGov) that it seems inevitable we'll outstrip phone polling at the next presidential." 0.01% 0% Media companies' shrinking budgets Look at what is happening to the traditional media: shrinkage on all front. Illinois "Predicting turnout in Chicago and suburban Cook County is an art, not a science." 12% Depends 27% Depends Not Applicable "I am most curious about the nature of the relationships between pollsters and media companies. We are paid for our research by our media client, but I know we have worked at-cost or for free in the past because it garners good publicity. Is this the case for others? I am also curious about how poll data is analyzed in these relationships. Are weighting and analytical decisions made by the client (media company) or by the pollster, or both?" "Commercial, Nonpartisan" No It's almost all confidential...have released some to various scholars and others with clients' permission. No It never occurred to me and no one ever asked. No "Since we do so many surveys for media clients in [one state] and provide question wording and methodology for our surveys to all clients, we are very transparent in our polling work." 50 It doesn't appear that Democrats are likely to perform well in the South (including Arkansas and Louisiana). Iowa is too close to call. I don't know that anybody has a clear handle on Alaska at the moment. Higher in 2014 Only slightly higher. Off-year elections make it more difficult to project turnout. 25% 35% AAPOR4 100% 60% 2020 "I'm uncertain about the future of the industry in general. People are getting bombarded by robocalls, web pop-ups and spam emails soliciting survey participation. Not too long ago, people would always tell me that they had never been polled before. Now they complain about all the robocalls they get on a daily basis. I personally get 5-6 junk calls per night. Down the road in 10-15 years it may reach a point where nothing reliable can be produced." 3% 1% Media companies' shrinking budgets Newspaper readership way down. Nevada That's where we poll. 14% Depends on area Depends Depends on area Not Applicable "I would ask who they buy lists from, how would they describe Internet panels, are they just emailing people, who are they getting voter lists from." Liberal No "Last time I checked, Roper would not accept data from our firm. Roper said it would need to call a board meeting to see if our data met their standards. Offended, I did not pursue it beyond that." No NCPP is the only organization dedicated to full disclosure of polls and polling data. Its guidelines are the most reasonable of any organization. No The requirement to provide weighting sources and parameters would divulge proprietary information. 51 "It's a guess, like everyone else's." Higher in 2014 "Polling midterms is just always harder than polling presidential elections because interest is lower and it's harder to get a clear gauge of what the electorate's going to look like. The key Senate battlegrounds this year are also places like Alaska, Arkansas, Kansas, Louisiana, etc. where most of the public pollsters don't have a ton of experience. It's not the Ohios and Pennsylvanias and Floridas of the world that we're all used to polling a lot." 35% 12% to 18% Completes divided by qualified contacts. Completes divided by numbers dialed is more like 5%-7%. 100% 15%-20% 2020 "Until someone can come up with a list of all email addressess, sampling online remains a conundrum." 3% 5% Media companies' shrinking budgets The current law on calling mobile phones. Market researchers are unable to use predictive dialers when calling mobile phones. It has to be a 1-1 call ratio which significantly increases the amount of time to complete a survey. New York Too much happens under the radar for opinion pollsters to pick up. Just ask the Las Vegas Review Journal. They have hired and fired four pollsters in the last 6 years. Nobody gets elections right in Nevada. Depends on the geo strata Depends on the state Depends on the state Depends on the state Not Applicable N/A No "Most of our work is done with private clients. Most of our public work is narrow (e.g., hyper-local) in scope." No Never been contacted by them. I would be open to releasing the information. No "Transparency crucial. Pollsters can ""weight"" data to whatever they want." 51 Just my current analysis Higher in 2014 Re-configuration of weighting formulae. 12% to 18% 2.9% approximately "For only our election-related surveys, the response rate is the percentage of respondents who answer the phone, meet the screening criteria to qualify for the survey, and then complete the survey among all qualified respondents." 100% 3x-4x the cost 2024 "We believe phones will not go away, supplementing part of the sample with cellphone-only households with Internet responses, text, or calling cellphone households." 5% 6% Media companies' shrinking budgets The disturbing trend for decreased readership of newpapers. Small states (population-wise) Depends on the state. Do not DK Do not Not Applicable "Opinions on automated polls in terms of accuracy, and the extent to which reweighting is required for gender, age, or race on typical polls conducted. Also, at what point does the N-size or weighting factor create manufactured data in the opinion of participants." No No one ever asked. No No particular reason. I was active in AAPOR for years. Yes 51 "The ""structural"" advantage they have this year with most of the contested seats in red states -- tempered by the low esteem for the GOP among voters." Higher in 2014 "The very term ""polling"" will soon fall out of the vocabulary altogether, or else be relegated to archaic usages such as candlepower and horsepower. Historians will write that ""polls,"" such as they now exist, lasted for 100 years, from about 1930 to about 2030. ""Polls"" were preceded by representative democracy and will be replaced by direct democracy. Viewed in this context, it is no wonder that polling in 2014 will be less accurate than was polling in 2012. Like chickens whose heads have been severed but who do not yet realize they are dead, pollsters continue to torment respondents by barging into respondents' lives unannounced. At present course and speed, pollsters will have obsoleted themselves before Nate Silver turns 50. When that day arrives, pollsters will have no one to blame but themselves." 3.2% approximately 6%-10% "For the purposes of this survey, simple completes/numbers dialed. Just FYI, in 2014, general-election polls have had a much higher response rate than primary polls." 100% 40%-70% Not Sure "Who can predict time frames when it comes to technological change? Like everyone else, I think it is a function of the reliability of phone polls (as measured and sanctioned by the academy) versus the decreasing reliability of phone polls plus their increasing cost. When do those lines cross?" 50% 33% Media companies' shrinking budgets Urban and southern states DK Unanswerable as written Unanswerable as written Unanswerable as written Yes Will you release EXACTLY what you did to create your likely voters or probable electorate? How much weight did you give to what? No "Since we do mostly Michigan surveys and not national polls, we have not felt it was necessary." No No reason Yes 51 Lower in 2014 "There are far too few quality polls in 2014 and that is saying something, given what we saw in 2012." 6%-10% N/A N/A - we do online polling and so work from panels and river sample. We don't calculate response rates because it is impossible. "Depends if we do an Internet panel. If we conduct an Internet panel, it's 20%-30% cellphone-only households." "You have mis-framed the question, and your answers will lack inspiration as a result. The battle is not between the phone and internet. The battle is between a small sample of respondents (chosen only because it is too expensive to talk to everyone), and talking to everyone, which by 2030 for sure, and possibly sooner, will be just as easy and no more expensive, than talking to just a handful. Polling can only thrive when the alternatives are worse. Once talking to everyone is effortless and affordable, we replace representative democracy with direct democracy (just as many of the Western U.S. states have already begun to do)." 50% 75% Rising cost of polls "Wisconsin, Hawaii and New Jersey" Unanswerable as written Yes No "They never asked, although we only started national polling in 2012 and have a limited number of national datasets as of now. All our state and national raw data sets are deposited with the Odum Institute via the National Network of State Polls." Yes Yes Lower in 2014 "There are two issues that might largely cancel each other out. On the one hand, accurate turnout projections for midterm elections (or any lower-turnout elections) are typically more difficult. On the other hand, voters who do turn out for midterms are more likely to belong to demographic groups that are more easily reachable through conventional telephone polling methods." N/A varies We use the AAPOR calculator. N/A 100% 100% Yes Yes Yes Yes Roughly the same Too many new and controversial data-collection methods. varies "Hard to calculate based on ""all"" revenue, but it's small." Yes Roughly the same "While not closely examining the variance in the polls in marginal seats, it does seem generally similar to 2012." Roughly the same Roughly the same Roughly the same Roughly the same \ No newline at end of file diff --git a/poll-of-pollsters/poll-of-pollsters.tsv b/poll-of-pollsters/poll-of-pollsters.tsv new file mode 100644 index 0000000..0483398 --- /dev/null +++ b/poll-of-pollsters/poll-of-pollsters.tsv @@ -0,0 +1 @@ +What is the name of your polling organization? What is your name? How would you describe your polling organization? Do you release raw data to the Roper Archives? Why do you or don't you release raw data to the Roper Archives? Are you a member of the National Council on Public Polls? Why are you or aren't you a member of the National Council on Public Polls? Have you signed onto the American Association for Public Opinion Research transparency initiative? Why have you or haven't you signed onto the AAPOR transparency initiative? How many seats do you expect Republicans will hold in the Senate after the election? Why do you expect Republicans to hold that many seats? How do you expect the average poll's error in 2014 Senate races to compare to the average error in 2012 Senate polls? Why do you expect that of average poll error? What was your average response rate across election polls in 2012? What is your average response rate across election polls in 2014? How do you define response rate? What percentage of your election polls include cellphones? By how many percentage points does including cellphones increase the cost of a poll for your organization? In what year do you expect Internet-based polling (using a probability or non-probability sample) to overtake phone polling as the primary method used across the industry in election polling? Why do you expect that timeframe for Internet-based polling to overtake phone polling? What percentage of your revenue comes from news-media clients? What percentage of your revenue came from news-media clients in 2010? What do you think is the biggest reason for the decline in political polling volume in recent national elections? Why do you think that is the biggest reason? Which states are the hardest to poll in? Why are those states the hardest to poll in? "What percentage of your respondents are black, on a typical poll?" By what factor do you typically have to overweight black respondents in your polls? "What percentage of your respondents are Hispanic, on a typical poll?" By what factor do you typically have to overweight Hispanic respondents in your polls? Do you oversample blacks or Hispanics on polls that focus on racial issues? How do you oversample blacks or Hispanics? Why don't you oversample blacks or Hispanics on polls covering racial issues? "What question or questions would you want to ask, or want us to ask, your fellow pollsters in future rounds of this poll?" Public Policy Polling Tom Jensen "Commercial, Liberal" No Because we are a Democratic pollster we only deposit raw data from our polls in the progressive poll data depositories. No We are primarily a private pollster and that organization has never reached out to us to join so I don't even really know what it does or entails. No "Since a large percentage of our polls that are released publicly are done for private clients, it is not feasible to commit to having all of their poll releases abide by AAPOR's guidelines." Around 10% Around 5% "Percentage of numbers called answering the poll. If we call 25,000 numbers and 1,250 respond it's 5%, etc. Also note while part of the decline from 2012 is probably part of the general decline, it also just reflects the lower interest in a non-Presidential year." "All of our public polls interview CPO respondents on the internet. Private polls, it's at clients' discretion whether to pay for that or not." 50% 2020 "I think there needs to be a presidential election where the media really takes Internet polling seriously -- which I think it likely will in 2016 -- and where the Internet polling also proves to be as good as the phone polling. If that happens, I think Internet polling will pretty much be accepted after that and will grow from there. Note I am only talking about this for public polling -- much of the campaign polling industry is in races like state legislative districts, municipal races, etc., and doing Internet polling in those much smaller geographies I think is still a ways off from really being feasible." Less than 1% 5% if Daily Kos counts as news media Rising cost of polls "We have significantly cut our public polling from 2012 because incorporating cellphone-only users into our polls through the Internet has increased our costs so much. There also just isn't really that much interest in this election compared even to 2010, so there's less incentive to do it." Nevada and Hawaii People in Las Vegas have unusual work schedules that make it harder to reach them and in both states it's hard to get a good sample of voters in key ethnic groups -- difficulties in polling Latinos accurately in Nevada cause Democrats to look worse in the polls there than they usually are and it's difficult to poll the various Asian groups in Hawaii. 8% on a national poll 1.5 4%-5% on a national poll 2 Not Applicable "Mason-Dixon Polling & Research, Inc." Brad Coker No All of our work is conducted for private clients. Data is their property. Yes Yes 53 Second midterm congressional elections typically favor party out of power. The Democrats involved in the key swing races are all running well under 50%. My gut feels there is a pretty strong anti-Obama vote building as his ratings fall and GOP will win the turnout battle. Lower in 2014 Change in methodology. Dumped voter lists for landlines and went back to RDD and also increased cell percentages. Varies from state to state Varies from state to state "The percentage of people who pick up the phone and then agree to take the survey. Note -- this doesn't mean they pass the screeners (incidence). As to above answers, response rates can be as high as 70% in places like North Dakota and Arkansas and lower than 10% in New York City or Los Angeles." 10% 35% Media companies' shrinking budgets It's really a combination of shrinking media budgets and the wide availability of free polls. Depends on universe Depends on universe Not Applicable To non-academic pollsters who give away free polls -- Who is paying for or subsidizing your costs? University of New Hampshire Andrew Smith Academic No Ownership issues with media clients No We are members of other organizations. Yes 51 Low Obama approval rating 25% "AAPOR #4. I have not calculated an average for 2014, but I suspect it is higher as we are not polling in Massachusetts but are in Maine. MA is a tough place to poll." 100% 10% "This is impossible to answer unless you define ""industry."" If this includes just media, especially state and local, it will be within three years as bad money drives out good. For respectable organizations, it will be many years, when someone figures out a way to do random sampling." 5% 5% See below "Some of each of the above, but primarily because 2008 may have been an aberration. We've seen ups and downs in this in the past." "On a scale of 1 to 10, where 1 is completely uncredible and 10 is mostly uncredible, how credible is a credibility interval?" ccAdvertising Gabriel Joseph Conservative No They do not ask. No I do not meet their requirements. We do automate polls and give the detailed data to our clients. No Not a member. Do not care. 54 We are working to make that happen. We are targeting the mobile phone channel of the key states. Higher in 2014 Pollsters are not focusing on mobile phones. 100% 100% How many people answer our surveys. 100% It doesn't. We charge the same. never. Mobile phones are more prevalent than the Internet. I do not. 0% 0% They are wrong "They are wrong. Customers want to invest in data. Traditional polling provides no data, just summaries that force you to spend money elsewhere to get results. In short, the cost does not justify the results." "North Dakota, Indiana" We cannot call there due to lawsuits we have pending. 10% I do not overweight blacks. 16% We do not overweight. Yes N/A How do you survey mobile phones? What is the cost? Muhlenberg College Christopher P. Borick Academic No We regularly release raw data to academic and public-opinion research organizations upon request but have not worked with the Roper Archives. No We simply haven't thought very much about joining the NCPP. It's an important organization and we probably should be a member. Yes We believe it's very important to fully disclose all relevant methodological information to the public and the AAPOR initiative has established clear and attainable guidelines that our organization believes are necessary to adequately evaluate public-opinion research. 51 "A combination of current state-level polling data, the president's fairly poor job approval and the location of open and competitive seats makes it likely that the GOP will pick up enough seats to get to 51." Lower in 2014 15% 13% AAPOR Response Rate 2 100% 15% to 20% 2022 "Declining response rates and increased costs of telephone surveys will make both the accuracy and cost of telephone election polls suboptimal and therefore you will continue to see an increase in both probability and non-probability internet polling. We are beginning to consider transition plans for our institute but since costs have remained acceptable and results very accurate, our transition is not imminent." 10% 15% Rising cost of polls I think cost is the largest factor. New York Our national surveys (nonelection) have shown the lowest response rates in New York. 10% 1.3 8% 1.6 No We traditionally use weighting to address undersampling since we are generally more interested in overall population estimates rather than comparisons across subgroups (i.e races). When we have a research question that is more focused on subgroup comparisons we have oversampled. I would love for you to get the opinions of academic pollsters on media use of polls that are devoid of transparency and use some sketchy methodologies. A partisan pollster put out a robopoll in Pennsylvania today that gave no details about the sampling frame or weighting but a number of media outlets ran with it and its outlier findings. Argghhhh! Franklin & Marshall College Berwood Yost Academic No "When I last considered this, it took a sizable amount of time to format data files and other information for submission." No "We are member of other similar and aligned organizations, including AAPOR and AAPOR's transparency initiative as well as AASRO." Yes We agree with the goals of the initiative and believe that organizations that release public polls should abide by specific rules of disclosure. Higher in 2014 "First, the lack of reliable, independent polling in many races may create problems. Second, turnout is likely to be lower than in 2012 and I expect the results will be more variable because of that." 31% 29% "We use AAPOR's RR3 calculation. Actual rates for our last six polls in 2014 were 28%, 25%, 25%, 30%, 36%, and 29%. Our cooperation rates (AAPOR COOP2) are much higher 90%, 87%, 90%, 91%, 93%, 91%. We also calculate R-indicators for each sample." 100% it becomes smaller every year can't predict "I think that Internet polls will have their place and as the methods evolve they will become more commonplace. We hope to experiment with these methods in the coming year, but there is a lot of work to do to overcome the limitations of Internet-based polls." 0% 0% All of the above "All of these reasons play a role. Costs are definitely high and ever-increasing, and poll aggregators lessen the value of individual polls to their sponsors. I also think that the partisan attacks on pollsters might reduce the willingness of organizations to do the polls." "most polls are in Pennsylvania, so can't comment" 10% 1 4% 1 Not Applicable Monmouth University Patrick Murray EPIC-MRA Bernie Porn Nonpartisan 50 "They are not holding seats currently held but will be picking up seats based on the polling conducted thus far and the political climate. While they lead in enough seats to take control now, and may end up taking control by a seat or two, the trendline has recently moved toward Democrats." 20% 25% 10-15 years "In looking at the trendline in recet years and making a best-guess projection, that is the timeframe that seams realistic." Media companies' shrinking budgets Loss of non-political advertising revenue Alaska "Have not polled in Alaska but several reports in recent years describe the difficulties polling in that state because of the geography, demographics and history of polling compared to results." 10% for governor years and 12% for presidential years Hardly ever required 2% Hardly ever required Yes "Randomly increase sample by the number agreed upon with clients within areas defined by the census as heavily African American or Hispanic, as well as precincts with very high Democratic percentages to help define African American geography." Gravis Marketing Doug Kaplan "Commercial, Nonpartisan" 51 The president is in his sixth year; historically the Republicans should do very well. The map favors the GOP. We believe the Democrats will have a harder time in states like Kansas. North Carolina appears to be a state the Democrats can hold. Colorado could be the surprise of the election. See Below "Depending on the amount of redials and if we conduct IVR or live polls. If we pull a random sample of 22,000 phone numbers on the first pass we will get 5,000 live answers, 15% will answer the first question, 9.6% will complete the poll. This is a general poll, this could change depending on the amount of questions and disqualification questions. When we do redials based on the 22,000 total calls I mentioned above, redialing the no answers, machines, and voters who hung up before answering question one, 8.9% will answer question one, 5.5% will finish the poll." Colorado and Nevada The minority population in these states that take the polls are much more conservative then the actual population. We find African Americans identify themselves as independents in the telephone polls at a higher percentage than the true electorate. Field Research Corporation (Field Poll) Mark DiCamillo Nonpartisan Yes Yes Yes 51 "Since we do not poll in states other than California, we are following the better-known polls conducted in the competitive states, and from time to time review the poll summaries and forecasting models compiled by FiveThirtyEight, Pollster.com, the Cook Report, Rothenberg, and others." Roughly the same "While the accuracy of pre-election primary polls may be declining for a variety of reasons, we haven't observed major changes in the accuracy of pre-general elections polls, and would be surprised if they showed significant declines in accuracy this year." 100% 20 25% Media companies' shrinking budgets Both the declining profitability of media properties and the consolidation of properties among fewer owners has significantly reduced both their ability to pay and the number of media companies we can approach to sell our services. N/A We only poll in California. 6% 1 23% 1 Yes "Since the voter samples of most Field Polls are drawn from listings of registered voters, the vendors of voter lists in California have gotten pretty sophisticated in their ability to identify voters' racial and ethnic backgrounds. This information can come from the official voter record, or from inspecting voter surnames. In addition, some of the better vendors are now cross-referencing the names and addresses of voters against other commercially available list sources that include a person's race and ethnicity. While we still need to ask screening questions to confirm that the voter is of the race or ethnic subgroup we are targeting, we have found them to be an increasingly accurate and reliable sampling source." "Question for the panel-based pollsters (of which I am not): As panel-based pollsters ask their pre-recruited panelists to respond to an increasing number of polls, when does the volume of poll requests begin to degrade the quality and reliability of information they provide?" Pew Research Center Scott Keeter Nonpartisan Yes We are committed to transparency and promoting wide access to our data. Yes Yes We have signed on as a supporter but have not yet submitted our application for membership. We are in the process of putting together an application. Don't know. Too much uncertainty in too many races at this point. Roughly the same "From what I can tell, the methods being used this year are not especially different from 2012. The one factor that is different, and may make polls more prone to error, is that turnout is less predictable this year than in 2012." 9% 8% We use a strict AAPOR RR3 method developed in conjunction with methodologists at our three major contractors. One of those individuals is a recent AAPOR Standards Committee chair. 100% of polls; 60% of interviews 25% relative to only landline interviewing Not sure -- 2020? "Telephone remains viable, if expensive, for many types of election polls. It provides good coverage (98%) and access to low-education and low-income populations due to the use of cellphones. It continues to perform reasonably well. Perhaps 5-6 years of development with online polls and non-probability sampling will reduce the uncertainties associated with them. And the non-internet population will shrink over that period." We do not have clients We do not have clients Rising cost of polls Good polling is costly and becoming more so as coverage and non-response problems grow. Don't know We rarely poll individual states. 10% 1.2 10% 1.4 Yes Typically by using stratification to oversample areas with larger concentrations of blacks and Hispanics. Siena College Don Levy "Academic, Nonpartisan" Yes No "AAPOR member but not NCPP, will check it out." Yes Merriman River Group Seth Rosenthal 80% 100% "Rise of poll aggregators such as FiveThirtyEight, Upshot, Real Clear Politics" "These reasons you provided are all highly correlated -- shrinking budgets is a very close second here. Outlets really have to stretch budgets to pay for polls, but they find that conducting their own polls has much less cachet now that readers can just use the aggregator sites. Our current media work is now conducted almost entirely with an outlets that poll about local issues that can't be found among the national aggregations." Hawaii "National voter databases (that we're aware of and have access to) do not use ethnicity categories that match the key voter demographics of the state. For instance, knowing the percentage of ""East Asians"" who voted in previous elections is less helpful in Hawaii than it would be in other states (because in Hawaii, for instance, Japanese ancestry voters and Chinese ancestry voters have very different voting patterns and are both key voting blocs, but are not differentiated in the databases. Also, voter databases tend to mislabel Filipinos -- another important voting bloc in Hawaii -- as ""Hispanic"")." SurveyUSA Jay H. Leve "Commercial, Nonpartisan" Elway Research Stuart Elway "Commercial, Nonpartisan" No "No particular reason. Since it is only Washington State data, doubt they'd be interested." No My membership lapsed and I have not renewed. I intend to rejoin soon. Yes I believe it is important to protect the integrity of the profession. Ipsos Julia Clark "Commercial, Nonpartisan" Yes "We have been in talks with Roper for the last two years to take our online data, but they haven't figured out a way to ingest it yet. So we're happy to provide it but are still working on logistics." Yes Yes "We believe it is critical to participate in this process to ensure that even as we move into an era of online poll work, a premium is still put on clarity and transparency." N/A Selzer & Company J. Ann Selzer "Commercial, Nonpartisan" No My clients have not wanted to pay us to put data in format Roper could use. Yes "It is/used to be the pre-eminent organization of pollsters in the country. In theory, I am on their board of trustees, but they are virtually dormant. Or they do some things in New York without enough notice that non-New Yorkers could arrange a trip that would include the event. I don't know that this organization is recruiting new members. So, I fear it may die off." No "I haven't done this yet. I have a couple of questions and the timing for signing up was unfortunate, given my schedule. I support transparency and have urged my clients to report methodology. I see so few other organizations explaining how they define ""likely voters,"" which seems very basic. So, I'm not sure this is working." 60% 60% Media companies' shrinking budgets "If media companies' budgets were growing, not shrinking, the rising cost might not matter so much. Polls break news and increasingly newspapers and other media outlets value their ability to do this, given 24-hour access to news feeds from a number of sources." "California, Washington, D.C., New York, Florida, Arizona" People move there and bring their phone numbers from their home state. They are virtually invisible to pollsters trying to capture a state. It's less of a problem for national polls. Anzalone Liszt Grove Research John Anzalone Commercial No Not sure we have ever been asked but it would also mean getting permission from our clients and a lot of work by our project staff which is difficult in the busy election cycle. We would certainly have an easier time doing it post-election when we have time to both get sign-off from clients and have the manpower to get data in the format needed to send to Roper. No I don't think I am. Not sure I even know what the hell it is. I am a member of AAPOR and AAPC. Yes I believe I have done that. Certainly support it. 50 "The math is solid for them to get to 50 and I really do think it may come down to how Kansas resolves itself including, if Orman wins, who he caucuses with." Higher in 2014 If pollsters are doing thorough methodology it should not change much. I do think that if the GOP does better sample modeling it will be better. 3% 2% 100% 10% Never I think there are still a lot of problems associated with online polling and it is going to be interesting to see if the New York Times switching over is going to be a Gallup-like problem for them come the 2014 election results. I do think there will be a hybrid of cell-land-online mostly through smartphone survey links. 0% 0% "It sure seems to be rising to me. There are polls everywhere, every day. I think what has happened is that there are more cheap polls out there being used by media outlets, primarily IVRs (robopolls) but also online polls." Alaska and Hawaii Bad voter files and hard-to-reach populations. Depends on the state Several percentage points. Depends on the state Several percentage points No "We oversample Hispanics in states like Florida to get a better look at the Hispanic universe. For example, we are polling for Charlie Crist, so we oversample Hispanics so we can see splits for Cubans and non-Cuban Hispanics and Hispanics by language and age, etc. But it is not driven by some racial issue. On African Americans we occassionally just do special polls of drop off voters." Are they using bilingual phone banks when polling in races where there is a universe of Hispanics over 10%? Marist College Barbara Carvalho Academic No "We provide aggregate data. There is some difficulty with the weight variables on the data set. We have both population weights and also use a probability model for likely voters. We work with Lois at the center to have the data available as quickly as possible from release. The director of our center, Lee Miringoff, is on their board." Yes The Marist Poll has been members of NCPP for many years and two of our directors have been officers and are currently trustees. It is an association of polling organizations rather than individuals. It was one of the first associations to advocate transparency for polling organizations. I believe they were the first to provide a guide to evaluate polls after each election.cycle. Yes As pollsters and especially as educators we adhere to the principles of transparency. The Marist Poll was one of the beta organizations to provide feedback to the committee. We also conducted a survey of organizations who polled in the 2012 presidential election to evaluate the level of transparency during the election cycle. The results were presented at AAPOR in 2013. This question wording is confusing. Do you mean have after the election or hold of their current seats? Hold all except possibly Kansas. Have 51 as a result of the election. "Hold - without a Democrat in the race in Kansas, the independent candidate has a greater probability of carrying that seat. Have - Democrats are defending 7 seats in states Romney carried. If the Republicans carry 6 of those 7, they are also competitive in three states Obama carried and could easily win 1. But, then there is Kansas." Roughly the same Poll error is statistically based on methods. Track record of most polls is decent. 11% 8% We use the AAPOR calculator. Understand response rate is a very different measure than cooperation rate. All of them Approximately 3 times the cost As it currently exists never I'm not sure how you're defining Internet polling or overtake phone polling. There are many innovative methods being tested and experimented with which include multi-modes. The future is likely to include some combination of opinion collection perhaps combined with analytics. Mobile is also promising because it's something nearly everyone has. Penetration approximates traditional landlines decades ago. It can become very inexpensive with changes in regulations for contact. 10% 5% Rising cost of polls It's expensive to do scientific probability based polling given regulations on cellphones. This is similar to the problems of early telephone surveys before access to phone numbers were allowed in the mid-1970s. "Washington, D.C." Proportion of cellphones is high and the population is very mobile. Depends on the geography being polled. Minimal weighting by race because of the proportion of cellphones in our samples Depends on the geography being polled Minimal weighting by race because of the proportion of cellphones in our samples and we have Spanish-speaking interviewers Yes Sometimes we screen for race; sometimes we combine results to the same questions of samples from multiple surveys. "Their assessment of the overall quality of polls, what types of polls do they watch and consider dependable." "Susquehanna Polling & Research, Inc." Jim Lee Commercial No Client confidentiality No Never heard of it. No "We are AAPOR members, but are not aware of initiative." 51 VTO favors Republicans. Obama referendum given low approval rating. Terrorism will be key wedge issue and breaking for GOP candidates. Lower in 2014 Lower turnout usually means lower error/variance in public polling. ? ? 10% - 25% depending on circumstances 5% to 25% "for national polls only, 2020" Less than 10% Less than 10% I kind of thought the opposite was true New England states "List issues, and level of mobility is higher." "0% to 30%, depending" Usually less than 5% adjustment 0% to 20% Usually 5% to 10% No Same reason we wouldn't UNDER sample them. We Ask America Gregg Durham 50 "We believe this will be a good, but not blockbuster year for Republicans. We see little evidence of a 1994-style tsunami that some have predicted." 10% 9% Percentage of participants who answer all poll questions. 26% 50% Blum & Weprin Associates Micheline Blum "Commercial, Nonpartisan" "Research & Polling, Inc." Brian Sanderoff Commercial 51 "1) Some of the seas that are up for election turned Democrat during the Obama wave of 2008, 2) Mood of the nation due to Obama 41% approval rating, 3) Historical trends of party not in the White House in midterm elections picking up seats." 100% 0% 10 years The commercialization of the business and companies looking for ways to save money. 2% 2% Media companies' shrinking budgets Newspaper budgets are shrinking. 2% in New Mexico No need to in New Mexico 40% 1.1 No No need to because we are able to get a representative sample of blacks without a signifianct oversample. Blacks comprise less than 2% of New Mexico. In voter polls we have included a Spanish-surname dictionary which enables us to add extra Hispanics in our probablility sample to avoid setting quotas or weighting. Hispanics with voter history can be reached at the same rate as non-Spanish whites. Princeton Survey Research Associates International Evans Witt Commercial 100% 50%-100% Yes "A variety of methods. Depends on project's objectives, resources and availability." Columbus Dispatch Darrel Rowland Nonpartisan No We publish just about everything we poll on in our newspaper. We'd certainly be open to sending it to Roper as well. No I have not heard of it. Yes "I was on the committee that helped prepare it. We have always believed in transparency -- a 10-inch ""explainer"" box has always accompanied our polls." Don't know. We don't poll out-of-state races. Roughly the same "In totality, there's not that much different about 2014 than 2012. On the other hand, it depends on what you mean by ""average poll."" Many are attempting to use Internet surveys with untested methodologies to determine likely voters. As often happens to pioneers, there could be some grim results." 15% 10% Number of polls mailed (minus return to sender) divided by number of polls returned. N/A N/A 2018 "As a practical matter, we'll need to see a track record of reliability before many will be willing to make the shift." N/A N/A Media companies' shrinking budgets "That's who was paying for a significant percentage of public opinion polls on political matters. In Ohio, for instance, we were part of the Ohio Newspaper Organization -- the state's largest eight papers -- that conducted polls for the past few statewide races (even-numbered years). Now, those papers are using their money simply to survive. A decade-and-a-half ago, our own paper used to conduct monthly polls through Ohio State University's Buckeye State Poll. That was dumped years ago because of expense. Even our own mail poll -- America's polling stepchild that has contrarily proved highly reliable over the past several decades -- has been reduced in frequency. We did three in 2012, only two this year." N/A N/A -- we only poll in Ohio. 9% 0 2% 0 Yes "Triple same predominantly black wards in Cleveland and Columbus. This occurs for all our polls, not just those involving racial issues." "How do those conducting electronic/Internet surveys make sure their sample matches that of the probable electorate, not just the census percentages?" \ No newline at end of file