On Friday the final seat-count poll before the election was published, but how much can we trust the numbers presented by pollsters? Over the past three years, seat-count polls have become a central factor in the political system, influencing parties’ decisions about whether to run in the election, withdraw from the race or unite with other parties. Poll results have led to attacks on Ofer Winter and a cool reception for the Reservists’ Party, and have also affected Benny Gantz’s political standing. At the start of the war, polls predicted he would win 36 seats, but his support subsequently declined, and for months the polls indicated that his party would not cross the electoral threshold. Ultimately, Gantz withdrew from the race.
After the final poll is published, no further polls will be released until Tuesday night, when the only poll whose results truly matter will take place: the election itself. In the meantime, seat-count polls also influence voters’ decisions, with some relying on the results to vote strategically or avoid undermining the chances of the bloc they support.
However, the discrepancies between polls raise questions about how reliable they are: How is it possible for different pollsters to produce such different results? How do they select participants and weigh their responses? What are the polls’ weaknesses, can they be biased, and why doesn’t a change of one or two seats necessarily indicate a real shift in support for a party?
An analysis of polling data and conversations with pollsters and researchers explain what lies behind the numbers, and how to distinguish statistical fluctuations from trends that may indicate the election results.
The Gap
A common claim about Channel 14 pollster Shlomo Filber is that he was the most accurate in predicting the results of the 2022 election. His polls consistently showed the Netanyahu bloc winning 62 seats or more, and it ultimately won 64 seats. By contrast, pollsters at other channels predicted a closer race between the blocs, with their polls giving the Netanyahu bloc between 59 and 60 seats.
However, an examination of the data paints a slightly different picture: the number of voters who voted for the parties that made up the coalition stood at 2.204 million, while the number who voted for the other parties (excluding parties that received less than 2% of the vote) stood at 2.330 million. However, around 289,000 of them voted for Meretz and Balad, which failed to cross the electoral threshold. This meant that, among parties that crossed the threshold in the anti-Netanyahu bloc, the actual number of votes was 2.011 million. The seat distribution changed, giving the Netanyahu bloc a significant advantage in forming a government that went on to serve its full term.
The gap between the different pollsters in the current election campaign is particularly striking. Over the past year, and even before that, Filber’s polls on Channel 14 have consistently predicted an especially high number of seats for Likud. The polls conducted by his former partner at Direct Polls, i24News pollster Tzuriel Sharon, also consistently give Likud an advantage, though not as pronounced as Filber’s. In contrast, all eight other pollsters working for media outlets place Likud at around 20 seats, and in recent months have put it close to Gadi Eisenkot’s Yashar party, and usually even below it.
An examination of the 118 election-poll reports submitted by pollsters and media outlets to the Central Elections Committee, which are open to public inspection, reveals differences in the methods used to conduct the polls, as well as in their depth and scope. However, it is not always possible to determine how much weight is assigned to each population surveyed (“weighting”), or whether the calculation accounts for its share of the general population, its share of eligible voters, or its share among those who voted in the previous election. A common breakdown in Filber’s polls distinguishes between “2022 coalition voters” and “2022 opposition voters”, but it is not reported whether this is used only for questions about policy issues or also for questions about seat counts and suitability for the premiership.
“None of this seems to meet a reasonable standard of scientifically accepted polling practice,” says Lior Sheffer, a professor at the School of Political Science at Tel Aviv University who researches the political behaviour of elites.
What is very clear, however, is the sampling method: while most pollsters use online sampling, with necessary supplements by phone and SMS for populations that do not participate in online polls, Filber conducts all his sampling online, using an independent online panel that does not work with any other polling institute.
According to Filber, however, it is all the other pollsters who are missing what is happening. At a conference held by the Israel Democracy Institute last June, he said that the number of viewers of Channel 14 had doubled twice since the current government took office: first, after the judicial overhaul began and, as he put it, Channels 12 and 13 amplified the protests; and a second time after 7 October, following what he described as the anti-government line adopted by those channels. The shift in viewers away from these channels also led to a shift between the panels used to conduct polls, leaving the panels used by the other outlets unrepresentative.
“To my understanding, we are the only panel that samples political demographics,” Filber said, explaining that he was referring to subgroups within the right-wing public, which he samples. According to him, Likud is made up of five groups: the “princes”, the state-minded bourgeoisie connected to Likud dating back to the Herut movement; lifelong Likud voters who have voted for the party following their parents and are deeply identified with it; the “Bibi-ists”, Netanyahu admirers who will follow him wherever he goes; the ideologues, who mainly come from the settlements or right-wing strongholds; and a group he calls the “middle generation”, who have moved between right-wing and centrist parties. The latter are referred to on Channel 14 as “crypto-Jews” — people living in the seventh to tenth income deciles who do not dare express their views.
The Bias
The question arises as to why Filber’s poll results are so unusual, and always favor Likud. For those who subscribe to the view that Filber supports Netanyahu as prime minister, as might be inferred from his comments on Channel 14 and Galei Israel and his extensive history alongside Netanyahu, the answer may seem obvious: the bandwagon effect.
“Research shows that these dynamics tend to feed into themselves,” says Professor Sheffer. “Parties that polls show are surging tend to generate a ‘bandwagon effect’ that encourages more voters to switch to them. There is, of course, a ceiling to this, but these are definitely circular processes. Naturally, we cannot infer what causes what simply by looking at the polls themselves, but it at least raises the possibility.”
The Fine Print and Major Errors: Under the Hood
One of the weaknesses of seat-count polls is the sampling error: even a sample that accurately represents Israeli society may present a different picture from the one we would get if we asked the entire public. In common polls, which sample 500 to 600 respondents, the maximum margin of sampling error is 4 to 4.5 percentage points in either direction, at a 95% confidence level. A poll provides only an estimate of public support. When the margin of sampling error is 4.5 percentage points in either direction, at a 95% confidence level, this means that in approximately 95 out of 100 samples, the difference between the estimate and the actual level of support is expected to be no more than 4.5 percentage points. In around 5 out of 100 samples, the difference may be greater.
“You have to understand that this is an exercise involving a great deal of uncertainty, even when it is done as well as possible. When you do it with so few respondents, the margin of error will be very large. There is no such thing as random sampling today in Israel or anywhere else in the world, and because that is the situation, you have to make a great many honest, non-devious adjustments to try to get closer to reality. But these things add a great deal of noise.”
In practice, most polls published each week by news channels and news websites consist of around 500 to 600 respondents, producing a sampling error of more than 4%. Even polls with 2,000 respondents have a margin of error of at least 2.2%.
“You can look under the hood of any poll and see exactly what weighting method is used, whether a panel is used, what is represented in it, and examine the questions and how undecided voters are classified. These are subjective choices that all pollsters make. It is a significant exercise involving a great deal of uncertainty. This is one reason why small polls, with 500–1,000 respondents, have a very high level of uncertainty in the estimates of seats they produce. There is a reason why, even under optimal conditions, the margin of error for such polls is between 3% and 4.5%. You could also argue that, in practice, the margin of error is much greater, because that figure of 3% to 4.5% applies at a 95% confidence level.”
To understand the drama behind the fine print referring to a 4% margin of error, for example, Sheffer and his colleague, Professor Michael Gilad, translate it into seats: the electoral threshold, equivalent to around four seats, is 3.25%, meaning that a 4% error amounts to almost five seats in total. But around the electoral threshold, the numbers produce much greater fluctuations.
“If a particular party receives six seats in a single poll, the pollster is essentially saying that, based on their data, they are fairly confident that the party’s actual result will fall somewhere between 3.6 and 8.4 seats,” Sheffer explains. “In practice, 3.6 seats translates into a flat zero, since the electoral threshold is equivalent to four seats. That is why some pollsters say that even a stable result of five seats, or even a rise to six seats in the polls, does not mean the party is out of danger, although the polls may also be predicting seven or eight seats for it.”
Among the larger parties, which are nowhere near the electoral threshold, a situation can arise in which one party receives 22 seats in one poll and 18 seats in another. “And that would not mean the poll was wrong,” says Sheffer. “It is within the margin of error.”
“Winter’s party could receive completely different results in two polls conducted at exactly the same time, even by the same pollster: in one, it could get nothing because it received three seats and failed to cross the electoral threshold; in the other, it could get seven seats. Both results would reflect the same reality,” says Sheffer. “Neither poll is wrong. That is what a 4% margin of error means.”
“So the big changes you see in the polls are usually a reflection of statistical noise,” he explains. “It is simply an exercise with a great deal of statistical noise, but the media do not present it as such. We think a particular party has gained or lost one or two seats — there is no such thing! No single poll can predict that a party has gained or lost one or two seats. It is all within the polls’ statistical margin of error.”
The Large Polls
In large polls that were conducted and submitted to the Central Elections Committee, the sampling error decreases as the poll grows in size, but even then, it is not always possible to establish its reliability. The largest poll conducted for a media outlet was carried out by Shlomo Filber for Channel 14 on 1 October this year. The poll sampled 2,145 people, with a margin of error of 2.1%. On the face of it, this was a poll that could have been considered highly reliable, but it did not publish the order of its questions.
“There is a method called priming,” says Sheffer, “where questions are asked that can create a particular inclination in the respondent and influence the results.” For example, if the pollster’s first question is “Who in the military leadership is responsible for 7 October?”, without addressing the political leadership, followed by another question or two that praise the prime minister’s decision-making, and only then the question “If the election were held today, which party would you vote for?”, then it is clear that this is priming intended to serve a particular party.
It is not possible to prove that priming took place in the poll in question, conducted for Channel 14, because the order of the questions about seat counts and suitability for the premiership was not published. However, additional questions about policy issues were asked alongside the seat-count question.
Polls with more than 1,000 respondents are generally considered somewhat more reliable. Since the beginning of August, 16 such polls have been conducted for media outlets, along with two independent polls.
However, even polls with higher levels of reliability still showed significant discrepancies. For example, in a Channel 14 poll conducted on 7 October, Likud received 31 seats compared with 23 for Yashar. Together received seven seats, while the Reservists’ Party and Amcha Israel failed to cross the electoral threshold. In a poll conducted for Channel 13 on the same day, Likud received 18 seats, Yashar received 18, Together received 12, and Amcha Israel and the Reservists’ Party crossed the electoral threshold.
Three polls conducted a week later, each with more than 1,000 respondents, also showed discrepancies. The polls for Channels 13 and 16 projected 49 and 52 seats for the Netanyahu bloc, respectively, and 50 for the Change bloc. By contrast, an i24 poll conducted on the same day projected 57 seats for the Netanyahu bloc and 50 for the Change bloc.
Political Bias in the Data
In a Next Data poll for Channel 14 conducted on 28 September, respondents were asked questions that appeared to contain semantic bias. For example: “Does the Attorney General’s instruction that construction in the E1 area should be postponed until after the election prove that she is acting out of political considerations?” (Question 13), or “What do you think is the motive behind the reports about the warnings Netanyahu received about a Hamas attack before 7 October?” (Question 4), with the possible answers being “electoral considerations” or “the desire to reveal the truth”. In other words, the question presupposes that there is a hidden motive behind the publication of the warnings Netanyahu received, and places the political explanation before the journalistic one.
The elephant in the imaginary room of polling is the agenda. There is a consensus among all pollsters that their role is to sample data, analyse it and present a reliable picture of the situation. A minority of pollsters also believe that some degree of interpretation should be incorporated in order to analyse the data properly.
The question takes on added significance given that Channel 14, which is identified with Likud, consistently shows the governing party winning every poll by a wide margin; on i24, the right as a whole leads, but Likud receives a more modest number of seats; and on Channels 12 and 13, which are identified with the sharpest criticism of the government, Likud has not led over Eisenkot for more than three months, and on Channel 13 it has been polling below 20 seats for more than a month.
What Should We Believe?
The known and inherent errors in polls, the margin of error, biased questions, the correlation between a media outlet’s agenda and the results of the polls it publishes, whether or not a party crosses the electoral threshold — which can change the entire distribution of seats — and other biases all raise the question: how can we trust polls at all, and which ones should we trust and which should we not?
“The main thing to do is distinguish between legitimate pollsters and those who are not,” says Sheffer, but he refrains from naming anyone he considers an illegitimate pollster. “Most pollsters are legitimate, and to overcome the inherent uncertainty of any individual poll, or of specific pollsters, you simply need to look at the averages, at the collection of polls, and at trends over time across all the polls and pollsters together, excluding those who are illegitimate pollsters. Then you get a picture about which you can have much greater confidence.
“If you look at the average of ten polls, you don’t have 500 respondents, you have 5,000. By definition, the margin of error decreases, and you have the subjective weighting decisions made by four or five different pollsters. If they don’t all work in exactly the same way, you create a kind of insurance for yourself against that.”