Analysis: Afrobarometer: Supporting or Influencing Public Policy Decisions? Part II
By Getu Teressa @GetuTeressa
Addis Abeba, October 02/2020 – This is part II of a two-part article. Part I assessed specific contending narratives surrounding Ethiopia’s federalism in order to provide a contextual framework for analyzing the Afrobarometer (AB) 2019/2020 Ethiopian study. Part II is focused on the A.B. survey itself. It explores a wide range of topics, including the timing of the survey results’ release, the intended goal of the study and its ramifications in the current fast-evolving Ethiopia’s political climate, and the critical factors that have implications in the survey’s validity and reliability.
Afrobarometer Survey
AB, which promotes itself as a pan-African, nonpartisan survey and research network, conducted the Ethiopian survey while noting that “[t]he fate of federalism has been intensely debated since the country launched a reform agenda two years ago.”
A.B.’s work spanning 38 countries gives it a pan-African appeal, but their survey research is conducted on the network’s behalf by partners in each of the survey countries. These partners are responsible for recruiting and training interviewers, leading and managing the survey, and data collection. Furthermore, AB delegates the task of generating country-specific questions (CSQs) on “hot” issues to its “National Partners” who consult with key stakeholders and potential users of the data. A.B. advises that some of the CSQs may be “… part of a specific strategy by the National Partners and stakeholder to influence the policy debate and policy.” This delegation of tasks to more-or-less independently operating local partners, therefore, undoubtedly complicates its assertion of being nonpartisan and risk introducing conflicts of interest at different stages of the survey research. The Afrobarometer Round 8 Survey Manual (obtained via email correspondence) does not provide any specific mechanisms to address this complex issue. This is undoubtedly critical when dealing with explosive topics such as federalism in Ethiopia.
The latest Round 8 (R8) survey, the second round involving Ethiopia, was conducted in December 2019 and January 2020 by A.B.’s National Partner in Ethiopia, ABCON Research & Consulting, directed by Mr. Mulu Teka.
The first Ethiopian survey in 2013, also conducted by ABCON-Research & Consulting, was deemed unreliable due to the results producing “a puzzling anomaly.” According to the report, which was coauthored by Mr. Mulu Teka, “[w]hile no expert assessment comes close to calling Ethiopia a democracy, 81% of Ethiopian respondents told Afrobarometer interviewers that the country was either a complete democracy or a democracy with only minor problems.” This was deemed to be in contrast to the opinions of ordinary citizens from other African countries who “tend to reach the same conclusions about the extent of democracy in their country as international expert rating systems devised by political scientists.” The paper was published under the title “Ethiopians’ views of democratic government: Fear, ignorance, or unique understanding of democracy?”
For the 2019/2020 survey, ABCON generated CSQs covering “issues that are topical or have currency in Ethiopia.” ABCON has not yet disclosed information regarding the CSQ selection process, the local stakeholders, or the potential survey users being consulted. However, according to the Afrobarometer Round 8 Survey Manual, survey questions are reviewed and approved by a Core Partner, the Institute for Development Studies, University of Nairobi, Kenya, and further commented on by a “questionnaire committee.” Such steps are important; however, the intricate responsibility of designing a “clean” survey questionnaire on concepts too complicated for most of the survey respondents will undoubtedly rest with the National Partners. They are better positioned to understand the specific national and subnational social and political contexts of the hyperpolarized society they are a part of. This, in turn, makes it all the more necessary for A.B. to have a robust mechanism to ascertain non-partisanship and conflicts of interest.
The partial results of the current A.B. Ethiopia survey were released to the public on August 25, 2020, seven months after the completion of the fieldwork on January 26, 2020
Timing of the results’ release
The partial results of the current A.B. Ethiopia survey were released to the public on August 25, 2020, seven months after the completion of the fieldwork on January 26, 2020. The AB R8 manual stipulates that National Partners conduct the first release event no more than one month (30 days) after the data set has been finalized. The AB Ethiopia R8 survey data set was finalized on April 27, 2020 (correspondence with A.B. publication manager). It is unclear why the initial release of the results, which should have been on or before May 27, 2020, was significantly delayed. Given that the August 25th press release and media briefing were conducted amid a rising (and possibly peaking) coronavirus pandemic, it is unlikely that the coronavirus pandemic and Ethiopia’s state of emergency were the reasons for the delay in releasing the results.
In one of its two-part press releases, A.B. concluded that “Ethiopians embrace federalism but are split over whether it should be ethnic or geographic.” The survey results have triggered a modest amount of debate and variable interpretations. However, some critical voices are missing from the debate owing to a dramatic authoritarian backsliding in recent months. Formidable political challengers to the incumbent party and fierce advocates of Ethiopia’s multinational federation are currently behind bars. A key opposition figure, Jawar Mohammed, who is also an undisputed proponent and arguably the most vital voice and “explainer in chief” of Ethiopia’s multinational federation, has been in detention since June 30, 2020, and is facing a political trial. Thus, the timing of the report’s release, which coincides with the relative vacuum felt as a result of the political incarceration of key pro-federalist politicians, notably leaders and members of the Oromo Federalist Congress and Oromo Liberation Front, may have diminished the diversity of interpretations of the report’s significance. In effect, the timing of the release may have altered or skewed the overall influence of the survey results and how they may be used by “potential survey users,” including the government.
Stated goal of the survey and cautionary note
As briefly mentioned previously, A.B. envisions its studies to influence “policymakers, policy advocates, civil society organizations, academics, news media, donors and investors.” A.B. defers to its National Partners the responsibility of “preparing and disseminating Media Briefs, Press Releases, Dispatches, and Summaries of Results, and play a key role in other communications activities.”
During the first media briefing at the survey’s release, A.B. National Investigator/National Partner Mr. Mulu Teka stated that A.B.’s Ethiopia survey results were not to be left on the shelf but to be disseminated widely. The dissemination strategies include, according to Mr. Teka, conducting briefings with the Office of the Prime Minister and other government officials, parliament, the executive branch, civil society, academics, political parties, and eventually, the diplomatic community, with the intention of influencing their policies and attitudes.
By means of a single survey, A.B.’s goal to influence policy on a combustible subject at the heart of Ethiopia’s ethnonational divide can be considered bold, audacious, and perhaps even very aggressive. The impact of the complex survey context on the survey’s validity and reliability cannot be overlooked. Such factors include the survey–surveyor language and ethnicity and the low literacy levels of a significant portion of the survey population who may not fully understand the survey’s concepts and are therefore vulnerable to leading survey questions or surveyor bias.
This concern is compounded by the fact that potential end users of political public opinion polls, as is often the case elsewhere, may not critically examine the actual survey questions or methodologies and may use such polls for self-serving purposes. Therefore, the potential for misuse or misguided use of opinion polls on highly divisive and politically charged subjects such as this one is very high and can be consequential. Furthermore, it is also concerning that A.B.’s overarching mission and overemphasis on influencing policy, by hook or by crook, exerts considerable pressure on the National Partners since it is stressed as the ultimate measure of their success. It also increases the probability of the misuse of the survey results, especially by those deemed to be stakeholders and are included in the development of the CSQs, as they may be actual users—and disseminators—of the survey results. These entities could well include authoritarian regimes that may exploit the polling results to maneuver around or manipulate the proper procedures needed for policy changes.
The stakes are very high in Ethiopia as concerns of a subversive plan to undo the multinational federal system are deepening. This is further compounded by the fact that the country is feared to be on a downward descent into authoritarianism. The current Ethiopian climate is one in which politicians, especially those who espouse multinational federalism, are imprisoned. The government is also seen as manipulating the timing of elections to pave the road for an uncontested electoral win and impose unpopular policies.
Survey language
The language in which the survey is conducted has significant implications on the quality of the data and its representativeness. This is particularly critical for A.B. as it tries to conduct a survey of representative samples in the multilingual state of Ethiopia.
A.B. conducted the interviews only in “officially translated” languages based on each respondent’s choice of a preferred language. The languages used for the Ethiopian survey were Amharic, Afaan Oromo, Somalia, and Tigrigna, covering approximately 75% of Ethiopia’s linguistic communities.
Since ad hoc field translations are not allowed, potential survey respondents are skipped if they cannot be interviewed in one of the aforementioned languages. Furthermore, potential respondents are also skipped if the interviewer is not fluent in the respondent’s preferred language, and another interviewer with the required skill cannot be brought in. This may have resulted in the undercoverage of the language groups in Ethiopia, thereby affecting the quality of the data and the survey’s representativeness. It is conceivable that this has impacted the survey outcomes since survey responses can potentially correlate with ethnolinguistic patterns.
Survey-respondent ethnicity
Surveys in different African countries have demonstrated that the interviewer and respondents’ co-ethnicity status significantly affect survey responses—by as much as 28%—depending on the political salience of ethnicity, the political sensitivity of the survey questions, and the social desirability of the responses, as discussed in “Who’s asking? Interviewer coethnicity effects in African survey data” by Adida et al. Using the Afrobarometer data from 14 African countries, Adida et al. confirmed that people’s response to survey questions is significantly different when interviewed by a non-co-ethnic —”always in the direction of the more politically correct or socially desirable answer.” This becomes even more important if the survey questions are framed to make one provide politically correct or socially desirable answers (discussed further in this article under “SURVEY QUESTIONS”). Surveys are therefore sensitive to the co-ethnicity status of the interviewer and the respondent.
It is also very important that the ethnic representativeness of the survey and the results of the survey according to ethnicity be released.
There are 80 distinct ethnic groups in Ethiopia, and there were 80 field personnel (16 supervisors, 64 enumerators) in the A.B. R8 Ethiopian survey. Given Ethiopia’s long history of ethnic marginalization and the pervasive imbalances in ethnic representation, it is likely that there was a significant interviewer–respondent non-co-ethnic dyad. Therefore, reporting on the ethnic characteristics of the interviewer–respondent could be informative in estimating the extent of their impact on the survey.
It is also very important that the ethnic representativeness of the survey and the results of the survey according to ethnicity be released. This factor may have a profound impact in Ethiopia since this survey goes to the heart of a subject that divides Ethiopians along ethnonational and ethnolinguistic lines. A.B. has released limited demographics data (gender, urban vs. rural residency, age, education, and religion), but not the critical data on ethnicity, which is fundamental to contextualize the current survey results. Given the apparent salience of this data to ascertain the representativeness of sampling, it is unclear why it was omitted during the initial survey results release event.
Survey questions
Two survey questions were administered sequentially to assess respondents’ views on federalism vs. a unitary form of government and geographic federalism vs. ethnic/multinational federalism.
Survey question 1. “Should Ethiopia remain federal or change to a unitary form of government?”
The survey respondents were asked:
Which of the following statements is closest to your view?
Statement 1: Because of Ethiopia’s diversity, some type of federalism with independent regional governments is still the best form of government for the country.
Statement 2: Federalism is too divisive and leads to conflicts; Ethiopia should change to a unitary government in which the central government has more authority in decision-making.
[Bold font and underlining are added only for emphasis.]The two questionnaire statements above are laden with descriptions outside the core survey questionnaires, which are “Federalism is the best form of government for the country” vs. “Ethiopia should change to a unitary government.”
In statement 1, the dependent clause “Because of Ethiopia’s diversity” (“በኢትዮጵያ ብዝሃነት የተነሳ”; “Itiyoophiyaan sababa sabdaneessummaatin,” as it appears in the Amharic and Afaan Oromo versions of the survey, respectively) seems like an appeal, albeit weak, to the respondents to choose federalism. Undoubtedly, Ethiopia’s diversity is one of the arguments, but it is by no means the strongest invoked in support of federalism. Ethiopia’s diversity is also not a point disputed by the detractors of federalism. Therefore, although the insertion of this mildly leading clause is risky, the potential biasing impact on statement 1, if there is any, is unlikely to be paramount. In other words, this dependent clause is less likely to “lead” those who would otherwise favor a unitary form of government.
However, statement 2 is quite glaringly problematic due to the strong framing effect of the leading independent clause: “Federalism is too divisive and leads to conflicts” (“ፌዴራሊዝም በጣም ከፋፋይና ለግጭቶች የሚዳርግ ስለሆነ”; “Federaalizimiin baay’ ee kan qoqoodu waan ta’ef gara walitti bu’insaatti geessa,” as it appears in the Amharic and Afaan Oromo versions of the survey, respectively). As discussed in part I of this two-part piece, the notion that federalism is inherently “too divisive and leads to conflicts” is contested but also the most potent argument deployed by detractors of the current federalism. Therefore, statement 2 has a robust and self-contained argument against federalism, and it has been deployed using strongly negative and emotion-evoking language. Consequently, it is a strongly leading statement. By essentially framing federalism as a morally or logically undesirable and “politically incorrect” choice, statement 2 plays into the psychology that leads to bias. As I argued earlier, this is also when the impact of the interviewer–respondent co-ethnicity dyad becomes more profound in biasing the survey outcomes.
Accordingly, the two survey statements were framed to variable degrees, leading the respondents to choose one side of the debate. The uneven effect of the two leading statements is noticeable. Regardless, including leading “… statements often used by proponents and adversaries/critics of federalism” (quoting my correspondence with Mr. Mulu, the National Investigator and Director of ABCON Consulting), whether the leading effects are “balanced” or not, is a very unconventional and erroneous approach to survey questionnaire design. Obviously, this would give the surveyors great latitude to frame the survey questions to manipulate the leading effects to variable degrees.
In part I of the article, I have discussed in detail the contending narratives surrounding Ethiopia’s federation. It is critically important to remember that in the Ethiopian political discourse and the ongoing debate on federalism, the description of federalism in statement 2 as “too divisive” and “leads to conflicts” is politically charged and specific to multinational federalism. Therefore, this description can be a clue to the respondent to provide a response other than multinational federalism. Although this first survey was about “federalism vs. a unitary [system of] government,” the biased survey statement 2 has virtually tagged “multinational federation” (the subject of the next survey question) in a suggestively negative way as a less moral and socially undesirable alternative. Therefore, as I will argue below, survey 2 is where the biasing effect is most manifested.
Survey question 2. “Type of federalism: multinational/ethnic vs. geographic”
In this part of the survey, the respondents were asked:
Which of the following statements is closest to your view?
Statement 1: If Ethiopia remains a federal system, then the current system of ethnic federalisms, where regions are defined based on ethnic groups, should be kept.
Statement 2: If Ethiopia remains a federal system, it should change to a system where regions are based only on [the] geographic features of the country, not on where different ethnic groups live.
Overall, these questionnaire statements describe “ethnic federalism” and “geographic federalism” in a neutral manner, except that the word ethnic is not used in Ethiopia’s Constitution and is a subject already criticized. Similarly, in the Afaan Oromo version of the survey, the term “gosaa” was repeatedly used to refer to what the Ethiopian Constitution refers to as “nations and nationalities and peoples.”
In the survey, “multinational federalism” or “ethnic federalism” was translated into Afaan Oromoo as “federaalizimii sabaa/gosaa.”
This is deeply problematic as gosaa is a pejorative word used by Amharic speakers in reference to “ethnic” groups or “nations and nationalities.” Therefore, “ye-gosa federalism” is meant to belittle the system. While in Afaan Oromo, gosaa (which refers to a kin group, clan, or tribe) is never used to define federalism, thereby providing little room for this to have been an “innocent” mistake; instead, this paints a picture of an explicit partisanship mindset or intent. This demonstrates the inadequacy of the current A.B.’s approach to oversee CSQ development.
Nonetheless, this survey 2 question followed a logical order from the first survey question about federalism vs. a unitary form of government (i.e., from general to specific). However, the validity of the survey was seriously impacted by the first question. As stated previously, in Ethiopia’s political context, the description of federalism as “divisive” and “leads to conflicts” is a politically charged description specific to multinational federalism. As such, the respondents would have already been primed to view multinational federalism negatively, thereby cuing the respondents to get their answers from the previous survey question. This phenomenon, called “order-effect bias,” could have affected the survey responses by up to 40 percentage points.
It is therefore incredibly likely that the entire survey questionnaire and its validity, mainly the results of survey 2, is highly compromised by bias.
Biased survey
There are several concerns regarding the validity and reliability of AB Ethiopia 2019/2020 survey on federalism. Bias may have resulted in the survey results being skewed, placing supporters of “geographic federalism” and “ethnic federalism” in a statistical tie (49% vs. 48%, respectively).
Although good evidence takes precedence over simple observations, it is essential to point out that the results of this survey are also difficult to reconcile with the prevailing realities of Ethiopia. For instance, the Southern Nations, Nationalities, and Peoples’ Region (SNNPR), one of the federal units of Ethiopia cobbled together from diverse nations, nationalities, and peoples, is heading toward ‘fragmentation’ as different ethnonational groups demand the establishment of their regional states to fully exercise their constitutional right to self-determination. Although detractors of multinational federation dismiss these demands as elite-driven, the Sidama referendum was a clear indication of the political tsunami that is pressuring Ethiopia to live up to its constitutional principle of self-determination and multinational federation, and not run away from it. The Sidama Regional State recently joined the Ethiopian Multinational Federation after a landslide vote of 98.52% in support of a new regional state in the referendum held in November 2019. The creation of Wolayta Regional State is imminent. At least 10 other ethnonational groups are demanding regional statehood, a right granted by the Constitution. This is a testament to the broad support for the current multinational federalism and the principle of self-determination in the SNNPR, which represents 21% of the Ethiopian population. Tigray Regional State, where 6% of citizens of the federation live, recently held regional elections in defiance of the federal government’s decision to postpone elections. The massive turnout of voters is largely seen as a show of Tigrayans’ commitment to multinational federalism and self-determination. Furthermore, based on the prevailing realities, it requires a suspension of common sense to believe that the Oromo, Somali, and many other historically marginalized communities would give up multinational federalism in favor of geographic federalism, which, as I argued in Part I, entails the abrogation of the right to collective self-determination. Therefore, the results of the survey, already fraught with several flaws and concerns, fails the plausibility test!
The flaws in the survey may therefore have underestimated the broad support for the current multinational federation and artificially moved the idea of geographic federalism more into the mainstream. This may just be the immediate effect of the result but the perception of momentum it generates can potentially manipulate the future. Public opinion polls do not just measure current views; they also manipulate the future by planting initial results that influence the balance between contending views and triggering a dynamic that eventually shifts the poll results in the desired direction. This could be one long-term consequence of the survey’s potentially flawed results- A.B.’s ‘gift’ to the proponents of geographic federalism.
Concluding remarks
Afrobarometer’s latest R8 survey, the second for Ethiopia, is intended to apply evidence from the public survey to influence policy. It is the first public opinion poll of its kind in Ethiopia to spark intense debate and has earned wide publicity in the country. As such, it is a noteworthy start.
Going forward, A.B. should focus more on promoting accountable and positive use of its survey results rather than the current aggressive and blind approach to influencing policy
However, it is essential to understand the limitations of this single study, not to overstate its significance, and not to open the door to the misuse of its findings at this time of political polarization and democratic backsliding that is drawing global concern, including from the U.S lawmakers.
Going forward, A.B. should focus more on promoting accountable and positive use of its survey results rather than the current aggressive and blind approach to influencing policy. A clear line needs to be drawn between generating evidence to support decision-makers and making aggressive attempts to influence decisions. The explicit instruction by A.B. to National Partners to ‘consult’ potential survey users in developing survey questions and dissemination of the results is a stunning invitation to use the survey for a particular agenda and erode confidence. Especially when this is done in the absence of transparency and a robust protocol to ascertain non-partisanship, the reputation of A.B. and the National Partner is seriously threatened.
The protocol in place to develop the CSQs and rules of engagement with stakeholders —potential survey users —should be reviewed as a matter of urgency because the consequences of bias can have significant effects on the political discourse in a country like Ethiopia. The delay in the first releasing event of the results of this survey and the glaringly misleading survey questions should also be independently investigated and the outcomes publicized to avoid possible negative associations with future survey results.
Potential consumers of survey results, including government officials, civil societies, politicians, the diplomatic community, and other stakeholders whom the survey was intended to influence, should be cautious in interpreting the results.
I believe that the arguments presented here are sufficiently compelling to cast doubt on the validity and reliability of the results of the survey, especially the part of the survey assessing support for “ethnic” vs. “geographic” federalism. The flaws are significant enough for A.B. to discuss the survey’s limitations and shortcomings and to take remedial actions because the biased results may have far-reaching consequences. AS
___________________________________________//____________________________
Editor’s Note: Getu Teressa is an assistant professor of clinical medicine based in United States. He can be reached at dabalaa@hotmail.com
