After many years of field research in Uganda, three economists outline how to prevent a powerful research tool such as randomized controlled trials from becoming too invasive for local populations and potentially leading to inaccurate results. Western researchers conducting experiments in developing countries should cooperate with local universities and scholars.
With the increased deployment of randomized controlled trials in low-income countries, researchers started to study human subjects in fragile contexts and potentially “not-protected” environments. Setting aside any ethical concern around experiments, there are other aspects of this method that deserve further discussion.
We base our comments on our own experience and personal backgrounds: we truly believe that field experiments are useful, sometimes unique in answering important development questions. Yet, development economists can definitely design more respectful and inclusive experiments.
Our discussion focuses on two stages: i) the project design phase, where feedback and co-authorship are dominated by the Western world, and where ethical review boards are generally not accompanied by local experts’ inputs. And, ii) the project implementation phase, where field experiments can interfere with local communities’ lives, politics and expectations beyond the scope of the project; and where team dynamics and power imbalances could benefit from local knowledge, discussions, and training on how history affects interactions in the “field” today.
Last year’s Nobel Prize in Economics went to three leading scholars in Development Economics and pioneers of the experimental method using “randomized controlled trials” (RCTs). Their major contribution has been to study how to identify the best strategies to alleviate poverty, and they spent their careers trying to do so. The assignment of this prize sparkled (again) a debate, with more or less extreme criticisms, on whether this method should be considered the “golden standard” to identify the causal impact of specific policies.
RCTs, first adopted in medical research and later in social sciences, are an important tool to understand what type of development programs or policies work in certain contexts. Most importantly, they inform bigger interventions in order to avoid investing in big developmental work based on “anecdotal evidence” only.
Yet, in different countries and contexts, as recently highlighted by Ankur Sarin of the Indian Institute of Management, experiments have also been “indecent”: seeking the randomization “at any cost”, Sarin argues, some scholars have sometimes come up with studies disrespectful of the local culture and environment.
Other economists such as Grieve Chelwa and Seán Muller have pointed out that many of these studies do not consider the “disruptions” they can have once the experiment is over, when “randomistas … leave behind once they’ve gone back to their cushy lives in the US and Europe”.
Although these allegations are strong, we believe there is some truth behind them. Based on lessons we learned and tough experiences we had as “field” Research Associates living for several years in Uganda, and now as Principal Investigators in the same country, we also believe there are solutions to design “decent” programs.
We write pondering over our different personal backgrounds and nationalities because we believe they have a big influence on how we analyze issues and propose solutions: two foreigners—one from Latin America and one from Europe—and one national from Uganda implementing research in a low-income country in Africa.
The Design Stage
Exploratory Work, Feedback, and Co-Authorship
When designing research projects, we are often interested in presenting our ideas to academic audiences to hear their feedback on how novel the idea is, how to go about implementing it, and so on. Yet, when designing RCTs in development economics—given that development projects are mostly implemented in the Global South—very few ideas are presented to national universities before implementation.
Take the acknowledgments to many published articles in this field: they mostly thank audiences from Western countries for feedback, and very few thank local universities or institutions for the same. How can researchers know that the question is of interest to local scholars or relevant for the country?
We believe it is not enough to read local news or history books. A simple solution would be to present at local universities/institutions when doing exploratory work—which can also serve to create networks, to know the work that has been done on the topic by local researchers and potentially and when possible, to co-author with nationals. This network can as well motivate the local universities and researchers to design RCTs and apply for grants.
Expertise in Local Institutional Review Board (IRB)
As mentioned in recent articles on the ethics of randomized control trials in low-income countries (read for instance this article by Johannes Haushofer and colleagues at Princeton), one way to make sure that the experiment design will not harm the participants in the study locations is to obtain a local Institutional Review Board (IRB).
The rationale is that local reviewers will have a better knowledge of the local environment where the study takes place and will be in a better position to say what is acceptable and what is not. Additionally, one should expect the review to be conducted by experts of a specific area in different realms of science. However, in practice, researchers tend to seek approval from just a handful of research ethics committees (perhaps the better-known ones) whose reviewers are not always experts of the topics they are called to review.
For instance, in Uganda, the most famous ethics committees are in the realm of public health or medicine. How well equipped are they to review the ethics of a project on the impact of policies on local labor markets on workers?
One way to deal with this dilemma is to first collaborate with local scholars such as academics or other experts residing in the country where the study takes place, as suggested by World Bank economist Oyebola Okunogbe.
Complex Consent Forms
Because of some IRB requirements, many of the consent forms are complex sheets with too much information for a respondent to process, especially if respondents have low education. We have not seen a consent form that is easy to understand and at the same time provides all the useful, summarized, and simplified materials that the respondents need (and want) to make a decision. Moreover, we do believe that the way consent forms are designed leads to misunderstandings between the population and lead to “researcher-approval bias”.
The Implementation Stage
Local politics
Many of the poverty-alleviating policies that development economists are interested in studying entail the provision of handouts, cash transfer, or other economic incentives, as well as payments for participating to (long) surveys. At times, economic stakes are particularly high (equal to days of paid work in rural areas).
For this reason, political elites and local leaders may see the arrival of a research project as a great opportunity for propaganda for themselves. This is because, in many contexts, such as in many areas of Uganda, researchers need to seek approval from these leaders to interview people from their villages. Often, these permissions are sought days before the start of the data collection or the implementation of the project.
Hence, in practice, we witnessed many times when local leaders took (or tried to take) full credit for the arrival of a research team in charge of providing money or goods to participants of the study. Then, in designing a research project in these contexts we should also ask: how does a research project indirectly affect local politics? As these are possible (although usually difficult to predict consequences) of experiments, we should train field managers and data collectors on how to deal with these consequences.
Some solutions that we think are (i) Researchers can always use media platforms accessible to the area of scope to inform how the project is purely research and disclaim any kind of political propaganda. Nowadays many rural areas in low-income countries have access to social platforms, and we believe it is a useful and cheap communication system; (ii) A meeting with village stakeholders can always be convened days before starting real data collection and participants can know that project is purely research. This may add to already expensive budget costs, but it is the responsibility of the researchers to avoid such harmful spillovers; (iii) The implementation of research projects should avoid taking place during election periods.
Moreover, not always development projects use cash incentives in the field. Yet in vulnerable contexts, influential people in a community can take advantage of a research project in different ways. One anecdotal but good example happened to us on one of our pilot studies, which took place in a refugee settlement in Uganda. We were interested in interviewing young refugee job seekers to understand challenges to labor market integration in the host country. We collaborated with several local leaders to collect these data.
A month after the data collection, one settlement leader we did not work with, leveraged on his influence over the community of Congolese refugees in the settlement to collect money from a few of the participants to our study, saying that he was in contact with a Swedish institution to organize a resettlement to Sweden (our consent form carried the name of the PI’s university in Sweden).
Alerted by our team and by local leaders with whom we created a strong collaboration, we collaborated with the NGO we worked with to make sure these rumors stopped before people had their hopes betrayed further.
Could have we prevented this from happening? Probably not. And probably this happens not only with research projects but also with other development work done in vulnerable areas. However, the dilemma remains strong: even if we did not make any promise during the data collection, perhaps the best way to avoid the problem would have been to abstain from collecting data in such a vulnerable context.
At the same time, though, we needed and continue to need data to propose better policies in many contexts, including the most fragile ones. So, we see some solutions: having close and truthful connections to the ground will definitely help to avoid any negative consequences associated with research work. And also, having a consent form that is “closer to the population” understanding will probably improve respondents’ understanding of the project.
Another potential solution would be to have more communication with research participants, not only during the data collection period. This could help in keeping research participants updated on the progress and scope of the project, as well as them reporting any issues happening in the field. This can be done through short text messages, or phone calling and it should not be done only when the researcher collects data from the respondent.
Team Dynamics
A trained team of data collectors will be led by experienced field managers and a complex system of checks and reports will be in place to address issues such as data quality and respondents tracking. Like a manager or CEO of a company, the researcher will then direct a team of different talents, roles, and cultures, and like a manager or a CEO she will need to oversee and solve potential frictions and misunderstanding among team members. For this reason, spending a significant amount of time in the country before and during implementation allows the researchers to understand how to deal with potential cultural clashes and find ways through.
We do understand that spending significant time abroad is not easy for many reasons, and again, we go back to the importance of having co-authors in the country, who can take the role of a manager when running these experiments.
Power Imbalances
Another issue that needs consideration is about power-imbalances and power-dynamics within teams and with respondents. We cannot ignore that most of these countries come from a strong history of colonization, and we cannot ignore how “white” people or foreign researchers/research organizations, more generally, are seen by the team members and by the respondents. For the first case, we just believe that researchers need good training of how power-imbalances affect everyday interactions to analyze some important but crucial things: are team members open with you? Are they telling you what you want to hear because of their culture? How do they consider you?
As far as we know, development economics courses in the Western world do not discuss this topic, which is of huge importance. Some advocacy groups from the Global South became recently more open about these issues and also some development practitioners and researchers are seriously discussing the implications (see Adhikari, Elorrieta, and Pomeranz.) As researchers from different personal backgrounds, we found it useful to be open about this sensitive issue and to have discussions about it during, for instance, training. Also, when we work with people that have never been in a country from the Global South we often ask them to have a look at useful pages on how to act in a respectful way, which can involve what type of pictures you take, or avoiding what is now called the “white savior” behavior.
The issue of complex consent forms, and the presence of foreign “white” researchers in local areas of study result to respondents giving misinformed consent. The respondents could just participate in the study for the sake of the existence of a “white” man/ “white” institution with the hope of getting aid. This, therefore, leaves a question on whether the responses got from participants reflect the true experience or it is a concoction of responses for the sake. An interesting CGD discussion between development practitioners Stephanie Kimou and Angela Bruce-Raeburn on consent, colonization, and development highlighted some of these concerns as well.
Finally, do local researchers (enumerators) take time to understand the concept of the research project or do they rush to go to the field for purposes of earning a living? This might also lead to coerced consent caused by local researchers because their performance is measured on output. Measures to have inclusive participation of the local researchers in planning (design) of the research project are also necessary.
We are positive about the changes that will take place in this field. Research practices in development economics are already more open to discussion than ever in the past. Yet, more can be done. We believe that the change could be driven by senior researchers, whose leadership, experiences, and extensive work already influence to a great extent the junior ones. In the meantime, we can all work towards making development economics research more respectful and inclusive.
Disclaimer: All pictures were taken with consent.
