The Abdul Latif Jameel Poverty Action Lab (J-PAL) is a network of 64 affiliated professors around the world who are united by their use of Randomized Evaluations (REs) to answer questions critical to poverty alleviation. J-PAL’s mission is to reduce poverty by ensuring that policy is based on scientific evidence. Below are the joint responses from Rachel Glennerster, the organization’s Executive Director, and Iqbal Dhaliwal, its Director of Policy.
What are the benefits and challenges of conducting randomized evaluations to alleviate global poverty? How do randomized evaluations contribute to evidence-based policy formulation?
Billions of dollars are spent on development programs around the world. But we have surprisingly little hard evidence on what are the most effective ways to reduce poverty. This is because it is very hard to measure the true impact of development programs in the field. Well-designed and implemented randomized evaluations can produce unbiased estimates of the true impact of a program or policy in the field.
By randomly selecting which of the potential beneficiaries will receive the program (treatment group), and comparing the change in the desired outcome with those potential beneficiaries who were not selected (control group), researchers can control for both observable and unobservable differences between the two groups, thus attributing the differences in the outcomes to the program.
In this way, evidence from randomized evaluations can help policymakers expand effective programs, while discontinuing or modifying the ineffective ones, thereby improving the impact of development policy.
The main challenge in conducting randomized evaluations is getting all the details right, both for the evaluation and the underlying program in the field. In a typical evaluation, researchers work closely with implementing partners to make sure the randomization and the original evaluation design is respected and local enumerators are trained well on how to collect large amounts of data (often in rural or very remote areas). Another challenge is to follow up or track those who participated in the evaluation over an extended period of time when trying to measure the long-term impacts of a program.
J-PAL partners with governments, NGOs, foundations, and other groups to turn research findings into effective policies and programs. How do you develop relationships with these groups? What are the greatest challenges you face as you seek to coordinate these efforts and scale-up development projects?
During J-PAL’s first few years, partnerships with policymakers and implementers were often initiated through the contacts that our affiliates or senior staff had at organizations that shared a desire to use evidence to inform their decisions. The last few years have seen an increasing demand from policymakers and practitioners towards rigorous evidence for policy decisions.
At the same time there is strong interest among academics to use randomized evaluations as a tool to answer key questions for development, sometimes in contexts and areas that have not traditionally been explored. As a result, J-PAL is now developing partnerships in many different ways. Three years ago we created a dedicated Policy Group that is responsible for creating and maintaining long-term relations with policymakers in governments, NGOs, international development organizations, and foundations to promote the use of evidence in decision making.
Partnerships can begin when an organization approaches J-PAL requesting an evaluation or asking for support identifying evidence regarding a particular policy challenge they are facing. Collaborations are also initiated after policymakers or practitioners participate in one of our capacity-building activities, evidence workshops, or matchmaking conferences.
The last are precisely designed to bring academics and practitioners together and to facilitate collaborations. J-PAL’s Policy Group also works to identify organizations or governments that may particularly benefit from the results of J-PAL research, and sets up meetings with key decision makers in those organizations to explain some of these findings and explore potential collaborations. Finally, partnership can also occur when a government or large development organization approaches us to set up special policy and research collaboration, as in the case of the Chilean Government in 2011.
Some of the challenges that we face in our policy outreach efforts include the inability to respond to all the requests that we receive from around the world, the long amount of time and intense effort required to build strong relationships with partners, and the challenge of convincing some policymakers to put greater weight on objective evidence, rather than their instincts or ideology when making key decisions.
How do you disseminate your analyses to governments, NGOs, foundations, and international organizations?
Research studies are often written as academic papers in a very technical format, which can be difficult for policymakers to access, interpret, or use. J-PAL’s Policy Group takes the results of field evaluations and creates publications that make it easier for policymakers to access the latest research results. This includes policy summaries of more than 200 J-PAL studies on our website, print publications that present research results in a concise and simple format, and cost-effectiveness analyses that synthesize the findings from multiple studies. Most of these publications are also translated into Spanish and French and made available on the J-PAL website. Many are printed, and mailed to key contacts of the J-PAL network around the world.
Can you explain the objectives and progress of the Governance Initiative?
The Governance Initiative (GI) is a multi-year effort to carry out a number of randomized impact evaluations of programs designed to improve citizens’ participation in the political and policy process, and reduce corruption and leakages in public programs.
GI began by conducting a comprehensive review of the research literature to identify what is currently known or not known about the drivers of good governance and identified a list of research priorities which then formed the basis for six monthly calls for proposals from J-PAL researchers working in the area. GI reviews the research proposals for their ability to increase our understanding of these identified priorities and provides funding to the most promising evaluations of governance programs. Once the results of these studies become available, GI will initiate a set of policy activities to disseminate the findings from its evaluations to policymakers, and will support the scale-up and replication of successful programs.
GI has already completed its first round of funding, providing grants to two full studies and six pilots, and is currently in the process of reviewing proposals for its second round.
How receptive are policymakers to your methodology and your findings? In your analysis of the need for police skills training, for example, how open were police and local governments to your analysis and subsequent scale-ups?
Although the degree of receptiveness from policymakers varies greatly, we see a growing demand and interest in randomized evaluations from governments, NGOs, foundations, and development organizations. An important reason for this is that randomized evaluations not only measure impact rigorously but also offer key insight from quantitative and qualitative data on why programs succeed or fail, provide practical information to help facilitate and guide scale ups, and are easier for nontechnical audiences to understand.
In the case of the police skills training, we had an amazing partner in the Rajasthan Police. In particular, Inspector General of Police, Nina Singh, was very open to new ideas and tireless in her efforts to convince her superiors, colleagues, and staff on the importance of introducing new innovations to improve policing and the need to evaluate these interventions to know what really worked—or not—and why.
Following the two-year pilot program in Rajasthan, statewide police reforms have been initiated. The apex body for policy formulation of Indian Police has sanctioned a special grant for the scale-up of both investigation and soft skills training across Rajasthan state. Training in soft skills (communication and mediation, stress management, leadership, and attitudinal and behavioral changes) has been scaled up to cover 10 percent of the 78,000 police personnel in the state. Training in scientific investigative methods has been scaled up to cover over 18 percent of the state’s investigating officers.
J-PAL works around the world. How do your findings and recommendations vary by region? For example, do your recommendations for clean water initiatives differ between Central America and Southeast Asia? Do regional nuances play a significant role in policy recommendations and project scale-ups?
J-PAL’s strategy for disseminating evidence and policy recommendations certainly recognizes the differences in economic, social, and political contexts across regions. Although many developing countries have similar problems, the source of those problems and the institutional contexts can be very different. Take, for example, the case of deworming, which has been shown to be a very cost-effective policy for increasing student attendance in Kenya and has now been scaled up in many other countries, including India. This is relevant evidence for countries in Africa or South Asia, where the prevalence of worms might be high, but perhaps less relevant in parts of Latin America, where sanitation infrastructure is better.
In those contexts, J-PAL will not emphasize deworming, but instead highlight alternative programs that have also been found to be effective in increasing student attendance. J-PAL’s regional offices also play a key role in identifying which research studies are well-suited for the local context and how best to communicate that evidence.
Similarly, in the case of clean water initiatives, a strategy like spring cleaning would be better suited for population clusters that rely on water from free-flowing streams rather than communities that rely on wells. For the latter, source chlorine dispensers may be more effective. Another key tool to help identify policy recommendations best suited for a particular context is cost-effectiveness analysis, and the sensitivity of relative cost-effectiveness to different parameters. For instance, by varying the population density assumptions in our cost-effectiveness calculations, policymakers can get a good sense of whether an intervention found to be cost effective in a densely populated region would still be cost-effective in a sparsely populated region.
Similarly, if it seems that contextual factors may play a large role in driving impacts, we often recommend that policymakers undertake a pilot or small-scale “policy evaluation” in their context before scaling up a program found to be effective in another context. This allows policymakers to understand the true cost and impact of the program, as well as draw key process and implementation lessons that can help strengthen the program that is ultimately scaled up.
Feature photo: cc/gfpeck