The power of RCTs: reflections & debates

By Zafeer Ravat

aen-blog-image

On Monday the 13th of March, the Africa Centre for Evidence (ACE) team attended a seminar held by the Abdul Latif Jameel Poverty Action Lab (J-PAL) foundation at the Wits University Business School. Under the theme ‘Drawing Policy Lessons from Randomised Evaluations’, the seminar aimed to start a conversation about evaluating public policies and programmes and how to use such information during policy design. The seminar was attended by many academics from various institutions, public sector officials, and civil society organisations. Here, Zafeer Ravat, a researcher at ACE, reflects on his impressions at the seminar:

I found the seminar to be enlightening and informative specifically in terms of how Randomised Controlled Trials (RCT) can be used by policy makers in a more pragmatic way.  The initial briefing of what RCTs are and their advantages was provided by Rachel Glennerster. She highlighted just how influential some of the projects by J-PAL have been in terms of informing policy and programme design. Following this, she presented on a few global projects that J-PAL has undertaken and the relevant results thereof.  One example focused on what type of incentives motivate Community Health Care workers in Zambia.  Following this, her colleagues Laura Poswell and Nilmini Herath went on to present J-PAL projects in South Africa. For instance, they presented on an RCT that evaluated the impact of prepaid electricity meters on low-income households in South Africa.

The presentations given by Rachel, Laura and Nilmini all provided a great form of justification for RCTs and why the methodology is often heralded as the gold standard in empirical social sciences. Following the presentations, there was a Q&A session that brought to light some of the criticisms and limitations of RCTs and the projects conducted by J-PAL. It should be noted that the Q&A was especially well handled by Rachel, Laura, and Nilmini. Furthermore, most of the criticism was largely directed toward the population selection and approaches chosen by the J-PAL researchers rather than methodologically related. However, on that note, there are limitations in terms of more macro-economic problems, for example, the impacts of changes in the interest rate on rural communities. In this case, it is not possible to limit the effect of a national interest rate change to a controlled sample. Nevertheless, RCTs do provide a more problem-specific approach with a great deal of application that perhaps allows for more targeted policy.

In sum, Randomised Control Trials (RCT) are a great tool that can be used to determine whether or not a program has an impact and better evaluate the extent of the respective impact in a rigorous and accurate manner. The J-PAL seminar was well executed in terms of providing awareness and understanding of RCTs and the potential it has in providing the right tools for policy makers.