Diary of a Systematic Review

What works to increase female labour market participation in LMICs: An ACE-led Systematic Review

review-team_eppi               (The ACE review team: Top row, left to right—Mary Opondo, Christa Heyneke, Luvuyo Zigana, Natalie Tannous; bottom row, left to right—Laurenz Langer, Ruth Stewart, Yvonne Erasmus, Charity Chisoro) 

The Africa Center for Evidence has been awarded a year-long grant to conduct a systematic review of the evidence on interventions aiming to support women’s participation in labour markets in LMICs. We have brought together a team of reviewers who will search for, screen, appraise, and synthesise all available evidence relevant to the review question. Our Johannesburg team is pictured above and we are further supported by Alison Bullen (Information Specialist), Dr Ekwaro Obuku (Africa Centre for Systematic Reviews and Knowledge Translation), Janice Tripney (University College London), and Dr Carina van Rooyen (University of Johannesburg). We will be working together over the next 12 months and hope to be able to provide a detailed synthesis of what interventions work, how and why, when it comes to increasing female labour market participation in LMICs. The systematic review will include a Meta-Analysis and Qualitative Comparative Analysis. Commissioned by DFID and overseen by the EPPI-Centre, the exact review question that we are answering is: What are the effectiveness and design features of interventions that aim to overcome barriers to women’s participation in the labour market in higher growth/male dominated sectors?

Here, we will provide a brief diary of our systematic review experience hoping to illustrate the review process and progress, ACE’s approach to systematic reviewing, as well as reflections from individuals within the review team. We hope this might give a better understanding of ACE’s work on evidence synthesis and how it overlaps with the remit of the AEN in supporting the use of evidence in Africa.


At the moment we are busy screening the 14,000+ citations that Alison’s scientific search of the academic literature has generated. Natalie, Mary, Charity, and Luvuyo are using EPPI-reviewer for the joyful task of assessing each citation’s title and abstract for relevance to our review. After 2 weeks, there is only 5,500 to go—piece of cake!


(A snapshot of the screening joy) 

Two and a half weeks and 15,185 citations later, we are finally done with screening! Naturally, the screening of the last study was duly celebrated. Thanks to EPPI-Reviewer, we have a detailed record of the inclusion and exclusion decisions for each citation and will be able to account for which study made it into the next stage of the review and why. After a quiet afternoon of post-screening recovery—each team member screened at least 500 studies per day over the last two weeks—we will regroup tomorrow and allocate tasks for Grey Literature searches and data cleaning for the included citations.


 (When the last citation was screened by the team)

Review team reflection




Luvuyo Zigana on working on his first systematic review: “The interesting thing about systematic reviews is that unlike processed food you know what goes into them. What I have learned so far, particularly in what I have done, is that the most important thing that one can do in order to change the world is to come with an intervention that will produce the outcome that will benefit people. The difficult thing about systematic reviews is that it requires an attentive mind and patience. My take home message after this week is that systematic review is like a methodology of your dissertation, you just never stop working on it.”





Review team reflection


Mary Opondo on Grey Literature searching for a systematic review:

Hello February!

This month started off with the searching for grey literature. This part of research resembles the analogy of broken glass, where all the pieces, including the tiniest shards are pieced up together to create beautiful art. And yes, research is art 🙂 I particularly enjoyed this stage of the research, as it is more involving in that I have to search for abstracts in various websites and screen them before uploading the relevant articles to EPPI reviewer.  This stage requires a lot of patience (especially for sites that reject Boolean phrases or for “hidden articles”), concentration, strategy and COFFEE!!! 🙂 One last thing: Did you know that empowerment is a simple, yet multidimensional concept?





Natalie Tannous on the joys of finishing the protocol and screening stage:

I had some fun moments with my tasks for this week. First I was helping out on the coding of studies at the abstract and title level. I can say that while coding I’ve been reminded of how frustrating vague abstracts are: I’ve seen more than my fair share of ‘entrepreneurship interventions’ and outcomes described as ‘economic lives’. The second major task I had this week was copyediting the protocol. While painfully slow at saving the large MS Word file each time changes were made, I enjoyed the satisfied feeling ones gets after having organised a room or tidied a less-than-perfectly-tidy desk. #successfulweek



Charity Chisoro on finishing the first review milestones and starting a new project: 


I will use the analogy of a puzzle to reflect on the tasks that I have been doing on this project. A puzzle has many pieces tpuzzle_charityhat fit together and every piece is very important as it completes the full picture. The different tasks that I did from the screening, coding, searching of grey literature to uploading articles onto EPPI represent the different pieces of this puzzle. By putting all the pieces together as a team, the puzzle was complete with a final product of a protocol and evidence map. We were able to meet our targets and reach the first milestones through hard work, commitment, support and teamwork. The late Nelson Mandela once said, “After climbing a great hill, one only finds that there are many more hills to climb.” We started this week with an introduction to ESPA project, I am looking forward to learn more as well as contribute to this “puzzle”.



Six weeks into our systematic review, we have now reached our first two milestones: the production of an evidence map of all research relevant to our review question as well as a draft review protocol. The protocol lays out the methods we intend to follow in the conduct of our systematic review. Usually, there is little room to later deviate from your protocol and the team invested a lot of effort in being as specific as possible in our approach. We are now awaiting peer-review on the protocol.

The evidence map, on the other hand, intends to facilitate a discussion between the review team and the review’s stakeholders. We see it as an engagement tool to ensure that our review’s scope fits users’ needs. In this, we first conceptualised a framework of all relevant interventions and outcomes that pertain to the research area (i.e. women’s labour market participation). You can have a look at the framework here. We then engaged key stakeholders of our review mainly from a policy context on the usefulness of this framework to their decision-making needs. For instance, we consulted on whether our interventions covered relevant programme areas that enjoy policy attention. Following this engagement, we then revised our framework and used it to map the evidence base on what works for women’s labour market participation. The result is the below evidence map, which highlights the size of the available impact evaluation evidence on different interventions that aim to support women’s labour market participation. Our next step is now to once more engage with all our stakeholders on the implications of this evidence map for the full review’s scope. This engagement will help us to identify which programme areas to focus on and what synthesis methods to use in order to ensure that our systematic review remains policy-relevant.




Our review team is currently engaged in discussions with review stakeholders to set the final scope of the research. We have shared a detailed review protocol with DFID, EPPI, and our advisory group. We have also disseminated the interactive evidence map to review stakeholders including the Department of Women and Department of Planning, Monitoring and Evaluation in South Africa. Our meetings and discussions with DFID and EPPI have been particular helpful. Following a joint viewing of the evidence map, we discussed possible review and synthesis options. The interactive evidence map allowed us to target the available evidence-base more effectively and helped to zoom in the most relevant intervention-outcome configurations. Having identified the configurations of most interest to a policy/decision-making audience, we then outlined what type of synthesis approaches we can offer (e.g. a review of reviews, a narrative synthesis, a meta-analysis, and so forth). We are currently signing off the proposed way forward balancing policy interests in the review scope and rigour within the conduct of the review. Our team in Johannesburg is using this quiet research time to tidy up the data collected for the map and to finetune the evidence mapping software that we developed for this project.