EEP/Shiree: Using adaptive programming to monitor change in Bangladesh

written by Salimah Samji

How do you effectively monitor an 8 year, £83.5 million (around USD$135 million) challenge fund that partners with NGOs to improve the livelihood of 1 million beneficiaries? A daunting task indeed.

The Economic Empowerment of the Poorest (EEP/Shiree) program is a partnership between the UK Department for International Development (DFID), the Swiss Agency for Development and Cooperation (SDC) and the Government of Bangladesh (GoB), whose objective is to lift 1 million people out of extreme poverty by 2015. The fact that it is a challenge fund that has managing partners, consortium academic partners and NGO partners, all with many moving pieces, made it crucial to have agile decision making tools in order to respond to the needs of their beneficiaries in real-time. Traditional M&E methods of baseline, midterm and endline surveys were deemed insufficient.

The need for real-time measures and iterative decision making created the space for experimentation and innovation. Armed with authorization from their donors, the EEP/Shiree team set out to explore, experiment and create Change Monitoring System 2 (CMS 2). There are a total of 5 CMS tools which include in-depth life histories. They crawled the design space to find and fit solutions that would work in their context (pilot-test-adapt-iterate). Here is a summary of the three pilot phases:

  • Phase 1: Optical reader technology: They first created a simple survey for the NGOs partners to administer and fill out. The surveys were then digitally scanned. They quickly learned that this was too cumbersome a process and it took 2-3 weeks to receive the surveys. The time-lag was too long, they needed something more efficient.
  • Phase 2: Java enabled phone: Since mobile penetration is high, they partnered with mPower to develop a ten minute, monthly census survey on the phone. They equipped up to 20 field officers (the front line personnel who work at the field level with beneficiary households) with simple mobile phone devices that used the Bangla script for the survey. It was meant to be a 6 month pilot but it lasted for 1.5 years by which time they had scaled to 100 devices, with surveys and simple visualization. Convincing NGO partners as well as the visualization and the development of an in-house feedback loop mechanism took much longer than had been anticipated.
  • Phase 3: Android smart phone: The dropping costs of smart phones in the market (android phones were $60-70) created a lucrative option. The smart phone allowed greater flexibility (field staff just update the app on their phone), more functionality and accountability (GPS location of households, photos and voice recording verify that the beneficiaries are being met regularly). mPower also built a dashboard that allowed the comparison and served as a litmus test to identify red flags that required further investigation, ultimately allowing the NGO partners and EEP/Shiree to tailor recovery plans to the beneficiaries needs and changing context.

Slide1

After the trial-and-error and incremental adjustments over three pilot phases, EEP/Shiree deployed a full roll out of the system towards the end of 2012 (3 years later) with the use of smart phones. EEP/Shiree project partners have over 700 smart phones equipped with an Android operating system, internet connectivity and GPS capability, and have been monitoring over 100,000 households every month across Bangladesh as well as accessing information through an online visualization dashboard that is updated in real time.

Here are some of the challenges they faced:

  • Bringing NGO partners on board: The NGO partners were reluctant and viewed the collection of data as an imposition from above. Asking, “why do we have to do it?” and saying “we don’t have time.” They did not understand that the data and the dashboard could serve as a management tool for themselves. NGO partners were then involved in the design of the questions and were included in the process. It took approximately eight months for data collection to cross the 100,000 per month mark which has since been consistently met and represents most households.
  • Infrastructure constraints: Accessing the dashboard from some areas still face connectivity issues. De jure, every field officer is supposed to visit once a month but de facto not all of them do. The sheer scale of the program makes it physically difficult to monitor. While changing the survey questions is easy – you just download the new form on your phone – the back end dashboard change costs are high. Furthermore, by changing questions you lose the ability to compare across time.
  • Effective use of existing data: While the data is used to respond to the needs of the beneficiaries, very little predictive/trend analysis is done. The data is not used to challenge assumptions of what works and to continuously refine their understanding of the dynamics of ascents out of and descents into extreme poverty. This is partly because no one is responsible for this task and so it doesn’t get done.

Complex problems do not have clear solutions. The fact that the donors were flexible and created the space for experimentation and innovation allowing several pilots to be tested (all with good reasoning) is commendable. Throughout the process, EEP/Shiree and mpower co-designed CMS 2 and their continuous cycle of partnership led to a virtuous cycle of action. The leadership on both sides meet every 2-3 months to discuss what is working and what is not, which helps adapt process to technology and technology to process. Together they built a dynamic monitoring tool, proving that this can be done at scale. This is a far cry from the usual case of consultant comes, builds an MIS system and then leaves.

3 thoughts on “EEP/Shiree: Using adaptive programming to monitor change in Bangladesh

  1. Thanks for the comment on CMS2 — I was involved from the start as former CEO of shiree between the period 2010-2014. I don’t disagree with anything written in the piece but, inevitably for something of this length, it rather oversimplifies the process. Briefly going back to the origins — CMS stands for Change Monitoring System – the essential precursor of the CMS2 system (and the other qual and quant weapons in our CMS arsenal) was our analysis that in order to defeat the problem of extreme poverty it is necessary to have an information system capable of capturing both the diversity and dynamism of this phenomenon in order to guide management actions (responses to identified poverty situations) that are sufficiently diverse and timely. I think I once came across this, perhaps in a totally different context, as “the principle of requisite complexity” – in any case the term seems to fit the bill!. In short one needs to be able to do the right thing for the circumstances facing a given household at the right time — not the wrong thing too late! Hence the requirement for virtual census level information available in real time — and the search for a technological, management and organisational system that would meet this requirement that over over a lot of trial, error, learning and adaptation, as outlined in the piece, led to CMS2.

    In fact the system served its purpose rather well for the project since it has been used to guide supplemental support interventions that are bringing the project impact up to and beyond the 1 million people out of extreme poverty target (what I like to term high impact for money). The question of the sustainability of this transition out of extreme poverty given the existence of residual vulnerabilities and the ever present threat of livelihood shocks is of course the moot point unanswerable within a time constrained project intervention – and relating more to fundamental issues of social protection reform and public service provision.

    It is true that, while designed and implemented for a project specific purpose that has been largely fulfilled, the system opens up many intriguing and attractive big data analysis possibilities that have only really been touched on — finding new poverty segments, examining correlations and trends and even predictive analysis – in what circumstances are people likely to descend into extreme poverty?. I spent 4 years during the roll out of the system thinking that clever and creative statisticians (of which i am not one) would grasp this fascinating challenge but in the end was rather disappointed. The questions I kept being asked were of the nature “Why a census not a sample? — obvious to anyone who understands the basis of the system . What about the bias of field workers, rather than trained enumerators, checking their own work? — pretty irrelevant as long as the bias does not change randomly. Where is the control group? – the questioner is stuck in the before/after impact analysis mindset which is not what this is about. — and other questions that served mainly to display the conservatism and resistance to change of the M and E and research community.

    I left shiree in July Last year and it seems likely that the CMS2 system and basic approach has run its course in that specific context (the ideas, ownership and excitement driving the system have dissipated). It remains for other big development interventions to learn from this large scale innovation and pick up the baton — perhaps even making a meaningful contribution to the big task ahead – the global eradication of extreme poverty!

    • Colin thanks for your comment. The fact that you built a management information system that was used to make timely decisions is a rare occurrence and a great example of Doing Development Differently. Clearly, you chose to take the red pill.

      I have also run into the conservatism and resistance you mention in my experience working in M&E, from both evaluators and donors. In fact we wrote our MeE paper specifically for large complex programs like EEP/Shiree where traditional M and E was deemed insufficient and experiential learning (or “e”) needed to be added. Since “e” is an internal decision making tool, having someone in-house whose sole responsibility is to analyze the data, look for trends, tweak the indicators (learn-iterate-adapt) is paramount. This analysis could then be used to demonstrate the power of real-time data to both donors as well as external evaluators – unfortunately it is not as obvious to them as it might be to us. I think CMS2 is an excellent example of “e” in action.

      There are no clear cut solutions. To quote Mike Tyson, “Everybody has a plan until they get punched in the mouth.”

  2. Woop woop for the red pill! Very excited to see a real program getting some attention – as you say often this exciting chat stops at theory and few programs that actually do get coverage. I also worked at EEP/shiree (with Colin) and was similarly pumped by CMS and its role in promoting adaptive management not just as an outcome but as an organizational culture. I wrote up EEP/shiree as a case study within a broader paper on adaptive management and change monitoring (through an extended football analogy, as I am incapable of talking about clever things without dipping into pub chat) which has just been published here: http://www.tandfonline.com/doi/abs/10.1080/09614524.2015.983460?journalCode=cdip20#.VMEl9EesUz0
    We’re also back on the crack trying to set up something similar across 78 partners and $12m in an Ebola response program in Liberia. It’s pretty tough to do in a short time (surprise)…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s