How do money, ideas and, reforms come together to produce better development outcomes?

Despite the considerable time, money, and effort that aid agencies, international organizations, and NGOs expend producing analysis and advice to inform or influence policymakers in developing countries, there is a remarkable lack of understanding about which of these instruments are most and least effective at spurring and sustaining reforms – and why.

In an attempt to answer these questions, AidData gathered firsthand experiences and insights from 6,750 policymakers and practitioners in 126 low-and middle-income countries. Through their analysis, they identify overall trends, attributes of influential assessments, and both intended and unintended effects of such assessments. These are summarized in their Marketplace of Ideas for Policy Change report.

Here’s an excerpt from the conclusion:

“The Minister’s adviser would probably tell her that, of the wide variety of assessments at her disposal, the policy analysis and advice contained in the assessments of large, global international organizations have proven to be particularly valuable to other reformers. He might also warn the Minister not to rely on the analysis and advice provided by assessment suppliers who lack an in-country presence. These suppliers may not be able to provide the contextual insights needed for the effective design of specific reform features. Additionally, after explaining that external performance assessments are generally associated with more successful reform efforts when they source their data from the governments they assess, the adviser would probably suggest that the Minister pay particularly close attention to assessments that draw upon the data that the government is already producing.

The Minister and her adviser would eventually confront the issue of whether to use country-specific performance assessments or cross-country benchmarking assessments. The adviser would likely explain that each type of assessment has its own advantages and disadvantages. Country-specific assessments contain in-depth diagnostic and advisory content, which may resonate with local stakeholders because they are more attuned to local needs and realities. These assessments tend to be more popular with policymakers across the developing world; however, they can also leave host government officials with a false sense of confidence in the analysis and advice they provide, thus bypassing critical internal processes of introspection, deliberation, and iterative problem solving.

Cross-country benchmarking assessments, on the other hand, are less attractive at first blush because they lack nuance and context specificity. However, when decision-makers confront complex issues of corruption, informality, and institutional dysfunction, they can provide greater policy flexibility and maneuverability than country-specific assessments, enabling the government to experiment with different reform strategies and iteratively adapt in pursuit of better, de facto outcomes.

Noting the Minister’s interest in Greek mythology, the adviser might remind her of the many perils Odysseus faced on his journey home from Troy and counsel the Minister to avoid steering her country into the “capability trap” that has ensnared other well-intentioned, would-be reformers. He would encourage her to focus on simple, solvable problems before undertaking more complex reforms. To underscore his point, he might even slip a copy of a short article by Lant Pritchett, Michael Woolcock, and Matt Andrews into the Minister’s read file, which describes the capability trap as a dynamic that enables [policymakers] to document instances of apparent reform and thus assure a continued flow of development resources to their country or sector, despite the fact that the reforms themselves may be generating few actual improvements in performance.”

Read the full report for more.

PDIA in Indonesia: The new frontline service delivery policy

How does the government of Indonesia make its presence felt by all 250 million citizens across the sprawling archipelago?

While decentralization provides district governments the authority to address local needs, effective execution of these functions relies heavily on the capacity of the local governments to analyze service gaps and drive more coordinated efforts to address them, as well as the capacity of communities to voice their needs, provide feedback and be part of the solution.

To address this, the Medium-Term National Development Plan 2015-19 includes a new policy to improve basic services for the poor and vulnerable. The approach focuses on enhancing interactions at the front line between government, service providers and citizens, as well as their collective ability to diagnose and solve service delivery bottlenecks at the community level.

A multidisciplinary team from 8 sectors conducted a series of field visits between September 2014 and January 2015, with a mission to identify local innovation and best practices for improving services for the poor and vulnerable. Using a PDIA approach, they engaged with a broad set of stakeholders, had enriching interactions, and were able to view the same problem from different angles. Vignettes from their field visits are captured in Catalyzing local innovation to improve services for Indonesia’s Poor. Some of the lessons they learned include:

  • Focus on fostering experimentation and learning at the local level , rather than fixating on sluggish reforms at the central level.
  • One size does not fit all. Instead of prescribing a set menu of interventions to improve service delivery, the approach should be to create a supportive policy and institutional environment that fosters innovation.
  • The locus of innovation also matters. The closer the innovation occurs to the community, the more potential for catalytic change.
  • Diffusion can happen organically but knowledge sharing and creating communities of practice can help the expansion of innovative ideas.

Watch Anna Winoto from the National Development Planning Ministry in Indonesia discuss the frontline service delivery policy at the Doing Development Differently Philippines workshop. The challenge in this work is to facilitate district governments to innovate, which requires multi-disciplinary district teams who can solve problems together, access flexible financing, leadership and change management, and diagnostic tools to allow for rapid feedback.

Getting real about development; It is hard

Written by Matt Andrews

I’m reminded so regularly that development is about change. If it’s done well it is about change that sticks, and even more about countries becoming adaptive (able to change continuously at the right pace and in the right way).

This requires learning and building a specific type of DNA in people, organizations, and countries. And this learning is hard. Often because learning is perceived as failure, and failure is feared.

The truth is that most key development breakthroughs happen out of the lessons of things gone wrong, but in the moment of going wrong it is hard to see how valuable the failure is; it seems like all is falling apart and critics come out of every window and door they can.

Keeping one’s head in these moments is crucial, and is required to let people see failure as learning and to see that learning itself is the key to success.

I wonder how often public policy schools teach students about these moments and how to manage yourself in the face of the turmoil these moments involve. I think this may be one of the most important lessons to learn if you want to work in development and not spend all the time writing safe reports no one uses or consult from a distance, or do stuff without bringing local folks along to learn how to do it themselves.

Making the case for case studies in development practice

Written by Michael Woolcock

The frequency and sophistication with which case studies are deployed by social scientists has greatly expanded in recent years. The goal now is not merely to document or describe, but to diagnose, explain, interpret, and inform a basis for action. Professional schools across the disciplines – from medicine and engineering to business and public policy – now routinely use ‘the case method’ not only to teach but to generate practical knowledge.

World Bank staff have been active contributors to and beneficiaries of these trends, especially as the role of institutions and governance has gained prominence in efforts to enhance development effectiveness. When complex places, processes, people and projects come together, they inherently yield a diverse range of outcomes. Mapping this variation with survey data and then explaining how it varies using targeted case studies can yield uniquely instructive insights for development policy and practice.

This twin approach forms the empirical foundations of a forthcoming report on public service delivery in the Middle East and North Africa region, and will be a central component of the new Global Delivery Initiative (which will focus on explaining and improving the quality of implementation systems).

Rather than seeking universal ‘best practice’ responses as a basis for policy advice, analysts use case studies to learn from ‘natural’ (or sometime overtly experimental) sources of intra-country variation. Everyone can agree in the abstract that context and high quality institutions matter for development, and that one size doesn’t fit all, but these truisms aren’t much help when trying to provide specific advice in response to a specific problem in a specific place, such as stemming urban violence in Rio de Janeiro or promoting more effective antenatal services for women in Cairo. Case studies are emerging as a useful strategy for eliciting not just uplifting success stories, but as unique data collection tools that can guide policy and practice by helping domestic actors adjust in real time as they seek solutions to emerging problems.

The recently completed ‘Institutions Taking Root’ (ITR) study is an exemplary instance of this new approach to discerning what works, and how success happens, in development. In places that initially seemed unlikely venues for successful development, the ITR team began from the premise that someone, somewhere, somehow had probably figured out how to make real gains where others had not.

In contexts ranging from rural electrification in Lao PDR to basic and secondary education in The Gambia, researchers sought out islands of success in seas of seeming failure, defining success as positive outcomes which were measurable, legitimate and durable (i.e., robust despite exogenous shocks or changes in political leadership). An institution was considered to have achieved results if it exhibited sustained, measurable improvements in key agency outputs and outcomes, doing so across prevailing social cleavages (e.g., rural-urban, between ethnic groups, etc.).

Having identified such outcomes, researchers sought to test hypotheses examining the institution’s interaction with its context, and the organization’s inner workings. This procedure was followed in each of the case studies to guide and standardize data collection, thereby making systematic comparison of the cases possible. Interviews and focus groups were conducted both outside-in (focusing on external stakeholders and constituents first before moving onto officials) and bottom-up (beginning with frontline service providers before moving to agency management and political leaders) to ensure that typically less dominant viewpoints were represented.

One memorable example from the ITR describes the ministry of public works in Lao. Most such ministries, especially in poor countries, have strong imperatives to focus on building new roads rather than maintaining existing ones: new roads look impressive, provide officials with elaborate opening ceremonies and flattering media coverage, and most have an immediately large economic impact.

Fixing old roads, by contrast, is boring, time-consuming and devoid of political pay-off, no matter how necessary. But in Lao, the minister of public works had managed to shape a broad public consensus on the moral, economic and political importance of keeping roads in good repair. Deploying a memorable slogan, he travelled the country reminding villagers and elites alike that “Making children is easy and fun; raising children is hard and costly!” In this way, he argued, roads were like people: they should be cared for if they are to make a lasting contribution to Lao society. No matter their level of education, everyone in Lao understood the minister’s analogy; his unique but explicit leadership on this issue enabled him to secure political credit for an important issue that is usually overlooked.

Is this then “the answer” for raising the profile of road maintenance in developing countries everywhere? Readers will doubtless have their own views, but I think it is an answer, demonstrating how a particular team of people found a way to solve a tough but widespread problem. As such, it can hopefully be a source of inspiration and ideas for others elsewhere seeking to find their own solutions.

(Read more posts about this report here)

This post previously appeared on the World Bank blog.

Rising to the challenge: Supporting PDIA in Nigeria

The importance of thinking and working politically in programs concerned with governance and institutional reform is widely recognized – but shifting from thinking politically to working politically is proving a major challenge. The State Accountability and Voice Initiative (SAVI), a governance reform program in Nigeria, supports local partners and stakeholders not through the usual accountable grants, but through mentoring, capacity building, relationship brokering and seed funding. This approach is making headway in enabling local partners and stakeholders to think and work in politically savvy ways around locally agreed problems – and its success is demonstrated in tangible examples of more responsive state governance in all the states where the program works. Read more here.

The SAVI program in Nigeria, a DFID-funded Empowerment and Accountability program operating in ten states, is included as a case study, among others, in ODI’s new report Adapting Development.

How do you keep 100 students awake on a Friday afternoon? Fast feedback and iterative adaptation seem to work

Guest blog written by Duncan Green

sleeping students
There’s a character in a Moliere play who is surprised and delighted to learn that he has been speaking prose all his life without knowing it. I thought of him a couple of weeks into my new role as a part-time Professor in Practice in LSE’s International Development Department, when I realized I had been using ‘iterative adaptation’ to work out how best to keep 100+ Masters students awake and engaged for two hours last thing on a Friday afternoon.

The module is called ‘Research Themes in International Development’, a pretty vague topic which appears to be designed to allow lecturers to bang on about their research interests. I kicked off with a discussion on the nature and dilemmas of international NGOs, as I’m just writing a paper on that, then moved on to introduce some of the big themes of a forthcoming book on ‘How Change Happens’.

As an NGO type, I am committed to all things participatory, so ended lecture one getting the students to vote for their preferred guest speakers (no I’m not publishing the results). In order to find out how the lectures were going, I also introduced a weekly feedback form on the LSE intranet (thanks to LSE’s Lucy Pickles for sorting that out), and asked students to fill it in at the end of the session. The only incentive I could think of was to promise a satirical video (example below) if they stayed long enough to fill it in before rushing out the door – it seemed to work. The students were asked to rank presentation and content separately on a scale from ‘awful’ to ‘brilliant’, and then offer suggestions for improvements.

It’s anonymous, and not rigorous of course (self-selecting sample, disgruntled students likely to drop out in subsequent weeks etc), but it has been incredibly useful, especially the open-ended box for suggestions, which have been crammed full with useful content. The first week’s comments broadly asked for more participation, so week two included lots of breakout group discussions. The feedback then said, ‘we like the discussion, but all the unpacking after the groups where you ask what people were talking about eats up time, and anyway, we couldn’t hear half of it’, and asked for more rigour, so week three had more references to the literature, and 3 short discussion groups with minimal feedback – it felt odd, but seemed to work.

At this point, the penny dropped – I was putting into practice some of the messages of my week two lecture on how to work in complex systems, namely fast feedback loops that enable you to experiment, fail, tweak and try again in a repeat cycle until something reasonably successful emerges through trial and error. One example of failing faster – I tried out LSE’s online polling system, but found it was too slow (getting everyone to go online on their mobiles and then vote on a series of multiple choice questions) but also not as energising as getting people to vote analogue style (i.e. raising their hands). The important thing is getting weekly feedback and responding to it, rather than waiting til the end of term (by which time it will be too late).

LSE content weeks 1-3The form is not the only feedback system of course. As any teacher knows, talking to a roomful of people inevitably involves pretty intense realtime feedback too – you feel the energy rise and fall, see people glazing over or getting interested etc. What’s interesting is being able to triangulate between what I thought was happening in the room/students’ heads, and what they subsequently said. Broad agreement, but the feedback suggested their engagement was reassuringly consistent (see bar chart on content), whereas my perceptions seem to amplify it all into big peaks and troughs – what I thought was a disastrous second half of lecture two appears to have just been a bit below par for a small number of students.

The feedback also helps crystallize half-formed thoughts of your own. For example, several complained about the disruption of students leaving in the middle of the lecture, something I also had found rather unnerving. So I suggested that if people did need to leave early (it’s last thing on Friday after all), they should do so during the group discussions – much better.

What’s been striking is the look of mild alarm in the eyes of some of my LSE faculty colleagues, who warned against too much populist kowtowing to student demands. That’s certainly not how it’s felt so far. Here’s a typical comment ‘I think that this lecture on the role of the state tried to take on too much. This is an area that we have discussed extensively. I think it would have been more useful to focus on a particular aspect, perhaps failed and conflict-affected states since you argue that those are the future of aid’. Not a plea for more funny videos (though there have been a few of those), but a reminder to check for duplication with other modules, and a useful guide to improving next year’s lectures.

What is also emerging, again in a pleasingly unintended way, is a sense that we are designing this course together and the students seem to appreciate that  (I refuse to use the awful word co-create. Doh.) Matt Andrews calls this process ‘Problem Driven Iterative Adaptation’ – I would love to hear from other lecturers on more ways to use fast feedback to sharpen up teaching practices.

You can also find this blog here.

EEP/Shiree: Using adaptive programming to monitor change in Bangladesh

written by Salimah Samji

How do you effectively monitor an 8 year, £83.5 million (around USD$135 million) challenge fund that partners with NGOs to improve the livelihood of 1 million beneficiaries? A daunting task indeed.

The Economic Empowerment of the Poorest (EEP/Shiree) program is a partnership between the UK Department for International Development (DFID), the Swiss Agency for Development and Cooperation (SDC) and the Government of Bangladesh (GoB), whose objective is to lift 1 million people out of extreme poverty by 2015. The fact that it is a challenge fund that has managing partners, consortium academic partners and NGO partners, all with many moving pieces, made it crucial to have agile decision making tools in order to respond to the needs of their beneficiaries in real-time. Traditional M&E methods of baseline, midterm and endline surveys were deemed insufficient.

The need for real-time measures and iterative decision making created the space for experimentation and innovation. Armed with authorization from their donors, the EEP/Shiree team set out to explore, experiment and create Change Monitoring System 2 (CMS 2). There are a total of 5 CMS tools which include in-depth life histories. They crawled the design space to find and fit solutions that would work in their context (pilot-test-adapt-iterate). Here is a summary of the three pilot phases:

  • Phase 1: Optical reader technology: They first created a simple survey for the NGOs partners to administer and fill out. The surveys were then digitally scanned. They quickly learned that this was too cumbersome a process and it took 2-3 weeks to receive the surveys. The time-lag was too long, they needed something more efficient.
  • Phase 2: Java enabled phone: Since mobile penetration is high, they partnered with mPower to develop a ten minute, monthly census survey on the phone. They equipped up to 20 field officers (the front line personnel who work at the field level with beneficiary households) with simple mobile phone devices that used the Bangla script for the survey. It was meant to be a 6 month pilot but it lasted for 1.5 years by which time they had scaled to 100 devices, with surveys and simple visualization. Convincing NGO partners as well as the visualization and the development of an in-house feedback loop mechanism took much longer than had been anticipated.
  • Phase 3: Android smart phone: The dropping costs of smart phones in the market (android phones were $60-70) created a lucrative option. The smart phone allowed greater flexibility (field staff just update the app on their phone), more functionality and accountability (GPS location of households, photos and voice recording verify that the beneficiaries are being met regularly). mPower also built a dashboard that allowed the comparison and served as a litmus test to identify red flags that required further investigation, ultimately allowing the NGO partners and EEP/Shiree to tailor recovery plans to the beneficiaries needs and changing context.


After the trial-and-error and incremental adjustments over three pilot phases, EEP/Shiree deployed a full roll out of the system towards the end of 2012 (3 years later) with the use of smart phones. EEP/Shiree project partners have over 700 smart phones equipped with an Android operating system, internet connectivity and GPS capability, and have been monitoring over 100,000 households every month across Bangladesh as well as accessing information through an online visualization dashboard that is updated in real time.

Here are some of the challenges they faced:

  • Bringing NGO partners on board: The NGO partners were reluctant and viewed the collection of data as an imposition from above. Asking, “why do we have to do it?” and saying “we don’t have time.” They did not understand that the data and the dashboard could serve as a management tool for themselves. NGO partners were then involved in the design of the questions and were included in the process. It took approximately eight months for data collection to cross the 100,000 per month mark which has since been consistently met and represents most households.
  • Infrastructure constraints: Accessing the dashboard from some areas still face connectivity issues. De jure, every field officer is supposed to visit once a month but de facto not all of them do. The sheer scale of the program makes it physically difficult to monitor. While changing the survey questions is easy – you just download the new form on your phone – the back end dashboard change costs are high. Furthermore, by changing questions you lose the ability to compare across time.
  • Effective use of existing data: While the data is used to respond to the needs of the beneficiaries, very little predictive/trend analysis is done. The data is not used to challenge assumptions of what works and to continuously refine their understanding of the dynamics of ascents out of and descents into extreme poverty. This is partly because no one is responsible for this task and so it doesn’t get done.

Complex problems do not have clear solutions. The fact that the donors were flexible and created the space for experimentation and innovation allowing several pilots to be tested (all with good reasoning) is commendable. Throughout the process, EEP/Shiree and mpower co-designed CMS 2 and their continuous cycle of partnership led to a virtuous cycle of action. The leadership on both sides meet every 2-3 months to discuss what is working and what is not, which helps adapt process to technology and technology to process. Together they built a dynamic monitoring tool, proving that this can be done at scale. This is a far cry from the usual case of consultant comes, builds an MIS system and then leaves.