Getting Best Fit in Development Projects is like buying a new suit

written by Matt Andrews

I often hear talk of moving from best practice to best fit in development. When I ask what people mean by this I seldom get the same answer. But the basic idea is that multiple solutions are considered  instead of a one-best-way solution. I like the idea in concept and have written on it as a way of ensuring that we get past the tyranny of one-best-way. But I don’t think having multiple solutions to choose from is enough.

Getting Best fit is not simply about looking at a variety of options and deciding which one works best in the place one is working. This is because it is really hard to know what led to and informed the ideas being considered, and it is hard to know what the contextual realities are in the place where one is doing the fitting. So: One may think that an idea from South Africa will work in Botswana, because there are contextual similarities, but one does not know if the South African idea depends on very localized political or capacity issues that are not present in Botswana.

The only way one truly does best fit is by trying stuff out and learning what works, hopefully why, and then adapting. Try the South African example in Botswana, ask where it is working and why, and then chisel the idea into a shape that fits Botswana better — simultaneously building some new capacities in Botswana to make the fit work.

It is not, therefore, about getting best fit in some conceptual manner, but about ‘fitting’ in practice. Kind of like being a tailor to someone looking for a new suit. How might it work? Let’s think of buying a new suit….

  • Identify the general type of suite that interests you…
  • Choose a variety of potential suites off-the-shelf
  • Try them on…with the tailor advising on their fit…
  • Learn what kinds of cuts and styles work best for your look (you may be surprised and find your assumptions were incorrect)
  • Decide on the style and cut and color you want (and other characteristics)
  • See what you can take off-the-shelf (you may have the entire suite, or a jacket, or a jacket with sleeves that need shortening, etc.)
  • Get the tailor to make alterations you need… as many as required
  • try it on again
  • make more alterations
  • pay that tailor and leave, to impress the world with your best fitted suit!

How do money, ideas and, reforms come together to produce better development outcomes?

Despite the considerable time, money, and effort that aid agencies, international organizations, and NGOs expend producing analysis and advice to inform or influence policymakers in developing countries, there is a remarkable lack of understanding about which of these instruments are most and least effective at spurring and sustaining reforms – and why.

In an attempt to answer these questions, AidData gathered firsthand experiences and insights from 6,750 policymakers and practitioners in 126 low-and middle-income countries. Through their analysis, they identify overall trends, attributes of influential assessments, and both intended and unintended effects of such assessments. These are summarized in their Marketplace of Ideas for Policy Change report.

Here’s an excerpt from the conclusion:

“The Minister’s adviser would probably tell her that, of the wide variety of assessments at her disposal, the policy analysis and advice contained in the assessments of large, global international organizations have proven to be particularly valuable to other reformers. He might also warn the Minister not to rely on the analysis and advice provided by assessment suppliers who lack an in-country presence. These suppliers may not be able to provide the contextual insights needed for the effective design of specific reform features. Additionally, after explaining that external performance assessments are generally associated with more successful reform efforts when they source their data from the governments they assess, the adviser would probably suggest that the Minister pay particularly close attention to assessments that draw upon the data that the government is already producing.

The Minister and her adviser would eventually confront the issue of whether to use country-specific performance assessments or cross-country benchmarking assessments. The adviser would likely explain that each type of assessment has its own advantages and disadvantages. Country-specific assessments contain in-depth diagnostic and advisory content, which may resonate with local stakeholders because they are more attuned to local needs and realities. These assessments tend to be more popular with policymakers across the developing world; however, they can also leave host government officials with a false sense of confidence in the analysis and advice they provide, thus bypassing critical internal processes of introspection, deliberation, and iterative problem solving.

Cross-country benchmarking assessments, on the other hand, are less attractive at first blush because they lack nuance and context specificity. However, when decision-makers confront complex issues of corruption, informality, and institutional dysfunction, they can provide greater policy flexibility and maneuverability than country-specific assessments, enabling the government to experiment with different reform strategies and iteratively adapt in pursuit of better, de facto outcomes.

Noting the Minister’s interest in Greek mythology, the adviser might remind her of the many perils Odysseus faced on his journey home from Troy and counsel the Minister to avoid steering her country into the “capability trap” that has ensnared other well-intentioned, would-be reformers. He would encourage her to focus on simple, solvable problems before undertaking more complex reforms. To underscore his point, he might even slip a copy of a short article by Lant Pritchett, Michael Woolcock, and Matt Andrews into the Minister’s read file, which describes the capability trap as a dynamic that enables [policymakers] to document instances of apparent reform and thus assure a continued flow of development resources to their country or sector, despite the fact that the reforms themselves may be generating few actual improvements in performance.”

Read the full report for more.

PDIA in Indonesia: The new frontline service delivery policy

How does the government of Indonesia make its presence felt by all 250 million citizens across the sprawling archipelago?

While decentralization provides district governments the authority to address local needs, effective execution of these functions relies heavily on the capacity of the local governments to analyze service gaps and drive more coordinated efforts to address them, as well as the capacity of communities to voice their needs, provide feedback and be part of the solution.

To address this, the Medium-Term National Development Plan 2015-19 includes a new policy to improve basic services for the poor and vulnerable. The approach focuses on enhancing interactions at the front line between government, service providers and citizens, as well as their collective ability to diagnose and solve service delivery bottlenecks at the community level.

A multidisciplinary team from 8 sectors conducted a series of field visits between September 2014 and January 2015, with a mission to identify local innovation and best practices for improving services for the poor and vulnerable. Using a PDIA approach, they engaged with a broad set of stakeholders, had enriching interactions, and were able to view the same problem from different angles. Vignettes from their field visits are captured in Catalyzing local innovation to improve services for Indonesia’s Poor. Some of the lessons they learned include:

  • Focus on fostering experimentation and learning at the local level , rather than fixating on sluggish reforms at the central level.
  • One size does not fit all. Instead of prescribing a set menu of interventions to improve service delivery, the approach should be to create a supportive policy and institutional environment that fosters innovation.
  • The locus of innovation also matters. The closer the innovation occurs to the community, the more potential for catalytic change.
  • Diffusion can happen organically but knowledge sharing and creating communities of practice can help the expansion of innovative ideas.

Watch Anna Winoto from the National Development Planning Ministry in Indonesia discuss the frontline service delivery policy at the Doing Development Differently Philippines workshop. The challenge in this work is to facilitate district governments to innovate, which requires multi-disciplinary district teams who can solve problems together, access flexible financing, leadership and change management, and diagnostic tools to allow for rapid feedback.

Getting real about development; It is hard

Written by Matt Andrews

I’m reminded so regularly that development is about change. If it’s done well it is about change that sticks, and even more about countries becoming adaptive (able to change continuously at the right pace and in the right way).

This requires learning and building a specific type of DNA in people, organizations, and countries. And this learning is hard. Often because learning is perceived as failure, and failure is feared.

The truth is that most key development breakthroughs happen out of the lessons of things gone wrong, but in the moment of going wrong it is hard to see how valuable the failure is; it seems like all is falling apart and critics come out of every window and door they can.

Keeping one’s head in these moments is crucial, and is required to let people see failure as learning and to see that learning itself is the key to success.

I wonder how often public policy schools teach students about these moments and how to manage yourself in the face of the turmoil these moments involve. I think this may be one of the most important lessons to learn if you want to work in development and not spend all the time writing safe reports no one uses or consult from a distance, or do stuff without bringing local folks along to learn how to do it themselves.

Making the case for case studies in development practice

Written by Michael Woolcock

The frequency and sophistication with which case studies are deployed by social scientists has greatly expanded in recent years. The goal now is not merely to document or describe, but to diagnose, explain, interpret, and inform a basis for action. Professional schools across the disciplines – from medicine and engineering to business and public policy – now routinely use ‘the case method’ not only to teach but to generate practical knowledge.

World Bank staff have been active contributors to and beneficiaries of these trends, especially as the role of institutions and governance has gained prominence in efforts to enhance development effectiveness. When complex places, processes, people and projects come together, they inherently yield a diverse range of outcomes. Mapping this variation with survey data and then explaining how it varies using targeted case studies can yield uniquely instructive insights for development policy and practice.

This twin approach forms the empirical foundations of a forthcoming report on public service delivery in the Middle East and North Africa region, and will be a central component of the new Global Delivery Initiative (which will focus on explaining and improving the quality of implementation systems).

Rather than seeking universal ‘best practice’ responses as a basis for policy advice, analysts use case studies to learn from ‘natural’ (or sometime overtly experimental) sources of intra-country variation. Everyone can agree in the abstract that context and high quality institutions matter for development, and that one size doesn’t fit all, but these truisms aren’t much help when trying to provide specific advice in response to a specific problem in a specific place, such as stemming urban violence in Rio de Janeiro or promoting more effective antenatal services for women in Cairo. Case studies are emerging as a useful strategy for eliciting not just uplifting success stories, but as unique data collection tools that can guide policy and practice by helping domestic actors adjust in real time as they seek solutions to emerging problems.

The recently completed ‘Institutions Taking Root’ (ITR) study is an exemplary instance of this new approach to discerning what works, and how success happens, in development. In places that initially seemed unlikely venues for successful development, the ITR team began from the premise that someone, somewhere, somehow had probably figured out how to make real gains where others had not.

In contexts ranging from rural electrification in Lao PDR to basic and secondary education in The Gambia, researchers sought out islands of success in seas of seeming failure, defining success as positive outcomes which were measurable, legitimate and durable (i.e., robust despite exogenous shocks or changes in political leadership). An institution was considered to have achieved results if it exhibited sustained, measurable improvements in key agency outputs and outcomes, doing so across prevailing social cleavages (e.g., rural-urban, between ethnic groups, etc.).

Having identified such outcomes, researchers sought to test hypotheses examining the institution’s interaction with its context, and the organization’s inner workings. This procedure was followed in each of the case studies to guide and standardize data collection, thereby making systematic comparison of the cases possible. Interviews and focus groups were conducted both outside-in (focusing on external stakeholders and constituents first before moving onto officials) and bottom-up (beginning with frontline service providers before moving to agency management and political leaders) to ensure that typically less dominant viewpoints were represented.

One memorable example from the ITR describes the ministry of public works in Lao. Most such ministries, especially in poor countries, have strong imperatives to focus on building new roads rather than maintaining existing ones: new roads look impressive, provide officials with elaborate opening ceremonies and flattering media coverage, and most have an immediately large economic impact.

Fixing old roads, by contrast, is boring, time-consuming and devoid of political pay-off, no matter how necessary. But in Lao, the minister of public works had managed to shape a broad public consensus on the moral, economic and political importance of keeping roads in good repair. Deploying a memorable slogan, he travelled the country reminding villagers and elites alike that “Making children is easy and fun; raising children is hard and costly!” In this way, he argued, roads were like people: they should be cared for if they are to make a lasting contribution to Lao society. No matter their level of education, everyone in Lao understood the minister’s analogy; his unique but explicit leadership on this issue enabled him to secure political credit for an important issue that is usually overlooked.

Is this then “the answer” for raising the profile of road maintenance in developing countries everywhere? Readers will doubtless have their own views, but I think it is an answer, demonstrating how a particular team of people found a way to solve a tough but widespread problem. As such, it can hopefully be a source of inspiration and ideas for others elsewhere seeking to find their own solutions.

(Read more posts about this report here)

This post previously appeared on the World Bank blog.

Rising to the challenge: Supporting PDIA in Nigeria

The importance of thinking and working politically in programs concerned with governance and institutional reform is widely recognized – but shifting from thinking politically to working politically is proving a major challenge. The State Accountability and Voice Initiative (SAVI), a governance reform program in Nigeria, supports local partners and stakeholders not through the usual accountable grants, but through mentoring, capacity building, relationship brokering and seed funding. This approach is making headway in enabling local partners and stakeholders to think and work in politically savvy ways around locally agreed problems – and its success is demonstrated in tangible examples of more responsive state governance in all the states where the program works. Read more here.

The SAVI program in Nigeria, a DFID-funded Empowerment and Accountability program operating in ten states, is included as a case study, among others, in ODI’s new report Adapting Development.

How do you keep 100 students awake on a Friday afternoon? Fast feedback and iterative adaptation seem to work

Guest blog written by Duncan Green

sleeping students
There’s a character in a Moliere play who is surprised and delighted to learn that he has been speaking prose all his life without knowing it. I thought of him a couple of weeks into my new role as a part-time Professor in Practice in LSE’s International Development Department, when I realized I had been using ‘iterative adaptation’ to work out how best to keep 100+ Masters students awake and engaged for two hours last thing on a Friday afternoon.

The module is called ‘Research Themes in International Development’, a pretty vague topic which appears to be designed to allow lecturers to bang on about their research interests. I kicked off with a discussion on the nature and dilemmas of international NGOs, as I’m just writing a paper on that, then moved on to introduce some of the big themes of a forthcoming book on ‘How Change Happens’.

As an NGO type, I am committed to all things participatory, so ended lecture one getting the students to vote for their preferred guest speakers (no I’m not publishing the results). In order to find out how the lectures were going, I also introduced a weekly feedback form on the LSE intranet (thanks to LSE’s Lucy Pickles for sorting that out), and asked students to fill it in at the end of the session. The only incentive I could think of was to promise a satirical video (example below) if they stayed long enough to fill it in before rushing out the door – it seemed to work. The students were asked to rank presentation and content separately on a scale from ‘awful’ to ‘brilliant’, and then offer suggestions for improvements.

It’s anonymous, and not rigorous of course (self-selecting sample, disgruntled students likely to drop out in subsequent weeks etc), but it has been incredibly useful, especially the open-ended box for suggestions, which have been crammed full with useful content. The first week’s comments broadly asked for more participation, so week two included lots of breakout group discussions. The feedback then said, ‘we like the discussion, but all the unpacking after the groups where you ask what people were talking about eats up time, and anyway, we couldn’t hear half of it’, and asked for more rigour, so week three had more references to the literature, and 3 short discussion groups with minimal feedback – it felt odd, but seemed to work.

At this point, the penny dropped – I was putting into practice some of the messages of my week two lecture on how to work in complex systems, namely fast feedback loops that enable you to experiment, fail, tweak and try again in a repeat cycle until something reasonably successful emerges through trial and error. One example of failing faster – I tried out LSE’s online polling system, but found it was too slow (getting everyone to go online on their mobiles and then vote on a series of multiple choice questions) but also not as energising as getting people to vote analogue style (i.e. raising their hands). The important thing is getting weekly feedback and responding to it, rather than waiting til the end of term (by which time it will be too late).

LSE content weeks 1-3The form is not the only feedback system of course. As any teacher knows, talking to a roomful of people inevitably involves pretty intense realtime feedback too – you feel the energy rise and fall, see people glazing over or getting interested etc. What’s interesting is being able to triangulate between what I thought was happening in the room/students’ heads, and what they subsequently said. Broad agreement, but the feedback suggested their engagement was reassuringly consistent (see bar chart on content), whereas my perceptions seem to amplify it all into big peaks and troughs – what I thought was a disastrous second half of lecture two appears to have just been a bit below par for a small number of students.

The feedback also helps crystallize half-formed thoughts of your own. For example, several complained about the disruption of students leaving in the middle of the lecture, something I also had found rather unnerving. So I suggested that if people did need to leave early (it’s last thing on Friday after all), they should do so during the group discussions – much better.

What’s been striking is the look of mild alarm in the eyes of some of my LSE faculty colleagues, who warned against too much populist kowtowing to student demands. That’s certainly not how it’s felt so far. Here’s a typical comment ‘I think that this lecture on the role of the state tried to take on too much. This is an area that we have discussed extensively. I think it would have been more useful to focus on a particular aspect, perhaps failed and conflict-affected states since you argue that those are the future of aid’. Not a plea for more funny videos (though there have been a few of those), but a reminder to check for duplication with other modules, and a useful guide to improving next year’s lectures.

What is also emerging, again in a pleasingly unintended way, is a sense that we are designing this course together and the students seem to appreciate that  (I refuse to use the awful word co-create. Doh.) Matt Andrews calls this process ‘Problem Driven Iterative Adaptation’ – I would love to hear from other lecturers on more ways to use fast feedback to sharpen up teaching practices.

You can also find this blog here.