Getting real about development; It is hard

Written by Matt Andrews

I’m reminded so regularly that development is about change. If it’s done well it is about change that sticks, and even more about countries becoming adaptive (able to change continuously at the right pace and in the right way).

This requires learning and building a specific type of DNA in people, organizations, and countries. And this learning is hard. Often because learning is perceived as failure, and failure is feared.

The truth is that most key development breakthroughs happen out of the lessons of things gone wrong, but in the moment of going wrong it is hard to see how valuable the failure is; it seems like all is falling apart and critics come out of every window and door they can.

Keeping one’s head in these moments is crucial, and is required to let people see failure as learning and to see that learning itself is the key to success.

I wonder how often public policy schools teach students about these moments and how to manage yourself in the face of the turmoil these moments involve. I think this may be one of the most important lessons to learn if you want to work in development and not spend all the time writing safe reports no one uses or consult from a distance, or do stuff without bringing local folks along to learn how to do it themselves.

Making the case for case studies in development practice

Written by Michael Woolcock

The frequency and sophistication with which case studies are deployed by social scientists has greatly expanded in recent years. The goal now is not merely to document or describe, but to diagnose, explain, interpret, and inform a basis for action. Professional schools across the disciplines – from medicine and engineering to business and public policy – now routinely use ‘the case method’ not only to teach but to generate practical knowledge.

World Bank staff have been active contributors to and beneficiaries of these trends, especially as the role of institutions and governance has gained prominence in efforts to enhance development effectiveness. When complex places, processes, people and projects come together, they inherently yield a diverse range of outcomes. Mapping this variation with survey data and then explaining how it varies using targeted case studies can yield uniquely instructive insights for development policy and practice.

This twin approach forms the empirical foundations of a forthcoming report on public service delivery in the Middle East and North Africa region, and will be a central component of the new Global Delivery Initiative (which will focus on explaining and improving the quality of implementation systems).

Rather than seeking universal ‘best practice’ responses as a basis for policy advice, analysts use case studies to learn from ‘natural’ (or sometime overtly experimental) sources of intra-country variation. Everyone can agree in the abstract that context and high quality institutions matter for development, and that one size doesn’t fit all, but these truisms aren’t much help when trying to provide specific advice in response to a specific problem in a specific place, such as stemming urban violence in Rio de Janeiro or promoting more effective antenatal services for women in Cairo. Case studies are emerging as a useful strategy for eliciting not just uplifting success stories, but as unique data collection tools that can guide policy and practice by helping domestic actors adjust in real time as they seek solutions to emerging problems.

The recently completed ‘Institutions Taking Root’ (ITR) study is an exemplary instance of this new approach to discerning what works, and how success happens, in development. In places that initially seemed unlikely venues for successful development, the ITR team began from the premise that someone, somewhere, somehow had probably figured out how to make real gains where others had not.

In contexts ranging from rural electrification in Lao PDR to basic and secondary education in The Gambia, researchers sought out islands of success in seas of seeming failure, defining success as positive outcomes which were measurable, legitimate and durable (i.e., robust despite exogenous shocks or changes in political leadership). An institution was considered to have achieved results if it exhibited sustained, measurable improvements in key agency outputs and outcomes, doing so across prevailing social cleavages (e.g., rural-urban, between ethnic groups, etc.).

Having identified such outcomes, researchers sought to test hypotheses examining the institution’s interaction with its context, and the organization’s inner workings. This procedure was followed in each of the case studies to guide and standardize data collection, thereby making systematic comparison of the cases possible. Interviews and focus groups were conducted both outside-in (focusing on external stakeholders and constituents first before moving onto officials) and bottom-up (beginning with frontline service providers before moving to agency management and political leaders) to ensure that typically less dominant viewpoints were represented.

One memorable example from the ITR describes the ministry of public works in Lao. Most such ministries, especially in poor countries, have strong imperatives to focus on building new roads rather than maintaining existing ones: new roads look impressive, provide officials with elaborate opening ceremonies and flattering media coverage, and most have an immediately large economic impact.

Fixing old roads, by contrast, is boring, time-consuming and devoid of political pay-off, no matter how necessary. But in Lao, the minister of public works had managed to shape a broad public consensus on the moral, economic and political importance of keeping roads in good repair. Deploying a memorable slogan, he travelled the country reminding villagers and elites alike that “Making children is easy and fun; raising children is hard and costly!” In this way, he argued, roads were like people: they should be cared for if they are to make a lasting contribution to Lao society. No matter their level of education, everyone in Lao understood the minister’s analogy; his unique but explicit leadership on this issue enabled him to secure political credit for an important issue that is usually overlooked.

Is this then “the answer” for raising the profile of road maintenance in developing countries everywhere? Readers will doubtless have their own views, but I think it is an answer, demonstrating how a particular team of people found a way to solve a tough but widespread problem. As such, it can hopefully be a source of inspiration and ideas for others elsewhere seeking to find their own solutions.

(Read more posts about this report here)

This post previously appeared on the World Bank blog.

Rising to the challenge: Supporting PDIA in Nigeria

The importance of thinking and working politically in programs concerned with governance and institutional reform is widely recognized – but shifting from thinking politically to working politically is proving a major challenge. The State Accountability and Voice Initiative (SAVI), a governance reform program in Nigeria, supports local partners and stakeholders not through the usual accountable grants, but through mentoring, capacity building, relationship brokering and seed funding. This approach is making headway in enabling local partners and stakeholders to think and work in politically savvy ways around locally agreed problems – and its success is demonstrated in tangible examples of more responsive state governance in all the states where the program works. Read more here.

The SAVI program in Nigeria, a DFID-funded Empowerment and Accountability program operating in ten states, is included as a case study, among others, in ODI’s new report Adapting Development.

How do you keep 100 students awake on a Friday afternoon? Fast feedback and iterative adaptation seem to work

Guest blog written by Duncan Green


sleeping students
There’s a character in a Moliere play who is surprised and delighted to learn that he has been speaking prose all his life without knowing it. I thought of him a couple of weeks into my new role as a part-time Professor in Practice in LSE’s International Development Department, when I realized I had been using ‘iterative adaptation’ to work out how best to keep 100+ Masters students awake and engaged for two hours last thing on a Friday afternoon.

The module is called ‘Research Themes in International Development’, a pretty vague topic which appears to be designed to allow lecturers to bang on about their research interests. I kicked off with a discussion on the nature and dilemmas of international NGOs, as I’m just writing a paper on that, then moved on to introduce some of the big themes of a forthcoming book on ‘How Change Happens’.

As an NGO type, I am committed to all things participatory, so ended lecture one getting the students to vote for their preferred guest speakers (no I’m not publishing the results). In order to find out how the lectures were going, I also introduced a weekly feedback form on the LSE intranet (thanks to LSE’s Lucy Pickles for sorting that out), and asked students to fill it in at the end of the session. The only incentive I could think of was to promise a satirical video (example below) if they stayed long enough to fill it in before rushing out the door – it seemed to work. The students were asked to rank presentation and content separately on a scale from ‘awful’ to ‘brilliant’, and then offer suggestions for improvements.

It’s anonymous, and not rigorous of course (self-selecting sample, disgruntled students likely to drop out in subsequent weeks etc), but it has been incredibly useful, especially the open-ended box for suggestions, which have been crammed full with useful content. The first week’s comments broadly asked for more participation, so week two included lots of breakout group discussions. The feedback then said, ‘we like the discussion, but all the unpacking after the groups where you ask what people were talking about eats up time, and anyway, we couldn’t hear half of it’, and asked for more rigour, so week three had more references to the literature, and 3 short discussion groups with minimal feedback – it felt odd, but seemed to work.

At this point, the penny dropped – I was putting into practice some of the messages of my week two lecture on how to work in complex systems, namely fast feedback loops that enable you to experiment, fail, tweak and try again in a repeat cycle until something reasonably successful emerges through trial and error. One example of failing faster – I tried out LSE’s online polling system, but found it was too slow (getting everyone to go online on their mobiles and then vote on a series of multiple choice questions) but also not as energising as getting people to vote analogue style (i.e. raising their hands). The important thing is getting weekly feedback and responding to it, rather than waiting til the end of term (by which time it will be too late).

LSE content weeks 1-3The form is not the only feedback system of course. As any teacher knows, talking to a roomful of people inevitably involves pretty intense realtime feedback too – you feel the energy rise and fall, see people glazing over or getting interested etc. What’s interesting is being able to triangulate between what I thought was happening in the room/students’ heads, and what they subsequently said. Broad agreement, but the feedback suggested their engagement was reassuringly consistent (see bar chart on content), whereas my perceptions seem to amplify it all into big peaks and troughs – what I thought was a disastrous second half of lecture two appears to have just been a bit below par for a small number of students.

The feedback also helps crystallize half-formed thoughts of your own. For example, several complained about the disruption of students leaving in the middle of the lecture, something I also had found rather unnerving. So I suggested that if people did need to leave early (it’s last thing on Friday after all), they should do so during the group discussions – much better.

What’s been striking is the look of mild alarm in the eyes of some of my LSE faculty colleagues, who warned against too much populist kowtowing to student demands. That’s certainly not how it’s felt so far. Here’s a typical comment ‘I think that this lecture on the role of the state tried to take on too much. This is an area that we have discussed extensively. I think it would have been more useful to focus on a particular aspect, perhaps failed and conflict-affected states since you argue that those are the future of aid’. Not a plea for more funny videos (though there have been a few of those), but a reminder to check for duplication with other modules, and a useful guide to improving next year’s lectures.

What is also emerging, again in a pleasingly unintended way, is a sense that we are designing this course together and the students seem to appreciate that  (I refuse to use the awful word co-create. Doh.) Matt Andrews calls this process ‘Problem Driven Iterative Adaptation’ – I would love to hear from other lecturers on more ways to use fast feedback to sharpen up teaching practices.

You can also find this blog here.

EEP/Shiree: Using adaptive programming to monitor change in Bangladesh

written by Salimah Samji

How do you effectively monitor an 8 year, £83.5 million (around USD$135 million) challenge fund that partners with NGOs to improve the livelihood of 1 million beneficiaries? A daunting task indeed.

The Economic Empowerment of the Poorest (EEP/Shiree) program is a partnership between the UK Department for International Development (DFID), the Swiss Agency for Development and Cooperation (SDC) and the Government of Bangladesh (GoB), whose objective is to lift 1 million people out of extreme poverty by 2015. The fact that it is a challenge fund that has managing partners, consortium academic partners and NGO partners, all with many moving pieces, made it crucial to have agile decision making tools in order to respond to the needs of their beneficiaries in real-time. Traditional M&E methods of baseline, midterm and endline surveys were deemed insufficient.

The need for real-time measures and iterative decision making created the space for experimentation and innovation. Armed with authorization from their donors, the EEP/Shiree team set out to explore, experiment and create Change Monitoring System 2 (CMS 2). There are a total of 5 CMS tools which include in-depth life histories. They crawled the design space to find and fit solutions that would work in their context (pilot-test-adapt-iterate). Here is a summary of the three pilot phases:

  • Phase 1: Optical reader technology: They first created a simple survey for the NGOs partners to administer and fill out. The surveys were then digitally scanned. They quickly learned that this was too cumbersome a process and it took 2-3 weeks to receive the surveys. The time-lag was too long, they needed something more efficient.
  • Phase 2: Java enabled phone: Since mobile penetration is high, they partnered with mPower to develop a ten minute, monthly census survey on the phone. They equipped up to 20 field officers (the front line personnel who work at the field level with beneficiary households) with simple mobile phone devices that used the Bangla script for the survey. It was meant to be a 6 month pilot but it lasted for 1.5 years by which time they had scaled to 100 devices, with surveys and simple visualization. Convincing NGO partners as well as the visualization and the development of an in-house feedback loop mechanism took much longer than had been anticipated.
  • Phase 3: Android smart phone: The dropping costs of smart phones in the market (android phones were $60-70) created a lucrative option. The smart phone allowed greater flexibility (field staff just update the app on their phone), more functionality and accountability (GPS location of households, photos and voice recording verify that the beneficiaries are being met regularly). mPower also built a dashboard that allowed the comparison and served as a litmus test to identify red flags that required further investigation, ultimately allowing the NGO partners and EEP/Shiree to tailor recovery plans to the beneficiaries needs and changing context.

Slide1

After the trial-and-error and incremental adjustments over three pilot phases, EEP/Shiree deployed a full roll out of the system towards the end of 2012 (3 years later) with the use of smart phones. EEP/Shiree project partners have over 700 smart phones equipped with an Android operating system, internet connectivity and GPS capability, and have been monitoring over 100,000 households every month across Bangladesh as well as accessing information through an online visualization dashboard that is updated in real time.

Here are some of the challenges they faced:

  • Bringing NGO partners on board: The NGO partners were reluctant and viewed the collection of data as an imposition from above. Asking, “why do we have to do it?” and saying “we don’t have time.” They did not understand that the data and the dashboard could serve as a management tool for themselves. NGO partners were then involved in the design of the questions and were included in the process. It took approximately eight months for data collection to cross the 100,000 per month mark which has since been consistently met and represents most households.
  • Infrastructure constraints: Accessing the dashboard from some areas still face connectivity issues. De jure, every field officer is supposed to visit once a month but de facto not all of them do. The sheer scale of the program makes it physically difficult to monitor. While changing the survey questions is easy – you just download the new form on your phone – the back end dashboard change costs are high. Furthermore, by changing questions you lose the ability to compare across time.
  • Effective use of existing data: While the data is used to respond to the needs of the beneficiaries, very little predictive/trend analysis is done. The data is not used to challenge assumptions of what works and to continuously refine their understanding of the dynamics of ascents out of and descents into extreme poverty. This is partly because no one is responsible for this task and so it doesn’t get done.

Complex problems do not have clear solutions. The fact that the donors were flexible and created the space for experimentation and innovation allowing several pilots to be tested (all with good reasoning) is commendable. Throughout the process, EEP/Shiree and mpower co-designed CMS 2 and their continuous cycle of partnership led to a virtuous cycle of action. The leadership on both sides meet every 2-3 months to discuss what is working and what is not, which helps adapt process to technology and technology to process. Together they built a dynamic monitoring tool, proving that this can be done at scale. This is a far cry from the usual case of consultant comes, builds an MIS system and then leaves.

Practicing Governance: PDIA and Basketball

Guest post written by Brad Cunningham

Basketball players everywhere are trying their best to shoot a ball through a hoop. In pursuit of this goal, players develop their own style of shooting. The image below shows three of the greatest basketball players just as they are about to shoot. At first glance, their form looks pretty similar. However, the differences intrigue me, and make me think about PDIA.

Slide1

From left to right: Steve Nash, Reggie Miller, and Larry Bird

With a closer look, differences in these players form becomes apparent. When Steve Nash shoots, the ball stays further in front of him, close to directly above his elbow, and slightly right of the center of his body. When Reggie Miller shoots, the ball is centered above his head (not his elbow) and further back. Larry Bird takes the ball even further behind his head and has it centered above his shoulder instead of his head.

If that all seems like insignificant nuance, consider that while Larry Bird shoots, the ball is nowhere in his field of vision. Contrast this with Steve Nash, who almost obscures his vision of the basket with the ball. Despite these differences of shooting form, all three of these players achieved the highest level of shooting functionality during their careers.

In sports, elements of form are often referred to as “mechanics”. Elite athletes using different mechanics happens in many sports. Top golfers develop unique swings that accommodate their physique and style of play. Baseball pitchers also develop unique pitching motions. In each of these sports, it is remarkable that different forms developed even though players are trying to achieve exactly the same function: put a ball through a hoop from up to 24 feet away, hit a golf ball a far distance in the desired direction, or throw a baseball past a batter.

I can only speculate at how a multiplicity of mechanics comes about based on my own (admittedly amateur) experience playing basketball. I was recently back on a basketball court for the first time in a while. An hour flew by while I was working through the seemingly endless combinations of elbow position, ball position, aiming cues, release angle, left hand position, etc. Yet by the end, the rapid feedback I got from how consistently the ball was going in had helped me find a combination of form-elements that worked for me.

In sports, athletes and teams improve their functionality and the integration of their elements of from through a process called “practice”. In governance, agencies improve through a process we call “re-form”. These words are interesting. In my experience, the activities that “practice” brings to mind (such as trial-and-error, coaching, and gaining skills through experience) are much more relevant to improving the functionality of agencies in developing countries. These also happen to be the ideas that PDIA is built around.

In contrast, “reform” implies re-adjusting forms. Without connecting newly adopted forms to their functionality through a process of “practicing”, the purpose gets lost. This is embodied in activities like changing bureaucratic rules and procedures that are easy targets for change but rarely seem to enhance functionality. To get where they need to go, government agencies need help and time practicing their craft. They need clear goals and quick and timely feedback on whether the form adjustments they make are helping to reach these goals. In other words, they need PDIA.

In addition to direct feedback during practice through a PDIA type process, professional athletes are also constantly scrutinized through detailed statistics of their functionality – shooting percentages, batting averages, driving accuracy, etc. This is a great analogy for the kind of metrics we need for government. As Matt Andrews has pointed out before, metrics of governance too often focus on forms. Instead, we need more and better metrics of functionality that will promote the practice of governance.

All of this confirms my suspicion that PDIA is not some new fad of a panacea that was invented by ivory tower elites. Rather, I see it as a mundane (and perhaps far too obvious) description of how people and organizations have always gone about getting better at accomplishing something in the real world… and I think that’s great.

Tacit Knowledge: What and How?

written by Matt Andrews

Tacit knowledge is an important focal point of my work. I think that many reforms fail because they try to transfer formal, codified knowledge only; when the key knowledge we need in governments and in the development process is tacit–knowledge that cannot be easily communicated in writing or even in words but that resides in our heads and organizations and helps us adapt to the unexpected and navigate the un-navigable.

The Problem Driven Iterative Adaption (PDIA) approach is designed to expand tacit knowledge as part of the development process. “But,” someone recently asked me, “what is tacit knowledge and how do you build it?”

Great question. And this weekend I read a fantastic article that helps me describe what tacit knowledge is and how it is built. The article centered on ‘The Knowledge: London’s Legendary Test for Taxi Drivers” (written by Jody Rosen in the New York Times Magazine). It starts with the following (a bit long for this blog but please read it…it’s great):

At 10 past 6 on a January morning a couple of winters ago, a 35-year-old man named Matt McCabe stepped out of his house in the town of Kenley, England, got on his Piaggio X8 motor scooter, and started driving north. McCabe’s destination was Stour Road, a small street in a desolate patch of East London, 20 miles from his suburban home. He began his journey by following the A23, a major thruway connecting London with its southern outskirts, whose origins are thought to be ancient: For several miles the road follows the straight line of the Roman causeway that stretched from London to Brighton. McCabe exited the A23 in the South London neighborhood of Streatham and made his way through the streets, arriving, about 20 minutes after he set out, at an intersection officially called Windrush Square but still referred to by locals, and on most maps, as Brixton Oval. There, McCabe faced a decision: how to plot his route across the River Thames. Should he proceed more or less straight north and take London Bridge, or bear right into Coldharbour Lane and head for “the pipe,” the Rotherhithe Tunnel, which snakes under the Thames two miles downriver?

“At first I thought I’d go for London Bridge,” McCabe said later. “Go straight up Brixton Road to Kennington Park Road and then work my line over. I knew that I could make my life a lot easier, to not have to waste brainpower thinking about little roads — doing left-rights, left-rights. And then once I’d get over London Bridge, it’d be a quick trip: I’d work it up to Bethnal Green Road, Old Ford Road, and boom-boom-boom, I’m there. It’s a no-brainer. But no. I was thinking about the traffic, about everyone going to the City at that hour of the morning. I thought, ‘What can I do to skirt central London?’ That was my key decision point. I didn’t want to sit in the traffic lights. So I decided to take Coldharbour Lane and head for the pipe.”

McCabe turned east on Coldharbour Lane, wending through the neighborhoods of Peckham and Bermondsey before reaching the tunnel. He emerged on the far side of the Thames in Limehouse, and from there his three-mile-long trip followed a zigzagging path northeast. “I came out of the tunnel and went forward into Yorkshire Road,” he told me. “I went right into Salmon Lane. Left into Rhodeswell Road, right into Turners Road. I went right into St. Paul’s Way, left into Burdett Road, right into Mile End Road. Left Tredegar Square. I went right Morgan Street, left Coborn Road, right into Tredegar Road. That gave me a forward into Wick Lane, a right into Monier Road, right into Smeed Road — and we’re there. Left into Stour Road.”

I liked this description of a man managing his way around London because it shows the blend of tacit knowledge and other, more formal, knowledge. The formal knowledge is what McCabe refers to when thinking about the map of London. He can look at it and see multiple routes from his start to his end point. But this knowledge has its limits. It doesn’t extend to know-how about when streets are most busy, or if locals call a street one thing when the map calls it another, or where the best fish and chips shop is en route… These things are tacit, things one learns by taking multiple routes multiple times and getting an imprint on one’s mind about what to expect, when and where… and what to do about what to expect… because one has been there before.

This kind of tacit knowledge is what London taxi drivers are apparently taught in ‘The Knowledge’–a multi-year testing process that requires drivers learning how to navigate the formal map and real-world grit and granularity of London. The tacit knowledge these drivers must build–where I use ‘tacit knowledge’ given Michael Polanyi’s work  in Personal Knowledge, 1958–is attained not by sitting in a room learning from someone. No: It cannot be learned this way, given that it is hard to verbalize, or to codify.

Instead, it must be earned, through actually going and seeing and touching and riding and engaging the streets of London…again and again and again. The ‘Knowledge Boys and Girls’ build this tacit knowledge over many years by riding around London on bicycles and motorbikes in a process called ‘pointing’–where they observe granular details of everything, memorizing details that most would take for granted but that are key to any adaptation they might need to make if a London rain storm starts unexpectedly, or if a lorry jams a key road, or if a passenger finds they urgently need to stop en route to see a doctor or buy flowers for a jilted lover!

The details are hard to pass onto others partly because they are so voluminous, partly because they might seem mundane, and partly because they become taken-for-granted in the taxi drivers head. But these details are the kind of knowledge that separate the taxi drivers who can adapt from those who cannot… the ones who know how to zig and zag past obstacles or towards uncertain destinations.

PDIA processes work much like the Knowledge Tests, and require that folks in governments undergoing reform work actively together doing the mundane and repetitive things that make up most days of work and life; stopping regularly to make mental notes of the unspoken lessons they are learning; doing similar things again…learning again….and storing the information about what they did. The process is arduous to some, and unnecessary to others who think that change should never involve recreating the wheel and is best done by getting external consultants to introduce a tried-and-tested external best practice and train locals in how to use such. As I wrote a few weeks ago, however, I think one always needs the process of doing, learning and adapting even if one does not reproduce the wheel… because those who use the wheel need the tacit knowledge built from finding and fitting to make it work. 

I believe that doing this kind of learning in groups is a possible way of inculcating the tacit knowledge in the DNA of organizations and not just the heads of individuals. It is a challenging prospect indeed, but I have seen enough of this learning in practice to think–with no hesitation–that it is key to development…the fundamental way of making sense of the complexity, uncertainty and senselessness of much we encounter in governments and societies struggling to progress. Consider the following from the New York Times article…where I replace mention of ‘London’ or ‘town’ with ‘development’…and which emphasizes the role of ‘The Knowledge’ (or process of acquiring both formal and tacit know-how):

[Development] bewilders even its lifelong residents. [Those in developing countries] are “a population lost in [their] own [system].” [Development’s] labyrinthine roadways are a symbol — and, perhaps, a cause — of the fatalism that hangs like a pea-soup fog over [development’s] consciousness. Facing the dizzying infinitude of streets, your mind turns darkly to thoughts of finitude: to the time that is flying, the minutes you are running late for your doctor’s appointment, the hours ticking by, never to be retrieved, on the proverbial Big Clock, the one even bigger than Big Ben. You can see it every day in Primrose Hill and Clapham, in Golders Green and Kentish Town, in Deptford and Dalston. A nervous man, an anxious woman, scanning the horizon for a recognizable landmark, searching for a street sign, silently wondering “Where am I?” — a geographical question that grades gloomily into an existential one. Which is where the Knowledge [and tacit knoweldge learning processes] comes in. It is [Development’s] weird solution to the riddle of itself, a training program whose graduates are both transit workers and Gnostics: chauffeurs taught by the government to know the unknowable.”

How often do development interventions provide such a training program?

Development is like London… “a mess: a preposterously complex tangle of veins and capillaries, the cardiovascular system of a monster.” You need to build tacit knowledge to navigate such system… by doing and learning and doing again…

Londonmessymap