EEP/Shiree: Using adaptive programming to monitor change in Bangladesh

written by Salimah Samji

How do you effectively monitor an 8 year, £83.5 million (around USD$135 million) challenge fund that partners with NGOs to improve the livelihood of 1 million beneficiaries? A daunting task indeed.

The Economic Empowerment of the Poorest (EEP/Shiree) program is a partnership between the UK Department for International Development (DFID), the Swiss Agency for Development and Cooperation (SDC) and the Government of Bangladesh (GoB), whose objective is to lift 1 million people out of extreme poverty by 2015. The fact that it is a challenge fund that has managing partners, consortium academic partners and NGO partners, all with many moving pieces, made it crucial to have agile decision making tools in order to respond to the needs of their beneficiaries in real-time. Traditional M&E methods of baseline, midterm and endline surveys were deemed insufficient.

The need for real-time measures and iterative decision making created the space for experimentation and innovation. Armed with authorization from their donors, the EEP/Shiree team set out to explore, experiment and create Change Monitoring System 2 (CMS 2). There are a total of 5 CMS tools which include in-depth life histories. They crawled the design space to find and fit solutions that would work in their context (pilot-test-adapt-iterate). Here is a summary of the three pilot phases:

  • Phase 1: Optical reader technology: They first created a simple survey for the NGOs partners to administer and fill out. The surveys were then digitally scanned. They quickly learned that this was too cumbersome a process and it took 2-3 weeks to receive the surveys. The time-lag was too long, they needed something more efficient.
  • Phase 2: Java enabled phone: Since mobile penetration is high, they partnered with mPower to develop a ten minute, monthly census survey on the phone. They equipped up to 20 field officers (the front line personnel who work at the field level with beneficiary households) with simple mobile phone devices that used the Bangla script for the survey. It was meant to be a 6 month pilot but it lasted for 1.5 years by which time they had scaled to 100 devices, with surveys and simple visualization. Convincing NGO partners as well as the visualization and the development of an in-house feedback loop mechanism took much longer than had been anticipated.
  • Phase 3: Android smart phone: The dropping costs of smart phones in the market (android phones were $60-70) created a lucrative option. The smart phone allowed greater flexibility (field staff just update the app on their phone), more functionality and accountability (GPS location of households, photos and voice recording verify that the beneficiaries are being met regularly). mPower also built a dashboard that allowed the comparison and served as a litmus test to identify red flags that required further investigation, ultimately allowing the NGO partners and EEP/Shiree to tailor recovery plans to the beneficiaries needs and changing context.

Slide1

After the trial-and-error and incremental adjustments over three pilot phases, EEP/Shiree deployed a full roll out of the system towards the end of 2012 (3 years later) with the use of smart phones. EEP/Shiree project partners have over 700 smart phones equipped with an Android operating system, internet connectivity and GPS capability, and have been monitoring over 100,000 households every month across Bangladesh as well as accessing information through an online visualization dashboard that is updated in real time.

Here are some of the challenges they faced:

  • Bringing NGO partners on board: The NGO partners were reluctant and viewed the collection of data as an imposition from above. Asking, “why do we have to do it?” and saying “we don’t have time.” They did not understand that the data and the dashboard could serve as a management tool for themselves. NGO partners were then involved in the design of the questions and were included in the process. It took approximately eight months for data collection to cross the 100,000 per month mark which has since been consistently met and represents most households.
  • Infrastructure constraints: Accessing the dashboard from some areas still face connectivity issues. De jure, every field officer is supposed to visit once a month but de facto not all of them do. The sheer scale of the program makes it physically difficult to monitor. While changing the survey questions is easy – you just download the new form on your phone – the back end dashboard change costs are high. Furthermore, by changing questions you lose the ability to compare across time.
  • Effective use of existing data: While the data is used to respond to the needs of the beneficiaries, very little predictive/trend analysis is done. The data is not used to challenge assumptions of what works and to continuously refine their understanding of the dynamics of ascents out of and descents into extreme poverty. This is partly because no one is responsible for this task and so it doesn’t get done.

Complex problems do not have clear solutions. The fact that the donors were flexible and created the space for experimentation and innovation allowing several pilots to be tested (all with good reasoning) is commendable. Throughout the process, EEP/Shiree and mpower co-designed CMS 2 and their continuous cycle of partnership led to a virtuous cycle of action. The leadership on both sides meet every 2-3 months to discuss what is working and what is not, which helps adapt process to technology and technology to process. Together they built a dynamic monitoring tool, proving that this can be done at scale. This is a far cry from the usual case of consultant comes, builds an MIS system and then leaves.

Practicing Governance: PDIA and Basketball

Guest post written by Brad Cunningham

Basketball players everywhere are trying their best to shoot a ball through a hoop. In pursuit of this goal, players develop their own style of shooting. The image below shows three of the greatest basketball players just as they are about to shoot. At first glance, their form looks pretty similar. However, the differences intrigue me, and make me think about PDIA.

Slide1

From left to right: Steve Nash, Reggie Miller, and Larry Bird

With a closer look, differences in these players form becomes apparent. When Steve Nash shoots, the ball stays further in front of him, close to directly above his elbow, and slightly right of the center of his body. When Reggie Miller shoots, the ball is centered above his head (not his elbow) and further back. Larry Bird takes the ball even further behind his head and has it centered above his shoulder instead of his head.

If that all seems like insignificant nuance, consider that while Larry Bird shoots, the ball is nowhere in his field of vision. Contrast this with Steve Nash, who almost obscures his vision of the basket with the ball. Despite these differences of shooting form, all three of these players achieved the highest level of shooting functionality during their careers.

In sports, elements of form are often referred to as “mechanics”. Elite athletes using different mechanics happens in many sports. Top golfers develop unique swings that accommodate their physique and style of play. Baseball pitchers also develop unique pitching motions. In each of these sports, it is remarkable that different forms developed even though players are trying to achieve exactly the same function: put a ball through a hoop from up to 24 feet away, hit a golf ball a far distance in the desired direction, or throw a baseball past a batter.

I can only speculate at how a multiplicity of mechanics comes about based on my own (admittedly amateur) experience playing basketball. I was recently back on a basketball court for the first time in a while. An hour flew by while I was working through the seemingly endless combinations of elbow position, ball position, aiming cues, release angle, left hand position, etc. Yet by the end, the rapid feedback I got from how consistently the ball was going in had helped me find a combination of form-elements that worked for me.

In sports, athletes and teams improve their functionality and the integration of their elements of from through a process called “practice”. In governance, agencies improve through a process we call “re-form”. These words are interesting. In my experience, the activities that “practice” brings to mind (such as trial-and-error, coaching, and gaining skills through experience) are much more relevant to improving the functionality of agencies in developing countries. These also happen to be the ideas that PDIA is built around.

In contrast, “reform” implies re-adjusting forms. Without connecting newly adopted forms to their functionality through a process of “practicing”, the purpose gets lost. This is embodied in activities like changing bureaucratic rules and procedures that are easy targets for change but rarely seem to enhance functionality. To get where they need to go, government agencies need help and time practicing their craft. They need clear goals and quick and timely feedback on whether the form adjustments they make are helping to reach these goals. In other words, they need PDIA.

In addition to direct feedback during practice through a PDIA type process, professional athletes are also constantly scrutinized through detailed statistics of their functionality – shooting percentages, batting averages, driving accuracy, etc. This is a great analogy for the kind of metrics we need for government. As Matt Andrews has pointed out before, metrics of governance too often focus on forms. Instead, we need more and better metrics of functionality that will promote the practice of governance.

All of this confirms my suspicion that PDIA is not some new fad of a panacea that was invented by ivory tower elites. Rather, I see it as a mundane (and perhaps far too obvious) description of how people and organizations have always gone about getting better at accomplishing something in the real world… and I think that’s great.

Tacit Knowledge: What and How?

written by Matt Andrews

Tacit knowledge is an important focal point of my work. I think that many reforms fail because they try to transfer formal, codified knowledge only; when the key knowledge we need in governments and in the development process is tacit–knowledge that cannot be easily communicated in writing or even in words but that resides in our heads and organizations and helps us adapt to the unexpected and navigate the un-navigable.

The Problem Driven Iterative Adaption (PDIA) approach is designed to expand tacit knowledge as part of the development process. “But,” someone recently asked me, “what is tacit knowledge and how do you build it?”

Great question. And this weekend I read a fantastic article that helps me describe what tacit knowledge is and how it is built. The article centered on ‘The Knowledge: London’s Legendary Test for Taxi Drivers” (written by Jody Rosen in the New York Times Magazine). It starts with the following (a bit long for this blog but please read it…it’s great):

At 10 past 6 on a January morning a couple of winters ago, a 35-year-old man named Matt McCabe stepped out of his house in the town of Kenley, England, got on his Piaggio X8 motor scooter, and started driving north. McCabe’s destination was Stour Road, a small street in a desolate patch of East London, 20 miles from his suburban home. He began his journey by following the A23, a major thruway connecting London with its southern outskirts, whose origins are thought to be ancient: For several miles the road follows the straight line of the Roman causeway that stretched from London to Brighton. McCabe exited the A23 in the South London neighborhood of Streatham and made his way through the streets, arriving, about 20 minutes after he set out, at an intersection officially called Windrush Square but still referred to by locals, and on most maps, as Brixton Oval. There, McCabe faced a decision: how to plot his route across the River Thames. Should he proceed more or less straight north and take London Bridge, or bear right into Coldharbour Lane and head for “the pipe,” the Rotherhithe Tunnel, which snakes under the Thames two miles downriver?

“At first I thought I’d go for London Bridge,” McCabe said later. “Go straight up Brixton Road to Kennington Park Road and then work my line over. I knew that I could make my life a lot easier, to not have to waste brainpower thinking about little roads — doing left-rights, left-rights. And then once I’d get over London Bridge, it’d be a quick trip: I’d work it up to Bethnal Green Road, Old Ford Road, and boom-boom-boom, I’m there. It’s a no-brainer. But no. I was thinking about the traffic, about everyone going to the City at that hour of the morning. I thought, ‘What can I do to skirt central London?’ That was my key decision point. I didn’t want to sit in the traffic lights. So I decided to take Coldharbour Lane and head for the pipe.”

McCabe turned east on Coldharbour Lane, wending through the neighborhoods of Peckham and Bermondsey before reaching the tunnel. He emerged on the far side of the Thames in Limehouse, and from there his three-mile-long trip followed a zigzagging path northeast. “I came out of the tunnel and went forward into Yorkshire Road,” he told me. “I went right into Salmon Lane. Left into Rhodeswell Road, right into Turners Road. I went right into St. Paul’s Way, left into Burdett Road, right into Mile End Road. Left Tredegar Square. I went right Morgan Street, left Coborn Road, right into Tredegar Road. That gave me a forward into Wick Lane, a right into Monier Road, right into Smeed Road — and we’re there. Left into Stour Road.”

I liked this description of a man managing his way around London because it shows the blend of tacit knowledge and other, more formal, knowledge. The formal knowledge is what McCabe refers to when thinking about the map of London. He can look at it and see multiple routes from his start to his end point. But this knowledge has its limits. It doesn’t extend to know-how about when streets are most busy, or if locals call a street one thing when the map calls it another, or where the best fish and chips shop is en route… These things are tacit, things one learns by taking multiple routes multiple times and getting an imprint on one’s mind about what to expect, when and where… and what to do about what to expect… because one has been there before.

This kind of tacit knowledge is what London taxi drivers are apparently taught in ‘The Knowledge’–a multi-year testing process that requires drivers learning how to navigate the formal map and real-world grit and granularity of London. The tacit knowledge these drivers must build–where I use ‘tacit knowledge’ given Michael Polanyi’s work  in Personal Knowledge, 1958–is attained not by sitting in a room learning from someone. No: It cannot be learned this way, given that it is hard to verbalize, or to codify.

Instead, it must be earned, through actually going and seeing and touching and riding and engaging the streets of London…again and again and again. The ‘Knowledge Boys and Girls’ build this tacit knowledge over many years by riding around London on bicycles and motorbikes in a process called ‘pointing’–where they observe granular details of everything, memorizing details that most would take for granted but that are key to any adaptation they might need to make if a London rain storm starts unexpectedly, or if a lorry jams a key road, or if a passenger finds they urgently need to stop en route to see a doctor or buy flowers for a jilted lover!

The details are hard to pass onto others partly because they are so voluminous, partly because they might seem mundane, and partly because they become taken-for-granted in the taxi drivers head. But these details are the kind of knowledge that separate the taxi drivers who can adapt from those who cannot… the ones who know how to zig and zag past obstacles or towards uncertain destinations.

PDIA processes work much like the Knowledge Tests, and require that folks in governments undergoing reform work actively together doing the mundane and repetitive things that make up most days of work and life; stopping regularly to make mental notes of the unspoken lessons they are learning; doing similar things again…learning again….and storing the information about what they did. The process is arduous to some, and unnecessary to others who think that change should never involve recreating the wheel and is best done by getting external consultants to introduce a tried-and-tested external best practice and train locals in how to use such. As I wrote a few weeks ago, however, I think one always needs the process of doing, learning and adapting even if one does not reproduce the wheel… because those who use the wheel need the tacit knowledge built from finding and fitting to make it work. 

I believe that doing this kind of learning in groups is a possible way of inculcating the tacit knowledge in the DNA of organizations and not just the heads of individuals. It is a challenging prospect indeed, but I have seen enough of this learning in practice to think–with no hesitation–that it is key to development…the fundamental way of making sense of the complexity, uncertainty and senselessness of much we encounter in governments and societies struggling to progress. Consider the following from the New York Times article…where I replace mention of ‘London’ or ‘town’ with ‘development’…and which emphasizes the role of ‘The Knowledge’ (or process of acquiring both formal and tacit know-how):

[Development] bewilders even its lifelong residents. [Those in developing countries] are “a population lost in [their] own [system].” [Development’s] labyrinthine roadways are a symbol — and, perhaps, a cause — of the fatalism that hangs like a pea-soup fog over [development’s] consciousness. Facing the dizzying infinitude of streets, your mind turns darkly to thoughts of finitude: to the time that is flying, the minutes you are running late for your doctor’s appointment, the hours ticking by, never to be retrieved, on the proverbial Big Clock, the one even bigger than Big Ben. You can see it every day in Primrose Hill and Clapham, in Golders Green and Kentish Town, in Deptford and Dalston. A nervous man, an anxious woman, scanning the horizon for a recognizable landmark, searching for a street sign, silently wondering “Where am I?” — a geographical question that grades gloomily into an existential one. Which is where the Knowledge [and tacit knoweldge learning processes] comes in. It is [Development’s] weird solution to the riddle of itself, a training program whose graduates are both transit workers and Gnostics: chauffeurs taught by the government to know the unknowable.”

How often do development interventions provide such a training program?

Development is like London… “a mess: a preposterously complex tangle of veins and capillaries, the cardiovascular system of a monster.” You need to build tacit knowledge to navigate such system… by doing and learning and doing again…

Londonmessymap

The DDD Manifesto finds a new home

written by Salimah Samji

Since we published The DDD Manifesto on November 21, it has been viewed over 5,000 times all around the world (in 100+ countries). It currently has over 400 signatories from 60 countries. It is an eclectic community with people from bilateral organizations, multilaterals, governments, academia, NGOs, private sector, as well as independent development practitioners. These are the founding members of The DDD Manifesto Community.

Today, we are delighted to launch the online platform of the DDD Manifesto Community which is the new home of the manifesto. We hope that this will be a place where you can come to share ideas, have conversations, question your assumptions, learn from others, offer support and be inspired. It includes a forum for discussion, blog posts written by community members and features video presentations from the recent DDD workshop (#differentdev).

To sign the manifesto and to participate in the forum, you can register here. Please contribute actively – this is a community website and you are the community. 

If you want to Do Development Differently but it sounds too hard…

written by Matt Andrews

Arnaldo Pellini recently wrote an interesting personal blog post about the Doing Development Differently workshop and manifesto. He concludes with, “I agree with these ideas and  I can share and discuss these ideas with the team with whom I work  but what difference can it make if the systems around us due to organizational culture, history, circumstances, and traditions struggle to embrace flexibility, uncertainty,  untested experimentation, and slow incremental changes?”

This is an honest reflection from a practitioner in the field; and one that I hear often–from folks working in multilateral and bilateral agencies, as contractors, and beyond. It captures a concern that the development machinery (organizations, monitoring and reporting devices, profession-alliances, government counterparts, etc.) is structurally opposed to doing the kind of work one might call DDD or PDIA.

It’s like this cartoon…where our organizations say “let’s innovate but stay the same.”

Change-is-hard-430x332

I have been thinking about this a lot in the last few years, ever since I wrote chapter ten of my book…which asked whether the development community was capable of changing. In that chapter I was not especially confident but (I hope) I was still hopeful.

Since then, I think I’m more hopeful. Partly because,

  • we have found many folks in the multilaterals, bilaterals, contractors etc. who are doing development in this more flexible way. We invited a range of them to the DDD Workshop and over 330 signed on to the DDD Manifesto. One of the goals of our work in the next while is to learn from these folks about HOW they do development differently even with the constraints they face. How do they get funders to embrace uncertainty? How do they get ministers in-country to buy-into flexibility and give up on straight isomorphism?
  • I am also working on research projects that tackle this question; doing PDIA in real time, in places where development is predominantly done through the incumbent mechanisms. It is hard work, but I am finding various strategies to get buy-in to a new approach (including showing how problematic the old approach is, by working in the hardest areas where one has a counterfactual of failed past attempts, and more). I am also finding strategies to keep the process alive and buy more and more space for flexibility (by iterating tightly at first, for instance, and showing quick wins…and telling the story of learning and of increased engagement and empowerment). So far, I have not experienced complete success with what I have done, but I have certainly not struggled in getting support from the practitioners and authorisers we work with. (In my world it is harder to get support from academics, who think action research on implementation is a hobby and consultancy work… indeed, anything that does not say ‘RCT’ is considered less than academic. Sigh.)

All this is to say that I think Arnaldo is emphasizing a really important constraint on those working in development agencies. But a constraint that we should work through if we really do agree that these more problem driven, flexible approaches are what is needed. To Arnaldo and others I would suggest the following:

  • Separate the conversation about which way we should do development from the conversation about how much our organizational realities ALLOW us to do it. The first conversation is: “Should we do DDD/PDIA?” The second conversation is: “How do we DDD/PDIA?” If we conflate the conversations we never move ahead. If we separate them then we can develop strategies to gradually introduce PDIA/DDD into what we do (in essence, I’m suggesting doing PDIA ourselves, to help change the way we do development…see an earlier blog).
  • I also constantly remind myself that we (external folks in development organizations) are not the only ones facing a challenge of doing new stuff in existing contexts–with all the constraints of such. This is what we are asking of our counterparts and colleagues in the developing countries where we work. Dramatic and uncomfortable and impossible change is in the air every time we are introducing and facilitating and supporting and sponsoring work in developing countries. I always tell myself: “If we can’t work it out in our own organizations–when we think that our own organizational missions depend on such change–then we have no place asking folks in developing countries to work it out.”
  • So, it’s a challenge. But a worthy one. And if we care about doing development with impact, I think it behooves us to face up to this challenge.

Good luck, Arnaldo, thanks for your honesty and for the obvious commitment that causes you to share your reality. It is really appreciated!

The PDIA Anthem

Need help decoding the acronym PDIA? Check out the PDIA anthem.

 

This Anthem uses the Instrumental from Mos Def – Mathematics. It was made by a very talented student as part of an assignment for Matt Andrews course entitled Getting Things Done in Development. We had never imagined that we could write a song about PDIA, let alone a rap. Thank you.

Let me hear you say P. D. I. A.

Introducing The DDD Manifesto

We are delighted to release The DDD Manifesto as an outcome of the 2014 Doing Development Differently (DDD) workshop.

In late October, a group of about 40 development professionals, implementers and funders from around the world attended the DDD workshop, to share examples where real change has been achieved. These examples employ different tools but generally hold to some of the same core principles: being problem driven, iterative with lots of learning, and engaging teams and coalitions, often producing hybrid solutions that are ‘fit to context’ and politically smart.

The two-day workshop was an opportunity to share practical lessons and insights, country experience, and to experiment first hand with selected methodologies and design thinking. In order to maximize the opportunity to hear from as many people as possible, all presenters were asked to prepare a 7:30 minute talk — with no powerpoints or visual accompaniments. The workshop alone generated a rich set of cases and examples of what doing development differently looks like, available on both Harvard and ODI websites (where you can watch individual talks, see the posters or link to related reports).

The aim of the event was to build a shared community of practice, and to crystallize what we are learning about doing development differently from practical experience. The workshop ended with a strong call for developing a manifesto reflecting the common principles that cut across the cases that were presented. Watch the closing remarks here.

DDD Closing Session

These common principles have been synthesized into The DDD Manifesto. We recognize that many of these principles are not new, but we do feel the need to clearly identify principles and to state that we believe that development initiatives will have more impact if these are followed.

As an emerging community of practice, we welcome you to join us by adding your name in the comment box of the manifesto.