Initiating PDIA: Start by running…and then run some more

written by Matt Andrews

“Once there is interest, how do you start a PDIA project?”

Many people have asked me this question. They are often in consulting firms or donor agencies thinking about working on PDIA with host governments, or in some central bureau in the government itself.

“We have an authorizer, know the itch that needs scratching (the problem), and have a team convened to address it,” they say. “But we don’t know what to do to get the work off the ground.”

I ask what they would think of doing, and they typically provide one of the following answers:

“We should do research on the problem (the itch)” or “We should hold a multi-day workshop where people get to analyze the problem and really used to a problem driven approach.”

I have tried starting PDIA with both strategies. Neither is effective in getting the process going.

  • When outsiders (donors, academics, or even central agencies responsible for making but not implementing policy) do the primary research on ‘the problem’, their product is usually a report that sits on shelves. If you start with such a product it is hard to reorient people to change their learned behavior and actually use the report.
  • When you hold an elaborate workshop, using design thinking, fancy analysis, or the like, it is very easy to get stuck in performance—or in a fun and exciting new activity. We find people in governments do attend such events and have fun in them, but often get lost in the discussion or analysis and stay stuck in that place.

Having tried these and other strategies to initiate PDIA interventions, we at Harvard BSC have learned (by doing, reflection, and trying again…) some basic principles about what does not work in getting started, and what does work. Here are a few of these findings:

  • It does not work when outsiders analyze the problem on behalf of those who will act to solve it. It works when those in the insider PDIA teams construct and deconstruct the problem (whether they do this ‘right’ or ‘wrong’). The insiders must own the process, and the outsiders must ‘give the work back’ to the rightful owners.
  • It does not work to stage long introductory workshops to launch PDIA processes, as participants either get frustrated with the time away from work or distracted by the workshop itself. Either way they get stuck and the workshop does not mobilize their action. It works if you convene teams for short ‘launchpad-type events’ where they engage rapidly and move as rapidly to action (beyond talk). We are always anxious to move internal PDIA teams to action. The meetings are simply staging events: they are not what ‘doing PDIA’ is actually about.

Acting on these principles, we now always start PDIA running.

We bring internal teams together, and in a day (or at most a day and a half) we ‘launch’ through a series of sessions that (i) introduce them to the PDIA method, (ii) have them construct and (iii) deconstruct their problems, (iv) identify entry points for action, and (v) specify three or more initial practical steps they can take to start addressing these entry points. At the end of the session they go away with their problem analysis and their next step action commitments, as well as a date when they will again meet a facilitator to discuss their action, and learn by reflection.

This is a lot to get done in a short period. This is intentional, as we are trying to model upfront the importance of acting quickly to create the basis of progress and learning. We use time limits on every activity to establish this kind of pressure, and push all team members to ‘do something’, then ‘stop and reflect’, and then do the next thing.

When we get to the end of each Launchpad event, the internal teams have their own ‘next step’ strategies, and a clear view that the PDIA process has now started: they are already running, and acting, and engaging in a new and difficult space. And they know what they need to do next, and what date in the near future they will account for their progress, be asked about their learning, and pushed to identify more ‘next steps’.

When I tell interested parties in donor agencies, consulting firms, etc. about our ‘start by running’ approach, they have a number of common responses:

“It does not sound like anyone is doing a proper diagnosis of the problem: what happens if the team gets it wrong?”

“What happens if the team identifies next steps that make no sense?”

“This strategy could be a disaster if you have the wrong people in the room—who don’t know what they are doing or who have a biased view on what they are doing…”

These concerns are real, but really don’t matter much in the PDIA process:

  • We don’t believe that initial problem diagnostics are commonly correct when one starts a program (no matter how smart the researchers doing the analysis).
  • We also don’t believe that you commonly identify the right ‘next steps’ from a study or a discussion.
  • And we also don’t believe that these kinds of processes are ever unbiased, or that you commonly get the right people in the room at the start of a process.

We don’t believe you address these concerns by doing great up front research. Rather, we aim to get the teams into action as quickly as possible, where the action creates opportunity for reflection, and reflection informs constant experiential learning—about the problem, past and next steps, and who should be involved in the process. This learning resides in the actors involved in the doing, and prompts their adaptation. Which leads to greater capability and constant improvement in how they see the problem, think of potential solutions, and engage others to make these solutions happen.

A final note:

When I discussed this strategy with a friend charged with ‘doing PDIA’ as part of a contract with a well-known bilateral donor, he lamented: “You are telling me the workshop is but a launching event for the real PDIA process of acting, reflecting, learning and adapting….but I was hired to do a workshop as if it was DOING PDIA. No one spoke of getting into action after the workshop.”

To this colleague—and the donors that hired him—I say simply, “PDIA is about getting people involved, and acting, and you always need to get to action fast. PDIA must start by running, and must keep teams running afterwards. Anything that happens one-off, or that promotes slow progress and limited repeated engagement is simply not PDIA.”

Learn more about initiating PDIA in practice in chapters 7 and 9 of our free book, Building State Capability: Evidence, Analysis, Action.

PDIA and Authorizers with an itch

written by Matt Andrews

“How do you decide where to work on a PDIA project?”  This is probably the most common question I have been asked with respect to PDIA.

After over 5 years of doing this work in a variety of countries and sectors, I have a simple answer: “When we find authorizers with an itch.”

“That sounds bizarre,” I hear you say. Or maybe you think I’m just being cute to fit in with a playful blogging technique.

No, authorizers with an itch are key to starting any PDIA initiative.

When I say we need counterparts with an itch, I mean that they are very aware of a problem they can’t solve. Like an itch you can’t scratch, or that you scratch again and again but to no avail. This is usually a policy problem that has come to the surface one too many times, usually where various prior reforms or policies or interventions have not provided effective solutions.

Stubborn itches create frustration and even desperation, which can create the space for doing things differently—and taking risks. PDIA needs this kind of space, and this motivating influence. Without it, we have found very little room to focus on the problem, and learn-by-doing towards a new solution.

There are downsides of working to scratch a stubborn itch. The fact that others have tried scratching it, to no avail, means that it is usually going to be ‘wicked hard’ to solve (so don’t expect an easy path to a solution). The fact that it seems to move around (sometimes itching here and sometimes there) reflects the many unseen and even dynamic factors that cause the itch itself (like nasty politics or bureaucratic dysfunction). Don’t expect these factors to go away just because you are tackling the problem with PDIA. You will hit the nastiness soon. Be ready.

When I say we need ‘authorizers’ to start, it is because the PDIA work we do is always in the public domain, where no real work (with action attached) is done without someone’s explicit authorization. The required authorizer is always, in my experience, someone inside the context undergoing change. This means the work cannot be ordered or organized or identified by an external agent (donor, consultant, or even academic).

My team at Harvard found this out the hard way. As you will read in a forthcoming working paper by Stuart Russell, Peter Harrington and I, we have experimented with PDIA initiatives where problems are identified in different ways.  We have had limited success whenever anyone from Harvard or an external entity (like a donor) has been a main identifier of the problem. In contrast, we have almost always had some success when the problem was identified by a domestic authorizer in the place undergoing change.

This is simply because the internal authorizer needs to have internal authority: at the least, to convene a group of internal people to start engaging with the problem, and beyond this to protect the PDIA process from threats and distractions. No external party can do this.

Beyond convening authority, we find that the authorizers need to provide three types of authorization: shareable authorization (where they allow the engagement of other authorizers in the process of scratching the itch), flexible authorization (which allows for an experimental process), and patient (or grit) authorization (where one can expect some continued support in the search for an effective ‘scratch’ solution).

These are big authorization needs, and one does not know if they will be met at the start of the PDIA process. But they tend to come when authorizers face an itch (making them willing to share, adaptive in demands, and patient for a real solution).

We find, therefore, that there is enough space to initiate a PDIA initiative if we find an authorizer with an itch she cannot scratch.  That’s where we start our work, buckling our seat belts and getting ready for a journey of, and to the unexpected.

Are you in a situation where an authorizer is facing a stubborn itch? Maybe you have space to ask, “What’s the problem…and can we mobilize a team to try something different to solve it?”

Learn more about engaging authorizers around problems that matter in chapters 6 and 9 of our free book, Building State Capability: Evidence, Analysis, Action.

 

Toward a new theory and practice of building state capacity

Guest blog by Archon Fung

I just want to begin by expressing my profound pride about the new book by Matt Andrews, Lant Pritchett and Michael Woolcock: Building State Capacity: Evidence, Analysis and Action. It is a wonderfully innovative volume that is full of insights about how to do development better.

As many of you know, Building State Capacity is the result of many years of practice, teaching, and research in the area of development. Indeed, it isn’t just a book or a research project, but a whole movement within the field of development and development studies — check out the website “doingdevelopmentdifferently.com” which offers a manifesto on PDIA and has many hundreds of signatories.

The book is about how to build more capable states — governments if you like — in developing countries. This is a topic that is distinctive to the Kennedy School and other public policy schools: their book is not about how to make economies grow, nor how to make democracy work, or even how to improve human develop. Rather, they focus clearly and deliberately on how to make state institutions more capable. They tackle the urgent question of how to make governments in developing countries better at doing the many things that we ask governments to do — deliver services like education and health care, to impose obligations like collecting taxes and keeping the peace, and to make policy and regulation that is based on good evidence and reasoning.

Their book is appropriately ambitious, aiming to break through old mistakes, establish a new paradigm of development practice and offer guidance to public leaders about how to implement that new paradigm. Let’s consider these three objectives in turn.

How do they break the old paradigms of development? What is it that they reject?

First, they aim to reject a two major pillars of common wisdom in current approaches to development. I’ll use my terms rather than their here. First, they seek to reject the teleology that is common in development thinking. A teleology is the notion that things are growing toward a set, fixed point. So, you might think that the telos of an acorn is to become an oak tree. Similarly, we often have it in our heads that the telos of states in developing countries is to become like those in developed ones — that the destiny, in the fullness of time and under favorable conditions — of the government of Burkina Faso is to become like that of the United States or Denmark. Matt, Lant, and Michael reject this view. Instead, like the families in Tolstoy’s Anna Karenina every unhappy government of a developing country is unhappy after its own fashion, and must find its own way out and perhaps even its own end point, which may not look at all like the developed states with which we are familiar.

Second, and relatedly, they break with the common wisdom that there are generalizable best practices — say in education, health delivery, anti-corruption, or participation  — much less best practices in the design of whole government institutions. In the attempt to adopt so-called best practices, countries engage in “isomorphic mimicry” so that their organizations assume the form of high-functioning governments with out actually functioning well. In chasing after those forms, governments and the donor and development agencies that offer best practices impose expectations that these governments cannot meet. The government organizations often stagnate or fail under such pressures. Best-practices thus creat “capability traps” that lock developing country states in to failure rather than facilitating their development.

This is a radical proposition. Much of what social science and development studies has been about is using qualitative or quantitative data, or techniques like randomized control trials, to identify best practices that developing country governments might adopt in order to, well, develop.

Whereas many research projects would be content to diagnose this major problem in development practice, this is just the first Act of Building State Capacity. Matt, Lant, and Michael go on to offer a new paradigm that they call Problem-Driven Iterated Adaptation (PDIA). If the intellectual ancestor of the existing paradigm is Max Weber, John Dewey — the most important philosopher of pragmatism and a great philosopher of education — is the progenitor of PDIA.

In reading the book, education is the analogy that came to my mind. Think of a grade school classroom or indeed any MPP/MPAID class. Each student faces different challenges to mastering capabilities like reading, communication, and numeracy. Assignments that are too difficult for any student produce frustration (the capability trap), while those they do not learn from assignments that are too easy (stagnation rather than development). Education requires exercises and problems that fall within each students “proximal zone of development” — each student ought to be challenged just enough to grow their mental muscles.

Poor teachers treat the whole class as if each student were exactly like all the rest (that’s the best practice approach). Effective teachers know that each student is different and they know how to create challenges that fall within each student’s “proximal zone of development.”

In Problem Driven Iterated Adaptation, government organizations ignore the best practice nostrums of development experts from Harvard and the World Bank. Instead, teams in those governments focus on particular problems — say of service delivery, public goods production, regulation, policy-making or other implementation — develop their own solutions to those problems, and seek to “fail forward” by understanding the errors in their own solutions, learning by iterating and adapting. Seeking out and solving problems in this way keeps government organizations in their “proximal zone of development” — building their “state capacity” muscles at an appropriate level of challenge that results in neither stagnation or over-use injuries.

The book is full of examples, showing both the catastrophic failures of the best practice / teleological approaches to development and the successes — from their own development practice, of the PDIA approach.

These paradigm breaking and paradigm making arguments are the book’s conceptual contribution. But understanding PDIA and being able to do PDIA are not the same thing. True to the book’s relentlessly practical orientation, the second part of the book is aimed squarely at development practitioners. It provides a wealth of diagnostic tools — even worksheets — that practitioners can use to begin to understand the most promising points of development intervention in their own contexts, how do marshall the authority and resources they will need to shift their organizations from best-practice approaches to PDIA, and how to begin to implement PDIA for specific challenges they face.

Though not everyone will embrace the PDIA approach, everyone should read this book. It is the clearest articulation — following in the traditions of Dewey, Lindblom, and other skeptics of synoptic public policy and management — of a pragmatic method of development and public management.

 

Register now for our free PDIA online course

written by Salimah Samji

We are delighted to announce that we will be offering our free PDIA online course once again. This is our third iteration of the course and we will use the recently published “Building State Capability: Evidence, Analysis, Action” book as the core reading.

We will offer two courses tailored to different audiences. Please read the descriptions below to determine which course is the right one for you. You cannot register for both courses.

  • The Practice of PDIA: Building Capability by Delivering Results (February 26- June 18, 2017). This is a 16-week course for practitioners who are in the weeds of development and actually want to learn how to do PDIA. In this course you will have the opportunity to work together as a team, on your nominated problem, using our tools. The course will include video lectures, required reading, assignments, reflection exercises, peer interaction as well as group work. We estimate that the weekly effort will be between 3-5 hours. If you are interested in this course, you will need to identify a problem you want to solve and a team of 4-6 people who will work with you. Enrollment is limited. Registration is closed.
  • Principles of PDIA: Building Capability by Delivering Results (February 26- May 7, 2017). This is a 10-week course for practitioners who are not directly involved in the implementation of programs but are interested in learning about PDIA. The course will include video lectures, required reading, assignments, reflection exercises as well as peer interaction. We estimate that the weekly effort required will be between 3-5 hours. Enrollment is limited. Registration is closed.

Here are some testimonials from students who have completed a similar version of the Practice of PDIA: Building Capability by Delivering Results.

The PDIA program faculty was truly exceptional, not only because of their expertise and individual intellect and knowledge and research, but also because they understand how to engage participants in different ways. If you are concerned about why and how countries are poor or mired in a vicious cycle of underdevelopment; then this course is just want you need to help unravel the answers to your questions and arm up with the principles and know-how to tackle them.” Abdulrauf Aliyu, Head of Business Development and Strategy, Inteliworx Technologies, Nigeria

A couple of years ago I joined the development industry as a program officer for a bilateral aid agency in Tanzania. Three years down the line I was frustrated: our partners in the government were “always committed” but things were not really moving in the way and pace we hoped they would. In short, nothing much was changing. If anyone asked me at the time who is at fault, I would have hastened to say it was the government. Having done the PDIA course, however, I can appreciate better why things were happening the way they were, and our responsibility as staff members of funding agencies in the reform failures. So I am thrilled that it is possible to do development differently, the PDIA way. It does not promise that it will be easier doing development this way, and it might never get any easier; but I believe it offers a better chance of bringing real and lasting change even if it comes slowly.” Rose Aiko, Independent Consultant, Tanzania

The course was terrific from both a theoretical and practical standpoint. I was amazed about how accurately the issues addressed in the course related to my day-to-day experiences working in development. In fact, our work plan for our upcoming technical assistance program is largely based on PDIA!” Team Leader, Asian Development Bank, Dili, Timor-Leste

“The PDIA course has been for me the learning highlight of this year. The course has given me the knowledge of a process and tools that I was looking since traditional approaches to projects with best practices from elsewhere, solution-based, blueprint-based, with fixed plan, aiming always at system change, etc. do not work in most cases. I have now a set of steps and, more importantly, questions that can guide me in the work with colleagues and partners to understand the context in which we try to introduce change, identify concrete problems that people want to solve, and try to solve them, one at a time.” Arnaldo Pellini, Research Fellow, Overseas Development Institute

As a Project Manager and Solutions Consultant in Nigeria, taking “PDIA: Building Capability by Delivering Results” opened paths to new possibilities for finding and fitting solutions that are based on specific contexts and current realities, by working with clients, communities and policy drivers. At the heart of these possibilities is the realization that no matter what the problem is or how complex it seems, we can start acting immediately. Most importantly, the interactions with peers and access to a growing PDIA Community of Practice provide unlimited potentials for the future.” Abubakar Abdullahi, Managing Principal, The Front Office NG, Nigeria

“Having worked in development for 35 years I recommend this course to all development practitioners. PDIA is a detailed process that will facilitate your design and implementation approach. PDIA has several steps. I believe the adoption of either all of these steps or just some selected steps will improve the design and implementation of your projects and programs, with improved benefits and results.”  John Whittle, Semi-retired and Consulting in Central Asia

“Through the modules of PDIA, I have had a mindset change on how development works and how it could work. It is an approach that has opened my eyes to many things that I had previously struggled to understand in my 15 years of development practice, where I have observed vicious cycles of problems like chronic poverty, corruption, and poor service delivery despite heavy investments by donors and recipient governments. I will continue to see my work with a PDIA lens and assess new projects in the same way. It is exciting to try and do things differently in an effort to get different results from the norm.” Cate Najjuma, Economist, Royal Danish Embassy, Kampala

“The PDIA course is perfectly designed for those who are currently trying to address real world issues. It has contributed to increase my value add on reform issues in Tunisia.  The course is very focused and practical, allowing it to fit into the busy schedule of professionals like me and to learn at an impressive pace.  I definitely recommend it to prospective applicants.” Gomez Agou, IMF Desk Economist, Washington DC

“The PDIA course showed how approaching and solving complex and challenging reform efforts are not pinned on rigid, structured frameworks but rather on a common sense approach bottled in a simple method all rooted on the fundamentals of understanding, clarifying, learning, experimenting and adapting.” Abubakar Sadiq Isa, Managing Director, Inteliworx Technologies, Nigeria

“The PDIA course represents an empirical reform prescription in building state capability by delivering results through theoretical and practical approaches geared toward sustained improvement and performance. Tom Tombekai, Liberia

“I enjoyed taking the course PDIA: Building Capacity by Delivering Results. I have been doing development work in Africa in the anti-corruption area. This course introduced me to some new concepts in terms of building acceptance for ideas and programs and especially understanding the environment in terms of what may be possible and how success should be measured. It has has changed how I will approach future development problems. I very much enjoyed the readings, lectures and interactions with other students from around the world.” Craig Hannaford, Independent Consultant, Canada

“I have also been taught that every problem has got a series of causes and sub-causes. You really have to be very critical in analyzing a problem in order to address it effectively. This is one of the products of PDIA. I find myself thinking outside the box when I have to solve a problem whether in the office, with vendors or even at home. It is in this course that I first heard “deconstruction of a problem”. Deconstruction and sequencing work has helped me to foster actions to solve a problem. Ultimately, through this course PDIA, I have learnt that in the development sector, before bringing solutions to the government, I have to understand the existing practice, positive deviance, latent practice and external best practice. Without this course, I would not be an improved reformer.” Doris Ahuchama, Finance and Administration Manager, Nigeria

 

PDIA Course: Alumni are already practicing what they learned

written by Salimah Samji

We offered 4 free PDIA online courses between November 2015 and June 2016. They were well received and 365 people, living in 56 countries, successfully completed the courses.

pdia-course-one-pager

In January 2017, we surveyed the 365 PDIA course alumni to learn whether (and how) they are using PDIA in their day-to-day lives. 113 (31%) of them, living in 36 countries, responded to the survey. This includes people who work for donors, governments, consulting firms, private sector firms and NGOs.

Here’s what we found:

  • 96% of the respondents have used the key concepts, ideas, and tools.
  • 91% have shared the ideas, concepts, and tools with others. They have shared with co-workers, bosses, and friends; led study and discussion groups;  conducted workshops and trainings; and one organization used the content to train others at an annual retreat!
  • 85% have achieved something by doing PDIA. 

The findings and concrete examples that were shared in the survey have been awe inspiring. People learned the key ideas/concepts/tools we taught, are using them in their work, and are teaching others.

We plan to offer another round of free PDIA online courses soon – stay tuned!


Here are some of the things the course alumni had to say.

I think just appreciating a more building block approach to issues has offered more practical and realistic ways of working.  It has meant accepting that progress may be slower than desired but likely to be more sustainable, because you are starting at the root of the problem and you are working with the grain of political support. – DFID Governance Adviser, Nigeria

The PDIA helped transformed the way I see development administration and governance. I now use a systems thinking frame of mind to see problems and not just throw solutions at them. As professor Clayton Christensen will say, “WHAT IS THE JOB TO BE DONE?” No matter how elegant or beautiful an introduced solution is, if it does not solve people’s problem then it is useless. – Head of business development, Inteliworx Technologies, Nigeria

Before the course, I was approaching problems (ie. corruption) as a large problem to be solved with a complex approach. PDIA taught me to look at the complexities of the problem, the different interests and barriers and how to focus efforts on areas that might actually be amenable to incremental change.  I learned that any program must assess the environment and devote resources where they will be effective.  Analysis of the problem, players and barriers is key before expending resources. – Development Consultant based in Canada

The PDIA course offered some variation in how to think through and act on development problems. As I said in my summing up of the source it is an approach that can be either used in full or parts of it can be merged in with other approaches depending on the context in which one is working/consultingDevelopment Practitioner based in Australia

Download the new PDIA book for free

written by Salimah Samji

We are delighted to inform you that our PDIA book entitled, “Building State Capability: Evidence, Analysis, Action” was just published by Oxford University Press. The book presents an evidence-based analysis of development failures and explains how capability traps emerge and persist. It is not just a critique, it also offers a way of doing things differently. It provides you with the tools you need to personalize and apply these new ideas to your own context.

Here is a review written by Francis Fukuyama

“Building State Capability provides anyone interested in promoting development with practical advice on how to proceed—not by copying imported theoretical models, but through an iterative learning process that takes into account the messy reality of the society in question. The authors draw on their collective years of real-world experience as well as abundant data and get to what is truly the essence of the development problem.”

In keeping with our commitment to provide free resources to help diffuse our PDIA approach to practitioners around the world, we have enabled an open access title under a Creative Commons license (CC BY-NC-ND 4.0). We hope you find the book useful and that it helps create a PDIA community of change that shares, learns and grows together. Visit the book webpage to download your free copy. Please share your thoughts on social media using the hashtag #PDIABook

Listen to what the authors have to say about the book:

 

Best Practice is a Pipe Dream: The AK47 vs M16 debate and development practice

written by Lant Pritchett

At a recent holiday party I was discussing organizations and innovations with a friend of mine who teaches at the Harvard Business School about organizations and is a professor and student about technology and history.  I told him I was thinking about the lessons for the development “best practice” mantra from the AK47 versus M16 debate.  Naturally, he brought out his own versions of both for comparison, the early Colt AR-15 developed for the US Air Force (which became the M16) and an East German produced AK-47.

screen-shot-2017-01-10-at-2-10-43-pm

Development practice can learn from the AK47.  It is far and away the most widely available and used assault rifle in the world.  This is in spite of the fact that it is easy to argue that the M16 is the “best practice” assault rifle.  A key question for armies is whether in practice it is better to adopt the weapon to the soldiers you have or train the soldiers you have to the weapon you want.  The fundamental AK47 design principle is simplicity which leads to robustness in operation and effective use even by poorly trained combatants in actual combat conditions.  In contrast, the M16 is a better weapon on many dimensions—including accuracy–but only works well when used and cared for by highly trained and capable soldiers.

One important criterion for any weapon is accuracy.  In the 1980s the US military compared the AK47 versus the M16 for accuracy at various distances in proving ground conditions that isolated pure weapon accuracy.  The following chart shows the single shot probabilities of hitting a standard silhouette target at various distances in proving ground conditions.  It would be easy to use this chart to argue that the M16 is a “best practice” weapon as at middle to long distances the single shot hit probability is 20 percent higher.

Figure 1:  At proving ground conditions the AK47 is a less accurate weapon than the M16A1 at distances above 200 yards

screen-shot-2017-01-10-at-2-11-05-pmSource:  Table 4.3, Weaver 1990.

The study though also estimates the probability of hitting a target when there are aiming errors of an actual user of the weapon.  In “rifle qualifying” conditions the shooter is under no time or other stress in shooting and knows the distance to target and hence ideal conditions for shooter to demonstrate high capacity.  In “worst field experience” conditions the shooter is under high or combat-like stress, although obviously these data are from simulations of stress as it is impossible to collect reliable data from actual combat.

It is obvious in Figure 2 that over most of range at which assault rifles are used in combat essentially all of the likelihood of missing the target comes from shooter performance and almost none from the intrinsic accuracy of the weapon.  The M16 maintains a proving ground conditions hit probability of 98 percent out to 400 yards but at 400 yards even a trained marksman in zero stress conditions has only a 35 percent chance and under stress this is only 7 percent.

Figure 2:  The intrinsic accuracy of the weapon as assessed on the proving is not a significant constraint to shooter accuracy under high stress conditions of shooting

screen-shot-2017-01-10-at-2-14-11-pm

Source:  Table 4.2, Weaver 1990.

At 200 yards we can decompose the difference from the ideal conditions of “best practice”–the M16 on the proving ground has 100 percent hit probability—and the contribution of a less accurate weapon, user capacity even in ideal conditions, and user performance under stress.  The AK47 is 99 percent accurate, but in in rifle qualifying conditions the hit probability is only 64 percent with the M16 and in stressed situations only 12 percent with the M16.  So if a shooter misses with an AK47 at 200 yards in combat conditions it is almost certainly due to the user and not the weapon. As the author puts it (it what appears to be military use of irony) while there are demonstrable differences in weapon accuracy they are not really an issue in actual use conditions by actual soldiers:

It is not unusual for differences to be found in the intrinsic, technical performance of different weapons measured at a proving ground.  It is almost certain that these differences will not have any operational significance.  It is true, however that the differences in the…rifles shown…are large enough to be a concern to a championship caliber, competition shooter.

Figure 3:  Decomposing the probability of a miss into that due to weapon accuracy (M16 vs AK47), user capacity in ideal conditions, and operational stress

screen-shot-2017-01-10-at-2-15-25-pm

Source:  Figure 2 above, Weaver 1990.

The AK-47’s limitations in intrinsic accuracy appear to be a technological trade-off and an irremediable consequence of the commitment to design simplicity and operational robustness   The design of the weapon has very loose tolerances which means that the gun can be abused in a variety of ways and not properly maintained and yet still fire with high reliability but this does limit accuracy (although the design successor to the AK-47, the currently adopted AK-74 did address accuracy issues).  But a weapon that fires always has higher combat effectiveness than a weapon that doesn’t.

While many would argue that the M16 in the hands of a highly trained professional soldier is a superior weapon, this does require training and adapting the soldier and his practices to the weapon.  The entire philosophy of the AK-47 is to design and adapt the weapon to soldiers who likely have little or no formal education and who are expected to be conscripted and put into battle with little training.  While it is impossible to separate out geopolitics from weapon choice, estimates are that 106 countries’ military or special forces use the AK-47—not to mention its widespread use by informal armed groups—which is a testament to its being adapted to the needs and capabilities of the user.

Application of ideas to basic education in Africa

Now it might seem odd, or even insensitive, to use the analogy of weapon choice to discuss development practice, but the relative importance of (a) latest “best practice” technology or program design or policy versus (b) user capacity versus (c) actual user performance under real world stress as avenues for performance improvement arises again and again in multiple contexts.  There is a powerful, often overwhelming, temptation for experts from developed countries to market the latest of what they know and do as “best practice” in their own conditions without adequate consideration of whether this is actually addressing actual performance in context.

The latest Service Delivery Indicators data that the World Bank has created for several countries in Sub-Saharan Africa illustrate these same issues in basic education.

The first issue is “user capacity in ideal conditions”—that is, do teachers actually know the material they are supposed to teach?  The grade 4 teachers were administered questions from the grade 4 curriculum.  On average only 12.7 percent of teachers scored above 80 percent correct (and this is biased upward by Kenya’s 34 percent as four of six countries’ teachers were at 10 percent or below).  In Mozambique only 65 percent of mathematics teachers could do double digit subtraction with whole numbers (e.g. 86-55) and only 39 percent do subtraction with decimals—and less than 1 percent of teachers scored above 80 percent.

Figure 4:  Teachers in many African countries do not master the material they are intended to teach—only 13 percent of grade 4 teachers score above 80 percent on the grade 4 subject matter

screen-shot-2017-01-10-at-2-15-54-pm

Source:  Filmer (2015) at  RISE conference.

A comparison of the STEP assessment with the PIAAC assessment of literacy in the OPECD found that the typical tertiary graduate in Ghana or Kenya has lower literacy proficiency then the typical OECD adult who did not finish high school.  A comparison of the performance on TIMSS items finds that teachers in African countries like Malawi and Zambia score about the same as grade 7 and 8 students in typical OECD countries like Belgium.

So, even in ideal conditions in which teachers were present and operating at their maximum capacity their performance would be limited by the fact that they themselves do not fully master the subject matters at the level they are intended to teach it.

The second issue is the performance under “operational stress”—which includes both the stresses of life that might lead teachers to not even reach school on any given day as well as the administrative and other stresses that might lead teachers to do less than their ideal capacity.  The Service Delivery Indicators measure actual time teaching versus the deficits due to absence from the school and lack of presence in the classroom when at the school.  The finding is that while the “ideal” teaching/learning time per day is five and a half hours students are actually only exposed to about 3 hours a day of teaching/learning time on average.  In Mozambique the learning time was only an hour and forty minutes a day rather than the official (notional) instructional time of four hours and seventeen minutes.

On top of this pure absence the question is whether under the actual pressure and stress of classrooms even the teaching/learning time is spent at the maximum of the teacher’s subject matter and pedagogical practice capacity.

Figure 5:  Actual teaching/learning time is reduced dramatically by teacher absence from school and classroom

screen-shot-2017-01-10-at-2-18-15-pm

Source:  Filmer (2015) at RISE conference

The “global best practice” versus performance priority mismatch

The range in public sector organizational performance and outcomes across the countries of the world is vast in nearly every domain—education, health, policing, tax collection, environmental regulation (and yes, military).  In some very simple “logistical” tasks there has been convergence (e.g. vaccination rates and vaccination efficacy are very high even in many very low capability countries) but the gap in more “implementation intensive” tasks is enormous.  Measures of early grade child mastery of reading range from almost universal—only 1 percent of Philippine 3rd graders cannot read a single word of linked text whereas 70 percent of Ugandan 3rd graders cannot read at all.

This means that in high performing systems the research questions are pushed to the frontiers of “best practice” both in technology and the applied research of management and operations.  There is no research or application of knowledge in improving performance in tasks that are done well and routinely in actual operational conditions by most or nearly all service providers.  That is taken for granted and not a subject of active interest.  There is research interest in improving the frontier of possibility and interest in practical research into how to increase the capacity of the typical service provider and their performance under actual stressed conditions—but in high performing systems these are both aimed at expanding the frontier of actual and achieved practice in the more difficult tasks.  This learning may be completely irrelevant to what is the priority in low performing systems.  Worse, attempts to transplant “best practice” in technology or organizations or capacity building that is a mismatch for actual capacity or cannot be implemented in the current conditions may lead to distracting national elites from their own conditions and priorities.

What are the lessons of the “best practice” successes of the Finnish schooling system for  Pakistan or Indonesia or Peru?  What are the lessons of  Norway’s “best practice” oil revenue stabilization fund for Nigeria or South Sudan?  What are the lessons of OECD “best practice” for budget and public financial management for Uganda or Nepal?  I am confident there are interesting and relevant lessons to learn, but the experience of the AK-47 should give some pause as to whether a globally relevant “best practice” isn’t a pipe dream.

Figure 6:  Potential mismatch of global “best practice” and research performance priorities in low performance systems.

screen-shot-2017-01-10-at-2-24-01-pm