Guest blog written by George Adam Sukoco Sikatan, Lanny Octavia, Sarah Ayu, Wahyu Setioko
This is a team of development practitioners who work for INOVASI and DFAT in Indonesia. They successfully completed the 15-week Practice of PDIA online course that ended in May 2019. This is their story.
It is at last the final week of the course, and we say this full of gratitude and relief. None of us had anticipated just how intense and demanding this course was going to be, from the essential (and optional) reading, individual and groups assignments, to reflection exercises and graded discussions; needless to say they were onerous! At the same time, the abundance of knowledge was exciting and overwhelming.
Working in the public/development sector, in a large, populous country such as Indonesia, the 4 of us often come across bewildering, deeply rooted problems that seem just impossible to resolve. The PDIA approach shines a positive light on this situation and more importantly, confidence to overcome them. We learned to deconstruct a problem into smaller pieces and find the root cause using a relatively simple, yet powerful, tool namely the 3A analysis (Authority, Acceptance and Ability).
Another key takeaway from our group is the importance of reflective process to help us look into failures, challenges and feedback as opportunity to grow and construct (or when necessary, deconstruct all over). This methodology taught us to become better listeners, to arrive in a situation with an open mind instead of a will to impose external practices. These reflections and adaptations to the local context, allow us to remain relevant both to the problem that we are trying to solve and towards our beneficiaries.
This course also reminded us of the importance of collaboration and coordination with a broad range of stakeholders. We understand now that multiple perspectives, incentives and even interests are actually useful in defining problems and formulating solutions. Sharing a common goal at the beginning of the work had founded a sense of belonging and motivation for all team members, even when the time is hard and problem becomes more challenging.
Guest blog written by Rina Arlianti, Stephanie Carter, Murni Hoeng, Siti Ubaidah Idrus, Susanti Sufyadi, Aaron W Watson.
This is a team of six development practitioners working for Indonesia’s Ministry of Education and Culture, the Tanoto Foundation, the Australian supported INOVASI program, and Australian Embassy, Jakarta. They successfully completed the 15-week Practice of PDIA online course that ended in May 2019. This is their story.
The Harvard BSC’s PDIA course has been an exciting journey for all of us. We began the course full of excitement and hope – with most of group members having not met before. We were one of four groups participating from Indonesia, all focused on the issue of education quality. Over the course of 15 weeks, we navigated the twists and turns of the PDIA process, putting key concepts to the test in the field of basic education in Indonesia.
Early on, once we had settled into our group dynamic, we settled on our problem statement:
Learning outcome quality in Indonesian primary schools is still low (low scores in international standardised student tests)
As we progressed, we gained several key insights and takeaways about our problem and the course. Through group discussion and debate, drawing on perspectives from working both within and outside the government system, we settled on the following six key sub-causes for low learning outcome quality in Indonesian primary schools:
Measures of learning are weak (including the use of formative assessment, due to low teacher knowledge)
Teaching/learning process is ineffective (with teachers lacking inadequate skills and knowledge of how to use learning media, to increase student engagement)
Parents are already satisfied with the status quo (there is often low demand for changes to the system, as parents do not know what good teaching looks like)
Lack of learning books for children (due to cumbersome book supply processes at the national level)
Many teachers don’t use digital technology in classrooms (creating missed opportunity for enhanced learning)
Policies that address education quality are not implemented well (and instead focus on physical infrastructure, or if they do exist, are not socialised well in a decentralised system)
Guest blog written by Nikhilesh Hari, Poona Verma, Sadashiv N., Vijay Siddharth Pillai
This is a team of four development practitioners working for the PMRDF in India and an M.Phil student in the UK. They successfully completed the 15-week Practice of PDIA online course that ended in May 2019. This is their story.
We began the course with a feeling that the approach which we are going to learn is going to be unique. As we progressed through the initial weeks, we realized that it’s a common sensical approach to solve problems. However we realized that the common sensical approach is rarely followed. We also realized while operationalizing the approach that it is not easy at all and requires a lot of perseverance.
Cara Myers is the Co-founder and Executive Director of the Mozambique School Lunch Initiative (MSLI). She learned about the PDIA approach by taking two courses at the Harvard Kennedy School as part of her Master’s in Public Administration in International Development (MPA/ID) program. She then began applying more of the concepts directly with the MSLI team. This is her PDIA story.
It was March of 2016 and the rains had completely failed for a second year in southern Mozambique. Farming families had no crops. Children were missing school to dig up river roots to eat. Teachers were sending students home because they were “too hungry to learn anything.” Even in normal years, child malnutrition and poor school participation are major issues in Mozambique. This is one of those big, complex problems that is caused by a myriad of interrelated causes and sub-causes that are difficult to disentangle and prioritize.
So there we were, myself, Talvina Ualane and Roberto Mutisse, all of us former colleagues who had worked together for a disaster relief nongovernmental organization in Mozambique in the past and felt deeply motivated to do something to help people affected by this crisis. But, where did we even begin?
We started with what we could do. This is one of the key aspects of the triple-A framework used in PDIA, which stresses that the space for change must include three key factors: authority, acceptance, and ability. PDIA also emphasizes moving to action quickly rather than taking a long time to try and plan everything out before starting to work. By deconstructing the problem into small, manageable bits, it creates points of entry whereby you can start addressing one of the causes or sub-causes of the problem and build the capacity to do more from there.
At a recent holiday party I was discussing organizations and innovations with a friend of mine who teaches at the Harvard Business School about organizations and is a professor and student about technology and history. I told him I was thinking about the lessons for the development “best practice” mantra from the AK47 versus M16 debate. Naturally, he brought out his own versions of both for comparison, the early Colt AR-15 developed for the US Air Force (which became the M16) and an East German produced AK-47.
Development practice can learn from the AK47. It is far and away the most widely available and used assault rifle in the world. This is in spite of the fact that it is easy to argue that the M16 is the “best practice” assault rifle. A key question for armies is whether in practice it is better to adopt the weapon to the soldiers you have or train the soldiers you have to the weapon you want. The fundamental AK47 design principle is simplicity which leads to robustness in operation and effective use even by poorly trained combatants in actual combat conditions. In contrast, the M16 is a better weapon on many dimensions—including accuracy–but only works well when used and cared for by highly trained and capable soldiers.
One important criterion for any weapon is accuracy. In the 1980s the US military compared the AK47 versus the M16 for accuracy at various distances in proving ground conditions that isolated pure weapon accuracy. The following chart shows the single shot probabilities of hitting a standard silhouette target at various distances in proving ground conditions. It would be easy to use this chart to argue that the M16 is a “best practice” weapon as at middle to long distances the single shot hit probability is 20 percent higher.
Figure 1: At proving ground conditions the AK47 is a less accurate weapon than the M16A1 at distances above 200 yards
Source: Table 4.3, Weaver 1990.
The study though also estimates the probability of hitting a target when there are aiming errors of an actual user of the weapon. In “rifle qualifying” conditions the shooter is under no time or other stress in shooting and knows the distance to target and hence ideal conditions for shooter to demonstrate high capacity. In “worst field experience” conditions the shooter is under high or combat-like stress, although obviously these data are from simulations of stress as it is impossible to collect reliable data from actual combat.
It is obvious in Figure 2 that over most of range at which assault rifles are used in combat essentially all of the likelihood of missing the target comes from shooter performance and almost none from the intrinsic accuracy of the weapon. The M16 maintains a proving ground conditions hit probability of 98 percent out to 400 yards but at 400 yards even a trained marksman in zero stress conditions has only a 35 percent chance and under stress this is only 7 percent.
Figure 2: The intrinsic accuracy of the weapon as assessed on the proving is not a significant constraint to shooter accuracy under high stress conditions of shooting
Source: Table 4.2, Weaver 1990.
At 200 yards we can decompose the difference from the ideal conditions of “best practice”–the M16 on the proving ground has 100 percent hit probability—and the contribution of a less accurate weapon, user capacity even in ideal conditions, and user performance under stress. The AK47 is 99 percent accurate, but in in rifle qualifying conditions the hit probability is only 64 percent with the M16 and in stressed situations only 12 percent with the M16. So if a shooter misses with an AK47 at 200 yards in combat conditions it is almost certainly due to the user and not the weapon. As the author puts it (it what appears to be military use of irony) while there are demonstrable differences in weapon accuracy they are not really an issue in actual use conditions by actual soldiers:
It is not unusual for differences to be found in the intrinsic, technical performance of different weapons measured at a proving ground. It is almost certain that these differences will not have any operational significance. It is true, however that the differences in the…rifles shown…are large enough to be a concern to a championship caliber, competition shooter.
Figure 3: Decomposing the probability of a miss into that due to weapon accuracy (M16 vs AK47), user capacity in ideal conditions, and operational stress
Source: Figure 2 above, Weaver 1990.
The AK-47’s limitations in intrinsic accuracy appear to be a technological trade-off and an irremediable consequence of the commitment to design simplicity and operational robustness The design of the weapon has very loose tolerances which means that the gun can be abused in a variety of ways and not properly maintained and yet still fire with high reliability but this does limit accuracy (although the design successor to the AK-47, the currently adopted AK-74 did address accuracy issues). But a weapon that fires always has higher combat effectiveness than a weapon that doesn’t.
While many would argue that the M16 in the hands of a highly trained professional soldier is a superior weapon, this does require training and adapting the soldier and his practices to the weapon. The entire philosophy of the AK-47 is to design and adapt the weapon to soldiers who likely have little or no formal education and who are expected to be conscripted and put into battle with little training. While it is impossible to separate out geopolitics from weapon choice, estimates are that 106 countries’ military or special forces use the AK-47—not to mention its widespread use by informal armed groups—which is a testament to its being adapted to the needs and capabilities of the user.
Application of ideas to basic education in Africa
Now it might seem odd, or even insensitive, to use the analogy of weapon choice to discuss development practice, but the relative importance of (a) latest “best practice” technology or program design or policy versus (b) user capacity versus (c) actual user performance under real world stress as avenues for performance improvement arises again and again in multiple contexts. There is a powerful, often overwhelming, temptation for experts from developed countries to market the latest of what they know and do as “best practice” in their own conditions without adequate consideration of whether this is actually addressing actual performance in context.
The latest Service Delivery Indicators data that the World Bank has created for several countries in Sub-Saharan Africa illustrate these same issues in basic education.
The first issue is “user capacity in ideal conditions”—that is, do teachers actually know the material they are supposed to teach? The grade 4 teachers were administered questions from the grade 4 curriculum. On average only 12.7 percent of teachers scored above 80 percent correct (and this is biased upward by Kenya’s 34 percent as four of six countries’ teachers were at 10 percent or below). In Mozambique only 65 percent of mathematics teachers could do double digit subtraction with whole numbers (e.g. 86-55) and only 39 percent do subtraction with decimals—and less than 1 percent of teachers scored above 80 percent.
Figure 4: Teachers in many African countries do not master the material they are intended to teach—only 13 percent of grade 4 teachers score above 80 percent on the grade 4 subject matter
A comparison of the STEP assessment with the PIAAC assessment of literacy in the OPECD found that the typical tertiary graduate in Ghana or Kenya has lower literacy proficiency then the typical OECD adult who did not finish high school. A comparison of the performance on TIMSS items finds that teachers in African countries like Malawi and Zambia score about the same as grade 7 and 8 students in typical OECD countries like Belgium.
So, even in ideal conditions in which teachers were present and operating at their maximum capacity their performance would be limited by the fact that they themselves do not fully master the subject matters at the level they are intended to teach it.
The second issue is the performance under “operational stress”—which includes both the stresses of life that might lead teachers to not even reach school on any given day as well as the administrative and other stresses that might lead teachers to do less than their ideal capacity. The Service Delivery Indicators measure actual time teaching versus the deficits due to absence from the school and lack of presence in the classroom when at the school. The finding is that while the “ideal” teaching/learning time per day is five and a half hours students are actually only exposed to about 3 hours a day of teaching/learning time on average. In Mozambique the learning time was only an hour and forty minutes a day rather than the official (notional) instructional time of four hours and seventeen minutes.
On top of this pure absence the question is whether under the actual pressure and stress of classrooms even the teaching/learning time is spent at the maximum of the teacher’s subject matter and pedagogical practice capacity.
Figure 5: Actual teaching/learning time is reduced dramatically by teacher absence from school and classroom
The “global best practice” versus performance priority mismatch
The range in public sector organizational performance and outcomes across the countries of the world is vast in nearly every domain—education, health, policing, tax collection, environmental regulation (and yes, military). In some very simple “logistical” tasks there has been convergence (e.g. vaccination rates and vaccination efficacy are very high even in many very low capability countries) but the gap in more “implementation intensive” tasks is enormous. Measures of early grade child mastery of reading range from almost universal—only 1 percent of Philippine 3rd graders cannot read a single word of linked text whereas 70 percent of Ugandan 3rd graders cannot read at all.
This means that in high performing systems the research questions are pushed to the frontiers of “best practice” both in technology and the applied research of management and operations. There is no research or application of knowledge in improving performance in tasks that are done well and routinely in actual operational conditions by most or nearly all service providers. That is taken for granted and not a subject of active interest. There is research interest in improving the frontier of possibility and interest in practical research into how to increase the capacity of the typical service provider and their performance under actual stressed conditions—but in high performing systems these are both aimed at expanding the frontier of actual and achieved practice in the more difficult tasks. This learning may be completely irrelevant to what is the priority in low performing systems. Worse, attempts to transplant “best practice” in technology or organizations or capacity building that is a mismatch for actual capacity or cannot be implemented in the current conditions may lead to distracting national elites from their own conditions and priorities.
What are the lessons of the “best practice” successes of the Finnish schooling system for Pakistan or Indonesia or Peru? What are the lessons of Norway’s “best practice” oil revenue stabilization fund for Nigeria or South Sudan? What are the lessons of OECD “best practice” for budget and public financial management for Uganda or Nepal? I am confident there are interesting and relevant lessons to learn, but the experience of the AK-47 should give some pause as to whether a globally relevant “best practice” isn’t a pipe dream.
Figure 6: Potential mismatch of global “best practice” and research performance priorities in low performance systems.