You cannot Juggle without the Struggle: How the USA historically avoided the “Tyranny of Experts”

written by Lant Pritchett

The period between the end of the American Civil War and the end of World War II saw a transformation of America with the rise of dominant large organizations in both the private economy and public life. The economic historian Alfred Chandler’s in The Visible Hand and Scale and Scope documents the rise of “managerial capitalism” with large economic bureaucracies in the railroads, oil, steel, automobiles, electricity, and telecommunications establishing new foundations of a productive economy.

Similarly, in nearly every domain of engagement the organization of the state became more centralized, more bureaucratic, more centrally controlled. Or, as this is typically described, the “Progressive Agenda” caused governance to become more “professional” more “scientific” more “efficient” as bureaucratic hierarchies displaced localism and Ostrom-esque “polycentric” systems.   Historians of the US tell this story about many fields and organizations.

Samuel Hays Conservation And The Gospel Of Efficiency: The Progressive Conservation Movement, 1890-1920 is summarized as: “Against a background of rivers, forests, ranges, and public lands, this book defines two conflicting political processes: the demand for an integrated, controlled development guided by an elite group of scientists and technicians and the demand for a looser system allowing grassroots impulses to have a voice through elected government representatives.” In the end the “modern” organizations of management of publicly owned lands—but only after having to struggle to prove the improved effectiveness of their methods against continued powerful opposition.

Daniel Carpenter’s The Forging of Bureaucratic Autonomy: Reputations, Networks, and Policy Innovation in Executive Agencies, 1862-1928 narrates the rise of (among other organizations) the “modern” Post Office as a centrally controlled civil service bureaucracy. It had to struggle itself into control of the post against powerful political forces and local resistance that supported the former Jacksonian system of locally appointed postmasters.

Education historian David Tyack’s The One Best System: History of American Urban Education tells of the rise of the urban school system as a “modern” and “scientific” organization that struggled—with mixed success—to consolidate and control the myriad of locally controlled schools and school districts. American began the century with 150,000 school districts, down to around 15,000 today.

Hernando de Soto, though not an academic historian like the others, tells “The Missing Lessons of US History” in chapter 5 of his Mystery of Capital. He shows that property rights in the US as the country expanded westward were a constant struggle between local systems of de facto recognition of use and attempts at de jure top-down ordered and rational systems. In this struggle the de facto usually won politically and the law changed to acknowledge the reality, rather than vice versa.

In each of these cases narratives of “scientific” and “efficiency” and “rational” and “ordered” and “modern” were used to justify placing ever greater power into the hands of centralized organizations. But these organizations had to struggle against counter-claims of people and grass-roots movements and local politics who knew they were losing power. Westerners strongly resisted the claims of the “conservation” movement that they should be given consolidate power over land use. Parents resisted being sidelined in their control of the schools.   People with de facto usufruct rights over land resisted the formal legal claims against their control. Local postmasters resisted the increasingly central control of the post office.

In this struggle they used their rights as citizens for a variety of modes expression of opposition and their voice in democratic processes. The result was a messy, protracted, conflicted struggle.

I take one of the main points of Bill Easterly’s new book The Tyranny of Experts: Economists, Dictators, and the Forgotten Rights of the Poor to be that the mainstream “development” process underestimated the necessity of the struggle. That is, it was thought that “modernization” could be achieved as a purely technical exercise in which the demonstrable successful organizations and institutions of the “developed” world could be transplanted to other countries. Why should the newly sovereign country in the 1950s and 1960s “struggle” towards an effective Post Office when there exist working models throughout the world (US Post Office, Royal Mail, Bundespost)? It was believed that “modern” police forces, schools, roads, courts, forest services, water companies—the organizations that make a state capable and deliver what the state promises—could be created without all of the bother of citizens being empowered to not just vote, but also resist, to subvert, to complain, to protest, to organize and agitate.

There are a variety of conjectures as to how and why this possibility of “transplantation” through development could actually, in some instances, make things worse.

One, organizations could gain legitimacy simply from mimicking the forms of rich country organizations without their function. Sociologists of organizations call this “isomorphism” and describe the pernicious impacts of allowing organizations to attract support without having to demonstrate superior functionality.

Two, the availability of resources from development agencies who were more familiar with and expected to see “modern” organizations meant that accountability to citizens could be attenuated.

Three, by changing the nature of the struggle the “experts” were not forced into a process of testing their ideas and notions and ways of “seeing like a state” against direct and immediate feedback—and push back—from local realities. Mistakes of mismatch between what the “experts” recommended and the reality of what could work in the local context could be larger and persist longer when insulated from the test of functionality.

But you cannot juggle without the struggle. The fact that someone else can juggle, and can show you how to juggle, and describe juggling in great detail does not mean that functionality is transferable. By changing the nature of the struggle many developing countries are stuck with state organizations that just cannot juggle.

Rigorous evaluation, but to what end?

written by Salimah Samji

Many development projects fail because of poor design. They have no clear roadmap of how they will get from A to B and therefore no way of knowing whether they are on the right track. However, design alone is not enough for success. In fact, many development projects that are well designed, fail because of last mile implementation problems. These include (but are not limited to) existing capacity constraints, difficulties with finding skilled staff, lack of funds, politics and misaligned incentives.

Spending time, effort and resources to rigorously evaluate projects like these can be a wasteful exercise. It would make more sense to begin by testing the logic of your intervention by using low-cost tools, like process audits or collecting and analyzing meaningful data on outputs and intermediate outcomes, that feed back into the design and implementation thus increasing the possibility of success. Does the project make sense in your context?

Several years ago when I worked for Google.org, I had suggested that a grantee of ours, one who conducted large-scale assessments, consider a process audit to identify gaps and to improve implementation. They were very open to the idea. I wanted to hire an external consultant to ensure impartiality, but also ensure that the consultant was seen as a team player who was hired to help them. The grantee was thrilled with the findings and told me that this simple process audit had been much more useful and cost-effective than the other Randomized Control Trials (RCTs) they had been part of. They invited the consultant to share the findings with their entire team and together brainstormed ways to improve implementation. It marked the beginning of a long collaboration between the two – but that is another story …

Recently, I was pleasantly surprised to read a paper by Diana Epstein and Jacob Alex Klerman entitled, When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model. They argue that a program that cannot achieve the intermediate goals specified by its own logic model, will not have the desired impacts and should therefore not proceed to Rigorous Impact Evaluation (RIE). And the determination of whether a project can achieve its own intermediate goals can be done using conventional process evaluation methods, that is, careful observation of program operation, at low-cost.

They highlight five common forms of failure of programs to satisfy their own logic models. These include:

  • Failure to secure required inputs: ability to establish inter-organization partnerships and to recruit and retain certain types of staff.
  • Low program enrollment: it does not attract the target number of clients/participants because there is no demand.
  • Low program completion rates: initially enroll in the program but do not complete the expected treatment.
  • Low fidelity: the program as implemented falls short of what was envisioned in the logic model.
  • Lack of pre/post improvement: sometimes clients show minimal or no progress on pre/post measures of the intermediate outcome which could be due to history and maturation.

Then, for each form of failure, they discuss how it could have been detected by a process evaluation – through the observation of the operation of a program, an experienced outsider can suggest direct and immediate ways to improve the program’s operation.

They also note, “it seems plausible that lessons learned in the first year or two of operation might lead to program refinements, improved program implementation, and improved client outcomes.” This fits nicely with our paper, It’s All About MeE: Using Structured Experiential Learning (‘e’) to Crawl the Design Space which helps develop a learning strategy that achieves more than monitoring, is cheaper than impact evaluations, has timely dynamic feedback loops built into the project and extends the insights gained into the design and management of development projects.

Can PDIA help to deliver services for the poor?

10 years ago, the World Development Report (WDR) 2004 entitled Making Services Work for Poor People, marked a watershed moment in the development agenda. It recognized that politics and accountability are crucial to improving services. Furthermore, it shifted the focus from measuring inputs to outputs.

Earlier this month, ODI and the World Bank jointly organized a 10th year anniversary conference to celebrate the achievements over the last decade and to discuss what remains to be done. You can browse the multimedia summary of the event.

In the opening plenary, Shanta Devarajan stated that the WDR 2004 changed the nature of the conversation by recognizing that: (i) services fail poor people, (ii) money is not the solution, and (iii) “the solution” is not the solution.

What have we learned?

  • Context Matters: We got better at describing service delivery problems but not at improving services. Ruth Levine in her interview acknowledged that we have learned how to measure how significant the problem is and to unpack the dimensions of service delivery quality. But we have not learned anything generalizable because context matters.
  • Politics Matters: Marta Foresti noted that ‘politics is not just a problem it’s also part of the solution.’ Working around politics rather than with it, does not work.
  • Connections Matter: In her reflections, Leni Wild wrote – we are dealing with systems and networks through which a much wider set of stakeholders are connected. So the nature of the connections matters, in terms of power balances, incentives and norms. This is similar to what Matt Andrews calls multi-agent leadership.
  • Motivating actors to do the right thing is much harder in practice, said Rakesh Rajani in his interview.
  • Individual capacity ≠ organizational capability: Lant Pritchett explained the difference in his interview.

So what will it take to deliver services for the poor?

  • Experimentation: Ruth Levine stressed the need to focus on organizations, individuals embedded in local circumstances/context and enabling local  providers to experiment and learn what works in their context. She added, “it is a long complicated road.”
  • Humility, Curiosity and Openness: Rakesh Rajani stated that we have to be able to know we don’t have all the answers. It is unlikely to work the first time and so we need to have the courage to tweak, listen to others and to learn from failure. Asking questions, trying, iterating, struggling and learning, rather than having solutions, is key.
  • Commitment at every level, political, organizational and individual.
  • Willingness to acknowledge and learn from failure.

These sound a lot like PDIA principles …

Muddling

Image reproduced from a blog on writing and inspiration:  http://inkspirationalmessages.com/2012/02/10371/

The 5 M’s of Development: Mobilizers matter (Part 5 of 5)

written by Matt Andrews

As I reflect on how change happens in development, 5 themes come to mind. I have written about the importance of moments, muddling, the mundane and multiple men and women. In keeping with the ‘m’s’, today I will emphasize the importance of mobilizers.

These are the people who bring multiple men and women together, encourage them to work beyond the mundane, muddle purposively, and take advantage of or create moments for change. They are people who convene small groups of key agents needed to play specific roles (often in teams or in small authorizing groups), or who connect distributed agents to each other (so the agents don’t even need to interact directly), or who motivate people across networks. These mobilizers are the key to effective leadership, if you ask me, because they bring all the different fucntional roles together. Here is an example of conveners and connectors in action, in a simplified version of a recent reform story I was engaged in. It was work in the judicial sector of a country. A donor had been supporting initiatives to introduce a statistical management system to the sector, so that resource allocation decisions could be more evidence based (everyone would know where case loads were high, where judges and prosecutors were present, where buildings were in place, etc.). After five years and millions of dollars no system existed. This was partly because different groups across the sector did not engage in the reform together. The donor had connections to the ministry of justice and although there were overlaps with the Supreme Court and prosecution, there were no direct connections. Thus the reform was not supported by the other agencies. See my diagram … Sorry it is a mess. Image
I started working on the issue, and had a local person work with some of the folks in the ministry of justice and the courts and the prosecution, as a convener (see M1 in the figure below). This person worked with me to hold meetings of people from these different agencies, and in this way created first degree relationships across the agencies that helped foster a common understanding of the problems warranting change and of the potential ways the agencies could work together to foster change. The role of one mobilizer created direct links between key agents and indirect links between all agents in the system. This opened up access to new ideas, functions, contributions, etc. Image
A second type of mobilization was also important, however, and involved another person working as a connector between distributed agents in the ministry of justice, courts and prosecution office (because it is not only important to convene the heads). This person (M2 in the figure below) created relationships between multiple people in the system and allowed them to connect with each other THROUGH him. This connector role ensured that all people in the system had a first or second degree link to other people, which made the network tighter than it was before, opening a path to agreement on reform and enhancing access to talents and ideas needed for reform to work. Image
The reform is still going step by step, but the connections are better than they have ever been. These connections are proving vital to reform and development and are only possible because of the role of mobilizers in the change process.

The 5 M’s of Development: Multiple Men and Women matter (Part 4 of 5)

written by Matt Andrews

As I reflect on how change happens in development, 5 themes come to mind. I have written about the importance of moments, muddling and the mundane. Today I will discuss the fourth one: multiple men and women matter. In my experience, development and governance reform is about people, not as targets of change, but as agents of change.

This is not a surprising observation but is an important one nonetheless, especially when one considers how little attention development initiatives commonly give to the men and women who have to risk and adapt and work to make change happen and ensure change is sustained. Development initiatives tend to emphasize ideas and money much more than people, even though it is the latter that actually come up with ideas, shape ideas to contexts, and use resources to foster change.

When people are considered in development initiatives, it is often with a narrow lens on ‘champions’ or ‘heroes’. That’s not the picture I see as relevant in the research and applied work I have been engaged in. This work shows me that development and change require multiple functions or roles: we need someone to identify problems, someone to identify solutions, someone to provide money, someone to authorize change activities, someone to motivate and inspire, someone to connect distributed agents, someone to convene smaller groups, someone to provide key resources other than money, and someone who can give an implementation perspective (of the implications of proposed change).

For a variety of theoretical reasons I don’t think we will often find these functions played by one person, or organization. My empirical research suggests that this is true in practice. What I see in my studies is that successful reform requires multiple people providing leadership in a coordinated and synergistic way, such that all the different functional roles are played in an orchestra of change (people who are familiar with Lee Kuan Yew’s view of leadership will relate to the idea of the orchestra).:

  • I wrote a paper on leadership in twelve interesting reforms, where I tested whether one person stood out as the major leader. I found that this was not the case at all. Many people were identified as leaders in the cases, all playing different roles in the change process.
  • In a review of 30 cases of successful reform (from Princeton University’s Innovations for Successful Societies repository) I found that an average of 19 agents were mentioned as playing the roles noted. They all took risks and stood out for providing an important part of the change puzzle.

    The research does find that ‘champions’ exist in many cases, however. But being a champion does not mean being multiple people (or wearing multiple hats, or playing multiple functional roles). Instead, my work showed that where a champion exists, he or she plays three specific functional roles: Authorizing, Convening, and Motivating. The champions do not typically play the other roles.

The 5 M’s of Development: Mundane matters (Part 3 of 5)

written by Matt Andrews

As I reflect on how change happens in development, 5 themes come to mind. I wrote about the importance of moments which are vital to foster change in complex contexts, and muddling which is important to find and fit reform and change content that fosters real development. Today I will discuss the third one: mundane.

The mundane matters in development. What I mean is simply that everyday, boring, taken for granted events, pressures, relationships, activities and such have a huge influence on prospects for change and development. We think these things are ordinary, banal, and don’t matter. But actually they dominate time and activity, and are the key to ‘getting things done’ and to prospects for change and development.

If mundane processes and pressures do not foster efficient activity, organizations are likely to be inefficient–there will be loads of meetings and people answering emails and writing papers and filling in time sheets and doing due diligence activities but these mundane activities will not foster effective results. Similarly, if the mundane does not support change then change and development will not happen: people will attend meetings but won’t follow-up with new activities because their time is already spoken for by the mundane.

I have seen this more than ever before in some of my reform experiments in 2013. The trouble they ran into had little to do with a lack of ideas or money. Instead, the challenges were mundane: getting people to ‘do’ new things in already full calendars, and to sit in meetings and engage purposively without looking down at the three cell phones on the table in front of them, and more. In all the experiences I have been part of, change only started when these and other mundane influences were managed or even altered.

The problem is two-fold:

  • First, development is full of mundanity. Governments and development organizations are the ones Andy Partridge (lead singer of the 80s band XTC) was talking about when he wrote: “We’re horrible mundane, aggressively mundane, individuals. We’re the ninjas of the mundane…”
  • Second, the mundane is mundane. What I mean is that most development specialists think it does not matter. “Too boring. Too unimportant. So easy to overcome. Surely not as important as rigorous empirical analysis and fancy new ideas.” But they often find that the mundane crowds out the new activities and empirics–again and again–to limit and undermine development initiatives.

mundane

It would be great to see development experts taking the mundane seriously. I think a strategy to identify, manage and even alter the mundane could be more important than most fancy development strategies. And infinitely more valuable than a fancy ‘Science of Delivery’. We need to rethink the mundane, seeing it as the key to getting things done and the key to change; less ordinary and banal and boring and more central to development.

The 5 M’s of Development: Muddling matters (Part 2 of 5)

written by Matt Andrews

As I reflect on how change happens in development, 5 themes come to mind. I wrote about moments yesterday. Today I will discuss the second one: muddling matters. What I mean is that developing countries need to muddle through if they want to improve governance; there are no quick answers to the complex challenge of governance reform.

However, I don’t mean that countries should be muddled. Muddling through–or purposive muddling as I like to call it–is an active, intentional and focused process that helps countries find and fit solutions that work in their context. I also don’t mean to say that muddling ‘should’ be part of a successful reform process. My research shows that muddling ‘is’ part of successful reform. There is ample evidence that successful change comes about through experimentation with multiple reform options, with an emphasis on solving problems, and a whole lot of iteration and learning. I’ve written a few papers in this regard and will continue to write more.

Many people in the development community tell me that they agree with the idea of muddling (conceptually), but don’t see how it can be done in developing countries or in governments where politicians are looking for solutions and want the solutions ‘yesterday.’ I keep telling these people, that purposive muddling is common, and necessary, and instead of saying ‘it isn’t possible’ we should be exploring the strategies others have adopted to make it possible (and to make it part of the DNA of some organizations).

Muddling

Image reproduced from a blog on writing and inspiration:  http://inkspirationalmessages.com/2012/02/10371/

The 5 M’s of Development: Moments matter (Part 1 of 5)

written by Matt Andrews

As I reflect on how change happens in development, 5 themes come to mind. The first is simple, but is one of the most important observations I continually make when observing successful change that fosters better government and development results: Moments matter.

‘Change events’ happen when contexts become ready for change. That is, when:

  • there is disruption that forces people to accept change,
  • incumbent structures are being questioned,
  • there are viable alternatives that local people are willing to try, and
  • the weight of agency shifts from the old and discredited ways to a search for new ways (that may be untested but promise better solutions to pressing problems).

Interestingly, I find that these moments are not always a product of lucky timing. In fact, I commonly see years of activity and engagement in advance of any ‘moment’ that seems to spawn change. Understanding the moment requires going back in time five or ten years (or longer) and learning about how coalitions were emerging, drawing attention to problems, and experimenting with new ways of doing things. These activities are often at the margin of the story until ‘the moment’ arrives… But without them there would probably have not been a moment at all.

The bottom line is that we should spend more time preparing for moments than we currently do. Moments of readiness matter more than the development solutions we try to stuff into contexts that are not ready for change.

PMRDF: an Innovative State Capability Experiment in India

written by Salimah Samji.

Anyone who has ever worked in India knows how hard it is to implement programs. The sheer size of the country makes it impossible for anyone but the government, who is the only one with infrastructure and reach, to provide public services to its citizens. Currently, every district administers typically 100+ development schemes. Each scheme has its own rules, regulations, reporting, and funding. Districts are continuously burdened with new schemes and more work without the requisite resources. They are often understaffed and have few (if any) skilled functionaries to be able to do their job. The reality is that 90% of the time is spent on routine day-to-day management leaving no time for strategic planning or experimentation.

To address the capacity and implementation gaps in backward districts which are isolated and less developed, the Ministry of Rural Development launched the Prime Minister’s Rural Development Fellowship (PMRDF) in 2012. The aim of the fellowship was to provide catalytic support to the district administration to help improve the delivery of programs and to develop a cadre of committed development professionals. The eligibility criteria was a minimum of a four-year degree from a recognized university. The fellows were to work for 2-3 years under the supervision of the District Collector, and paid INR 75,000 ($1,210) per month with a 10% increment each year subject to performance. That is a very generous amount. The rationale was to attract younger people who were passionate about development and willing to go to remote areas, but could not afford the low wages. In essence this was a way to bring in new blood into the system.

816 applicants were shortlisted from a total of 8,560 applicants for a group discussion and a personal interview. 156 fellows were selected to work across 82 districts in 9 states. Each state decided on the placement of fellows with 1-3 per district. Currently there are 137 fellows from the first batch and a second batch has just been selected. For more details see the TEDx talk.

Some of the challenges faced during the implementation of the 2012-13 PMRDFs include:
  • Building the trust of the States/District Collectors: When the program was initially announced, it was viewed suspiciously by some (i.e. was the center sending monitors to the states/districts?). Many meetings were held to explain and clarify the objectives of the program and to bring the states on board.
  • Ineffective use of the fellows: Depending on the District Collector, some fellows were welcomed and given a lot of responsibility and authority to experiment, while some others were used as glorified executive assistants – which is also understandable given the lack of human resources at the district level.
  • Lack of authority to issue instructions or to sanction funds: The fellows are not part of the government bureaucracy and therefore have no signing authority and cannot issue orders to lower level functionaries. As a result, they need to go to the District Collector with all those requests leading to time delays and inefficiencies. To address this, the state of Andhra Pradesh has delegated a budget head specifically for the fellows.
  • High monthly salary is attracting applicants who are not necessarily passionate about development: This has been an unintended consequence and they have tried to be more mindful in the interviews for the 2nd batch of fellows.
  • What happens after the fellows leave? One criticism is that they are building temporary capacity. To address this, the second batch will be offered one year of public service after their fellowship. However, this issue remains a challenge.
Here are some testimonies of the fellows, for more read their profile book or visit their blog.
  • “I still remember an incident from my initial days, when a village welcomed and garlanded me saying: We are receiving our first government official in our village after Independence.
  • “… working with the field staff in implementation of these schemes was not always easy. My efforts at convincing them and the higher authorities did not yield fruit every time. The convergence of different schemes has also been a challenge as each scheme has different criteria for beneficiary selection and process of implementation. However, struggling with these issues every day helped me understand the minutest details of these schemes and the complexities involved in the development sector.”
  • “This last one year as a PMRDF has helped me develop both personally and professionally. It has given me a chance to work and interact with people at every level of the hierarchy, starting with the panchayat secretary and going up to state officials.”
  • “The last one year has been a period of immense pleasure, satisfaction, frustration, learning and un-learning … There were difficulties: sluggishness, winning the trust of various actors and bringing them onto common platform to get the ball rolling, getting bogged down with too many operational formalities needing to continuously establish yourself within the system. I learnt that things will happen if there is strong will. I learnt of a number of unsung heroes who keep things going—unappreciated and unrecognized—but making a difference each day with their integrity.” This comment is a great illustration of Matt Andrew’s multi-agent leadership paper, Who really Leads Development?

The PMRDF is a work in progress with a lot of learning, iterating and adaptation. There are no easy solutions when you are working on implementation intensive programs. It will be interesting to see what other lessons are learned from this dynamic experiment of trying to solve capacity issues at the last mile of service delivery.

Rigorous Evidence Isn’t

written by Lant Pritchett

Currently, there are many statements floating around in development about the use of “rigorous evidence” in formulating policies and programs. Nearly all of these claims are fatuous. The problem is, rigorous evidence isn’t.

That is, suppose one generates some evidence about the impact of some programmatic or policy intervention in one particular context that is agreed by all to be “rigorous” because it meets methodological criteria for internal validity of its causal claims. But the instant this evidence is used in formulating policy it isn’t rigorous evidence any more.  Evidence would be “rigorous” about predicting the future impact of the adoption of a policy only if the conditions under which the policy was to be implemented were exactly the same in every relevant dimension as that under which the “rigorous” evidence was generated.  But that can never be so because neither economics—nor any other social science—have theoretically sound and empirically validated invariance laws that specify what “exactly the same” conditions would be.

So most uses of rigorous evidence aren’t.  Take, for instance, the justly famous 2007 JPE paper by Ben Olken on the impact of certain types of monitoring on certain types of corruption. According to Google Scholar as of today, this paper has been cited 637 times.  The question is, for how many of the uses of this “rigorous evidence” is it really “rigorous evidence”?  We (well, my assistant) sampled 50 of the citing papers with 57 unique mentions of Olken (2007).  Only 8 of those papers were about Indonesia (Of course even those 8 are only even arguably “rigorous” applications as they might be about different programs or different mechanisms or different contexts.)  47 of the 57 (82%) of the mentions are neither about Indonesia nor even an East Asia or Pacific country—they might be a review of the literature about corruption in general, about another country, or methodological.  We also tracked whether the words “context” or “external validity” appeared within +/- two paragraphs of the mention. In 34 of the 57 (60%) mentions, the evidence was not about Indonesia and did not mention that the results, while “rigorous” for the time, place and programmatic/policy context, have no claim to be rigorous about any other time, place, or programmatic/policy context.

Another justly famous paper, Angrist and Lavy (1999) in the QJE uses regression discontinuity to identify the impact of class size on student achievement in Israel.  This paper has been cited 1244 times.  I looked through the first 150 citations to this paper (which Google Scholar sorts by the number of times the citing paper has itself been cited) and (other than other papers by the authors) not one mentioned Israel  (not that surprisingly, as Israel is a tiny country) in the title or abstract while China, India, Bangladesh, Cambodia, Bolivia, UK, Wales, USA (various states and cities), Kenya and South Africa all figured.  Angrist and Lavy do not, and do not claim to, provide “rigorous” evidence about any of those contexts.

If one is formulating policies or programs for attacking corruption in highway procurement in Peru or reducing class size in secondary school in Thailand, it is impossible to base those policies on “rigorous evidence” as evidence that is rigorous for Indonesia or Israel isn’t rigorous for these other countries.

Now, some might make the argument that formulation of policies or programs in context X should rely exclusively/primarily/preferentially on evidence that is “rigorous” in context Z because at least we know that in context Z in which it was generated the evidence is internally valid.  This is both fatuous and false as a general proposition.

Fatuous in that no one understands the phrase “policy based on rigorous evidence” to mean “policy based on evidence that isn’t rigorous with respect to the actual policy context to which it is being applied (because there are no rigorous claims to external validity) but rather based on evidence that is rigorous in some other context.”  No one understands it that way because that isn’t rigorous evidence.

It is also false as a general proposition.  It is easy to construct plausible empirical examples in which the evidence suggests that the bias from internal validity is much (much) smaller than the bias from external validity as the contextual variation in “true” impact is much larger than the methodological bias from lack of “clean” causal identification of simple methods.  In these instances, better policy is made using “bad” (e.g. not internally valid) evidence from the same context than “rigorous” evidence from another context (e.g. Pritchett and Sandefur 2013).

Sadly perhaps, there is no shortcut around using judgment and wisdom in assessing all of the available evidence in formulating policies and programs.  Slogans like “rigorous evidence” are an abuse, not a use, of social science.