There is currently a pervasive emphasis on the use of “rigorous evidence” in shaping policies and programs in the development sector. However, the reality is that rigorous evidence isn’t as straightforward as it seems.

Consider this: suppose we have evidence from a meticulously conducted study that demonstrates the impact of a specific policy intervention in a particular context. While this evidence may be deemed rigorous based on methodological criteria for establishing causal claims, its applicability becomes questionable the moment it is used to inform policy decisions. Truly rigorous evidence would only accurately predict the future impact of a policy if the conditions under which the policy is implemented mirror those under which the evidence was generated. However, such exact replication of conditions is virtually impossible because social sciences like economics lack theoretically sound and empirically validated laws specifying identical conditions across different contexts.

Therefore, in many cases, the use of so-called “rigorous evidence” falls short. For example, let’s examine the renowned 2007 JPE paper by Ben Olken on the impact of monitoring on corruption. Despite its acclaim, a substantial portion of citations of this paper fail to consider its limitations in different contexts. Similarly, another influential paper by Angrist and Lavy (1999) fails to provide rigorous evidence for contexts other than the specific one in which it was conducted.

When formulating policies or programs for different contexts, relying solely on evidence from unrelated contexts is flawed. It’s impractical and misguided to base policies on evidence that lacks external validity for the context in question. While some may argue for prioritizing evidence from one context because of its internal validity, this approach overlooks the nuanced complexities of different contexts and the potential biases introduced by relying on evidence from unrelated settings.

In reality, there’s no substitute for exercising judgment and wisdom when evaluating evidence to inform policy decisions. Slogans like “rigorous evidence” oversimplify the complexities of social science research and can lead to misguided policy choices. Ultimately, effective policymaking requires a nuanced understanding of the available evidence and its applicability to specific contexts.