Can we learn from system-level social protection interventions?

Influencing change at the system level is often very difficult to do, requiring different inputs in different contexts. It is also very difficult to evaluate, as the effects of interventions that seek to make changes at system and policy levels are difficult to disentangle from changes that occur outside the intervention.

An attempt at providing an answer to this methodological challenge is being made in the case of the ‘Adaptive Social Protection’ programme in West Africa.

This rigorous theory-based evaluation, which is one of the first of its kind in the domain of social protection interventions, will provide DFID and the World Bank implementing team with a clearer understanding of the potential and effective contribution of ASP programming in general and the ASPp particular in strengthening the overall capacities of national institutions in the countries where the programme is being implemented.

Adaptive Social Protection (ASP) is a new, integrated approach, with the ambitious objective of reducing the vulnerability of people to the detrimental impacts of disasters, economics shocks and other forms of climate change-related extreme events. To achieve this, ASP seeks to harness and build on the potential synergies to be gained from bringing together social protection, disaster risk reduction and climate change adaptation.

Early examples from different parts of the world have highlighted the benefits of integrating social protection into climate adaptation programmes. There is also increasing evidence that social protection interventions can reduce vulnerability to disasters by enabling adaptive capacity and providing a greater range of livelihood choices. These ideas are at the core of the concept of Adaptive Social Protection.

Recognising these emerging (yet still relatively untested) insights, DFID has recently decided to explore empirically whether the concept of ASP programming is delivering its promises. Can adaptive social protection effectively contribute to strengthening the resilience of the hundreds of millions of vulnerable people that are affected by climate-change-related adverse events in different places in the world?

To answer this question DFID is funding a UK £50 million pilot ASP programme (ASPp) under the wider umbrella of the BRACED programme. The objective of the ASPp, which is implemented by the World Bank in six countries in the Sahel (Burkina Faso, Chad, Mali, Mauritania, Niger and Senegal), is to foster the creation of national level adaptive social protection systems with the ambition to enhance the resilience of poor and vulnerable populations within the course of the 5 years of the programme.

With a strong learning objective, DFID commissioned an independent evaluation team composed of experts from ITAD and CIAT, to conduct a comprehensive evaluation exercise of the ASPp.

The focus of the evaluation is on the 'lesson-learning' elements that can be derived from the ASPp implementation. In particular the team will document the changes occurring at the system level in the policy formulation processes and institutional arrangements in each of the countries where the World Bank is operating, and will aim to establish the contribution of the ASPp to those changes.

Recognizing the difficulties within conventional evaluation techniques to provide an adequate evaluation framework for interventions implemented at system level, the evaluation team adopted a theory-based evaluation approach.

Central to the theory-based approach is the testing of the theory of change (ToC) that underpins the ASPp and the understanding of how contextual factors in each of the six countries shape and influence the extent to which the desired changes identified in the ToC will occur given a specific context. The overall design is centered on a contribution analysis approach to structure the assessment and to generate rigorous lessons from the system level changes.

The baseline survey of the evaluation was completed in four of the six countries (June 2016) included in the ASPp (Senegal, Mauritania, Niger and Mali). The exercise, which followed an evidence-based analysis principle (where every statement and ‘data point’ presented in the analysis can be traced back to one or more specific source(s) of information), was designed to provide an initial 'snapshot' of the situation in the four countries at the time the ASP programme was starting its different activities.

Particular attention was paid to three elements: (a) the respective ToCs developed by the World Bank teams in each country; (b) the potential contextual and external challenges/constraints and drivers/enablers which could be affecting, positively or negatively, the implementation of the ASPp activities in those countries; and (c) the initial behaviours, attitudes, skills and knowledge (BASK) of the key actors (especially government officials) targeted by the ASPp.

A similar exercise will be repeated at endline. The changes observed in the key actors' BASK scores between the baseline and endline, will give an indication of the influence of the ASPp on these key actors.

The different constraints and enablers that have been identified during the baseline will be further explored and used to ‘explain’ the potential discrepancies between the initial ToCs recorded at the baseline and the actual performance of the programme at the endline stage, thus helping the evaluation team to assess the contributions (both positive and negative) of the ASPp to the changes observed between the baseline and endline. The evaluation is to be completed by March 2018.