Working Effectively in Conflict-affected and Fragile Situations: Briefing Paper I: Monitoring and Evaluation

This 13-page briefing paper from the United Kingdom (UK) Department for International Development (DFID) is focused on monitoring and evaluation (M&E) in situations of conflict and fragility. As indicated here, M&E is often neglected in these contexts, for a number of reasons; to cite one example, in insecure environments, implementing agencies may lack the staff resources or expertise for effective monitoring.

Despite the challenges, this briefing note suggests that M&E is imperative in situations of conflict and fragility for a series of interrelated purposes:

• to assess whether programmes are achieving their objectives, including objectives intended to reduce conflict and instability;
• to monitor progress towards peace-building and state-building objectives;
• to assess whether programming is conflict-sensitive;
• to assess whether the risks presented by conflict and fragility are having a negative impact on programme delivery; and
• to monitor changes in the country context, and to determine whether programme objectives, modalities, and partnerships remain appropriate.

Part I discusses what should be monitored at different levels:

• Common strategic frameworks: Even in a low-capacity environment, joint monitoring processes involving donors and partner governments - for instance, Transitional Results Frameworks (TRFs) designed to support monitoring of short-term objectives - can provide a mechanism for engaging country partners in the coordination and oversight of complex interventions.
• DFID country plans: Country plans in fragile contexts should include objectives, with indicators and data sources, that are explicitly designed to address sources of conflict and fragility, in addition to poverty reduction and other development goals. DFID notes that strategic conflict assessments should be updated periodically to provide a picture of changing conflict dynamics. A wide range of data sources and reports from independent agencies can be used and supplemented with surveys on issues such as human rights, social exclusion, or human security.
• Sector-level engagement: According to DFID, where there is an agreement with country counterparts on a set of broad sectoral objectives, a programme of joint or independent progress reviews can create a useful platform for dialogue and build a shared understanding of policy and institutional priorities. For example, in Zimbabwe, DFID's Expanded Support Programme for HIV and AIDS is helping to strengthen the national HIV/AIDS M&E system. "In a very difficult political environment, it has helped to identify joint objectives, such as the need for measures to prevent the drain of qualified personnel out of the health sector." In DFID's experience, joint monitoring at the sectoral level also helps to stimulate demand for data.
• Project level: An example provided in this section involves DFID's work in Nepal, where a Risk Management Office (RMO) undertakes continuous monitoring of security risks in conflict-affected areas. The RMO encourages partners regularly to analyse threats, vulnerability, and risks to projects, their relationships with local counterparts, and their impacts (direct and indirect) on the conflict. Among the questions partners are asked to review are: (i) Are their programmes, staff and partners seen as neutral or aligned with stakeholders or parties to the conflict? (ii) How are they perceived by the insurgents and security forces? (iii) With whom and where are they working - and not working? (iii) What human and financial resources do their programmes transfer, and who benefits? (iv) Whose power, interests, or needs are they challenging? 

Part II provides practical guidance on particular M&E challenges in fragile contexts, including:

• Theories of change: DFID notes interventions aimed at conflict reduction, state-building, or peace-building should articulate a clear theory of change that can be tested through evaluation - although this does not happen in practice. A box on page 8 of the briefing paper sets out some examples to illustrate the importance of unpacking broad objectives and translating these into clear, logical theories about how such objectives can be achieved. DFID notes that, when testing theories of change, untangling the chains of cause and effect can be very difficult. When attributing change to the programme is problematic, a broader evaluation approach that asks "what happened, and why?", rather than "did X cause Y?" can be used. An analytical narrative describing the events and processes, based on the accounts of stakeholders and informed observers, may be more useful than a strict social science method.
• Conflict-sensitive indicators: Suggestions offered here for identifying indicators from the start include: (i) considering the sources of data available (it may be useful to commission additional surveys on issues such as social exclusion); (ii) including both indicators of drivers of conflict and fragility (e.g. mortality rates, human rights violations, perceptions of insecurity, indicators of social capital) and of successful peace-building and state-building (e.g. reintegration of displaced persons, levels of marginalised minority groups utilising government services, levels of access to justice); (iii) monitoring intangible qualities, such as trust or confidence among groups, through proxy measures, such as freedom of movement, numbers of intergroup organisations and process, or intermarriage rates; (iv) using a mix of qualitative and quantitative indicators; (v) disaggregating data to expose differential dimensions or effects of the conflict (e.g. by region, gender, age, disability, religion, or ethnic origin).
• Setting baselines: Where appropriate M&E arrangements have not been put in place at the outset, it is important to develop them as soon as circumstances allow.
• Data collection in insecure environments: "...conducting sound M&E means locating 'good enough' data so that it is possible to draw useful conclusions about programme impact. Where travel is constrained, other options may be available for data collection, including drawing on secondary sources, changing the geographical sample for monitoring purposes, or identifying proxies (e.g. representatives able to speak for minority groups in conflict-affected areas, parents of child soldiers). Alternatively, it may be possible to bring representatives of key groups out of the conflict zone, in order to conduct focus groups..."
• Monitoring by beneficiaries: Increasing the transparency of assistance (i.e., publicising exactly what should be delivered, where, and to what standard), and assisting local communities to monitor delivery through community-based organisations or other civil society mechanisms can provide an additional source of data, as well as help to strengthen the accountability of local service providers and implementing agencies to their own communities.
• "Real-world" evaluations: The World Bank's July 2005 guide "Designing Quality Impact Evaluations under Budget, Time and Data Constraints" [PDF] is referenced here. DFID also notes that biases may result from pragmatic approaches; for example, for quantitative analysis, combining several analytical methods may help to limit methodological bias. It is also important to be aware that there may be high levels of unreliable information and bias among interlocutors, particularly in situations of conflict where disinformation may be a tactic used by the combatants.
• Evaluation criteria in conflict situations: (i) Relevance: Does the programme address key actors and issues? Is the theory of change credible? Has implementation responded flexibly to changing circumstances? (ii) Impact: Has the programme helped prevent violence or address long-term drivers of conflict? (iii) Coherence: Is the programme coherent with the overall national or international strategy for conflict resolution, and with the activities of other actors? (iv) Linkages: Are peace-building efforts at the political level linked to the grassroots, or vice versa? (v) Coverage: Are the communities or geographical areas in which the programme is operating the right ones, given the conflict dynamics? (vi) Consistency: Is the programme being implemented in a way consistent with UK policies and values (e.g. respect for human rights, support for democracy)?