Roll Back Malaria Communication Community of Practice Working Paper: Checklist for Reporting on Malaria Communication Evaluations

"Good-quality reporting is all the more crucial as malaria control efforts seek to keep up with shifts in epidemiology and communication technologies. Well-written reports can lead to a better understanding of 'what works' in different contexts and are crucial to maximizing investments in research."

The premise of this resource is that behaviour change communication (BCC) programmes, which promote individual behaviour change, social norms, and supportive environments, are a key component of many malaria programmes. BCC campaigns often include interpersonal communication components, such as those between a community health worker and her client, as well as mass media activities like radio, TV, and SMS (text) messaging. Furthermore, well-designed and well-executed evaluations of the use of BCC to combat malaria provide reliable evidence for policymaking; however, the authors contend that there are few published articles about their effectiveness. Many are incompletely or inconsistently described in evaluation reports and papers. "There is a need for researchers to publish their work and to present it in ways that help us understand (a) the outcomes on which communications programs have the greatest effect, (b) the magnitude of these effects, (c) the elements of a communication program and the context in which it operates that contributes to its effectiveness, and (d) the cost-effectiveness of communication in malaria control."

To fill that void, this checklist offers strategies for reporting on evaluations of BCC programmes. Developed through a collaborative process that took place from 2012-2014, it emphasises description of the BCC intervention and rationale for the BCC strategy, choice of BCC outcomes, methods of creating comparisons, and a discussion about the effects, causal mechanism, and future implications and generalisability of the results. The goal is to: "improve the transparency of reporting, increase the efficiency of the writing and review process, and ultimately, facilitate synthesis of which BCC approaches work in different contexts."

Checklist items proposed to be included in evaluation reports and manuscripts for BCC interventions are grouped into 3 domains:

1. BCC intervention - 9 questions are offered to help an evaluator explore how the intervention was designed and to provide a description of the intervention. For instance, when carrying out formative research, the evaluator might note that: "In one district, the majority of women may be not aware of the importance of malaria prevention in pregnancy. In another district, most women may feel that taking a medicine on an empty stomach is more dangerous than malaria during pregnancy. Formative research methods such as focus groups, key informant interviews, surveys and observations can be used to understand the beliefs, preferences, constraints, motivations, current behaviours, and communication channels accessed by target audiences. This information, along with behavioural theories and materials testing, provide the basis for strategically selecting messages and approaches with the best chances of producing measurable change."
2. Study design - 6 questions are offered to help assess the selection of outcomes and method of comparison, such as: "Did the authors use the recommended outcome indicators from the RBM malaria BCC indicators guide [see Related Summaries, below] (exposure to the BCC intervention, changes in malaria behaviours, intermediate outcomes such as knowledge, norms, attitudes, risk and efficacy)?" To cite only one strategy outlined in this domain: "One powerful and advanced approach, multivariate causal attribution, combines theory-based structural equation modelling, propensity score matching analysis, and sensitivity analysis to create an evaluation approach that is both theory and method-driven. The structural equation modelling allows researchers to test causal pathways; propensity score matching creates statistically matched control groups; and sensitivity analysis tests the effect of unmeasured confounders."
3. Discussion - 7 questions, such as "[w]hat are the implications for future research, BCC campaigns and policy?", are offered to facilitate interpretation of the results, factoring in strengths, limitations, and/or weaknesses of the study. One focus here is on determining causality through a list of conditions, including: strength, consistency, specificity, theoretical coherence, plausibility, dose, temporality, and responsiveness to experiment.

The concluding section of the resource offers observations such as the fact that word limits may restrict writers' ability to report on all the recommended items that have been included in this checklist. It is suggested that "[a]uthors can provide as much description as possible within the limits and then provide a link to the intervention's details online; this can take the form of a toolkit of communication materials, program and formative research reports as well as the full evaluation reports or supplemental analysis. Similarly, authors can create elegant and concise ways of presenting the additional data, such as flowcharts to map the link between exposure, intermediate outcomes, and behaviors, or provide a table or textbox with the intervention details..."