Guidance for Evaluating Mass Communication Health Initiatives: Summary of an Expert Panel Discussion

"When national broadcasts contain health or other pro-social messages, exposure to the messages cannot be ensured for a pre-determined intervention group and denied to a control group. Thus, Entertainment Education is an example of a "full coverage" communication campaign. As Valente (2002) and others have lamented, it's hard to be an evaluator of full coverage programs in a world where the gold standard research design is the true experiment (or Randomized Controlled Trial, as we often say in public health)..."

To discuss and address these and other evaluation challenges, in May 2004 the United-States-based Centers for Disease Control and Prevention (CDC) convened a panel of experts from private, academic, and governmental sectors for a 2-day meeting in Atlanta, Georgia (USA). This report is intended to articulate and explore some of the key issues in health communication programme evaluation that emerged from the discussion, and to encourage others to join in the dialogue.

The 54-page report is divided into the following major sections:

• I. Theory
  
  The panelists offered their views about issues such as: when evaluators should use formal theory and when they should use a logic model; the contribution a theorist makes to an evaluation effort; underutilised theories; the advisability of using one intact theory vs. components of several theories; and what to do when outcome data suggest a breakdown in the causal pathway...
• II. Methods
  
  Selected key points include:
  • When a campaign cannot be evaluated with a randomised controlled trial (RCT), experimental procedures may be used to to pilot-test the efficacy of the planned communication strategy in a small geographic area prior to the formal campaign launch. While such strategies are "ethically and fiscally responsible" they "have been omitted in the past because of political pressure to respond rapidly and at full scale to health issues of widespread concern."
  • Quantitative but non-randomised designs that can be used for evaluating large communication campaigns include cross-sectional, cohort, time-series and quasi-experimental designs; particular methods are described here. One suggestion: "Consistent findings across studies that used multiple methods with complementary strengths can be compelling."
  • With regard to the necessity of making programme corrections based on process data: "If desired program outcomes cannot be specified at the earliest stage (e.g., if they depend on political processes that have not yet played out), one course of action is to include all likely outcomes in the logic model and baseline measures."
  • Evaluators need to address secular trends, or changes over time that are based on behavioural or other changes not attributable to the programme itself. "A good way to refute a charge that a secular trend was wholly responsible for an observed behavior change may be to employ a 'switch-back' design. Experimental and control conditions are switched after more change is observed in the experimental community than in the control community. If change levels off in the new control site and accelerates in the new experimental site, then attributing cause solely to a secular trend can be ruled out."
  • "Many mass communication efforts not only affect individuals but also reverberate at multiple levels of the socio-political environment." These "higher-order effects" have been measured in a variety of ways; a table provided here illustrates each strategy and its drawbacks. Other methods include: mining archival records for advocacy, policy agenda and funding patterns; qualitative analysis; and tracking commodity availability.
  • "Large-scale communication programs often disseminate messages through a variety of channels....However, assessing the contribution of each channel to a campaign outcome means measuring exposure to each channel accurately and that is difficult." Given the challenges that panelists articulated - e.g. the puzzle of trying to tease apart the contribution of communication from the impact of other kinds of programme components in a multi-component intervention - the report urges that "the potential of qualitative methods should be mined...[but] expectations for success in components analyses should be tempered."
  • Timing Exposure, Awareness, and Effects - "an understanding of the factors that converge to affect the shape of exposure and outcome curves can inform an evaluator's decisions about when to begin to collect data, how many survey waves to conduct, and when to terminate data collection." The report goes on to identify and discuss a number of the factors that should influence timing decisions...
  • "Campaign messages compete with 'background noise' from other messages for the attention of target audiences." The report shares methods for detecting an environmental noise factor (e.g., environmental scanning through key informant interviews and monitoring news media coverage) and strategies for reducing the effects of noise (e.g., "using propensity scores to handle several covariates simultaneously, being sensitive to overinterpretation and the problem of reciprocal causation of exposure and outcome").
  • "Where possible, exposure should be manipulated to create an objective measure with which to confirm self-reports [of media exposure]. New web evaluation techniques should be a boon in this regard."
  • "[A]lthough imperfectly suited to the evaluation needs of most programs, indicator data can complement other evaluation procedures and stand in for custom data collection when evaluation resources are scarce. The challenge is finding good time-series data that are sufficiently relevant to specific campaign objectives."
  • "In general, the duration of an evaluation should depend on its goals..."
  • Statistical issues were largely beyond the scope of this discussion, but a few general recommendations about statistical are shared here.
• III. Resource-Scarce Settings
  
  When evaluation funds are scarce, evaluation should be limited in focus to the highest priority questions; rank-ordering programme aims and related research questions on the basis of a theory-derived logic model is considered best practice...
• IV. Addressing Racial/Ethnic Health Disparities
  
  If a communication programme addresses minority audiences, members of the audience should be involved in programme and evaluation planning from the outset. The expert evaluation panel also urged paying special attention to theory selection, dedicating resources to formative research and cognitive testing, using special means to track message dissemination, and employing adequate subgroup sample sizes...
• V. Policy Issues
  
  The panelists: (1) articulated a need for more efficacy trials before taking campaigns nationwide, (2) recommended extending the knowledge base for the practice of "message bundling" prior to launching national or large-scale campaigns that rely on this strategy, (3) called for centralised identification of secular trends, (4) listed other specific areas in which communication research or research guidance is needed, and (4) recommended ways to obtain additional expert advice on evaluating large health communication programmes and campaigns...