In this example you will read about how STOP Malaria Community Champions initiative used a quasi-experimental study design to compare intervention areas with non-intervention areas (with cross-sectional surveys), with the goal of assessing overall coverage of key malaria case management SBCC activities, and to show the differential impact of the SBCC intervention on behaviors.
Use this example and the links included to think about how use a quasi-experimental study design to evaluate malaria case management SBCC activities.
Case Study
The Communications Support for Health (CSH) was a 5-year USAID-funded project designed to provide technical assistance to the government of Zambia to strengthen national health communication activities. The primary objective was to strengthen SBCC activities and interventions. A key component of project activities was the STOP Malaria campaign, which included case management behaviors such as increasing the number of people who were tested for malaria and received treatment. A participatory community mobilization approach called Malaria Champion Communities was combined with mass media channels in eight districts in four provinces. After 1.5 years of implementation CHS conducted an assessment of program activities.
Study design: Cross-sectional household survey in Champion Communities and in similar communities that did not receive the program.
Objectives: Assess campaign exposure and its effect on behaviors in Champion Communities compared with similar communities.
Sampling: 20 communities were selected and surveyed, 10 exposed (selected randomly) and 10 not exposed (selected with the help of NMCP). 25 households were recruited per community for a total of 500 households. Heads of household, pregnant women, or mothers of children under 5 were surveyed. Selection criteria for non-intervention communities stated that they must fall within the same health center catchment area, must not have received Champion Communities activities or be aware of the program, and had to have at least 50 households.
Data collection and analysis: Two teams of six interviewers and a supervisor led data collection. Four days of training for interviewers included an orientation to the campaign, clarification of roles and responsibilities, procedures for household selection, interviewing skills, and procedures for obtaining consent forms. Data was entered using CSPro software and exported to STATA for analysis.
Outcome 1: Campaign exposure
- 90.4% of those surveyed in intervention areas had heard of the STOP Malaria campaign or Champion Communities initiative, compared to 21.6% of non-intervention areas.
- 99.1% of those familiar with the campaign had attended at least one malaria event in the last year. 88% of respondents in intervention areas had been visited by a community health worker, compared to 22.4% in non-intervention areas.
Outcome 2: Case management outcomes (among many other behaviors measured)
- 90% of intervention area survey respondents reported discussion malaria prevention, testing or treatment, compared to 25% in non-intervention areas.
- 80% of households with a febrile member sought care. Caretakers of children under 5 years of age were more likely to have sought care if they lived in intervention areas (97.1 intervention areas, 80% non-intervention areas).
- Children under 5 in intervention areas were 25% more likely to receive a diagnostic test in intervention areas
- Treatment administered for patients with malaria did not differ significantly.
Evaluation conclusions
This evaluation found exposure to the STOP Malaria campaign to be widespread in the intervention area. No causal attribution could be established but evaluators found results notable.
This case study demonstrates both merits and limitations of a quasi-experimental survey designed to measure exposure and behaviors. CHS did go on to conduct a more thorough evaluation of the STOP Malaria campaign after this initial study. The more thorough study compared a 2013 baseline survey with a 2014 end line survey, finding improvements in all behaviors measured.
Zambia Communications Support for Health: Stop Malaria Champion Communities Program Evaluation
Replicate the scenario and learn
A five-year USAID-funded SBCC project has begun work in a Sub-Saharan African country with areas of stable and interrupted malaria transmission. Program activities include a multi-channel campaign including mass media, print materials, interpersonal communication training for health workers, community mobilization including group talks, village-level talks, school programs, home visits, and talks at health facilities. Several months into program implementation project personnel are informed that a mid-term outcome evaluation is to be designed and conducted in 18 months time. The evaluation will consist of two cross-sectional surveys, one to measure exposure to SBCC, knowledge, attitudes, and behaviors in areas of project intervention, the other to measure the same variables among people living in areas far from project intervention. Project personnel are expected to design the cross-sectional survey, implement it, and submit results in the form of a mid-term report.
Quantitative measurement tool
Question to answer: Reach and exposure of messages among the intended population; knowledge, attitudes and practices among those reached; the differences between key behaviors in intervention communities compared to those in other similar communities where the intervention was not done or that did not receive the intervention.
Data collection method: Sub-national cross-sectional household surveys
Conduct cross-sectional household surveys in intervention areas and non-intervention areas. Select an equal number of intervention and comparison districts using specific, pre-determined criteria. Ensure that the communities are similar in demographic and socio economic status. Use simple random sampling techniques to determine households in each community in the selected district.
The survey can be administered to one member of each household (in most cases, the head of household). Ensure to gather information on age, gender, and any other relevant demographics of respondents. Work through the sampling size to try and ensure a large enough sample to show statistical differences. Use statistical software such as STATA or SPSS to analyze your data.
Limitations: Cannot claim causal attribution (you will not know definitively whether or not changes that took place are direct results of your SBCC activities), only correlations (trends that make it appear your activities are associated with improvements in knowledge, attitudes, and practices in intervention areas but not control areas). This is because you don't have a true control group and no baseline to compare your data with. If behaviors your activities promote start becoming the norm over the project lifecycle, direct exposure may not be needed to be affected by the campaign as those not directly exposed could be influenced by what they hear and see around them (indirect effects explained in theories like the diffusion of innovation).
The use of mediation analysis to assess the effects of a behaviour change communication strategy on bed net ideation and household universal coverage in Tanzania: A household survey in three districts where community change agents were active was conducted to collect information about a number of malaria prevention behaviors. A mediation analysis was conducted to see the impact exposure to a community change agent only, mass media and community messaging only, or exposure to both. This manuscript focuses on bed net use, however, the data set also examined care seeking for fever.