Quantitative data collection is numerical and is often used to count the variable of interest. Quantitative data collection tools used for monitoring include program logs, attendance sheets, and other tools that collect counts for referrals made, referrals completed, tools developed, products distributed, workshops conducted, broadcast, number of RDTs completed, number of ACTs prescribed etc. Quantitative data collection for evaluation purposes can be surveys, censuses and health records. Quantitative studies can be designed to be representative of the entire population of interest so that the findings are generalizable, and reflective of the entire population – not just those individuals included in the study. For malaria case management, for example, quantitative data might use a household survey to count the number of people who recall seeing or hearing a message about prompt care seeking for fever. Another example of a way to use of quantitative data would be to count how many people believe something or behave a certain way. 

From an SBCC evaluation standpoint it is important to measure reach and exposure of materials and messages among the intended population. Once you have measured exposure, it is possible to measure changes in knowledge and behavior among those exposed or unexposed. It is also helpful to measure how many messages respondents recall. An effective SBCC campaign will show a dose response, or a strong correlation between the number of messages recalled and increases in knowledge, attitudes, and behavior. Note that when attempting to show a dose response between exposure to messages and increases in knowledge, attitudes, or behavior, it is important to control for ownership of commodities related to your messaging. For example, it would be inappropriate to compare exposure to radio messages among those who do and do not own radios.

Examples of Quantitative Data Sources

Dose Response Analysis: Caution

Dose Response Analysis: Caution

As mentioned above, quantitative data collection makes it possible to show that the more messages audiences recall, the greater increases in knowledge, attitudes, and behavior. A randomized cluster controlled trial conducted in Burkina Faso attempted to show a dose response between radio broadcasting intensity and reported behavior change (including possible increases in prompt care seeking for fever). Radio ownership, or access to a radio is clearly an important consideration. To adjust for this, study designers broke audiences down in to three categories, those with: no radio in the compound, radio in the compound, and radio in the household. A difference in difference analysis was used to adjust for these levels of radio ownership/access. Mid-term results from this study actually did not find a dose response relationship between intensity of radio messages and prompt care seeking for fever. Higher exposure did not result in proportionally higher care seeking for fever. As the only communication channel in this study used was radio, perhaps this article's most important contribution to understanding SBCC for malaria case management is that it demonstrates the importance of multi-channel communication: reliance on a single channel failed to improve prompt care seeking for fever.

 

Examples of Qualitative Methods

National/Sub-National Level

Community/Health Center Level

In order to improve the design of SBCC interventions in malaria case management, formative research program design is critical. The results from the formative research are then used to design a more effective SBCC program. Conducting formative research includes reviewing existing information through literature reviews or secondary data analysis as well as collecting and analyzing qualitative and/or quantitative data to better understand the audience and proper framing of the SBCC messaging. The goals of formative research can include:

• Identify previous SBCC programs that have been designed for similar audiences for similar issues and learn from them
• Understand the media habits of the audience and identify what channels they have access to, use, and trust
• Identify the behaviors, perceptions, and information to promote (For example, what do caregivers of children under 5 years of age in the target area know, think, and feel regarding prompt care seeking for febrile children?)
• Identify the factors that hinder or motivate the behavior

Adapted from Evidence-based Malaria SBCC: From Theory to Program Evaluation 

Routine monitoring, including process evaluation, is used to track program activity progress towards expected goals. It will identify areas of excellence and deficiency which should then inform midcourse corrections. Preliminary successes have the added benefit of boosting morale and the commitment of program staff. Finally, SBCC monitoring can help inform future programs. 

For the M&E plan, it is helpful to describe the various sources of monitoring data and how often these data will be collected. It is important that the monitoring plan includes plans to monitor both activities and audiences. It is also important to describe how the data will flow up the system, where it will be stored, and who is responsible for it.

Activity Monitoring

• Activity report forms provide information on trainings and community mobilization activities to track how many activities were conducted and how many people participated. The SBCC program needs to create a system for collecting these forms regularly from implementers and checking to ensure they are filled out correctly. Mobile reporting, supervision visits, and data review meetings can bolster these channels.
• Media monitoring reports are created by third-party agencies who track which radio or TV materials are being aired, at what time, and how often. This allows the program to negotiate “make goods” or airings to make up for under-broadcasting. When media monitoring services are not available, broadcast logs can be requested from stations. Station logs can be verified by having community based listeners also listen to and log the dates and times of broadcasts.
• Passive or recorded observations, checklists, mystery clients, client exit interviews, and record reviews may be useful to track service provider behaviors
• Community-based household surveys, SMS questionnaires, and omnibus surveys track changes in knowledge and reach at the community level

Audience Monitoring

• Health management (HMIS)/logistics management information systems (LMIS) can be helpful for tracking service utilization, sales, and stock-outs. Since they are often used and managed by other parties - the SBCC program may need to invest funds into working with these parties to improve this system. Such activities may include providing modems for data uploads, training staff on using forms correctly, and conducting data quality meetings.

Adapted from: A Guide to Developing M&E Plans for Malaria SBCC Programs

Impact, or outcome, evaluation takes place after the activity is finished, however, it must be planned for at the beginning of the project as it is most useful when the data are compared to a baseline survey conducted prior to program implementation.

An important concept to consider when thinking about impact evaluation is the difference between correlation and causation. Causation is the measurement of an input of interest, for example – a program activity, that occurs in time before the outcome of interest, for example – prompt care seeking activity for children under 5 by caregivers. In contrast, correlation assesses the exposure of the project activity input at the same time as the desired outcome behavior. As a result, if two variables are correlated this does not necessarily mean that the input has caused the behavior output. There are three main types of impact evaluation research designs: experimental, quasi-experimental, and non-experimental. Only with the experimental design can the team assess causation.

Impact evaluations for SBCC need to be designed to answer three questions: a) Was the program effective? B) Did it change behavior? and c) How did it work? SBCC program messages influence behaviors indirectly through knowledge, attitudes, and beliefs that drive behavioral decisions. Understanding the specific attitudes through which messages affected behavior is important since this helps take the lessons from a successful program and apply them elsewhere.

One way to establish a link between exposure and behavior is to use self-reported exposure to SBCC messages in household surveys to construct the groups of exposed and unexposed individuals. In this approach, a series of questions in a household survey are constructed to ask each respondent about their exposure to SBCC messages and to specific program elements such as logos and slogans.

A brief description of three evaluation designs can be found below:

It is important for programmers to be aware of potential pitfalls of any research design and to take these into account when designing and drawing conclusions from the M&E outcomes. The Research Methods Knowledge Base provides a comprehensive overview of these methods.

Adapted from: A Guide to Developing M&E Plans for Malaria SBCC Programs and Evidence-based Malaria SBCC: From Theory to Program Evaluation