Step 4: Data Collection

Malaria Case Management SBCC M&E: Step by Step

DATA COLLECTION METHODS

Health programmers use qualitative and quantitative research to monitor and evaluate their programs. The choice of whether to use quantitative or qualitative data collection techniques, or a mixed methods approach using both, depends on a number of factors, such as:

The type of problem identified and outcome being assessed (some issues are best assessed with qualitative, some with quantitative)
The theory guiding design
The purpose of data collection. For example, is the assessment for formative research, monitoring or process evaluation, or impact evaluation?

Quantitative data collection is numerical and is often used to count the variable of interest. Quantitative data collection tools used for monitoring include program logs, attendance sheets, and other tools that collect counts for referrals made, referrals completed, tools developed, products distributed, workshops conducted, broadcast, number of RDTs completed, number of ACTs prescribed etc. Quantitative data collection for evaluation purposes can be surveys, censuses and health records. Quantitative studies can be designed to be representative of the entire population of interest so that the findings are generalizable, and reflective of the entire population – not just those individuals included in the study. For malaria case management, for example, quantitative data might use a household survey to count the number of people who recall seeing or hearing a message about prompt care seeking for fever. Another example of a way to use of quantitative data would be to count how many people believe something or behave a certain way.

From an SBCC evaluation standpoint it is important to measure reach and exposure of materials and messages among the intended population. Once you have measured exposure, it is possible to measure changes in knowledge and behavior among those exposed or unexposed. It is also helpful to measure how many messages respondents recall. An effective SBCC campaign will show a dose response, or a strong correlation between the number of messages recalled and increases in knowledge, attitudes, and behavior. Note that when attempting to show a dose response between exposure to messages and increases in knowledge, attitudes, or behavior, it is important to control for ownership of commodities related to your messaging. For example, it would be inappropriate to compare exposure to radio messages among those who do and do not own radios.

Examples of Quantitative Data Sources

Dose Response Analysis: Caution

Qualitative data is textual and is often used to determine the why to a question, but can also explore who, where, and what. An example of one type of qualitative data would be to find out why some people think or act the way they do. Examples of qualitative methods of data collection include focus group discussions, in-depth interviews, key informant interviews, and participant observation. While qualitative studies are not representative or generalizable (they are intentionally purposeful), they can provide a more in-depth analysis of issues related to specific behaviors and practices that may not be captured through quantitative surveys. The following section goes into detail on a variety of qualitative data collection techniques.

Examples of Qualitative Methods

Collection Technique	Description	Resources
Focus Group Discussions	A group discussion often using a semi-structured guide asking participants to share their opinions and experiences. This form of inquiry is helpful in drawing out social and cultural norms among a group of people who are similar in characteristics that are associated with the study interest.	Qualitative Research Methods: A Data Collector’s Field Guide
In-depth Interviews	Asking questions of one person in a private setting in order to understand their perspective on a topic. This data collection technique is an appropriate way to explore a sensitive issue that respondents are not likely to speak about openly in a group setting.	Guide for Designing and Conducting In-Depth Interviews for Evaluation Input
Key Informant Interviews	Key informant interviews are a way to gather first-hand knowledge about a topic from an individual who is deemed an authority on the topic.	Conducting Key Informant Interviews
Participant Observation	Participant observation is a way to collect information about people while spending time in their presence. Participant observation is unique in that is allows researchers to observe verbal as well as non-verbal communication as well as interactions between individuals.	Participant Observation as a Data Collection Method
Most Significant Change	Collect a series of stories about change and systematically selecting the most significant example.	The MSC Technique: A Guide to its Use

National/Sub-National Level

National/Sub-national level	Quantitative and qualitative tools	Types of questions answered
National: Multiple channel mass media (e.g. radio, television, billboards)	Cross-sectional household survey Observational pre, mid, post intervention Pre and post KAP surveys Focus group discussions	What is the reach of the intervention? Is there an increase in knowledge of behaviors linked to the intervention? What is the prevalence of healthy malaria prevention behaviors in intervention areas?
Subnational: Multimedia, multi messages based on barrier analysis, community groups	Barrier analysis to design SBCC strategies Participatory mixed quantitative/ qualitative methods Participatory consensus-building analytical methods; Baseline, midline and end line KPC surveys Methods of triangulation with data from key informant interviews and focus group discussions	What were accomplishments and contributions to increasing demand for health behaviors and services with multi-media multi messenger SBCC?
Subnational: Training of mother and child health community volunteers in health promotion	Document review Assessment of volunteers' knowledge and performance Focus group discussions and/or in-depth interviews	Did the interventions contribute to the use of services in the intervention zones? What are the determinants of use of services for different childhood illnesses?
Subnational: Care Groups	Periodic surveys Baseline and end line KPC surveys Mini KPCs approximately every 6 months Focus group discussions	What effect does the intervention have on specific child health outcomes? As compared to national level estimates? How has the coverage level for different indicators changed over the life of the project? Was the intervention carried out as planned? Perceived benefits? Is this intervention cost effective?

Community/Health Center Level

Community/Health Center Level	Quantitative and Qualitative Tools	Types of questions answered
Community, health center: IEC, mass media, counseling	Baseline, KPC, Rapid CATCH surveys Midterm evaluation Health facility survey Focus group discussions	What are the results of the project activities on iCCM practices in project areas? What areas need strengthening?
Community: Mass media, posters, distribution of messaging materials, message dissemination through road-shows and established community-based organizations	Pre-post intervention cluster sample survey, one district Randomly select households within each cluster using the WHO EPI random walk method Cross-sectional household survey, photo-illustrated guide to help in recognition of forms of antimalarials available Process evaluation	Does the SBCC intervention have an effect on the proportion of children under five years of age with fever in the last 14 days accessing AL within 48 hours of fever onset? Did it have an effect on treatment adherence?
Health Center: In-service training, Job aids	Pre-post comparison Cross-sectional health facility survey	Does enhanced training with job aids affect health workers’ malaria case management practices among febrile children in accordance with national guidelines? Does the training improve coverage of trained health workers?
Health facility, community: SMS/Mobile messaging	3-arm Randomized Control Trial Literature review to inform design Pretesting using qualitative methods Logistic regression	Do short SMS messages from health workers to the community increase adherence to ACT? What is the impact on completion of ACT treatment?

In order to improve the design of SBCC interventions in malaria case management, formative research program design is critical. The results from the formative research are then used to design a more effective SBCC program. Conducting formative research includes reviewing existing information through literature reviews or secondary data analysis as well as collecting and analyzing qualitative and/or quantitative data to better understand the audience and proper framing of the SBCC messaging. The goals of formative research can include:

Identify previous SBCC programs that have been designed for similar audiences for similar issues and learn from them
Understand the media habits of the audience and identify what channels they have access to, use, and trust
Identify the behaviors, perceptions, and information to promote (For example, what do caregivers of children under 5 years of age in the target area know, think, and feel regarding prompt care seeking for febrile children?)
Identify the factors that hinder or motivate the behavior

Adapted from Evidence-based Malaria SBCC: From Theory to Program Evaluation

Routine monitoring, including process evaluation, is used to track program activity progress towards expected goals. It will identify areas of excellence and deficiency which should then inform midcourse corrections. Preliminary successes have the added benefit of boosting morale and the commitment of program staff. Finally, SBCC monitoring can help inform future programs.

For the M&E plan, it is helpful to describe the various sources of monitoring data and how often these data will be collected. It is important that the monitoring plan includes plans to monitor both activities and audiences. It is also important to describe how the data will flow up the system, where it will be stored, and who is responsible for it.

Activity Monitoring

Activity report forms provide information on trainings and community mobilization activities to track how many activities were conducted and how many people participated. The SBCC program needs to create a system for collecting these forms regularly from implementers and checking to ensure they are filled out correctly. Mobile reporting, supervision visits, and data review meetings can bolster these channels.
Media monitoring reports are created by third-party agencies who track which radio or TV materials are being aired, at what time, and how often. This allows the program to negotiate “make goods” or airings to make up for under-broadcasting. When media monitoring services are not available, broadcast logs can be requested from stations. Station logs can be verified by having community based listeners also listen to and log the dates and times of broadcasts.
Passive or recorded observations, checklists, mystery clients, client exit interviews, and record reviews may be useful to track service provider behaviors
Community-based household surveys, SMS questionnaires, and omnibus surveys track changes in knowledge and reach at the community level

Audience Monitoring

Health management (HMIS)/logistics management information systems (LMIS) can be helpful for tracking service utilization, sales, and stock-outs. Since they are often used and managed by other parties - the SBCC program may need to invest funds into working with these parties to improve this system. Such activities may include providing modems for data uploads, training staff on using forms correctly, and conducting data quality meetings.

Adapted from: A Guide to Developing M&E Plans for Malaria SBCC Programs

Impact, or outcome, evaluation takes place after the activity is finished, however, it must be planned for at the beginning of the project as it is most useful when the data are compared to a baseline survey conducted prior to program implementation.

An important concept to consider when thinking about impact evaluation is the difference between correlation and causation. Causation is the measurement of an input of interest, for example – a program activity, that occurs in time before the outcome of interest, for example – prompt care seeking activity for children under 5 by caregivers. In contrast, correlation assesses the exposure of the project activity input at the same time as the desired outcome behavior. As a result, if two variables are correlated this does not necessarily mean that the input has caused the behavior output. There are three main types of impact evaluation research designs: experimental, quasi-experimental, and non-experimental. Only with the experimental design can the team assess causation.

Impact evaluations for SBCC need to be designed to answer three questions: a) Was the program effective? B) Did it change behavior? and c) How did it work? SBCC program messages influence behaviors indirectly through knowledge, attitudes, and beliefs that drive behavioral decisions. Understanding the specific attitudes through which messages affected behavior is important since this helps take the lessons from a successful program and apply them elsewhere.

One way to establish a link between exposure and behavior is to use self-reported exposure to SBCC messages in household surveys to construct the groups of exposed and unexposed individuals. In this approach, a series of questions in a household survey are constructed to ask each respondent about their exposure to SBCC messages and to specific program elements such as logos and slogans.

A brief description of three evaluation designs can be found below:

Experimental

Description: Assess causation: program input proceeds behavior outcome

Example: Randomized controlled trial: subjects randomized to treatment or control group; removes bias in factors affecting exposure to intervention

Pros: Intervention and control groups are similar – do not differ on measured and unmeasured variables, so relationship between program input and behavior outcome is isolated

Cons:

Expensive
Can be unethical
Difficult to implement

Quasi-Experimental

Description: Exposure not randomized –exposed and un-exposed to intervention may not be similar on background characteristics. Association between exposure to SBCC intervention and behavior change is weaker than with experimental designs

Example:

Longitudinal Study: repeated measure on same study participants over time
Pretest, Post-test: assess changes in scores for one group
Post-test: compare results between those exposed to SBCC intervention and those not exposed

Pros:

Ethical
Not difficult to implement
Can assess change over time

Cons:

Difficult to ensure causality is measured
Expensive
Time-intensive
Lost to Follow-up a concern
Limited generalizability

Non-Experimental

Description: Exposed and un exposed groups are not compared. Only one point of data collection.

Example: Cross-sectional: data gathered at one point in time

Pros:

Fast
Less expensive

Cons:

Differences observed between groups may be due to differences between study samples and not exposure to SBCC intervention
Bias due to non-response

It is important for programmers to be aware of potential pitfalls of any research design and to take these into account when designing and drawing conclusions from the M&E outcomes. The Research Methods Knowledge Base provides a comprehensive overview of these methods.

Adapted from: A Guide to Developing M&E Plans for Malaria SBCC Programs and Evidence-based Malaria SBCC: From Theory to Program Evaluation