Evaluation Methods
An evaluation can use quantitative or qualitative data, and often includes both. Both methods provide important information for evaluation, and both can improve community engagement. These methods are rarely used alone; combined, they generally provide the best overview of the project. This section describes both quantitative and qualitative methods, and Table 7.1 shows examples of quantitative and qualitative questions according to stage of evaluation.
Quantitative Methods
Quantitative data provide information that can be counted to answer such questions as “How many?”, “Who was involved?”, “What were the outcomes?”, and “How much did it cost?” Quantitative data can be collected by surveys or questionnaires, pretests and posttests, observation, or review of existing documents and databases or by gathering clinical data. Surveys may be self- or interviewer-administered and conducted face-to-face or by telephone, by mail, or online. Analysis of quantitative data involves statistical analysis, from basic descriptive statistics to complex analyses.
Quantitative data measure the depth and breadth of an implementation (e.g., the number of people who participated, the number of people who completed the program). Quantitative data collected before and after an intervention can show its outcomes and impact. The strengths of quantitative data for evaluation purposes include their generalizability (if the sample represents the population), the ease of analysis, and their consistency and precision (if collected reliably). The limitations of using quantitative data for evaluation can include poor response rates from surveys, difficulty obtaining documents, and difficulties in valid measurement. In addition, quantitative data do not provide an understanding of the program’s context and may not be robust enough to explain complex issues or interactions (Holland et al., 2005; Garbarino et al., 2009).
Qualitative Methods
Qualitative data answer such questions as “What is the value added?”, “Who was responsible?”, and “When did something happen?’’ Qualitative data are collected through direct or participant observation, interviews, focus groups, and case studies and from written documents. Analyses of qualitative data include examining, comparing and contrasting, and interpreting patterns. Analysis will likely include the identification of themes, coding, clustering similar data, and reducing data to meaningful and important points, such as in grounded theory-building or other approaches to qualitative analysis (Patton, 2002).
Observations may help explain behaviors as well as social context and meanings because the evaluator sees what is actually happening. Observations can include watching a participant or program, videotaping an intervention, or even recording people who have been asked to “think aloud” while they work (Ericsson et al., 1993).
Interviews may be conducted with individuals alone or with groups of people and are especially useful for exploring complex issues. Interviews may be structured and conducted under controlled conditions, or they may be conducted with a loose set of questions asked in an open-ended manner. It may be helpful to tape-record interviews, with appropriate permissions, to facilitate the analysis of themes or content. Some interviews have a specific focus, such as a critical incident that an individual recalls and describes in detail. Another type of interview focuses on a person’s perceptions and motivations.
Focus groups are run by a facilitator who leads a discussion among a group of people who have been chosen because they have specific characteristics (e.g., were clients of the program being evaluated). Focus group participants discuss their ideas and insights in response to open-ended questions from the facilitator. The strength of this method is that group discussion can provide ideas and stimulate memories with topics cascading as discussion occurs (Krueger et al., 2000; Morgan, 1997).
The strengths of qualitative data include providing contextual data to explain complex issues and complementing quantitative data by explaining the “why” and “how” behind the “what.” The limitations of qualitative data for evaluation may include lack of generalizability, the time-consuming and costly nature of data collection, and the difficulty and complexity of data analysis and interpretation (Patton, 2002).
Mixed Methods
The evaluation of community engagement may need both qualitative and quantitative methods because of the diversity of issues addressed (e.g., population, type of project, and goals). The choice of methods should fit the need for the evaluation, its timeline, and available resources (Holland et al., 2005; Steckler et al., 1992).