Disseminating the EBP ProjectVivibelle
Stage 3: Writing an Evaluation Report
Table of Contents SECTION 1: Executive Summary
SECTION 2: Program Description
SECTION 3: Evaluation Methodology
SECTION 4: Findings
SECTION 5: Interpretation and Reflection SECTION 6: Recommendations
This stage of the guide provides a suggested format for your final evaluation report, along with ideas for developing each section of the report. While this stage covers the basic content for any evaluation report, your report may vary from the model presented here. How you decide to organize your report will depend on the purpose of the report, your audience, and the requirements of the funding agency. (See Appendix H for alternative evaluation report formats.)
An “Executive Summary” (sometimes called a Summary) is a short document of one or two pages that appears at the beginning of the final evaluation report. The Executive Summary provides an overview of the program and highlights key findings and recommendations from the evaluation, giving the reader a sense of the report’s content without having to read the entire document.
Why write an Executive Summary? • The Executive Summary outlines what the reader should expect
to find in the report.
• A Summary may be used separately from the report. For instance, it may serve as an efficient means of sharing key findings of the evaluation with a large audience or a potential funder.
What is included in the Executive Summary?
The exact length and components of a summary may vary depending on the purpose of the report. Typically, an executive summary is an overview of the report and includes:
• Purpose of the program • Program activities, setting and population served • Purpose of the evaluation • Overview of findings or outcomes • Overview of recommendations
Tips for writing an Executive Summary • Do not include technical details in the Executive Summary. That
is, do not include details about data collection methods used. • Write the Executive Summary last, after all other sections of the
report are completed. • Write the Executive Summary in a way that allows the reader to
learn about the most salient aspects of the evaluation without reading the full report.
SECTION 1: Executive Summary
The “Program Description” section introduces readers to your program. It should contain a succinct description of the program being evaluated, present program goals and objectives, and explain how program activities were intended to meet the objectives. Depending on your audience, report requirements, and whether you have comparative information about the program from a previous evaluation, you may choose to include additional information about your program and its history in this section.
What is included in the Program Description section?
• Explanation of how the program originated: Provide the rationale for the program in relation to the agency’s mission, research literature, community needs assessment, and/or the political climate.
• Program overview: Focus on the program’s purpose and key program activities. Describe the program’s target population (who is served by the program), when and where activities took place, and why the program was set up the way it was (program design).
• Program goals and objectives: List the program’s goals and objectives.
• Significant program revisions: Describe any changes to the program’s objectives or activities that occurred prior to or during the evaluation, and provide a rationale for those changes.
Additional information to consider including in the Program Description section:
• Relationship of this program to CWIT’s mission and broader organizational efforts.
• History of the program’s development or changes in the program since its initial implementation, including prior accomplishments the current program builds on or gaps its seeks to address. This is especially relevant for programs that have been in existence for several years, and/or if the program received funding from the same agency in prior years.
• Comparison of the program evaluated to similar programs sponsored by CWIT or other agencies.
SECTION 2: Program Description
The “Evaluation Methodology” section of your final evaluation report describes the research methods you used in the evaluation and describes why those particular methods were the most appropriate for the evaluation. The purpose of this section is to explain how the evaluation was designed and carried out and to let your reader assess the quality of your evaluation design.
Why include an Evaluation Methodology section in your final report?
A clear description and justification of the evaluation methods used has several advantages for the program staff and other stakeholders:
• It demonstrates that the evaluation and procedures for collecting data were carefully and systematically planned.
• It tells readers how the evaluation team gathered the information presented in the report. This allows readers to assess the quality of data-collection procedures.
• It provides documentation that program staff can use to repeat procedures if they want to collect comparable data in the future.
• It documents your methods, providing a framework for staff with similar programs to draw on as they design or improve their evaluation procedures.
• It assesses whether or not the data collection tools used were appropriate for the group served by the program.
Remember You may have already described the your evaluation methods in another document, such as a grant application. If so, you may be able to draw on that text for your final report, editing it to reflect any changes in methods used and challenges faced.
SECTION 3: Evaluation Methodology
What should be included in the Evaluation Methodology section of an evaluation report?
• Types of information collected Did you collect quantitative data? If so, what types and why? Did you collect qualitative data? If so, what types and why? Did you collect both kinds of data? If so, what types and why?
• How and why information was collected Describe the data collection tools used and include examples as
appendices. Explain whether the data collection tools existed prior to the
evaluation or if they were developed in-house or by an outside evaluator.
Explain how the data collection tools were intended to answer the research questions.
• Who information was collected from and how participants were selected to provide information
Was information collected from program participants? Program staff? Community members? Other stakeholders? How and why were individuals chosen to provide information for the evaluation? Or did they volunteer? (To review sampling, see Appendix B)
Was a comparison group used? How was the group chosen? (For a review of comparison groups, see Appendix C)
• Who collected the information Was the information collected by program staff? By outside
evaluators? By program participants?
• Limitations in the evaluation design or implementation Were you able to collect the information you needed using your
data collection tools? Were the data collection tools appropriate for those who provided
information? What challenges did you face in carrying out the evaluation? What changes did you make in your evaluation methods over the
course of the project?
Tips for writing your Evaluation Methodology section
• It is not necessary to reproduce the data collection instruments you used in the main text of your evaluation report. However, you may want to include them as appendices.
• The Methodology section of your final evaluation report is important, but need not be long. Be concise!
The "Findings" section presents an organized summary of the information you have collected in a way that describes whether and how well each program objective has been met. In a sense the “Findings” section is the core of the evaluation report, because this is the section in which you provide a coherent account of what you have learned about your program from the data you collected.
How do I “present” my “findings”? Presenting the findings of your evaluation includes organizing the data you have collected, analyzing it, and then describing the results. The first step in presenting your findings is to organize the quantitative and qualitative information you have collected in a way that will allow you to address your objectives. After it is organized, the information can be used to evaluate your objectives. The second step is analyzing the data. “Data analysis” sounds daunting, but all it really means is determining what the information you collected can tell you about whether and how well you met your program objectives. This entails using the information you have collected to answer your research questions. The third step is describing the analysis of your findings. Data should be presented in condensed, summary form. For example, to describe quantitative data you can use frequencies, percentages or averages. To present qualitative data you might summarize patterns you detected in observational records or clusters of answers in interview responses. Try to avoid interpreting your findings in this section. Interpretations of the findings are comments about why the objectives were or were not met. Interpretations will be presented in the next section of your report. (See Section 5 of this Stage for more information on interpretation.)
Remember It may be tempting to display data in the form of charts, tables or quotations. Be sure to do so only if it will make the results easier for the reader to understand.
SECTION 4: Findings
Example The following is an example of how data may be summarized and analyzed for presentation in the Findings section of a final report. Objective: “60% of women completing the training program during program year 2001 will have acquired the skills need to pass the carpenter’s entrance exam.” Research questions: 1. What percentage of women who completed the training program
passed the carpenter’s entrance exam? 2. After the training program, did women have more skills to pass
the carpenter’s entrance exam than before the program? 3. Did participating women receive the training they expected? Data collected:
• Pre- and post-training assessments of participants’ skills to pass the carpenter’s entrance exam.
• Pre-training focus groups with participants to determine their expectations for the training and assess skill levels.
• Post-training focus groups with participants to assess satisfaction with training, whether expectations were met, and whether skills were obtained.
Presentation of Findings: 1. To answer the first research question, provide the number of
training participants. Present the percentages of participants that did and did not pass the carpenter’s entrance exam.
2. To answer the second research question, summarize the results of pre-training and post-training skills assessments. Describe how pre-training results differed from post-training results. In addition, use qualitative data from focus groups to present participants’ views about whether they felt prepared to pass the exam based on what they learned in the training.
3. To answer the third research question, summarize qualitative responses from focus groups that assess whether the training met participants’ expectations.
Steps for presenting evaluation data 1. Review your program objectives and research questions to
determine how the information you have collected will be used. Think about the types of information you will need to summarize to determine whether or not you have met your objectives.
• Make sure notes taken during interviews and focus groups have been transcribed, and surveys have been entered into a computer database, if appropriate.
2. Read through the data you have collected to be sure it is
understandable and to determine whether any information is missing.
• When you are checking for missing data, keep in mind that missing data may be physical data such as attendance sheets or meeting minutes, or specific information that is needed to evaluate specific objectives.
• If information is unclear, determine whether clarification or additional information is necessary, or whether the information you have should not be included in the report.
• If information is incomplete, determine whether you can and should obtain the missing data.
• Determine how missing data will affect your results.
3. Analyze data and organize analyzed data in the form of charts, lists, graphs or summaries that can be used to report on each objective.
• Address each program objective in your Findings section. If there are insufficient data to determine whether or not the objective was met, then indicate that this was the case and account for the lack of data.
• A conventional method of reporting findings is to state the objective and present data pertaining to the objective immediately below. However, you may also report on clusters of related objectives, particularly process objectives.
• Make sure data are presented clearly. Share your "Findings" section with co-workers to make sure the findings are presented clearly, without interpretation, and in a way that is honest, concise, and complete.
Tips for analyzing qualitative data Qualitative data may be used to answer research questions that require in-depth answers. Examples are questions about individual experiences, program processes, or the relationship between program outcomes and the wider context in which the program took place. Analyzing qualitative data involves looking for patterns, categories, and themes. When examining qualitative data, it is important to put aside what you already know, and allow themes to emerge from the data on their own. Be careful: if you only focus on what you already know to be true about the program and what you have learned in prior evaluations or from personal experience working with the program you may not see other patterns that emerge.
Strategies for analyzing qualitative data: • Content analysis: Look for themes that emerge from data such
as interview or focus group responses, and organize responses according to themes. Themes should relate to your research questions. Be cautious about using interview or focus group questions as themes for organizing data because they are topics imposed on the discussion by those conducting the evaluation, and may not have been discussed otherwise. You may find it useful to ask a co-worker to organize your data according to the themes you have developed. Compare how each of you organized the responses to ensure they were categorized in a similar way. Lack of agreement may indicate different understandings of the data. This does not mean that one of you was wrong, but rather that you should have a discussion the data and the themes you have generated to come to a common understanding. Once you’ve organized the data into categories or themes, write a statement or short paragraph that summarizes all the responses in each category.
• Case Studies: If you have collected in-depth information about aspects of the program, you may wish to create summary case records from the extensive documentation, and then write descriptive accounts of each person, company, program, or initiative.
Tips for analyzing quantitative data Quantitative data may be used to answer numerical research questions, such as questions about attendance, retention, test results, and comparisons with a control group. Statistical techniques may be used to convert responses from large surveys or other instruments into averages, frequencies, and percentages, and may also be used to calculate simple correlation. Obtaining expert assistance may be necessary when using advanced statistical techniques. Quantitative data may be tallied by hand or by computer. Hand tallying involves going through each data record and counting the information you are seeking, and is appropriate when total numbers are small. Computers may be useful for compiling data from surveys or long forms. This involves entering responses using a spreadsheet program such as Excel. Staff members who plan to manage data using a computer should be familiar with spreadsheet programs or obtain support from someone with experience managing quantitative data.
Strategies for analyzing quantitative data: • Frequencies: Frequencies are counts of the number of times
an event occurred. For example, frequencies can be used to count the number of times an answer was given on a survey, the number of people who attended a meeting, or the number of people who completed a training program.
• Means: Means are averages that allow you to describe a group of people or items using a single value. For example, you may wish to calculate the average attendance of a monthly meeting over the course of a year. Averages may also be used to describe salaries of program graduates or numbers of women enrolled in partnering training programs.
• Percentages: Percentages can be used to illustrate how part of a group compared to a whole. For example, you may wish to determine the percentage of trainees who were placed in manufacturing jobs, or the percentage of graduates who stay in their placements for a determined amount of time.
The “Interpretation and Reflection” section of your final evaluation report is where you give meaning to the information you have collected, relating your program’s successes and challenges to factors within and outside of the program and organization.
What is interpretation? Interpretation is a step in the evaluation process in which you make sense of the findings you have presented in the previous section. The interpretation section is a narrative account that gives meaning to the information you have collected. In a sense, you are telling a story with the data in which you draw on your findings to explain the program’s successes and challenges in meeting its objectives. The Interpretations section of your evaluation report includes conclusions about the program that you draw based on the data presented in the Findings section of your evaluation report. When making interpretations for your evaluation report:
• Determine what your results say about your program. Compare results obtained with what was expected.
• Discuss which objectives were and were not met and give possible reasons why.
• Determine whether circumstances or events unrelated to the program contributed to or limited its success.
• Consider how different stakeholders would interpret the results. • Discuss whether different data sources yielded different results,
why this might be the case, and the implications for the evaluation.
• Discuss unexpected findings, including anything learned in the evaluation that was relevant to the program but not directly related to the objectives. These types of interpretations may be useful in determining how the program may be changed in the future.
• Compare your results with results from previous evaluations of your program or similar programs.
• Determine whether there is anything you would like to know about your program that your results do not tell you. This may help you to understand the limitations to your evaluation, and assist in developing research questions for an evaluation in the future.
SECTION 5: Interpretation and Reflection
What are reflections and how do they differ from interpretation?
Reflection is a phase in the evaluation process in which you step back from a narrow focus on the program’s objectives and use your data to examine the relationship between the program and the broader context in which it has taken place. The Reflections section is a place to relate the program’s successes or challenges to factors outside the program itself. You may choose to consider the broader social, political, economic, or cultural issues that had an impact on the program. While most funders do not require a Reflections section, reflection allows you to think beyond your interpretations – that is, to put them in a broader context in order to give them more significance. Reflecting on the evaluation may lead to a better understanding of the program and how the program may be improved in the future.
Suggestions for reflecting on your findings • Set program objectives and data aside and think about what you have
learned. Think about the program, what happened, what worked well, how it could have worked better.
• Reflect alone and also with other staff members and program participants for multiple insights about the program.
• Consider whether there are particular program experiences or stories you can tell that might help explain the program’s successes and challenges by linking them to issues beyond the program itself.
• Be creative. Reflections allow you to move beyond the data to make connections between the program and the wider context in which it took place.
Tips for interpreting and reflecting
• Interpretations and reflections should be presented honestly, in a way that will be helpful to the program. The program will not benefit if you present program strengths and successes with no attention to weaknesses or challenges. The aim is to draw on your data to account for both successes and challenges. This will allow you make meaningful recommendations, and to use the evaluation to continue improving your program.
• Data can be interpreted in many different ways. As with your Findings section, it is often helpful to confer with co-workers and other stakeholders such as program participants about your interpretations and reflections. This will help to ensure the Interpretation and Reflection section is presented clearly, carefully, and honestly.
The “Recommendations” section of your final evaluation report is a narrative list of suggestions for modifying or supplementing the program in the future in order to improve its ability to meet its objectives and increase its success. As you list your recommendations, it is useful to explain the basis for each recommendation and why you believe a particular recommendation will improve the program.
Tips for developing recommendations • Recommendations should draw directly on what you have
learned in your evaluation and what you know about the program. They should reflect what was reported in your findings and interpretations sections.
• Recommendations may be directed toward the specific program being evaluated. They may also be directed toward the organization, the community being served, the funding agency, or other stakeholders.
• Think about the objectives you did not meet. Are there program
changes that would help you to meet the objectives? Were the objectives unrealistic? Recommendations can suggest changing how the program is organized or staffed; they can also suggest the modification of objectives.
• Recommendations can be creative, as long as they can be realistically implemented.
• Program staff and other stakeholders may be able to provide useful recommendations for the program, or may be helpful in determining whether recommendations are realistic.
• Focus on a reasonable number of recommendations that will have a significant impact on the program. Avoid listing numerous recommendations that require minor changes or that have no context.
• Be specific in explaining how implementing the recommendation will improve the program.
SECTION 6: Recommendations