ETAC Evaluation Resources

Program Evaluation Overview

(Download PDF – 234 KB)
The document provides a summary of program evaluation by first distinguishing among the different types of evaluations, next laying out a detailed outline on how to plan for an evaluation, and finally describing the process of completing an evaluation. Understanding the process and carefully planning and executing the evaluation will allow for a thorough evaluation that will yield rich information.

Evaluation Reporting

(Download PDF – 184 KB)

This document provides a snapshot of how to effectively write comprehensive evaluation reports. Evaluators should align their reports to the overall purpose of the evaluation and keep in mind the needs of their stakeholders. The reports need to fully address all the questions raised in the evaluation plan and detail both the outcome data and process data in order to fully capture a program's progress.

Annual Program Evaluation Report – Qualitative Data

(Download PDF – 141 KB)

This sample evaluation report contains data collected during the evaluation of the Red Apple Connect program. The report aligns the implementation, process, and outcome data to draw comprehensive conclusions. The data collection methods selected for the outcomes framework includes both qualitative and quantitative tools and methods including surveys, interviews, activity logs, observations and document review. The mixed-method approach provides for increased validity of measurement through triangulation, as well as insight into project activities and outcomes.

Annual Program Evaluation Report – Quantitative Data

(Download PDF – 162 KB)

This sample evaluation report assesses the fidelity of implementation of elements of the Red Apple Connect and it includes professional development modules. The relative effectiveness of the individual intervention programs is also assessed to inform both process improvements and sustainability plans. By aligning implementation, process, and outcome data the evaluator is able to draw comprehensive conclusions.

Program Sustainability

Checklist for Sustainability Planning

(Download PDF – 232 KB)

This Checklist, presented at the 2011 Kick-off Conference, includes action steps for developing and implementing effective sustainability plans and practices. The action steps summarize the essential activities that should be performed to identify, and maintain support for, successful program components.

Sustainability Self-Assessment

(Download PDF – 208 KB)

This self-assessment will help projects determine progress in preparing for the transition from grant supported to self-sustaining programs. Each component required to build support for sustainability is presented as an activity for a specific stakeholder group. The survey elements allow the team to rate progress on each activity and determine priorities for future action.

Program Sustainability in Challenging Economic Times

(Download PDF – 1.5 MB)

The Office of Safe and Drug Free Schools (OSDFS) devoted an entire issue of The Challenge to sustaining school-based initiatives begun under five OSDFS grant programs. It contains examples of successful strategies from actual programs, and provides several references and useful resources to use in planning for sustainability.

External Resources

Evaluation

Answering the Questions that Count

(Download PDF – 750 KB)

Although the amount of data available in schools has increased, many educators do not know how to analyze and use all the different types of data available to them, especially formative data. This article outlines “The Essential-Questions Approach”, a process that managing data use around essential questions. This approach is rooted in three components of the systemwide data use: data quality data capacity, and data culture.

Why Evaluation Matters: Determining Effective School Counseling Practices

This article highlights the importance of evaluation in promoting program accountability. The only way program evaluators can tell whether their programs are successful or not is through evaluation; thorough evaluations highly contribute to effective and efficient programs.

A Guide for Education Personnel: Evaluating a Program or Intervention

(Download PDF – 804 KB)

The Guide for Education Personnel lays out the basics of the evaluation process for educators, administrators, state officials, and other members of the school community. It provides an overview of how an evaluation is planned, conducted, and analyzed, rather than a complex or comprehensive study of the field. This guide is intended for education practitioners new to the evaluation field.

Teacher Quality

Approaches to Evaluating Teacher Effectiveness

(Download PDF – 1.2 MB)

The authors of this study evaluated the current research on teacher effectiveness and the various methods and tools used to measure it. The resulting research synthesis provides a thorough discussion of findings as well as implications current evaluations have on public and education policy. The authors consider an expanded definition of teacher effectiveness to include their overall impact on their students, colleagues, and environments, in hopes of providing substantial guidance in this vital field.

What Can Mixed-Methods Designs Offer Professional Development Program Evaluators?

(Download PDF – 152 KB)

This paper considers the role of mixed methods designs and their applicability to evaluations of professional development programs in education. The authors study the literature regarding mixed methods designs and argue that professional development, with its emphasis on accountability and decision-making driven by data. The advantages of mixed methods designs are compared to the additional demands they require, and the authors find their potential benefits to be worthwhile for pursuit in future research.

Evaluation Research to Sustain and Expand an Established PDS

(Download PDF – 114 KB)

The research, conducted with pre-service teachers in the UI-PDS and graduates of the UI-PDS in their first year of teaching, indicates teachers prepared within a PDS model are well equipped to meet the challenges of urban settings. They plan and implement lessons relative to students' diverse backgrounds, interests, and skills while simultaneously engaging in advocacy and collaboration to advance their students' achievement. This research also indicates components of the UI-PDS program that best supported pre-service and novice participants' learning to teach that they specified through interviews and surveys. Components included taking responsibility Evaluation Research 3 for teaching students, daily on-site support, collaborative practice, and personal attributes of participants.

Fidelity of Implementation

A Model for Fidelity of Implementation in a Study of a Science Curriculum Unit:
Evaluation Based On Program Theory

(Download PDF – 114 KB)

This paper proposes a conceptual framework for fidelity of implementation and its measures in the context of a quasi-experimental effectiveness study of a middle school science curriculum unit, Motion and Forces (M&F). In this study of M&F, measures of fidelity were based upon the intervention's program theory, and included the intervention's processes and structure from the points of view of both teacher and students. By using multiple measures of fidelity of implementation for treatment and comparison classrooms and correlating fidelity measures with student outcomes, fidelity of implementation not only provides evidence for the internal validity of the study, but also provides a rich view of how and why the implementation "works", and supports the intervention's program theory.

Implementation: Measuring and Explaining the Fidelity of CSR Implementation

(Download PDF – 1.9 MB)

This study examines data collected by the National Longitudinal Evaluation of Comprehensive School Reform to determine how implementation of different comprehensive school reform (CSR) models varies according to what factors. The authors find variances between different components within certain CSR models as well as between different CSR model-implementation keys. Implementation can be predicted by several factors, and the study notes that implementation levels are closely connected to the leadership style of the principal.

The Evolving Definition, Measurement, and Conceptualization of Fidelity of Implementation in Scale-up of Highly Rated Science Curriculum Units in Diverse Middle Schools

(Download PDF – 464 KB)

In this paper, we report on our progress in defining, conceptualizing, and measuring fidelity of implementation as we complete the fourth year of a six-year study on the scale-up of highly rated middle school science curriculum units in a large diverse public school system within the metropolitan area of Washington, DC (Lynch, Kuipers, Pyke, & Szesze, in press). To begin, we describe our initial attempts to examine fidelity of implementation within our scale-up study. Given that evaluation studies are prerequisite to scale-up studies, we then discuss the dilemma we faced when studying fidelity of implementation of curriculum materials that had not yet been "proven" effective through evidence-based research designs. In addition, the focus of our research is on improving outcomes for diverse student populations, so it is critical that the effects of the curriculum materials on subgroups of students are determined to guide scale-up.