Jump to:Page Content
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
Selected Evaluation Terms
Priscilla M. D. Little
Evaluation terminology can be confusing to even the most seasoned of evaluators. This resource provides commonly accepted definitions to evaluation terms frequently used in the out-of-school time field. For “real life” examples of how these terms are used, check our Out-of-School Time Evaluation Database, currently offering detailed evaluation profiles of over 20 out-of-school time programs nationwide.
Accountability means that a public or private agency, such as a state education agency, that enters into a contractual agreement to perform a service, such as administer 21st Century Community Learning Center programs, will be held answerable for performing according to agreed-on terms, within a specified time period, and with a stipulated use of resources and performance standards.
(1) An intermediate target to measure progress in a given period using a certain indicator. (2) A reference point or standard against which to compare performance or achievements.
Data collection methods:
Secondary Source/Data Review:
Formative evaluations are conducted during program implementation in order to provide information that will strengthen or improve the program being studied—in this case, the out-of-school time program or initiative. Formative evaluation findings typically point to aspects of program implementation that can be improved for better results, like how services are provided, how staff are trained, or how leadership and staff decisions are made.
An indicator provides evidence that a certain condition exists or certain results have or have not been achieved. Indicators enable decision-makers to assess progress towards the achievement of intended outputs, outcomes, goals, and objectives.
Performance Measurement (also called Performance Monitoring):
“The ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals”¹ (sometimes also called outcomes). Performance measurement is typically used as a tool for accountability. Data for performance measurement is often tied to state indicators and is part of a larger statewide accountability system.
Summative evaluations are conducted either during or at the end of a program's implementation. They determine whether a program's intended outcomes have been achieved—in this case, the out-of-school time program or initiative's outcomes. Summative evaluation findings typically judge the overall effectiveness or “worth” of a program based on its success in achieving its outcomes, and are particularly important in determining whether a program should be continued.
¹ U.S. Government Accounting Office, April 1998.
1. What is the difference between performance measurement and program evaluation?
|Performance Measurement||Program Evaluation|
|Purpose||Provides a broad, shallow snapshot of program functioning. Typically answers the question of whether a program has achieved its objectives, expressed as measurable performance standards.||Provides a narrower, deeper examination of program functioning. Typically answers questions of why a program worked, unintended benefits or consequences of a program, and how a program might be improved or changed.|
|Components||Identification of program goals or outcomes, indicators to measure progress, and regular collection and reporting of data.||Collection of broader range of information on program performance and its context. Information often includes both qualitative and quantitative data.|
|Scope||Usually involves data collection from all sites.||Usually involves data collection from only a subset of sites.|
|Timeframe||Annually, or at least at pre-determined intervals.||As needed.|
|Uses||To examine progress over time, to compare sites, to understand progress toward pre-established outcomes. Can serve as an early warning system to management and a tool for improving accountability to the public.||The more in-depth nature of program evaluation allows for an overall assessment of whether the program works and identification of adjustments that may improve its results. Program evaluation is also used to determine whether a program “caused” outcomes to be achieved.|
2. What are the main features and trade-offs in design choice?
|Experimental Design||Random assignment of individuals to either treatment (i.e., an out-of-school time program) or control groups (i.e., no out-of-school time program); groups are usually matched on general demographic characteristics and compared to each other to determine program effects.||The strongest design choice when interested in establishing a cause-effect relationship. Experimental designs prioritize the impartiality, accuracy, objectivity, and validity of the information generated. They allow for causal and generalizable statements to be made about a population or impact on a population by a program.|
|Quasi-Experimental Design||Features non-random assignment of individuals to treatment and comparison groups, as well as the use of controls to minimize threats to the validity of conclusions drawn. Often used in real-life situations when it is not possible to use random assignment.||Quasi-experimental designs prioritize the impartiality, accuracy, objectivity, and validity of the information generated. However, non-random assignment makes causal and generalizable statements harder to ascertain than when using an experimental design.|
|Non-Experimental Design||No use of control or comparison groups; typically relies on qualitative data sources such as interviews, observation, and focus groups.||Non-experimental designs are helpful in understanding participants' program experiences and in learning in detail about program implementation. No causal or generalizable conclusions can be drawn using a non-experimental design.|
3. What is the difference between quantitative and qualitative data?
|Definition and Uses||Methods|
Free. Available online only.