Jump to:Page Content
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
Volume XI, Number 2, Summer 2005
Issue Topic: Evaluation Methodology
Robert Penna and William Phillips from the Rensselaerville Institute’s Center for Outcomes describe eight models for applying outcome-based thinking.1
Over the past decade, outcomes has moved from being just a buzzword in the nonprofit, government, and foundation worlds to becoming a full-fledged movement. As the outcomes movement and outcome-based decision making have grown, many models or frameworks for applying this thinking have emerged. While evaluators and practitioners have benefited greatly from the development of various tools to guide outcomes thinking, understanding the unique advantages of each model and how to select the right one is challenging for many.
Responding to this challenge, the Rensselaerville Institute’s Center for Outcomes published in 2004, Outcome Frameworks: An Overview for Practitioners, a book that offers insights into which model might be appropriate to the particular needs of a program at a given point in time. It captures what the outcomes movement means, where it came from, the major models now in use, and the movement’s probable future. The models described in Outcome Frameworks fall into three main categories: program planning and management, program and resource alignment, and program reporting. In addition, most models can be used as an evaluation tool.
Program Planning/Management Tools
Program planning or management tools are outcome models that assist in an effort’s proposal, funding, and implementation phases. They illustrate the logic, theory of change, and anticipated flow of an intervention, providing markers against which both incremental and ultimate progress may be measured. Examples include the following models:
Model 1: The Logic Model. Logic models, the most widely used of these models, provide a graphic overview of a program, outlining the outcomes to be accomplished along with how they are to be achieved and for what groups.2 A logic model generally includes the target group, the resources to be used, activities, and objectives. Best used for describing a program in the broadest strokes, it can be an extremely useful tool, particularly at the earliest stages of a project.3
Model 2: Outcome Funding Framework. This model stresses key shifts in the thinking that traditionally has influenced human service programs. It encourages funders to think like investors4 and encourages programs to shift from emphasizing service activities, to focusing on performance targets, defined in terms of client changes gained. The model also uses milestones, or sequential steps toward achieving ultimate targets, to allow for ongoing assessment and mid-course program corrections.
Model 3: Results-Based Accountability (RBA). This model starts with the desired ends and works backward toward the means to achieve them. RBA first describes what a desired result would look like, then defines that result in measurable terms, and, finally, uses those measures to gauge success or failure. RBA asks and answers three basic questions: What do we want? How will we recognize it? What will it take to get there? This model distinguishes between population accountability and program accountability. Its inclusion of the crosswalk, a tool for matching RBA with other outcome models, is a unique and useful aspect of the framework.5
Model 4: Targeting Outcomes of Programs (TOP). This model is based on a hierarchy of sequential steps in planning, implementing, and evaluating programs. It helps answer four basic questions: Why have a program? How should it be conducted? Has the program design been implemented? What are the benefits delivered?6
Program and Resource Alignment Tools
Program and resource alignment tools ensure that resources and effort are expended in support of organizational goals; one such is the Balanced Scorecard, explained below:
Model 5: Balanced Scorecard. Initially designed as a corporate management framework, the balanced scorecard synthesizes multiple measures, reflecting a range of processes, and links them to a consistent and mutually reinforcing whole.7 The model’s use of a resource or target matrix makes it particularly well suited to organizational alignment.
Program Reporting Tools
Program reporting tools allow organizations to capture and communicate the fullness of the results they have achieved. They include the following:
Model 6: Scales and Ladders. This model offers a matrix-based system popularly associated with the implementation of the Results Oriented Management and Accountability system within community-service-block grant-funded programs.8 The model’s essential concept centers on a series of anchored scales and their placement within a matrix that describes different states or conditions of client status along a continuum.
Model 7: Results Mapping. Results mapping is an outcome-based evaluation tool that systematically captures anecdotal evidence and uses the information to present a results-based conclusion.9
Model 8: Program Results Story. Currently under development, this approach applies the power of the story format to capture organizations’ achievements and present them in a results-based format.10
Overview of Eight Outcome Models
Well Suited For
|1. Logic Model||Diagrammatic representation of a program, showing what it is supposed to do, with whom, and why||Inputs, outputs, outcomes; arrows show relationships between elements in the model||Easy to use; provides easily understood representation of program’s theory of change||Program overview; presentations; program and evaluation planning|
|2. Outcome Funding Framework||Key management focus on the achievement of specific, sequential results for customers of services; emphasis on results, not activity||Investor return, results, customers, milestones, performance targets, outcome statement||Highly disciplined approach that serves both program investors and implementers; Web-based software has strengthened usability||Government and philanthropic grantmaking; program and organization management|
|Real-time approach that describes what desired results look like, defines results in measurable terms, and uses measures to drive action plans for improvement||Results, experience, indicators, baselines, strategy, action plan and budget, accountability||Thorough system for planning community-
change efforts and improvements in program, agency, or system performance; uses lay language and provides direct link to budgeting; useful for integrating different outcome systems
|Project planning and start-up; development of community report cards; program/agency improvement plans and budgets; grantmaking and evaluation design|
4. Targeting Outcomes of Programs
|Tracking progress toward achievement targets; evaluating degree to which programs impact targeted conditions||Knowledge, attitude, skills, and aspiration; process, outcome, and impact evaluation||Fairly easy to use; helps integrate program development and evaluation; implementers and managers can use same concepts||Program design and evaluation|
|5. Balanced Scorecard||Business-based model designed to provide integrated management and accounting for multiple variables impacting organization performance by connecting them to a set of performance indicators||Strategy, alignment, short- and long-term objectives; financial and nonfinancial measures; lagging and leading indicators; performance measures and drivers; internal and external indices of success||Allows for a graphic assessment of the degree to which an organization’s resources and efforts support its goals||Monitoring either a single program with several associated initiatives or multiple programs within an organization; analyzing alignment of resources and initiatives to strategic targets|
|6. Scales and Ladders||Graphic tool that centers around a series of scales and their placement within a matrix designed to illustrate progress along a continuum of stages||Scales; mutually exclusive, multiple, and floating indicators||Places a client, community, or program on a continuum; shows incremental and relative progress, stabilization, or decline; individual data together tell a complete story; behaviorally anchored description of levels of change||Demonstration of aggregate progress; measuring concepts that are not easily quantified|
|7. Results Mapping||Outcome-based evaluation tool designed to systematically capture otherwise nonquantifiable anecdotal evidence||Causal and synchronistic attribution; levels and milestones||Way to systemize, standardize, gather, and utilize lessons embedded in anecdotal information||Turning anecdotal information into a useful tool for program presentation, evaluation, and assessment|
|8. Program Results Story||Uses stories to capture organizations’ achievements and present them in a results-based format||Results, stories, anecdotal evidence||Easily understood approach for presenting results; brings outcomes to human interest level; captures and conveys richness of information||Presenting program and results to multiple audiences|
1 The Rensselaerville Institute is a national nonprofit specializing in outcome-tool creation and assisting government, foundations, and nonprofits to put them into practice. Known as “the think tank with muddy boots,” the Institute develops new outcomes approaches and tools through its Center for Outcomes, and then applies them through partnerships and independent initiatives. www.rinstitute.org
2 Kirkpatrick, S. (2001). The program logic model: What, why and how? Retrieved June 8, 2005, from http://www.charityvillage.com/cv/research/rstrat3.html
3 MacNamara, C. (2000). Guidelines and framework for designing basic logic model. Retrieved June 8, 2005, from http://www.managementhelp.org/np_progs/np_mod/org_frm.htm
4 Williams, H. S., Webb, A. Y., & Phillips, W. J. (1991). Outcome funding: A new approach to targeted grantmaking. Rensselaerville, NY: The Rensselaerville Institute.
5 Friedman, M. (2001). The results accountability implementation guide. Retrieved June 8, 2005, from http://www.raguide.org
6 Bennett, C., & Rockwell, K. (1995). Targeting outcomes of programs (TOP): An integrated approach to planning and evaluation. Washington, DC: Cooperative State Research, Education, and Extension Service. Learn more about it at citnews.unl.edu/TOP.
7 Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard: Translating strategy into action. Boston: Harvard Business School Press.
8 Learn more about it at www.roma1.org.
9 Pacific Institute for Research and Evaluation. www.pire.org
10 The program results story is not included in Outcome Frameworks. Contact the authors for more information on this approach.
Robert M. Penna, Ph.D.
William J. Phillips
Center for Outcomes
The Rensselaerville Institute
63 Huyck Road
Rensselaerville, NY 12147