Jump to:Page Content
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
Volume IX, Number 3, Fall 2003
Issue Topic: Evaluating Community-Based Initiatives
Stephen Bagnato, Robert Grom, and Leon Haynes describe an evaluation design for Pittsburgh’s Early Childhood Initiative that provides scientific rigor in a community setting.
Little agreement exists about how evaluations of social intervention programs should be conducted. Traditional social scientists argue for the use of laboratory-based, control group, randomized designs as the gold standard, but this approach lacks generalizability to real-life settings. Alternative evaluation designs are necessary to document the elements of intervention programs that predict outcomes in natural community settings. Yet critics charge alternative methods with a lack of experimental rigor.1 An evaluation approach known as authentic assessment and program evaluation research meets the demand for rigor while addressing the community setting context.
Pittsburgh’s Early Childhood Initiative
In 1994, as part of the Early Childhood Initiative (ECI), the Heinz Endowments organized the business, corporate, agency, and foundation sectors in Pittsburgh to expand quality early care and education programs and options for unserved children in high-risk neighborhoods. The overarching mission of ECI is to foster preschool and school success for children of poverty, whose typical retention and special education placement rates in kindergarten have ranged between 18% and 40%.
A consortium of business, community, and foundation leaders designed the goals, approach, and expected outcomes of ECI. This design was based on seven core features of successful early childhood programs for children at developmental risk that were identified by Craig Ramey and Sharon Ramey in their article, Early Intervention and Early Experience.2 The seven core features include: (1) longitudinal interventions starting in infancy and monitored through functional benchmarks; (2) intensive, comprehensive, and individualized programs and supports; (3) integral parent participation; (4) high program quality and frequent monitoring; (5) direct child interventions; (6) community-directed programs and integrated services; and (7) follow-through of child and family supports and program evaluation into the primary grades.
Several Pittsburgh urban neighborhoods have participated in this collaboratively designed and privately funded joint venture. Braddock’s 4 Kids Early Childhood Initiative and the Wilkinsburg ECI are two of the most distinctive of these community-driven ventures. A community leadership council established in Braddock forged a relationship between Woodland Hills School District, Head Start, and various formal and informal resources in the community (e.g., churches, U.S. Department of Housing and Urban Development community councils, and local hospital networks) to link services for children and families. In Wilkinsburg, Hosanna House, a broad community service center, incorporated family support programs as central features of their early care and education programs. In fact, these communities have lead efforts to incorporate the School Readiness Group, a nonprofit early childhood consortium, in order to harness the influence of cross-community partners to advocate for government, foundation, and agency funding.
SPECS Authentic Program Evaluation Research Model
In 1996 the Heinz Endowments and the ECI Management Council, composed of business, corporate, foundation, and community members, selected an interdisciplinary research team from Children’s Hospital of Pittsburgh and the UCLID Center at the University of Pittsburgh known as SPECS (Scaling Progress in Early Childhood Settings), as the winners of a national competition to conduct ECI’s longitudinal evaluation.
SPECS’ evaluation approach—authentic assessment and program evaluation research—helps community-based programs demonstrate “how good they are at what they do.” It has been validated in the field through evidence-based research conducted through “natural experiments” in real-life community settings rather than laboratory settings.3 SPECS’ strategies are unique and effective because they:
SPECS’ research methods track progress and interrelationships among multiple factors like children’s development (e.g., basic concepts, literacy, social skills, and self-control behaviors), parenting and family strengths, the standards and “best practices” of early childhood programs, and neighborhood resources and interagency partnerships in systems reform efforts.
The Results of the Early Childhood Initiative
The SPECS evaluation team carefully tracked the progress of 1,350 enrolled children between 1997 and 2003. The team observed and profiled progress three times each year, focusing on thinking, language, early literacy, social, behavioral, and play skills. They regularly provided feedback to teachers and parents to guide their teaching and care. They also conducted program quality evaluations in 25 programs in nine Pittsburgh neighborhoods (Braddock, Wilkinsburg, Sto-Rox, East Liberty, South Side, Highlands, Hill District, Homewood, and Steel Valley).
SPECS’ research on ECI’s impact showed major outcomes in four areas (for details see the box):
To download the full SPECS report or executive summary go to www.uclid.org:8080/uclid/ech_specs.html.
How Children Benefited From the Early Childhood Initiative (ECI)
Social and Behavioral Progress
Early School Success
1 Yoshikawa, H., Rosman, E. A., & Hsueh, J. (2002). Resolving paradoxical criteria for the expansion and replication of early childhood care and education programs. Early Childhood Research Quarterly, 17(3), 3–27.
2 Ramey, C. T., & Ramey, S. L. (1998). Early intervention and early experience. American Psychologist, 53(2), 109–120.
3 Bagnato, S. J., Suen, H. K., Brickley, D., Smith-Jones, J., & Dettore, E. (2002). Child developmental impact of Pittsburgh’s Early Childhood Initiative (ECI) in high-risk communities: First-phase authentic evaluation research. Early Childhood Research Quarterly, 17(4), 559–580.
4 HLM (Hierarchical Linear Modeling) is an analysis that estimates the effects of social units—groups, neighborhoods, schools, workplaces, organizations, communities, social networks, or whole social systems—on individuals. Path analysis refers to the method by which the path of the cause and effect relationship among variables is determined.
Stephen J. Bagnato, Ed.D.
Professor of Pediatrics & Psychology
Director, Early Childhood Partnerships
Children’s Hospital of Pittsburgh
The UCLID Center at the University of Pittsburgh
President and CEO
Heritage Health Foundation, Inc.
Greater Braddock Early Childhood Network
President and CEO
Wilkinsburg Early Childhood Initiative