You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.


Program Description

Overview The New York City (NYC) Department of Youth and Community Development’s (DYCD) Out-of-School Time (OST) Programs for Youth Initiative provides funds to support OST programs for elementary, middle, and high school students across NYC. This initiative is designed to address a broad range of developmental objectives for youth and to serve the needs of NYC’s families and communities.
Start Date September 2005
Scope local
Type afterschool, weekend, summer/vacation
Location urban
Setting public school, community-based organization
Participants elementary through high school students
Number of Sites/Grantees 622 programs (2008)
Number Served 50,618 youth (2005–06); 68,449 youth (2006–07); 81,213 youth (2007–08)
Components Programs operate in NYC schools or community-based organizations (called center-based programs). Trained staff provide a variety of activities including homework assistance, academic support activities, sports/recreational activities, and arts and cultural experiences.

The initiative funds three program options. Option I programs serve elementary through high school students and also includes 15 “Priority Middle Schools” in which OST programs collaborate with state-approved Supplemental Educational Services providers to offer intensive services to youth. Option II programs serve students of any grade level and use private match funds to subsidize at least 30% of their OST budgets. Option III programs collaborate with the NYC Department of Parks and Recreation on three different types of programs: therapeutic recreation and educational services for youth with disabilities, academic support programs, and life skills programs that help youth learn how to manage their finances.
Funding Level over $44 million in 2005–2006; over $117 million in 2008–2009
Funding Sources NYC DYCD
Other DYCD contracted with the Partnership for After School Education (PASE) to provide technical assistance and professional development workshops for OST program staff.


Evaluation

Overview Designed to report on the initiative’s first 3 years, the evaluations examine program characteristics and attendance; participants’ social, emotional, and academic outcomes; and the initiative’s capacity to assist working parents and improve community-level capacities to serve youth during the OST hours.
Evaluators Policy Studies Associates, Inc.
Evaluations Profiled Report on the First Year

Implementation of Programs for High School Youth

Report on the Initiative’s First Three Years
Evaluations Planned Policy Studies Associates will continue to evaluate the program, including an evaluation of the OST Transition to High School program.
Report Availability

Russell, C. A., Reisner, E. R., Pearson, L. M., Afolabi, K. P., Miller, T. D., & Mielke, M. B. (2006). Evaluation of DYCD’s Out-of-School Time Initiative: Report on the first year. Washington, DC: Policy Studies Associates.

Pearson, L. M., Russell, C. A., & Reisner, E. R. (2007). Evaluation of OST programs for youth: Patterns of youth retention in OST programs, 2005–06 to 2006–07. Washington, DC: Policy Studies Associates. Available at: www.policystudies.com/_policystudies.com/files/Year_2_Report.pdf

Russell, C. A., Mielke, M. B., & Reisner, E. R. (2008). Evaluation of the New York City Department of Youth and Community Development Out-of-School Time Programs for Youth Initiative: Results of efforts to increase program quality and scale in year 2. Washington, DC: Policy Studies Associates. Available at: www.nyc.gov/html/dycd/downloads/pdf/ost_evaluation_year_2%20_report.pdf

Russell, C. A., Vile, J. D., Reisner, E. R., Simko, C., Mielke, M. B., & Pechman, E. (2008). Evaluation of the New York City Department of Youth and Community Development Out-of-School Time Programs for Youth Initiative: Implementation of programs for high school youth. Washington, DC: Policy Studies Associates.

Russell, C. A., Mielke, M. B., & Reisner, E. R. (2009). Evidence of program quality and youth outcomes in the DYCD out-of-school time initiative: Report on the initiative’s first three years. Washington, DC: Policy Studies Associates. Available at: www.wallacefoundation.org/KnowledgeCenter/KnowledgeTopics/CurrentAreasofFocus/Out-Of-SchoolLearning/Pages/evidence-of-program-quality-and-youth-outcomes.aspx

Contacts

Evaluation Elizabeth Reisner
Founder and Principal
Policy Studies Associates, Inc.
1718 Connecticut Avenue, NW, Suite 400
Washington, DC 20009
Tel: 202-939-5323
Fax: 202-939-5732
Email: ereisner@policystudies.com
Program Denise Williams
Acting Commissioner
Out of School Time Programs
The City of New York
Department of Youth & Community Development
161 William Street
New York, NY 10038
Tel: 212-676-9845
Email: dwilliams@dycd.nyc.gov
Profile Updated May 9, 2011


Evaluation 1: Evaluation of DYCD's Out-of-School Time Initiative: Report on the First Year



Evaluation Description

Evaluation Purpose To examine: (a) the characteristics of the initiative’s programs; (b) who participates in these programs and their patterns of attendance; (c) whether programs affect participants’ social, emotional, and educational outcomes; and (d) whether programs meet the city’s needs for assistance to working parents and for improvement in community-level capacities to serve youth during OST hours.
Evaluation Design Quasi-Experimental and Non-Experimental: Evaluators collected data on all sites from an online participant tracking system and from surveys of program directors and executive directors of provider organizations. Evaluators administered participant surveys in a random sample of 133 Option I programs and in the 15 Priority Middle School programs. Fifteen Option I “in-depth” sites were selected at which to conduct site visits and interviews with staff, program directors, and youth participants, and to collect additional surveys from staff and parents.

Online program data were collected on 40,818 youth in 417 Option I programs; 9,139 youth in 100 Option II programs; and 1,021 youth in 11 Option III programs. Surveys were completed in spring 2006 by:
  • 483 program directors from 543 programs
  • 161 of 190 executive directors
  • 114 staff members at 12 of the 15 in-depth study programs (response rates from 14% to 94% per program)
  • 283 parents at 12 of the 15 Option I in-depth study programs (response rate of 23%)
  • 3,614 youth from 95 programs (71% of the sampled programs and 59% of consented participants across the 95 responding programs)
  • 562 participants at 11 of the 15 Priority Middle School programs (50% of consented participants from the 11 programs)
Interviews with program directors, staff, and youth participants were conducted at the 15 in-depth sites.
Data Collection Methods Interviews/Focus Groups: Interviews focused on understanding program implementation and stakeholders’ perceptions of the programs, as well as the program’s potential benefits to participating youth.

Observation:
Evaluators used a structured observation instrument to observe program activities during 2-day site visits to each of the 15 Option I in-depth study programs in spring 2006.

Secondary Source/Data Review: Evaluators reviewed data from OST Online, DYCD’s program information system that OST programs use to record and maintain program information including participant characteristics and OST participation.

Surveys/Questionnaires: Surveys of the program directors asked about program goals, activities, and schedules; staff recruitment and qualifications; participant outreach, recruitment, needs, and preferences; and linkages with schools, communities, and families.

Surveys of the executive directors asked about how the OST program influenced the fulfillment of the provider organizations’ core missions, how OST programs linked to other services delivered by the provider organization, and the cost and funding of specified program elements.

Participant surveys asked about program perceptions, experiences, perceived benefits, etc.

Program staff surveys asked about staff’s background, qualifications, and experience working in the program.

Parent surveys asked about program perceptions, perceived benefits, and how and whether the program met parents’ needs.

Tests/Assessments: Through observations, evaluators recorded and assessed: (a) staff instructional strategies that foster mastery (e.g., staff communicate goals, purposes, and expectations); (b) the quality of the activity content and structure (e.g., the activity requires analytic thinking); and (c) positive relationships displayed in activities (e.g., youth respect one another). Indicators within these scales were rated from 1 to 7, with higher numbers indicating greater presence of the desired features.
Data Collection Timeframe Data were collected during the 2005–2006 school year.


Findings:
Formative/Process Findings

Activity Implementation According to program director surveys, almost all Option I elementary school programs offered homework help (98%), and the majority offered visual arts and crafts activities (84%), free time for physical play (59%), and unstructured time for socializing (57%). Middle school programs offered a similar set of activities, although organized team sports and dance/movement activities were more common in these programs than in elementary programs. Findings suggest that high school programs tended to be more specialized and civically oriented.

Across observations, the average scale score of the degree to which staff instructional strategies supported mastery was 3.06, on a scale from 1 to 7. Notable differences were evident in the extent to which staff delivered mastery-oriented instruction by activity type and grade level. Homework help and tutoring scored significantly lower on this scale (average score of 2.68, p < .05) compared to all other activity types (3.26). Academic enrichment activities were rated significantly higher (3.62, p < .05) than were other activity types (2.87). Activities in which certified teachers or activity specialists were present were rated significantly higher (3.72, p < .05) than other activities (2.92). Activities designed to promote skill building were rated significantly higher (4.07, p < .05) than were activities designed to allow students to practice existing skills (3.32) or complete homework (2.55). Activities in high school programs were rated significantly higher (4.24, p < .05) than were activities in elementary (2.90) or middle (2.73) school programs.

The average rating on the activity content and structure scale was 4.26 out of 7. Ratings varied based on activity content. Arts activities scored significantly higher on this scale (average score of 4.88, p < .05) than did other activities (4.00). Unstructured activities were rated significantly lower on this scale (2.98, p < .05) than were other activities (4.46). Activities in which certified teachers or specialists were present scored significantly higher on this scale (4.93, p < .05) than did other activities (4.13).

Option II program directors most frequently reported offering discussion-based activities, compared to other types of possible activities. More than half of Option II program directors reported offering the following discussion activities to the majority of participants on an ongoing basis: peer discussion of topics that are important to youth; discussion of diversity issues; mentoring; discussion of issues, events, or problems in the community; and discussion of current events.

Option III program director reports of activity offerings demonstrate a focus on academic enrichment activities. At least 7 of the 10 responding program directors reported regularly offering the following activities: peer discussion of topics important to youth, organized writing activities, organized reading activities, and recreational reading.

About half of OST participants surveyed “strongly agreed” that their program exposed them to new and interesting activities, and more than half “agreed a lot” that the program gave them a chance to do “a lot of new things” (53%) and that program activities “really got them interested” (56%). Further, the majority of youth surveyed agreed “a lot” that, in the program, they “had a good time playing with other kids” (70%); had “a lot of friends” (69%); and “got to know other kids really well” (64%).

High school participants were more likely to report playing leadership roles in the OST program than were middle school participants. In particular, high school participants were more likely than middle school participants to report that they had (a) been asked by staff for their ideas about the program (67% vs. 56%); (b) led a program activity (58% vs. 51%); (c) helped plan a program activity or event (57% vs. 37%); (d) helped out in the office (54% vs. 26%); (e) helped out on a youth council, advisory group, or leadership team for the program (49% vs. 30%); and (f) helped with meetings for parents or community members (39% vs. 20%).

Nearly three quarters of parents reported that the academic activities were most important to them: 43% selected homework help and 28% selected academic enrichment as the most important activity offered.

Youth who participated in programs with more sports and fitness activities were less likely to report new and engaging experiences in the OST program than did youth participating in programs with fewer such activities.
Costs/Revenues DYCD awarded over $44 million to the 528 programs managed by 187 provider organizations included in the evaluation’s participation analyses, with an average of $84,000 per program, ranging from $3,000 to $340,000.

For the most part, provider organizations had sizable budgets: 52% of executive directors reported that their annual organizational budget was more than $3 million, and an additional 10% had an annual budget of more than $2 million.

On average, organizations drew 59% of their OST budgets from DYCD funding. Twenty percent of organizations relied exclusively on DYCD funds for their OST budget.

Salaries reported by Option I program directors ranged from below $30,000 to above $55,000 per year, with 23% of directors making less than $35,000 per year, and 24% making over $50,000 per year. Nine percent of program directors reported earning an hourly wage, ranging from $14 to almost $40 per hour, with an average hourly wage of $22.82. Program directors most frequently reported a staff wage of $11 to $15.99 per hour (41% of programs); 20% of programs reported a lower staff wage of $6 to $10.99 per hour.

More than one third of executive directors reported that DYCD participation had increased their opportunities to access city funding sources other than DYCD (38%).
Parent/Community Involvement Three quarters of program directors reported that outside organizations, in addition to the provider, offered special programs, activities, or services for youth at their program. More than half of programs reported that outside organizations donated materials or supplies (58%), provided funding through grants or contracts (57%), referred youth to the OST program (56%), and provided special programs, activities, or services to families (56%).

Programs reached out to parents to varying degrees. Over a third of Option I programs (35%) reported having a parent liaison or parent outreach coordinator. Program directors reported most frequently communicating with parents through phone conversations and meetings with parents (55% and 46%, respectively) and reported doing so at least weekly. Program directors also reported offering parents opportunities to attend events at the program (90%) and to attend cultural or recreational events in the community (75%).
Program Context/ Infrastructure Among all sites included in the evaluation’s analyses of participation, 314 were based in NYC schools, and 214 were center-based programs.

Almost all Option I program directors reported in surveys that providing a safe environment for youth was a major objective of their OST program (98%).

Option I participants reported relatively high levels of attachment to their OST program; the majority “agreed a lot” that they felt safe (74%), and that they “belonged” (60%) in the program and that the program was “a good place to hang out” (60%).

Middle school participants were significantly less likely (p < .05) to report that, in the program, they felt that (a) their ideas counted (44% as compared to high school participants at 50%), and (b) they were safe (71% as compared to elementary school participants at 76%).

Center-based program participants were significantly more likely (p < .05) to report feeling a strong attachment to their OST program. For example, youth in center-based programs were more likely than youth in school-based programs to report feeling like (a) they were safe (78% vs. 73%), (b) they belonged (64% vs. 59%), (c) they mattered (60% vs. 55%), and (d) their ideas counted (51% vs. 45%) in the program. These differences were accentuated for the high school participants.

More than three quarters of surveyed parents (81%) reported working at least 20 hours per week and about a third (34%) reported that they were enrolled in school.

The challenges to implementing high-quality programming reported most often by program directors were that families were not adequately involved in the program (64%) and that programs lacked sufficient funds (52%). Program directors also noted low participation rates as a challenge: 42% said that youth dropped out due to lost interest, and 40% said that youth did not attend OST activities regularly enough to have enriching experiences.
Program–School Linkages In general, Option I program directors reported inconsistent communication with school-day principals, teachers, and other key school staff. Center-based program directors reported significantly less communication with school staff than did directors of school-based programs (p < .05). Among school-based programs, more than half of program directors reported communicating with school staff at least once a month on the following topics: individual students’ needs or progress, issues related to classrooms/sharing space, homework assignments, planning OST program content, and student discipline policies.

Program line staff reported minimal interaction with school-day staff. Less than a third reported communicating with school staff on a monthly basis about any of the topics on the survey. Among those engaging in such communication, the most frequent topic was homework assignments (reported by 32% of staff).

Some schools and programs implemented formal communication structures. For example, in one school, all OST service providers met twice a month with the principal to discuss issues related to the use of space and the coordination of their various activities. In another program, teachers of grades K–2 gave program staff a weekly homework log to help staff and participants use program time effectively. In many other programs, staff communicated informally through conversations with teachers about student behavior and academic progress.
Recruitment/ Participation Across the board, programs successfully enrolled participants. According to OST Online data, 528 programs served 50,978 youth during the 2005–2006 school year. In addition, Option I programs served more youth than were specified in their contracts, with programs funded to serve 31,668 and actually serving 40,818. Nearly three quarters of these programs met or exceeded their enrollment targets.

Option I program directors overwhelmingly reported that they offered open enrollment to all youth interested in attending the program (91%). In addition, more than half (59%) of directors reported recruiting youth who were recommended by school-day teachers.

Parents reported a host of reasons for enrolling their children in the OST program. The most frequently reported reasons included: “I believed the program would help my child do better in school” (86%); “I wanted my child to participate in new types of activities” (83%); “I wanted my child to be safe after school” (82%); and “I wanted my child to get help with homework” (79%).

According to OST Online, the majority of program participants were in the elementary (44%) or middle (32%) grades. Programs served approximately equal numbers of boys and girls. Hispanics/Latinos were the largest racial/ethnic group served (39% of participants).

Of Option I programs, elementary school participants attended programs an average of 311 hours during the year, compared to the expected average of 432 hours; middle school participants attended an average of 154 hours, compared to the target of 216 hours; and high school participants surpassed their targeted attendance of 76 hours, attending an average of 97 hours. A quarter of elementary and middle school participants met the targeted number of hours, while 39% of high school participants met attendance expectations. Evaluators caution that these calculations of attendance levels may have been suppressed by program-level difficulties in entering data into OST Online.

Center-based programs were more likely to enroll older youth than were school-based programs. In center-based programs, almost half of enrolled youth were in high school, with another 18% in middle school and 35% in elementary school. In contrast, 48% of youth enrolled in school-based programs were in elementary school, 39% were in middle school, and 14% were in high school. Center-based programs also enrolled a higher percentage of African American youth than did school-based programs (40% vs. 33%).

Programs serving younger youth differed from those serving older youth in how they targeted participants. Elementary school programs were significantly more likely than high school programs to seek out youth who scored “below proficient” on academic assessments (43% vs. 27%, p < .05), according to program director surveys. Both elementary and middle school programs were significantly more likely than high school programs to recruit youth who were identified by their school as needing special assistance in reading and/or math (48% and 50% vs. 30%, p < .05). Similarly, high school programs were significantly less likely than elementary or middle school programs to target youth who received free or reduced-price lunch (21% vs. 45% and 40%, p < .05), or to target youth with siblings attending the program (25% vs. 59% and 49%, p < .05). Elementary school programs were also more likely to target English-language learners than were high school programs (43% vs.19%, p < .05).

School-based program directors reported recruitment approaches that differed from those used by center-based program directors. School-based programs were significantly more likely than center-based programs to seek to serve youth who were (a) recommended by school-day teachers or counselors (63% vs. 49%, p < .05) and (b) identified by their school as needing special assistance in reading and/or math (46% vs. 36%, p < .05). Center-based programs were more likely than school-based programs to serve youth who participated in other programs sponsored by the organization (45% vs. 34%, p < .05).

Of Option II program directors, 64% reported that they offered open enrollment for all interested youth and 43% reported seeking to serve youth who were recommended by school-day teachers or counselors. All 10 surveyed Option III program directors reported that they allowed open enrollment for all interested youth. Half of these directors reported trying to serve youth who scored below proficient on their city or state assessments or who were identified by their school as needing special assistance in reading and/or math.

Participants, particularly high school students, reported participating in other programs or activities after school on occasion: 64% of high school participants reported participating in a different afterschool activity at least once a week vs. 38% of elementary and 37% of middle school participants. Participants also reported spending afterschool time supervised by an adult in a home at least once a week (61% of elementary school participants, 70% of middle school participants, and 75% of high school participants). In addition, 57% of high school participants and 21% of middle school participants reported going to a job after school at least one afternoon a week.
Satisfaction Overall, parents were satisfied with the quality of their child’s OST program; 62% rated the overall program quality as excellent. Parents were particularly satisfied with the program’s ability to provide a safe space for youth to participate in activities and interact with other youth. About two thirds of parents strongly agreed that their child was able to join activities that they would not otherwise experience and felt their child was safer in the OST hours as a result of the program, and 62% reported that their child made new friends.
Staffing/Training Activities demonstrated an average score of 5.31 out of 7 on the positive relationships scale. There were few significant differences in the extent to which evaluators observed a positive atmosphere in the activities. Most notably, homework and tutoring were associated with a significantly lower atmosphere rating, compared to other activity types (4.97 vs. 5.49, p < .05).

More than half of Option I OST program directors reported in surveys that they had directed an OST program prior to the OST initiative (60%), as did similar percentages of Option II and III program directors. In general, program directors were well educated: 86% had completed a 4-year college degree or higher, as did similar percentages of Option II and III program directors. On average, OST program directors reported high levels of satisfaction with their jobs and the level of support they received: 91% of Option I program directors “agreed a lot” that they found the work at their program to be rewarding.

Fifty-six percent of Option I program directors reported hiring a master teacher or educational specialist, 62% of directors reported employing staff with teaching certificates, and 78% said that at least some staff had a college degree. More than three quarters (78%) of Option I program directors reported hiring college students, and less than half (45%) employed teen staff. Option II programs’ staff tended to have fewer experienced professionals than Option I programs while Option III programs tended to employ staff similar to those in Option I programs.

In surveys, all program directors acknowledged challenges in hiring program staff: 33% reported that it was a major challenge to offer competitive salaries, and 25% reported that it was a major challenge to find volunteers with the necessary time and expertise. Greater percentages of Option II and III program directors reported these issues as challenges.

In interviews, program directors described strategies they used to recruit and hire quality staff. For example, one program director asked teen staff candidates to submit their report cards and to write an essay in order to test their ability to provide support to middle school youth. Another described a preference for recruiting college students studying education. In one instance, a program hired former participants who had a strong connection to the program.

OST programs provided support to their staff through program-level supervision and professional development as well as through opportunities to attend professional workshops offered through the OST initiative and the provider organization. Almost all Option I program directors reported holding meetings with their OST program staff at least monthly (98%), with 39% reporting doing so at least weekly. A majority of program directors required at least some staff to submit lesson plans on a regular basis (68%) and used a published or externally developed curriculum (57%). Program directors received technical assistance (TA) most frequently in the use of OST Online (83%), followed by program design and implementation (54%) and program management and administration (51%). Similar patterns were seen in Option II and III programs.

Most providers (over 80%) offered program directors paid vacation/sick time, training/ professional development, and staff meeting/conference attendance. The majority also offered program directors health insurance (77%) and a retirement savings plan (59%).

Part-time program staff received few fringe benefits. The main benefits reported by executive directors were paid training or professional development (60%) and paid attendance at staff meetings and conferences (58%). Only 19% offered paid time off, 15% offered health insurance, and 10% offered a retirement savings plan

Program directors reported high jobs satisfaction levels. Of Option I program directors, 91% “agreed a lot” that they found their work rewarding, 88% reported that they enjoyed working at the program, and 77% reported that they got the support and feedback they needed from their supervisor. Program directors’ job satisfaction did not vary by the grade level served or by whether the program was center-based or school-based.

The majority of executive directors reported offering program staff both professional development opportunities to advance their careers (90%) and opportunities to move to other positions within the organization (77%). Somewhat fewer executive directors reported opportunities for promotions (67%) or raises (58%) within the OST program.

In general, participants reported positive interactions with program staff. About two-thirds of participants “agreed a lot” that staff treated them with respect, and reported that staff thought that they could learn new things, although only 44% of youth “agreed a lot” that staff always kept their promises. Middle school participants were less likely to report positive feelings toward staff than were participants in either elementary or high school programs. Center-based programs participants were more likely to report certain types of positive interactions with staff, especially among high school participants.

In programs offering higher levels of academic activities, youth reported lower quality interactions with staff. High levels of physical fitness activities were negatively associated with participant reports of positive interactions with program staff.
Systemic Infrastructure In the 1st year of the initiative, program directors worked to create program policies and procedures that ensured compliance with city and state regulations and that provided a foundation for a positive program environment. More than three quarters of Option I program directors reported “to a great extent” that their program had policies in place to report suspicions of child abuse or neglect, ensure enough time for program activities, and deal with participant behavior. Among elementary school Option I program directors, 99% reported that policies were in place to report suspicions of child abuse or neglect.

When asked how their OST contract affected the organization’s operations, executive directors reported the most impact on staff training and TA opportunities: 74% responded that DYCD increased their opportunities to participate in training “to a great extent” or “somewhat.” For the majority of providers, DYCD also increased their opportunities to partner with city agencies (62%), a public school (58%), and cultural organizations (55%).

The majority of provider organizations had experience operating OST programs prior to the DYCD initiative. Of executive directors surveyed, 87% said that their organization had previously operated an OST program. According to program directors, 77% of provider organizations had operated programs since at least the 1997–1998 school year, although almost half (44%) reported that the initiative brought OST programming to their location for the first time. Seventy-seven percent of Option II program directors reported that their programs existed prior to the DYCD grant. More than half (55%) had been in operation for 6 or more years. In contrast, 9 of 10 Option III program directors reported that OST programming was not offered prior to the DYCD grant.

In comparison to their organization’s other OST programs, approximately three fourths of executive directors reported that DYCD OST program staff more often tracked program attendance, 46% reported that DYCD OST programs offered programming on weekends and holidays more often, and 34% reported that their organization’s DYCD OST programs offered academic programming and enforced minimum attendance policies more often.

Of Option I program directors, 26% reported that the training and TA through the OST initiative served their purposes completely, and another 64% reported that it was a good start. Thirty-nine percent of program directors reported that implementing the ideas and strategies presented in training and TA had improved their project; 43% were in the process of implementing the strategies. Of program directors who reported that their program did not implement these strategies, the most common obstacle was a need for further training (reported by 50% of program directors). More than half of program directors reported that additional training on the following topics would benefit their staff: youth development (75%), academic enrichment and literacy development (59%), classroom management (57%), and fine and performing arts (51%). Program directors most frequently reported that the training topics they found most useful were staff supervision (54%), program design (52%), and program management (51%).


Summative/Outcome Findings

Academic Elementary grade participants were more likely to report academic benefits compared to either middle or high school participants. For example, significantly more elementary (67%) than middle (58%) and high (33%) school participants “agreed a lot” that the program helped them finish their homework more often (p < .05). Elementary school participants were also significantly more likely to report that the program helped them feel better about their schoolwork (53%), compared to middle (35%) and high (31%) school youth (p < .05). A similar pattern emerged in youth reports of academic self-esteem.

For specific academic areas, just over half of elementary school youth reported that the program helped them solve math problems better and get better grades in school. Only 37% of middle school youth agreed with each item. A quarter of high school youth reported that the program helped them solve math programs better, and almost a third agreed that the program helped them get better grades in school. In addition, just over half of elementary school participants felt that the program helped them read and understand better, compared to only a third of middle and high school participants. Forty-five percent of elementary school youth reported that the program helped them write better, compared to about a third of middle and high school youth.

High school participants were significantly more likely than elementary school participants to report that the program helped them learn to use computers to do schoolwork better (35% vs. 29%, p < .05).

School-based program participants were more likely than center-based program participants to report that the program helped them solve math problems better (44% vs. 36%, p < .05) and finish their homework more often (60% vs. 48%, p < .05).

Participants who attended programs with a strong academic focus reported more academic benefits from participation and higher academic self-esteem. Higher levels of arts activities were also positively associated with self-reported academic benefits and academic self-esteem.

Frequency of OST attendance, especially in school-based programs, was related to greater youth reports of academic self-esteem.

About half of responding parents “strongly agreed” that their child benefited academically from program participation: 55% felt their child was getting needed academic help, 54% felt that their child talked to them more about what was going on in school, and 54% felt that their child was doing better in school overall as a result of program participation.
Family Seventy-one percent of parents “strongly agreed” that the program hours fit their needs, 63% reported that they missed less work than before the program, and 61% reported that the program made it easier for them to keep their job.

 


Evaluation 2: Implementation of Programs for High School Youth



Evaluation Description

Evaluation Purpose To describe youth engagement in OST high school programming, program features and approaches associated with social and educational benefits for youth, and strategies used to build program capacity.
Evaluation Design Non-Experimental: Data were collected on 122 programs serving high school youth in each of the 2005–06 and 2006–07 school years and on 111 programs in operation in fall 2007. Surveys were completed by 110 Option I high school program directors and 1,238 Option I high school participants in spring of 2007. To survey high school participants, evaluators first randomly selected 43 high school programs. Participant data represent the responses of high school participants attending 29 of these 43 randomly selected high school programs in the evaluation’s participant-survey sample (67% of sampled programs).

In addition, the evaluators selected eight Option I high school programs as “in-depth” sites: three were part of the randomly selected in-depth sample of 15 Option I programs identified during the first year of the OST evaluation, and the other five were selected based on information available from DYCD Online and DYCD program managers indicating that the programs used varied program models and activity strategies. During visits to the in-depth sites, evaluators conducted interviews with program directors, staff, and youth participants, and observed program activities.
Data Collection Methods

Interviews/Focus Groups: Interviews focused on understanding program implementation and quality, and the factors influencing youth engagement in their OST programs.

Observation: Observations examined program implementation and quality.

Secondary Source/Data Review: Evaluators analyzed patterns of enrollment and participation in high school OST programs that had entered data into DYCD Online, the agency’s participant tracking system, over the first 2 years of the initiative and in the fall of the 3rd year. Evaluators also examined program-level data from DYCD Online on the types of activities that programs offered and the number of hours available for each type of activity.

Surveys/Questionnaires: Program director and youth surveys focused on perceptions of the program environment, quality and activity implementation, and participation benefits.

Data Collection Timeframe Data were collected from the fall of 2005 to early 2008. Observations were conducted in 2007–2008.


Findings:
Formative/Process Findings

Activity Implementation Over half of high school OST programs offered a wide range of program activities, while the rest tended to focus on a single area that most often reflected the sponsoring organization’s core mission or expertise.

Across all OST high school programs, academic enhancement activities and recreational activities were most commonly offered. These and other activities frequently incorporated opportunities for both formal and informal youth leadership and input.

Programs provided opportunities for youth to learn about careers and college, participate in internships, and gain understanding of both their local and global communities.
Program Context/ Infrastructure According to interviews, programs focused on two main areas in their efforts to build internal capacity to successfully serve high school youth: (a) improving staff skills and expertise, and (b) maximizing program resources through funds development and external partnerships.

Participants reported that their OST programs exposed them to new, beneficial experiences. More than two-thirds of participants “agreed a lot” that they “feel safe” in their program (69%), and more than half felt that “they belong” (59%), they “matter in their program” (56%), and they “are successful in the program: (53%). More than half “agreed a lot” that they “get along well with other participants in the program” (52%), “have a lot of friends in the program” (52%), and “get to know the other participants in the program well” (51%).
Recruitment/ Participation OST programs serving high school students enrolled 8,332 participants in the 2005–06 program year and 13,097 participants in the 2006–07 program year across 122 programs. At the beginning of 2007–08 program year, 8,790 participants were enrolled in 111 programs serving high school. From the initiative’s inception through December 2007, a total of 24,944 high school students have participated in the initiative’s programs.

On average, high school participants attended 97 hours of OST programming during the first year of the initiative and 105 hours in the second year.

The evaluation identified four features that were typically present in programs that were especially successful in reaching and serving older youth: (a) creative, age-appropriate strategies to recruit youth and encourage their continued participation; (b) staff who could relate to youth and staff with expertise in activity content areas; (c) activities designed to meet the developmental needs of older youth, (e.g., career- and college-oriented activities and leadership opportunities); and (d) program partnerships to increase the fiscal and other resources available to the program.

Overall, high school participants in center-based OST programs attended more hours (116 on average) of OST programming than did participants in school-based OST programs (79), consistent with the greater number of program hours offered by center-based programs (373 hours vs. 286 hours, respectively).

Program directors reported varied strategies for recruiting and retaining high school youth in OST programs, including the use of stipends, the encouragement of strong staff relationships with participants, youth-friendly outreach methods, and development of strong relationships with schools.
Staffing/Training Program activities were typically led by professionals, including teachers and program specialists, and were supported by young staff members trained to work with high school youth and to deliver enriched activity content. In order to provide diverse programming by qualified staff, program directors often brought in people from their partner organizations to provide staffing.


Summative/Outcome Findings

Academic Participants reported in interviews and surveys that their OST programs contributed to improvements in their grades and educational performance. Specifically, majorities reported that the program helped them get better grades in school (77%), feel better about schoolwork (76%), finish homework more often (68%), read and understand better (71%), write better (67%), better use computers for schoolwork (64%), and solve math problems better (58%).

 


Evaluation 3: Report on the Initiative’s First Three Years



Evaluation Description

Evaluation Purpose To explore the associations among program quality, patterns of youth participation, and youth outcomes.
Evaluation Design Quasi-Experimental and Non-Experimental: Evaluators collected data on all 622 sites from an online participant tracking system and surveys of program directors and executive directors of provider organizations. Evaluators administered participant surveys in a random sample of 133 Option I programs. Fifteen Option I “in-depth” sites were randomly selected at which to conduct site visits and interviews with staff, program directors, and youth participants, and to collect additional surveys from staff and parents.

Online program data were collected on 40,584 youth in Option I programs in Year 1; 56,742 youth in Option I programs in Year 2; and 67,524 youth in Option I programs in Year 3. Surveys were completed in spring of each year by
  • 483 program directors from 543 programs in Year 1, 470 program directors of 547 programs in Year 2, and 555 program directors from 630 programs in Year 3
  • 161 of 190 executive directors in Year 1, 148 of 191 executive directors in Year 2, and 169 of 203 executive directors in Year 3
  • 114 staff members at 12 in-depth sites (response rates from 14%– 94% per program) in Year 1, 191 staff members at 13 in-depth sites in Year 2, and 193 staff members at 14 of the 15 in-depth sites in Year 3
  • 283 parents at 12 of the 15 in-depth sites in Year 1 (response rate of 23%), 500 parents at 12 in-depth sites in Year 2, and 450 parents at 13 in-depth sites in Year 3
  • 3,614 youth from 95 programs in Year 1; 5,336 youth from 100 programs in Year 2; and 6,301 youth from 108 programs in Year 3
In each in-depth site, evaluators conducted interviews with the program director, 2–3 staff members, and approximately 5 participants, in each year.  They also interviewed 2–3 parents, mostly in elementary sites.

To analyze the impact of OST participation on academic achievement and school engagement, participants were compared with similar students who did not participate in an OST program. For each OST program, the evaluation identified a primary feeder school (i.e., the school that most of the OST program participants attended). Characteristics of the primary feeder school (e.g., proportion of English language learner [ELL] students, and English language arts [ELA]/math results) were used to identify similar schools without OST or other city-run youth programs from which to select nonparticipants. The evaluation then matched a sample of students from the comparison schools with OST participants, based on grade in school, gender, race, free or reduced-price lunch eligibility, and ELL status. For two high school programs that attracted participants from across the city, similar nonparticipants were selected from a set of schools in the same district without OST programs. This process yielded an evaluation sample of 3,093 participants and 3,093 nonparticipants for the quasi-experimental sample.
Data Collection Methods

Interviews/Focus Groups: Interviews with staff, program directors, and youth gathered information on their perceptions of the program.

Observation: Evaluators used a structured observation instrument to observe program activities during site visits to each of the 15 Option I in-depth study programs.

Secondary Source/Data Review: Evaluators reviewed data from OST Online, DYCD’s program information system, which OST programs use to record and maintain program information, including participant characteristics and OST participation. Evaluators also collected data on all youth in the quasi-experimental sample from Department of Education databases. These included academic performance data on school attendance, math and ELA test scores, and total credits earned.

Surveys/Questionnaires: Surveys of the program director asked about program goals, activities, and schedules; staff recruitment and qualifications; participant outreach, recruitment, needs, and preferences; and linkages with schools, communities, and families.

Surveys of the executive directors asked about how the OST program influenced the fulfillment of the provider organizations’ core missions, how OST programs linked to other services delivered by the provider organization, and about the cost and funding of specified program elements.

Participant surveys asked about such issues as youth participants’ perceptions of the program including its benefits and their experiences in the program.

Program staff surveys asked about staff’s background, qualifications, and experience working in the program.

Parent surveys asked about program perceptions, perceived benefits, and how and whether the program met parents’ needs.

Tests/Assessments: Through observations, evaluators recorded and assessed: (a) staff instructional strategies that foster mastery (e.g., staff communicate goals, purposes, and expectations); (b) the quality of the activity content and structure (e.g., the activity requires analytic thinking); and (c) positive relationships displayed in activities (e.g., youth respect one another). Indicators within these scales were rated from 1 to 7, with higher numbers indicating greater presence of the desired features.

Evaluators constructed various scales from participant survey questions, including scales tapping youth participants’ sense of belonging in the program, engagement in pro-social behaviors such as helping others, interacting with peers and staff, academic motivation, and perceptions of academic benefits from the program.

Based on all correlations between program features (as measured by director surveys, the OST Online system, and aggregated youth surveys) and youth outcomes over the three years, evaluators created a program quality index as a tool for assessing the overall quality of an OST program. Final components of the program quality index included exposure to new experiences, youth interactions with peers, youth interactions with staff, wide mix of staff, presence of a master teacher, and presence of a parent liaison.

Evaluators also collected Regents exam test score data from the New York City Department of Education. These exams are mandatory state-level subject-area tests administered to all students in the state of New York in grades 3–12.

Data Collection Timeframe Data were collected in Year 1 (2005–2006), Year 2 (2006–2007), and Year 3 (2007–2008).


Findings:
Formative/Process Findings

Activity Implementation Nearly all elementary and middle school OST programs offered academic enhancement, arts and culture, and recreational activities. About two thirds of these programs offered life skills activities, and about half offered community-building activities. Consistent with the age ranges they serve, relatively few elementary and middle school OST programs offered any career or work activities (15% and 24%, respectively).

High school OST programs often focused on a narrow set of activities or topic areas. Reflecting that focus, the activity patterns in high school programs differed from those of elementary and middle school programs: while more than two thirds of high school programs offered academic enhancement, arts and culture, and recreational activities, only about half of these programs offered life skills, community-building, and career or work activities.

Overall, participants reported a high degree of exposure to new experiences through the program (average score of 3.20 out of 4 on the youth survey scale). Elementary school youth were somewhat more likely to report a high level of participation in new experiences (3.34 out of 4) than were middle school youth (3.22) and high school youth (3.27).

On average, observed activities rated highly on clear activity goals (5.07 out of 7) and the extent to which activities developed personal and social skills (4.21 out of 7). In contrast, observations revealed less evidence of activity quality as defined by the extent to which the activity built on skills and content previously learned (3.60, out of 7) or engaged youth in active, hands-on learning experiences (1.75, out of 7).

Parents especially valued the academic support features of OST programs: 47% cited homework help as the most important activity in the OST program, and an additional 26% cited academic enrichment as the most important activity.
Parent/Community Involvement Nearly all program directors (91%) had conversations with parents over the phone at least a few times a month; 83% met in person with parents with similar frequency.

Programs relied on family or parent liaisons to engage families and encourage greater participation: 45% of programs employed someone for this role.
Program Context/ Infrastructure The initiative expanded the number of programs offering OST services over the 3 years of the evaluation, with the addition of 111 new elementary school OST programs in the third year.

Participants reported an average score of 3.38 out of 4 on the youth survey scale of “sense of belonging” in the program. In particular, more than two-thirds of participants “agreed a lot” that they “feel safe” in the program (68%) and 58% said that the program was a “good place to hang out.” High school students were especially likely to report a strong sense of program connection (with an average score of 3.48, compared to 3.32 for elementary school participants and 3.38 for middle school participants).

Analyses of youth reports revealed positive relationships between the extent to which a program exposed youth to new experiences and of the youths’ sense of belonging in the OST program (p < .05).

Measures of a supportive OST environment, including youth reports of interactions with their peers and with staff members in the program, were positively and significantly correlated with youth reports of their sense of belonging (p < .05).

Analyses found positive and significant correlations between overall program quality and aggregate youth reports of their sense of belonging in the program (p < .05).
Program–School Linkages Program directors reported communicating with school administrators or staff at least monthly about the needs or progress of individual students (61%), issues related to sharing classroom space (56%), homework assignments (56%), and student discipline policies (50%).

In surveys, 61% of program directors said that receiving responses to requests to coordinate services or resources with school staff was not a challenge; only 6% reported that this issue was a major challenge.
Recruitment/ Participation The OST initiative grew over its first 3 years to serve increasingly large numbers of youth each year. In Year 1, 50,618 youth enrolled in OST programs throughout the city, including 40,584 participants in the Option I programs. In Year 2, this number had increased to 68,449 participants overall with 56,742 in Option I programs, and by Year 3 the initiative had grown to serve 81,213 youth, including 67,524 Option I program participants.

A total of 6,371 youth participated in all 3 years of OST school-year programming. Almost 22,000 youth participated in 2 years of OST programming, either in Years 1 and 2 or in Years 2 and 3, while 102,837 participated in a single year of programming (Year 1, 2, or 3). Approximately 22,000 youth participated only in summer programming.

In the third year of the OST initiative, programs on average exceeded their targeted enrollment levels (measured by the number of slots available for participants as established in the program’s contract with DYCD). Option I programs had a target enrollment overall of approximately 63,000 youth; programs actually served a total of about 64,500 students during the school year. Even so, some individual programs could not meet their targeted enrollment: 31% of elementary school youth programs did not meet or exceed their enrollment targets, nor did 30% of middle school youth programs or 41% of high school youth programs.

On average, elementary school youth participants attended 377 hours of OST programs during the year, compared to the 432 hours they were expected to attend. This attendance represents an average of 87% of targeted hours, exceeding average participation rates of 72% and 83% attained in Year 1 and Year 2, respectively. Middle school youth participants as a group achieved their targeted number of hours of participation: on average, middle school participants attended 218 hours of the 216 hours expected at the middle school level. High school participants also exceeded their target of 76 hours of participation, attending on average 92 hours in Year 3.

A program’s mean rating of exposing youth to new experiences was negatively correlated with mean hours of participation of individual youth in OST; this correlation was significant (p < .05).

The breadth of content in OST programs, measured by the number of different activity types offered, was positively and significantly associated (p < .05) with the total number of hours of youth participation.

Measures of communication with schools and with parents were positively and significantly associated (p < .05) with the number of hours of youth participation in OST programming.

Analyses found a positive and significant relationship between the program quality index and whether the program succeeded in meeting its targeted enrollment level (p < .05).

Parents’ reports of their reasons for enrolling their child in the OST program reflected an emphasis on seeking academic support: 76% believed the program would help their child do better in school, and 72% wanted their child to get help with homework. Seventy-five percent of parents also said that they enrolled their child in an OST program to provide them with the opportunity to participate in new activities.
Satisfaction Youth reported a high level of social interactions within the program (average score of 3.32, out of 4), although both middle and high school youth responded somewhat more positively than did elementary school youth participants (3.35 and 3.37 vs. to 3.28).
 
About three quarters of parents rated the OST program that their child attended as either excellent (43%) or very good (33%).
Staffing/Training The majority of programs (82%) hired college students, and 69% of programs hired at least one professional specialist (e.g., a professional artist, coach, or dancer). In addition, 63% of programs had at least one certified teacher on staff, and 61% hired teen staff members. Fifty percent of OST program directors reported that a staff member was assigned to be a master teacher or educational coordinator within the program.

OST programs were strategic in the roles they assigned to certified teachers and specialists within the programs. Certified teachers were employed mainly to provide academic support to programs by leading academic activities (72% of programs) and tutoring (71%). Specialists were hired primarily for non-academic enrichment activities such as arts and sports (88%). College students, in contrast, played roles across program activities, including tutoring youth (88%) and assisting with enrichment (82%) and academic (78%) activities.

OST participants responded with high ratings on a scale measuring the quality of their interactions with program staff (average score of 3.35 out of 4), with high school youth responding somewhat more positively than either elementary or middle school youth (3.46 for high school youth vs. 3.31 for both elementary and middle school youth).

Program directors reported that their staff received professional development through staff meetings at the program (86%), internal staff orientations (66%), and off-site workshops (62%).


Summative/Outcome Findings

Academic Participants reported academic benefits on a survey scale (average score of 3.06 out of 4). The most common academic benefit reported by participants was that the program helped them to finish their homework more often (54% “agreed a lot”).

No significant differences were found between participants and matched nonparticipants on the measures of academic performance that are maintained by the Department of Education (attendance, test scores, or credits earned). Consistent with citywide trends, both groups showed small improvements in performance over the course of the OST initiative, with no significant differences in the size of the gains between the two groups.

For high school youth in the sampled sites, analyses found no significant differences between the program and comparison groups on the cumulative number of course credits accrued after each year of OST participation.

Analyses found no significant differences in the number of New York State Regents exams that participants and matched nonparticipants had taken and passed.

Analyses of school attendance patterns of participants and matched nonparticipants found no significant differences in attendance changes over time.

Analyses revealed positive and significant relationships (p < .05) between youth reports of the extent to which a program exposed youth to new experiences and their perceptions of academic benefits and rates of school attendance.

The breadth of the type of content in OST programs, measured by the number of different activity types offered, was positively and significantly associated (p < .05) with the number of credits earned by high school participants.

Measures of a supportive OST environment, including average youth reports of their interactions with their peers and with staff members in the program, were positively and significantly correlated with youth reports of the program’s academic benefits (p < .05).

Analyses found positive correlations between overall program quality and youth’s academic motivation and perceptions of academic benefits.
Family Across all surveyed parents, 74% agreed that the program made it easier for them to keep their job, and 73% agreed that they missed less work than they had previously because their children attended the OST program. In addition, 71% of parents reported that they were able to work more hours because their children were in the program.
Systemic More than half of executive directors reported that the initiative increased “somewhat” or “to a great extent” the organization’s capacity to reach out to serve more youth and families (83%); provide staff training and technical assistance (73%); partner with a public school (71%), cultural organizations (65%), or city agencies (63%); offer programming on weekends and holidays (59%); and provide a career ladder for OST staff (57%).
Youth Development Analyses found a positive and significant correlation between overall program quality and youth’s engagement in pro-social behaviors as reported on the youth survey (p < .05).

 

© 2014 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project