2014

College Club Program Evaluation PREPARED FOR: JOSHUA WRIGHT

EDUCATION COORDINATOR, COLLEGE CLUB

BOYS & GIRLS CLUB OF DANE COUNTY

LA FOLLETTE SCHOOL OF PUBLIC AFFAIRS KATE AUSTIN

ISAAC HEDTKE  

 

i  

Table of Contents Executive Summary .................................................................................................................................... ii 

I: Introduction ............................................................................................................................................ 1 

Purpose of Evaluation ............................................................................................................................... 1 

Literature Review ...................................................................................................................................... 1 

College Club ............................................................................................................................................. 3 

Research Question and Hypotheses .......................................................................................................... 5 

II: Evaluation Design: ................................................................................................................................ 5 

Logic Model .............................................................................................................................................. 5 

Research methodology ............................................................................................................................. 8 

Target population and sampling plan ..................................................................................................... 13 

Data sources and collection procedures ................................................................................................. 14 

Data analysis procedures ....................................................................................................................... 16 

III: Quality Control and Human Subjects Protection ....................................................................... 1817 

IV: Evaluation Implementation .............................................................................................................. 18 

Timeline .................................................................................................................................................. 18 

Implementation Factors ...................................................................................................................... 2019 

Proposed Budget ..................................................................................................................................... 20 

V: Anticipated Results ......................................................................................................................... 2221 

Remaining Questions .......................................................................................................................... 2322 

Stakeholders ............................................................................................................................................ 23 

Appendix A: Logic Model ........................................................................................................................ 25 

 ............................................................................................................................................................ 25 

Outcomes -- Impact ................................................................................................................................... 25 

Appendix B: Power Calculation .............................................................................................................. 26 

Appendix C: BGCDC Contract ............................................................................................................... 27 

Appendix D: Parent/Guardian Survey ................................................................................................... 29 

Appendix E: Student Survey ................................................................................................................ 3739 

Appendix F: Data Collection Table ..................................................................................................... 4344 

Appendix G: Timeline .............................................................................................................................. 45 

Appendix H: College Club Evaluation Budget ...................................................................................... 47 

References:................................................................................................................................................. 48 

 

ii  

Executive Summary

Achievement gaps in educational attainment and college attendance across racial, ethnic,

and socio-economic groups are well documented across the country and in the state of

Wisconsin. In Dane County, educational disparities between students are even more pronounced.

To locally address this growing problem, the Boys & Girls Club of Dane County (BGCDC) has

developed the College Club program to support students in elementary and middle school and to

prepare them for high school success and beyond. This multi-component program has been in

existence for five years. To demonstrate programmatic impacts, we have developed an impact

evaluation plan with the following research question: does participation in the College Club

program improve academic performance, increase successful entrance into a high school college

preparatory program, and improve college readiness when compared to students with comparable

demographics who have not participated in the program.

We have proposed a difference-in-difference, quasi-experimental methodology for this

evaluation. Sixth to eighth grade members of the College Club afterschool programs at the Taft

and Allied BGCDC Centers will be followed longitudinally for 10 years, along with a

comparison group of their peers who participate at BGCDC Centers, but are not involved with

College Club. Data will be collected using mixed methods, at the outset of the evaluation,

quarterly and yearly while the students are in middle-school, and upon entry and exit of high

school.

With this evaluation we expect BGCDC and College Club Coordinators to be able

articulate short, intermediate and long-term impacts for those students who participated in

College Club compared to those who did not. We also recommend collection of implementation

data to allow for an improved utilization of services.

1  

I: Introduction

Purpose of Evaluation

The achievement gap in Dane County is growing. The Race to Equity Report for Dane

County illustrates this problem and its effect on education. In 2010-2011, 50 percent of non-

Hispanic Black students at the Madison Metropolitan School District (MMSD) did not graduate

with a regular diploma in four years of high school. The same is true for 16 percent of white

students. Both numbers are higher than the Wisconsin rates of 36 and 9 percent, respectively, and

there exist additional disparities beyond the black/white gap (WCCF 2013). Helping improve

students’ chances of successfully completing high school and college can begin by providing the

structure and support for changing learning habits and behaviors. One way to target these skills is

through afterschool programs. The Boys & Girls Club of Dane County (BGCDC) provides

afterschool services to youth in the area, including the College Club program. Conducting an

evaluation of College Club will illustrate if the program is effective, and highlight potential areas

for improvement. Given the current need to improve the education of youth in Dane County, this

evaluation could not be more timely nor relevant.

Literature Review

College Club is based on the reasoning that time dedicated to academic learning outside

of the classroom will help improve a student’s academic performance and educational

attainment. Afterschool programs gained acceptance for two main factors. First, students have

unsupervised time after the school day ends and before parents return home from work, which

needs to be structured. Second, the opportunity exists for academic learning and social

development during this time (Power 2008). Research has found that students who attend

afterschool programs most often perform better on standardized tests, and have higher self-

2  

efficacy scores and educational aspirations than those students who attend less frequently

(Cosden et al. 2001).

Literature often separates afterschool programs into two categories: extended learning

and enriched learning. Extended learning is aligned with a student’s schoolwork, directly helping

them with tutoring, homework, and furthering their understanding of the key topics from class.

Enriched learning may be aligned with the schoolwork, but focuses more on outside projects to

develop further learning (Britsch et al. 2005). Both types of academic afterschool programs

provide indirect support towards academic achievement. Participation in these programs

increases the amount of engaged learning, educational equity, and key skill development for

academic success (Britsch et al. 2005).

A review of relevant literature conducted by the Northwest Regional Education

Laboratory showed a positive effect on reading scores for low achieving students who completed

out of school learning. The five studies focusing on tutoring all showed a positive effect, with

greater results for students who received one on one tutoring and those who had the lowest

reading scores (Britsch et al. 2005). Additionally, a study in California used regression

discontinuity design to evaluate the effectiveness of using computer-based programs to improve

literacy rates. This study found statistically significant effects of using computer based programs

outside of the school day to improve reading comprehension and a student’s overall academic

performance (Yancsurak 2013).

Additional studies have found similar results. While Granger (2008) did not find positive

results in all programs reviewed, he concluded that afterschool programs can have a positive

effect on student achievement in reading and math. Other studies have found greater

participation in afterschool programs and longer duration increase the benefits received by the

3  

students (Afterschool Alliance 2006). And Hollister (2003) found mentoring and tutoring

afterschool programs are associated with positive outcomes in school activities and a reduction

in negative out of school outcomes.

College Club

College Club is the afterschool college prep program of the Boys & Girls Club of Dane

County, and it utilizes the successful techniques discussed above. It serves 300 students from 1st

through 12th grades in the Madison area and works to increase college readiness through

academic monitoring, tutoring and case management (BGCDC 2014). There are currently five

College Club sites located across the Madison area. These are at the Allied and Taft Boys &

Girls Clubs, at Wright and Cherokee Middle Schools, and at Verona High School. College Club

primarily targets low-income students, students of color, and students who will be first

generation college attendees with their program. The program is in its fifth year, with the cohort

size growing each year.

This proposed evaluation focuses on the programs at the Allied and Taft Boys & Girls

Clubs, specifically on middle school age children, grades six to eight. The Allied and Taft Clubs

were chosen because these are similar programs; the participant demographics are similar

between Allied and Taft, and both are after-school centered programs. While College Club

members and Coordinators at these centers interact with the schools and teachers, they are not

based in the school like the other three College Club programs.

All College Club participants must be Boys & Girls Club members, and there are three

ways to be selected to participate in the program. The first is through student interest. A student

who shows initiative through regular attendance at the Boys & Girls club and seeks help on their

homework and requests entrance can be accepted. The second path to selection is through a

4  

parent or guardian request. Upon request the staff will assess the student’s interest and needs for

acceptance into the program. The third way a student is identified for participation is through a

teacher or school request for participation. Upon this request, the student is verified as a Boys &

Girls Club member, and his/her interest is assessed. Academic performance is not considered for

entry into College Club. The club attempts to retain current participants first, but has a goal of

filling every available slot and currently has a waiting list for participation.

College Club uses eight components to achieve its goal of increasing academic

performance and increasing participants’ chances of completing higher education. The program

is overseen by the Education Director, and Education Coordinators are at each site to help

orchestrate the eight components. The components are tutoring, case management, college

exposure, mentoring, school/teacher engagement, family engagement, summer academic

programs and eliminating barriers to academic achievement. Certified volunteers conduct the

tutoring. Education coordinators conduct the case management, work to eliminate barriers, and

coordinate with the schools and teachers for their involvement. The Education Coordinator also

works with the tutors to help focus tutoring sessions. Students are matched with a professional

with college experience for their mentoring. The college exposure, family engagement and

summer programming are planned at the program level. Additionally, College Club students

have access to Stride Academy, a computer-based training program focused on math and reading

skills. While this service is available to all members at the BGCDC, it is more structured for

College Club students when paired with the other components of the program.

The College Club program has received national recognition for its innovation and

improvement in reading and math scores; however, there has not been a formal study evaluating

its impact (BGCDC 2014). Conducting the evaluation now will help identify the areas of the

5  

program that are succeeding in meeting goals and those that are struggling. The evaluation

results will allow the program coordinator to improve the program. Additionally, the evaluation

will help strengthen future grant applications to increase funds for College Club.

Research Question and Hypotheses

Our evaluation of the College Club program at the Boys & Girls Club of Dane County

(BCGDC) will attempt to answer the following research questions: does participation in the

College Club program improve academic performance, increase successful entrance into a high

school college preparatory program, and improve college readiness when compared to students

with comparable demographics who have not participated in the program? Additionally, we

recommend collection and analysis of implementation data to assess whether students are

reaching program benchmarks, and to improve utilization of services.

There are two potential results from this evaluation: an effect of participating in College

Club or no discernable difference between those who participated and those who did not. Results

showing an increase in academic performance, entrance into a college preparatory program,

and/or improved college readiness are signs of an effective program. However, no effect, or a

negative effect is also possible. There could be multiple factors attributing to a result of no effect.

The design of this evaluation will help illustrate how the services are utilized by participants.

Utilization results could help identify an implementation issue with the program services.

II: Evaluation Design:

Logic Model

College Club as implemented at the Taft and Allied BGCs is comprised of eight key

components (please see Appendix A for detailed logic model).

6  

1. Tutoring: College Club members receive weekly, individualized tutoring sessions

from certified Wisconsin teachers or graduate students in education at the University

of Wisconsin-Madison. The Education Coordinator creates an individual

development plan for each member, based on communication with the member, their

family, and teachers. The tutor then works with the member to improve in core

content areas identified in the development plan and tracks progress using tutor logs.

2. Case Management: The Education Coordinator at each BGCDC site is responsible

for the case management of the College Club members. Coordinators work with

members to set yearly academic goals and individual development plans. They are

also responsible for identifying social, psychological and emotional barriers that may

impede academic success, good attendance and healthy behavior. Case management

is documented and progress is tracked by Education Coordinators through case

management logs.

3. Addressing and Eliminating Barriers to Academic Achievement: After the

Education Coordinator identifies psychological and socioeconomic barriers to

academic achievement, individualized attention is given to each member to attempt to

address and remove the barriers. Types of barriers members encounter can vary

widely, with some more easily resolved than others.

4. College Exposure: Education Coordinators plan at least two college visits for all

members each school year. Trips are designed to give members a better

understanding of the college environment and show attending college is attainable.

5. Mentoring: At each site, 10 members are matched with an adult mentor from the

community. For those members with mentors, they meet for a minimum of 60

7  

minutes of individualized time per week, which will be tracked and measured through

mentoring logs. Mentors focus on improved academic performance through

homework completion and problem-solving, but may spend time in recreational

settings as well.

6. School and Teacher Engagement: Education Coordinators tailor individual

development plans and tutoring to align student needs with school and teacher

curriculum. They also arrange for teacher, principal and counselor visitors to the

BGCDC to create a strong support network for members.

7. Family Engagement: Educational Coordinators maintain communication with the

families of College Club members and host at least two academic conferences or

recognition events throughout the year. They work to establish sustained relationships

with families, and keep families engaged in the academic achievement of their child.

8. Summer Academic Programming: Academic programming is planned by

Education Coordinators for members during the summer at the BGCDC sites.

Reading and math skills are stressed, and field trips and library visits are organized as

well. Summer academic programming is designed to mitigate summer learning loss

that can negatively impact academic achievement.

The combination of the eight College Club key components will contribute to short-,

intermediate-, and long-term expected outcomes. In the short term, these activities are meant to

increase Stride Academy scores, lead to the development of action plans to reduce barriers, and

improve attendance and behavior in the schools. Over the course of middle school, the College

Club activities are intended to help improve core and cumulative GPA, increase educational

aspirations, and improve academic performance. Over the long term, program components are

8  

designed to improve college readiness, establish a college-going culture at the BGCDC, increase

educational aspirations, and increase AVID/TOPs (Advancement Via Individual Determination/

Teens of Promise) acceptance. AVID/TOPs is a high school level college prep program

coordinated jointly by the Madison Metropolitan School District (MMSD) and the BGCDC.

AVID is a nationally recognized program that is designed to help students excel in high school

and subsequently attend and graduate from college; the TOPs program is a BGCDC initiative

intended to complement the AVID curriculum in MMSD schools (WISCAPE 2014).

There are several assumptions on which we base the logic model:

a) BGCDC has the financial and human resources necessary to implement program activities.

b) Once activities are provided, members in the target population will take advantage of them.

c) Once club members participate in programming, activities will lead to improved academic performance.

d) Improved academic performance will lead to increased likelihood of participation in high school level college preparatory programs.

In addition, there several external factors that have the potential to impact the successful

implementation of the logic model:

a) Level of parental or familial engagement in the lives of the members;

b) Uncertain socioeconomic situation of participating families;

c) Funding of BGCDC programs; and

d) Access to additional academic support outside of school and College Club.

Research methodology 

Our proposed evaluation uses the quasi-experimental design of a difference-in-difference

study. Difference-in-difference is a technique to simulate an experimental design by measuring

the difference in average outcome in the treatment group before and after treatment, minus the

9  

difference in average outcome in the control group before and after treatment (Alhoudy [2014]).

The design will follow three cohorts of students in distinct treatment and comparison group

through time. Cohorts are determined by grade-level at initiation of the study, (either sixth,

seventh, or eighth grade), as the program is only for middle school students. Short-term

outcomes will be measured quarterly, while the cohort is in middle school; intermediate

outcomes will be measured yearly, while the cohort is in middle school; and long-term outcomes

will be measured at entry and exit of high school. The proposed evaluation will last 10 years in

its ideal implementation. We propose using mixed methods for data collection throughout the

process to capture quantitative changes in academic performance and qualitative differences in

aspirational outcomes that may change with the program.

This design is both feasible to implement for our client and will address the need for

short-term results to give to funders. Additionally, it is a balance of a more rigorous, but more

expensive method, and a feasible, but much weaker design. The difference-in-difference

approach reduces excess burden on students and education coordinators, while maintaining a

level of scientific rigor through the comparison group and multiple pre- and post-tests. We will

utilize a concurrent comparison group of students consisting of members of the BGCDC, but

who do not participate in College Club. Baseline data will be established through collecting

initial GPA and attendance figures, Stride Academy reading and math skills tests scores,

Qualtrics survey results, and by conducting qualitative interviews with students and parents.

The difference-in-difference design removes the bias that may occur due to changes in

time. This is particularly applicable when researching educational programs, as one assumes

maturation effects as students move through school. Additionally, through pre-post analysis

within each group and across study groups, this design eliminates the bias that may be due to

10  

stable differences between those in the program and the comparison group. A related design

limitation is that it is necessary to assume that unobserved characteristics of the College Club

members and students in the comparison group are fixed over time, which may or may not hold

(Shager 2014).

Using quasi-experimental evaluation methods presents several threats to validity. Specific

to this evaluation plan, we are concerned about internal validity, statistical conclusion validity,

and external validity. With regard to internal validity, we are concerned that our comparison

group is unmatched to the treatment group. We will address this by selecting individuals for the

comparison group from the same geographical area, who attend the same schools, who

participate in BGCDC programs, and who are roughly matched on demographic characteristics

to the treatment group. We assume that if all the students are involved in some way in BGCDC

programs, we will minimize selection bias effects. Some selection bias may still exist with this

design due to the selection into the program. We also acknowledge that comparing changes in

achievement between these two groups will yield different results than an evaluation studying

the effects of College Club compared to students with similar characteristics who have no

affiliation with BGCDC. It is important to keep this distinction in mind when interpreting effect

sizes, because this is a more rigorous design, but comparing to students who already are seeking

extra-curricular services will help reduce validity issues. This may be a good area for further

research. We are also concerned with attrition of the comparison group, whether due to the

students leaving the BGCDC or because the student is accepted into the College Club program.

This further complicates the comparison group, but it is the best available solution. If funding

increases and more slots become available, it is likely some of our comparison group members

could be entered into the College Club program. This is most likely for the sixth grade cohort.

11  

Due to this evaluation focusing on the program in the middle school years, this is only a minor

concern.

The small sample size presents a threat to statistical conclusion validity from low

statistical power. Initially, we planned to only evaluate College Club at the Allied Center;

however, as previously mentioned, to increase sample size we have chosen to include the

program at the Taft Center in our evaluation as well. Further increasing the sample size is not

feasible at this point, because that would include growing the College Club program. Our

intention with this evaluation is to serve the internal needs of BGCDC, providing a plan for data

collection and analysis to justify sustained and increased investment in the program. A power

calculation can determine the minimum program effect size that will be able to be detected given

the fixed sample. Further information can be found in the upcoming power analysis section.

The limited scope of the existing College Club program presents threats to the external

validity of the evaluation. We plan to assess only the club-based, after school portion of the

College Club program. Will these results be valid with respect to the middle-school and high

school based College Club programs? Or programs in other communities? To address this

concern, we will outline differences between the afterschool and club based programs, and

provide guidance for our clients on what conclusions will be appropriate to assume given the

data. It is also important to consider the conditions specific to Madison that have resulted in the

one of the widest educational achievement gaps and largest black/white arrest and incarceration

disparities in the country. The extent to which poverty and “disadvantage” has become racialized

in Dane County makes it distinct from other communities that may implement a similar program

(WCCF 2013).

12  

Rationale

We considered several study designs and evaluation methods prior to selecting our

evaluation plan. First, we examined the feasibility of a rigorous, randomized experimental

method, as this design is the most effective in reducing biases and differences between treatment

and control groups, and in establishing causal links. This method proved not to be appropriate for

our client for several reasons. It is expensive, we have a small sample size, and some of the

students have been in the program for a couple of years, which would not only diminish effects

seen between groups, but ethically we do not want to randomize them out of the program.

Because there has not yet been an impact evaluation done on the College Club program, we plan

to start with a smaller trial to demonstrate potential effects, and a next step could be scaling up

the program and conducting a randomized controlled trial (RCT) across sites.

An additional method we considered was a simple one group, pre-test/post-test design.

While this evaluation method would be much more feasible for our client, it would only

demonstrate progress in students receiving College Club services over time and would not show

the impact of the College Club programming against the counterfactual, a comparison group who

did not receive services. It is important to note that the pre-test/post-test analysis of achievement

among College Club members is still possible within the framework of the proposed evaluation

plan. Standardized data collection tools and intervals (discussed in detail below) will give our

client the flexibility to analyze pre-post measures for internal or external needs.

Complementary evaluations

In additional to the impact evaluation, we recommend conducting an implementation

evaluation to answer the question: are College Club members efficiently utilizing services and

reaching program benchmarks? Efficiently utilizing services will mean maximizing the resources

13  

available to College Club in relation to having the greatest impact on the participating students.

The designed evaluation plans to track program outputs and collect implementation data after

each semester while the students are in middle school to determine how they are utilizing

services.

Target population and sampling plan

As previously mentioned, our target population consists of sixth through eighth grade

students who participate in the College Club program at either the Taft or Allied BGCDC

Centers and those students of the same age who participate in other BGCDC programs. Many of

these students are low-income students, students of color and potential first-generation college

students. An important shared characteristic to take into consideration between the groups is that

both the College Club members and those in the comparison group are motivated enough or have

the family and social support to participate in BGCDC programming. While this will not allow

an evaluation of the effect of College Club versus no participation in any extracurricular

activities at the BGCDC, it isolates the unique added value of College Club for students.

Our sample size is relatively fixed. There are 25 members of each the Taft and Allied

College Clubs, giving a treatment population of 50. We will recruit 25 students to serve in the

comparison group at each Center, which will be convenience sampling, using students who are

on the waiting list for College Club and those who are participate in other BGCDC programs.

The overall study population will be approximately 100 individuals.

Power analysis

The statistical power in impact evaluations allows researchers to be able to detect

whether a program has had an effect on the target population (McKenzie 2011). Using the

Optimal Design Software, we can calculate the minimum effect size that will be able to be

14  

detected for our population given the standard power threshold of 0.8 (Raudenbush et al. 2011).

For a single-level trial conducted at the individual level, given a sample size of 100 and power of

0.8, the minimal detectable effect size that would be statistically significant is 0.57. The literature

evaluating similar programs shows a wide variation in effect sizes, making it difficult to

accurately predict a potential effect size of College Club.

Due to the fixed nature of the sample for this evaluation, we are limited to using

statistical analysis techniques to improve power and lower the minimal detectable effect size.

Our collection and analysis of repeated post-tests and measures will allow for greater power as

well. Please see Appendix B for power calculation.

Data sources and collection procedures

Data to determine any potential impact of the program as well as the data for the

implementation analysis will be collected at varying points over the 10-year time horizon of the

proposed evaluation. As previously mentioned, baseline data will be established at the beginning

of year one through collecting initial grade point average (GPA) and attendance figures for both

College Club members and those in the comparison group. BGCDC has access to both of these

measures (please see Appendix C). Additional baseline data includes a preliminary Stride

Academy reading and math skills test that will be administered to all participants. This program

allows for stratification of data according to the student and the subject matter. The final baseline

measure will be level of family engagement and student’s educational aspirations, gathered

through Qualtrics survey results (please see Appendix D for preliminary survey tool), and by

conducting qualitative interviews with students and parents. All baseline measures will be

collected for both the treatment and control groups.

15  

Implementation data will be collected quarterly for all program outputs relating to the

treatment groups while the students are in middle school. Standardized reporting of tutoring, case

management and mentoring using logs will allow for measurement of the utilization of services

among members, including number of meetings and meeting minutes per week. College

exposure, summer academic programming and field trips will be measured through attendance

logs, and time logged on Stride Academy by students will be captured by the software. School,

teacher, and family engagement will be tracked by the Education Coordinators in their case

management logs. Establishing standard procedures to record and document activities beginning

in year one will improve the reliability of the reporting tools and improve internal validity.

Short term expected outcomes will be measured at the end of each academic quarter

while the students are in middle school. Attendance will be measured by the school each student

attends, and access to this information will have been granted for all BGCDC participants. Stride

Academy scores will be measured for math and reading, and generated using the software.

Action plans to reduce academic barriers will be measured as a percentage of members with

action plans, and this will be tracked using the case management logs.

Increased academic performance will be measured using both core and cumulative GPA.

BGCDC obtains permission to access students’ grades through the participation agreement each

signs with their families. These will be collected and documented at the end of each year of

middle school. Grades and GPA are only one measure of success and there are challenges to

reliability of this measure, because students across the evaluation will attend different schools.

This information will be documented and controlled for by using dummy variables, whose use is

described in more detail in the following section. Educational aspirations as an intermediate

outcome will be assessed through a survey tool (please see Appendix E) administered by the

16  

Education Coordinator. Because this tool will be used in repeated years, there is the potential for

students to adjust responses based on prior knowledge of the question or due to social

desirability bias to align their responses with the perceived most desirable answer.

The long-term outcome of improved college readiness will be measured using

components that are predictors of college eligibility. Components to be measured include the

American College Testing (ACT) test, a standardized college readiness assessment; overall grade

point average; and completion of the Free Application for Federal Student Aid (FAFSA), an

application to determine eligibility for student financial aid. Increased AVID/TOPS acceptance

will be determined using AVID/TOPS administrative data upon high school entry. Additionally,

increased educational aspirations will be measured through in-person interviews at high school

entry and exit. The final long-term outcome, to establish a college going culture at BGCDC, will

be a qualitative measure assessed by the Education Coordinator. Appendix F is the proposed data

collection table, providing a snapshot view of this information.

Data analysis procedures

The basic notation we will use for our data analysis of this difference-in-difference model

is as follows, where Y is the sample average of the indicated measure, subscript 0 is the pre-test,

subscript 1 is the post test, T is treatment and C is control group:

Pre Post Post-Pre Difference

Treatment Y0T Y1

T Y1T – Y0

T

Control Y0C Y1

C Y1C – Y0

C

T-C Difference Y0T – Y0

C Y1T – Y1

C (Y1T – Y0

T) – (Y1C – Y0

C)

(Albouy [2014])

17  

This formula allows for analysis first within the treatment or the comparison group over

the course of the program and then to assess the differences between the groups. Because there

will be different treatment durations between the sixth, seventh, and eighth grade cohorts, we

propose using “dummy” variables to detect potential differences in effect sizes across the grades.

Dummy variables are variables that take the values of only 0 or 1 and serve as numerical stand-

ins for qualitative or categorical values that may be expected to shift the outcome of analysis

(Chinn 2014). There are three categories of interest (sixth, seventh, or eighth grade at initiation

of evaluation) which means two dummy variables will need to be created and compared to the

“base case.” In this situation we propose using eighth grade as the base case and measuring

differences for sixth and seventh graders against their average outcome measures. The three

cases can be defined as follows:

Dummy Variable One (D1) Dummy Variable One (D2) Eighth grade (base case) 0 0 Seventh grade 1 0 Sixth grade 0 1

(Chinn 2014)

We propose creating two more variables to control for additional differences between

participants. First is to control for differences in the schools the students attend. Second is to

control for students who have previously participated in the College Club program. This method

will be used to measure the intermediate expected outcomes of improved core and cumulative

GPA.

18  

III: Quality Control and Human Subjects Protection

This evaluation requires obtaining the test scores and grades of College Club participants

and of the BGCDC members who are not participants in College Club to accurately assess the

program. This requires consent to gather the data from the schools. The BGCDC gathers consent

from parents for accessing school grades upon a child’s entrance into the club. Since Stride

Academy is also implemented club wide, there is no need for additional consent to be gathered.

The form currently used is provided in Appendix C. This form obtains consent for access to

transcripts, attendance records, schedules, and achievement tests (WKCE and school specific).

We recommend adding a line to the consent form informing parents of the program and the need

for evaluation. Proposed language to add to the form is also included in Appendix C. As long as

this remains the standard practice, no further information is required to be gathered.

The results from this study are intended for local use, staying within the program. If the

intentions of the evaluation shift toward a broader academic focus with the goal of being

published, the Institutional Review Board for the governing body of the program evaluators

should be consulted to identify further consent requirements.

IV: Evaluation Implementation

Timeline

Implementing the program evaluation of College Club requires two timelines to follow.

The first is a short-term, one-year timeline, and the second is a long-term, multi-year timeline.

Examples of both can be found in Appendix G. The short term timeline starts in August, prior to

the beginning of the school year and ends in June, after the completion of the school year.

August is the proposed start of preparatory work for the evaluation with the College Club staff,

19  

ensuring understanding of how the evaluation will work and the data that needs to be collected

and reported. After Labor Day, when the school year begins, the members of the BGCDC are

placed into their respective groups (College Club or not), and initial baseline testing is

conducted. From this point until the end of the semester in January, the Club runs as normal.

During this time, data is collected in case management logs, tutoring logs, Stride Academy

reports, mentoring notes and attendance logs. At the end of the semester all participants take the

mid-year tests. The data collected so far is reviewed for completeness, not analysis. The second

semester mirrors the first, with final tests occurring near the end of the school year in June.

Surveys are also administered at this point. Next, the school year data is compiled and analyzed

for results and recommendations. Summer programs begin, and the relevant data is collected

throughout the summer.

The long-term timeline provides the framework to follow College Club and BGCDC

members into high school to see who is accepted and completes a college preparatory program.

High school graduation, college acceptance and college completion data is also collected. High

school records will provide high school graduation information, while college acceptance and

completion will be collected through survey responses. This information will be collected twice a

year, at the start and end of the academic calendar. It will take four years to see the first College

Club participants be eligible for acceptance into college. We recommend following the members

for at least 10 years. This will allow sufficient time for three cohorts of members and students to

have completed a two or four year degree. The data will be gathered through follow-ups

conducted via email by the program.

20  

Implementation Factors

Implementation of the evaluation will be hindered by a few factors. At this time there is a

spending freeze for the BGCDC. Without available funds it will be difficult to resource the

evaluation. The Education Director position is currently vacant as well. The Director is the

logical person to oversee the evaluation since it involves two locations. Conducting the

evaluation without the Director is not impossible, but will pose additional challenges. The final

implementation challenge is the school year. This evaluation is designed to begin with a new

school year to most accurately capture results. During the waiting period to begin, materials and

data collecting systems should be prepared to ensure a smoother implementation.

Proposed Budget

The Kellogg Foundation recommends designating 5 to 7 percent of a program’s total

budget to conduct an evaluation (Curnan et al. 2004). The evaluation of College Club will cost

time and money. Fiscal amounts will attempt to be minimized because of the current fiscal

situation. Fiscal costs will be discussed first, followed by time costs.

The majority of fiscal costs for this evaluation are already accounted for in the BGCDC

and College Club budgets. For example, this evaluation relies heavily on access to Stride

Academy software, currently purchased by the BGCDC. The budget should reflect this cost to

ensure it, or comparable software, remains available. The surveys completed by participants,

teachers, staff, and family members can be accessed via computers at the clubs. A few can be

printed out and sent home with members if needed. We assume the minimal printing costs can be

covered by the BGCDC printing budget. Obtaining the long-term data from former participants

in the evaluation can also be accomplished by email. This saves on costs and also will likely be

an easier way to maintain contact as people move.

21  

Two fiscal costs should be discussed by the program organizers prior to starting the

evaluation. The first is for an outside analyst to compile the data and provide results. It is

reasonable to hire an analyst for 80 hours a year to review the data and analyze for results. This

gives the analyst time to review the data for completeness after each semester, analyze the data

and answer questions. Using an hourly wage between $27 and $36 this total is between $2,160

and $2,880. This is the amount for the first 3 years when there is the largest quantity of data. The

last 7 years of the proposed evaluation, we estimate the need for an analyst to be 20 hours for one

week each year, for a range between $540 and $720 a year.

The other fiscal cost is any incentives for participating. We recommend having pizza

twice a year to help coordinate participation in baseline and follow-up Stride Academy testing

for all students in the evaluation.. This cost accrues for the first 3 years. To feed 100 people

pizza, an estimated 25 pizzas should be sufficient. At $14 a pizza, the total is $350. A yearly

pizza incentive budget is $700.

The majority of the costs to implement this evaluation are in time. These costs are

incurred by the participants taking tests, tutors filing reports, education coordinators filing logs

and conducting interviews, and mentors and teachers providing feedback to the program. Time

estimates are difficult to measure. The activities listed above are already conducted by College

Club; the evaluation will only increase requirements minimally at the beginning and end of the

school year. People who actively assist College Club can expect less than a 10% increase in their

work load to successfully implement this evaluation. Therefore, if one volunteers 10 hours a

week, s/he can expect to volunteer 11 to ensure proper implementation. The largest time effect

falls on whomever is coordinating the evaluation; this increase in time will vary by week, but is

22  

sizable and should be taken into account when identifying the evaluation coordinator. A brief

overview of the budget is in Appendix H.

V: Anticipated Results

Successful implementation of this program evaluation will likely allow a few key lessons

to be learned by College Club. Based on relevant literature, we expect the program to have a

positive effect on the achievement of its participants. However, if no effect is shown, there are a

few possible explanations. A likely explanation is that the program services are being under-

utilized. The implementation portion of the evaluation will help identify areas to focus efforts

and resources to improve program adherence to obtain expected results. Viewing emerging best

practices by similar programs might help generate ways to improve implementation shortcoming

identified in this evaluation.

If a positive effect emerges, it allows for key areas of organizational learning. The first

three are focused on the participants and the effects of the program, while the fourth is focused

on the program itself. First, this will allow clients to say participation in College Club is

associated with an increase in academic performance by X% across various measures. These

measures are GPA, Stride Academy scores, state test scores, and attendance ratios. Secondly,

participation in College Club is associated with being X times more likely to enter into a college

preparatory program in high school. After four years, this can be furthered to show an

association between College Club participants and college attendance compared to non-

participants. The third result to be learned from the evaluation is the effect on college readiness.

In particular, College Club will be able to say their participants have X% of predictors of college

eligibility compared to non-participants. Finally, this evaluation will allow the administrators of

23  

College Club to identify areas for improvement in terms of program adherence. This information

allows the program to continue to develop areas that need greater attention in the program, adjust

resources to maximize positive results, and identify additional areas for improvement.

Remaining Questions

A few questions will remain after this evaluation is complete. The most prominent

question remaining is if participation in the program is associated with increased high school

graduation and college graduation rates compared to non-participants. For example, just because

a student begins a college preparatory program and is eligible for college does not mean s/he will

obtain a degree from a higher education institute. To answer this question, a longer, more

rigorous evaluation will need to take place. The evaluation presented here is the first step.

Other questions will likely arise from the results of the evaluation as well. One potential

question will be which component of the program is most effective? This evaluation will be

unable to answer this question, but descriptive information can help identify which component is

being utilized the most. The design of the proposed evaluation is quasi-experimental and not a

randomized control trial; therefore, the results will be unable to show causal relationships. The

descriptive data can be used to help infer associations, but nothing more. A more rigorous

randomized controlled trial would be required to more accurately determine which components

of the program are having an effect.

Stakeholders

The results of this evaluation will need to be presented to the stakeholders of the

organization. First, we would recommend a formal presentation to the BGCDC Board and the

College Club Program Director. These stakeholders will be able to make decisions for the

program based on the results and should have the opportunity for a detailed presentation with

24  

discussion. The second group of stakeholders include all of the families of the BGCDC and key

personnel from the community and schools. The results of the study should be summarized into

one or two pages for this group and disseminated via email to reduce costs. A point of contact

should be provided for additional information. A town-hall style presentation on the program and

the results of the evaluation could be beneficial to improve awareness and participation. The

final group of stakeholders is the funders. And results should be used in future proposals and

updates. Most of the funders receive at least quarterly updates; this is the ideal opportunity to

highlight pertinent results from the study.

Based on the duration of this evaluation, we recommend disseminating information in

three phases for the first two groups of stakeholders. After the first year of the evaluation is

complete, the descriptive data and initial results from the analysis should be disseminated. After

all three cohorts are out of middle school, the second round of disseminating information takes

place. The final results should be disseminated at the end of the evaluation. Due to the nature of

the third group of stakeholders, the funders, results should be disseminated to them based on the

timing of funding requirements.

25  

Appendix A: Logic Model Program: College Club, Boys & Girls Club of Dane County, Logic Model

Inputs Outputs Outcomes -- Impact Activities Outputs Short Medium Long

Tutors (certified); materials for instruction; classroom environment; Case Managers (certified); meeting space; information re actions of student) Schools supplies; financial support; food support; Transportation; location to visit; adult supervision; food support; Mentors (volunteers); meeting space; specific need materials BGCDC Visits; Teacher and Principal time Family time, resources for conferences and events Transportation; supplies

1. Tutoring; Education

Coordinator, Tutors, members

2. Case Management;

Education Coordinator, members

3. Address/Eliminate

Barriers to Academic Success; Education Coordinator, members

4. College Exposure;

Education Coordinator, members

5. Mentoring; Education Coordinator Mentors, members,

6. School and Teacher

Engagement; Education Coordinator, members, Teachers, Principals, Counselors

7. Family Engagement; Education Coordinator, families, members

8. Summer Academic

Programming; Education Coordinator, members, library staff

Meetings, Stride Academy sessions and tests

Meetings, Individual Development Plan

Interviews, Action Plans

Field trips, College visits

Meetings

Individual Development Plan Family Conferences, recognition events Field trips, library visits, reading lists

A. Increased Stride

Academy scores

B. Improved attendance C. Development of

action plans to reduce barriers

D. Improved behavior at

school

A. Increased cumulative

GPA

B. Increased core GPA

C. Increased educational aspirations

D. Improved academic

performance

A. Improved college

readiness

B. Increased AVID/TOPS Acceptance

C. Established college going culture at BGCDC

D. Increased educational

aspirations

Assumptions

External Factors BGCDC has the financial and human resources to implement program activities Once activities are provided, students in the target group will participate Once accessed, the activities will lead to improvements in GPA Improved middle school performance will increase likelihood of participation in

high school college prep programs

Parental involvement; SES situation; BGCDC program funding, Geographical area for access to resources

Rev. 12/15//2014

Comment [IH1]: Do we need to add a Part E for Family Engagement (append ix D findings) 

26  

Appendix B: Power Calculation   

 

27  

Appendix C: BGCDC Contract

PARENT AUTHORIZATION AND RELEASE OF INFORMATION FORM I provide consent to allow The Boys & Girls Club of Dane County staff electronic access to the following information about my child using the Madison Metropolitan School District information systems and/or the Verona Area School District’s data: Academic progress reports, report cards, unofficial school transcripts,

assignments, teacher comments, teachers’ names, dates of courses taken, student course schedules, daily attendance, absences, tardy data, test scores (including Wisconsin state achievement tests and MMSD achievement tests) and fee payments due and paid.

In addition, to access to my child’s records electronically I authorize MMSD and/or The Verona Area School District to provide The Boys & Girls Club of Dane County with other demographic and program service eligibility information for my child. This information includes Limited English Proficiency status, primary language (other than English), disability status and primary disability and qualification for federal free and reduced lunch program. I understand that this information will remain strictly confidential and will only be used for the improvement of educational services and resources rendered to my child. The Boys & Girls Club of Dane County will not further disclose the data to any third party, researcher or others without obtaining a separate written permission from you. I understand that this release of information agreement and participation agreement will remain in effect until my child completes high school, until my child resigns from the program or until I revoke this consent in writing.

28  

PARENT SIGNATURE DATE (PRINT NAME) TELEPHONE Proposed Feedback on Contract: We recommend the BGCDC adds the following to the contract used by all BGCDC members: 

‐ The Boys & Girls Club of Dane County is currently running an evaluation of one of its programs, College Club. The purpose of this evaluation is to better understand how the Club can best support the academic learning of our participants, to identify areas where the Club can improve our role, and to provide further justification/evidence of the positive effects of this program. Your child’s test data will be used in this evaluation, but no personal identifying information will connect scores to your child. 

29  

Appendix D: Parent/Guardian Survey   The purpose of the survey on family engagement and student educational aspirations is to identify how involved the participants’ family members are in their learning and how family members view the potential education attainment level of the student. The following survey questions will help obtain answers to these questions. The National Center for Educational Statistics (www.nces.ed.gov) is an available resource to help identify lines of questioning used by other surveys to answer similar questions. One study on the NCES website is the Education Longitudinal Study of 2002. This study used a specific survey for parents of 10th graders. Questions 52 through 60 and 69 through 83 are appropriate questions to include on a survey for this College Club evaluation. The questions below are school-specific; however, they can be tailored to reflect opinions regarding BGCDC Centers. The complete survey can be found at the following link: http://nces.ed.gov/surveys/els2002/pdf/ParentQ_baseyear.pdf  Q1 Since your student's school has opened last fall, how many times have you or your spouse/partner been contacted by the school about the following? 

  None (1)  Once or twice (2)  Three or four times (3)  More than four times (4) 

Your student's poor performance in school (1) 

       

Your student's school program for this year (2) 

       

Your student's plans after leaving high school (3) 

       

Your student's course selection for entry into college, vocational, or technical school after completing high school 

(4) 

       

Your student's poor attendance record at 

school (5)         

Your student's problem behavior in school (6) 

       

Your student's positive or good behavior in school 

(7)         

Participating in school fund‐raising activities or doing volunteer work 

such as supervising lunch or chaperoning a field 

trip (8) 

       

Information on how to help your student at 

home with specific skills or homework (9) 

       

Obtaining information for school records such as your address or work telephone number (10) 

       

 

30  

Q3 Since your student's school has opened last fall, how many times have you or your spouse/partner contacted the school about the following? 

  None (1)  Once or twice (2)  Three or four times (3)  More than four times (4) 

Your student's poor performance in school (1) 

       

Your student's school program for this year (2) 

       

Your student's plans after leaving high school (3) 

       

Your student's course selection for entry into college, vocational, or technical school after completing high school 

(4) 

       

Your student's poor attendance record at 

school (5)         

Your student's problem behavior in school (6) 

       

Your student's positive or good behavior in school 

(7)         

Participating in school fund‐raising activities or doing volunteer work 

such as supervising lunch or chaperoning a field 

trip (8) 

       

Information on how to help your student at 

home with specific skills or homework (9) 

       

Obtaining information for school records such as your address or work telephone number (10) 

       

 

Q2 In this school year, do you or your spouse/partner do any of the following? 

  Yes (1)  No (2) 

Belong to the school's parent‐teacher organization (1) 

   

Attend meetings of the parent‐teacher organization (2) 

   

Take part in the activities of the parent‐teacher organization (3) 

   

Act as a volunteer at the school (4)     

Belong to any other organization with several parents from your student's school (5) 

   

 

 

31  

Q4 How often do you... 

  Never (1)  Seldom (2)  Usually (3)  Always (4) 

check that your student has completed all of their 

homework (1)         

discuss your student's report card with her/him 

(2)         

know where your student is when s/he is not at home or in school (3) 

       

make and enforce curfews for your student 

on school nights (4)         

 

32  

Q5 In the first semester or term of this school year, how often have you and/or your spouse/partner provided advice or information about the following to your student? 

  Never (1)  Sometimes (2)  Often (3) 

Selecting courses or programs at school (1) 

     

Plans and preparation for college entrance exams such as ACT, 

SAT, or ASVAB (2)       

Applying to college or other schools after high school (3) 

     

Specific jobs your student might apply for after high school (4) 

     

Community, national and world events (5) 

     

Things that are troubling your student (6) 

     

 

Q6 Looking back over the past year, how frequently did you and your student participate in the following activities together? 

  Never (1)  Rarely (2)  Sometimes (3)  Often (4) 

Attending school activities (1) 

       

Working on homework or school projects (2) 

       

Attending concerts, plays, or movies outside of 

school (3)         

Attending sporting events outside of school (4) 

       

Attending religious services (5) 

       

Attending family social functions (6) 

       

Taking day trips or vacations (7) 

       

Working on a hobby or playing sports (8) 

       

Going shopping (9)         

Going to restaurants/eating out 

(10)         

Spending time just talking together (11) 

       

Doing something else fun together (12) 

       

 

 

33  

Q7 How much do you agree or disagree with the following statements? 

  Strongly Agree (1)  Agree (2)  Neither Agree nor Disagree (3) 

Disagree (4)  Strongly Disagree (5) 

Most people can learn to be good at 

math (1)           

You have to be born with the ability to be good at math (2) 

         

 

Q8 Looking back over the past year, how many times did the following occur?  The parent of one of my student's friends... 

  None (1)  Once or twice (2)  Three or four times (3)  More than four times (4) 

gave me advice about teachers and/or courses at my student's school (1) 

       

did me a favor (2)         

received a favor from me (3) 

       

supervised my student on an educational outing or 

field trip (4)         

 

Q9 Are there family rules that are enforced for your student about any of the following activities? 

  Yes (1)  No (2) 

Maintaining a certain grade average (1)     

Doing homework (2)     

Doing household chores (3)     

Watching television (4) 

 

Q11 In a typical week how many days do you eat at least one meal with your student? 

0 days per week (1) 

1 day a week (2) 

2 days a week (3) 

3 days a week (4) 

4 days a week (5) 

5 days a week (6) 

6 days a week (7) 

7 days a week (8) 

 

Q12 Do you have a computer in your home that your student may use? 

Yes (1) 

No (2) 

 

Q13 Does this computer have access to the internet? 

Yes (1) 

No (2) 

 

34  

Q14 Do you or your spouse/partner use a computer to communicate with or get information about your student's school? 

Yes (1) 

No access to a computer (2) 

Don't use a computer for these purposes (3) 

 

Q15 how often do you or your spouse/partner use a computer in the following ways? 

  Never (1)  Once or twice a year (2) 

Several times a year (3) 

At least once a month (4) 

Weekly (5) 

To communicate with your student's 

teachers and administrative staff via e‐mail about your student (1) 

         

To find out what events and activities are happening at your student's school (2) 

         

To express concern to the school 

principal or teachers over school practices and policies (3) 

         

To let school staff, counselors, teachers 

or the principal know which courses you would like your student to take (4) 

         

To find out about homework and other school 

assignments and projects (5) 

         

 

Q16 Does your student's school have a voice‐messaging/e‐mail system that you can use for information about school events, activities, and programs, or leave messages for your student's teachers, school administrator or other staff? 

Yes (1) 

No (2) 

Don't know (3) 

 

35  

Q17 During the school year, how often do you use the school's messaging system? 

Never (1) 

Once or twice a year (2) 

Several times a year (3) 

At least once a month (4) 

At least once a week (5) 

 

Q18 How much do you agree or disagree with each of the following statements concerning your student's school? 

  Strongly agree (1)  Agree (2)  Neither Agree nor Disagree (3) 

Disagree (4)  Strongly Disagree (5) 

The school assigns too little homework 

(1)           

My student is challenged at school 

(2)           

My student is working hard at 

school (3)           

The school is preparing students well for jobs in the workplace (4) 

         

The school is preparing students well for college (5) 

         

The school is a safe place (6) 

         

Parents have an adequate say in 

setting school policy (7) 

         

Parents work together in 

supporting school policy (8) 

         

My student's teachers are well 

trained (9)           

Drinking on school grounds is a 

problem at my student's school (10) 

         

Drug use on school grounds is a 

problem at my student's school (11) 

         

The sale or use of drugs on the way to or from school is a 

problem (12) 

         

Theft on school grounds is a 

problem at my student's school (13) 

         

Violence on school grounds is a 

         

36  

problem at my student's school (14) 

The lack of discipline in classrooms is a problem at my 

student's school (15) 

         

 

 

Q19 How satisfied are you with the education your student has received up to now? 

Very Dissatisfied (1) 

Somewhat Dissatisfied (2) 

Somewhat Satisfied (3) 

Very Satisfied (4) 

 

Q20 How far in school do you want your student to go? Please mark only the highest level that applies. 

Less than high school graduation (1) 

High school graduation or GED only (2) 

Attend or complete a 2‐year school course in a community of vocational school (3) 

Attend college, but not complete a 4‐year degree (4) 

Graduate from college (5) 

Obtain a Master's degree or equivalent (6) 

Obtain a Ph.D., M.D., or other advanced degree (7) 

 

37  

Q21 How important is each of the following to you in your student's choice of a school to attend after high school? 

  Very important (1)  Somewhat important (2)  Not important (3) 

Low expenses (tuition, books, room and board) (1) 

     

Availability of financial aid, such as a school loan, scholarship or 

grant (2)       

Availability of specific courses or curriculum (3) 

     

Strong reputation of the school's athletic programs (4) 

     

Active social life at the school (5)       

Ability to attend school while living at home (6) 

     

Chance to live away from home (7) 

     

A religious environment (8)       

A low crime environment (9)       

A good record for placing graduates in jobs (10) 

     

A good record for placing graduates in graduate school 

(11)       

Strong reputation of the school's academic programs (12) 

     

Easy admission standards (13)       

Racial/ethnic composition of the school (14) 

     

Size of the school (15)       

 

 

Q23 How far in school do you think your student WILL go? Please mark only the highest level that applies. 

Less than high school graduation (1) 

High school graduation or GED only (2) 

Attend or complete a 2‐year school course in a community of vocational school (3) 

Attend college, but not complete a 4‐year degree (4) 

Graduate from college (5) 

Obtain a Master's degree or equivalent (6) 

Obtain a Ph.D., M.D., or other advanced degree (7) 

 

Q22 Have you or your spouse/partner done anything specific in order to have some money for your student's education after high school? 

Yes (1) 

No (2) 

 

   

Appendix E: Student Survey

38  

The purpose of the survey on educational aspirations is to identify what level of educational attainment the student believes s/he can achieve. The following survey questions will help obtain answers to these questions. The National Center for Educational Statistics (www.nces.ed.gov) is an available resource to help identify lines of questioning used by other surveys to answer similar questions. One study on the NCES website is the Education Longitudinal Study of 2002. This study surveyed 10th graders on a variety of topics, to include what level of education they believe they could obtain. The questions in Part III (questions 54 to 66) are relevant for this College Club Evaluation. The questions below are school-specific; however, they can be tailored to reflect opinions regarding BGCDC Centers. The complete survey can be found at the following link: http://nces.ed.gov/surveys/els2002/pdf/StudentQ_baseyear.pdf Student Educational Aspirations 

Q1 How important is each of the following to you in your life? 

  Not Important (1)  Somewhat Important (2)  Very Important (3) 

Being successful in my line of work (1) 

     

Finding the right person to marry and having a happy 

family life (2)       

Having strong friendships (3)       

Being able to find steady work (4) 

     

Helping other people in my community (5) 

     

Being able to give my children better opportunities 

than I've had (6)       

Living close to parents and relatives (7) 

     

Getting away from this area of the country (8) 

     

Working to correct social and economic inequalities (9) 

     

Having children (10)       

Having leisure time to enjoy my own interests (11) 

     

Becoming an expert in my field of work (12) 

     

Getting a good education (13) 

     

 

 

39  

Q2 Have you taken or are you planning to take any of the following in the next two years? 

  I haven't thought about it (1) 

No, I don't plan to (2) 

Yes, this school year (3) 

Yes, next school year (4) 

Yes, in 12th grade (5) 

Pre‐SAT test (PSAT) or Preliminary 

American College Testing Test (PACT) (1) 

         

College Board Scholastic 

Assessment Test (SAT) or American College Testing Service (ACT) (2) 

         

Advanced Placement (AP) 

test (3)           

Armed Services Vocational 

Aptitude Battery (ASVAB) (4) 

         

 

Q3 As things stand now, how far in school do you think you will get? 

Less than high school graduation (1) 

High school graduation or GED only (2) 

Attend or complete a 2‐year school course in a community or vocational school (3) 

Attend college, but not complete a 4‐year degree (4) 

Graduate from college (5) 

Obtain a Master's degree or equivalent (6) 

Obtain a Ph.D., M.D., or other advanced degree (7) 

Don't know (8) 

 

Q4 Do you plan to continue your education right after high school or at some time in the future? 

Yes, right after high school (1) 

Yes, after staying out of school for one year (2) 

Yes, after staying out of school for over a year (3) 

Yes, but I don't know when (4) 

No, I don't plan to continue my education after high school (5) 

I don't know if I will continue my education after high school (6) 

 

Q5 Which of the following do you plan to attend? 

Four‐year college or university (1) 

Two‐year community college (2) 

Vocational, technical or trade school (3) 

 

40  

Q6 Where have you gone for information about the entrance requirements of various colleges? 

Guidance counselor (1) 

Teacher (2) 

Coach (3) 

Parent (4) 

Friend (5) 

Sibling (6) 

Other relative (7) 

College publications or websites (8) 

College representatives (9) 

College search guides, publications, websites (10) 

None of the above (11) 

 

Q7 Would you like to participate in athletics (not intramurals) at the collegiate level? 

Yes (1) 

No (2) 

 

Q8 Do you hope to receive an athletic scholarship to pay for all or part of your college expenses? 

Yes (1) 

No (2) 

 

Q9 Which of the following are reasons why you have decided NOT to continue your education past high school? 

  Yes (1)  No (2) 

I do not like school (1)     

My grades are not high enough (2)     

I will not need more education for the career I want (3) 

   

I cannot afford to go on to school (4) 

I'd rather work and make money than go to school (5) 

   

I plan to be a full‐time homemaker (6)     

I do not feel that going to school is important (7) 

   

I need to help support my family (8)     

 

 

Q10 Write in the name of the job or occupation that you expect or plan to have right after high school 

Click to write Choice 1 (1) ____________________ 

I don't plan to work right after high school (2) 

I don't know (3) 

 

41  

Q11 Write in the name of the job or occupation that you expect or plan to have at age 30. 

Click to write Choice 1 (1) ____________________ 

I don't plan to work when I'm 30 (2) 

I don't know (3) 

 

Q12 How far in school do you think your Mother or Father or Guardian want you to go? 

  Mother (1)  Father (2)  Guardian (3) 

Less than high school graduation (1) 

     

High school graduation or GED only (2) 

     

Attend or complete a 2‐year school course in a 

community or vocational school (3) 

     

Attend college, but no complete a 4‐year degree (4) 

     

Graduate from college (5)       

Obtain a Master's degree or equivalent (6) 

     

Obtain a Ph.D., M.D., or other advanced degree (7) 

     

Don't know (8)       

Does not apply (9)       

 

42  

Q13 What do the following people think is the most important thing for you to do right after high school? 

  Does not apply (1) 

Go to college (2) 

Get a full‐time job (3) 

Enter a trade school or an 

apprenticeship (4) 

Enter military service (5) 

Get married 

(6) 

They think I should 

do what I want (7) 

They don't care (8) 

I don't know (9) 

Your mother (1) 

                 

Your father (2) 

                 

Your friends (3) 

                 

A close relative (4) 

                 

School counselor 

(5)                   

Your favorite teacher (6) 

                 

Coach (7)         

 

 

43  

Appendix F: Data Collection Table  

Output/ Outcome 

Measure/ Indicator

Data Source

Measurement Interval 

Target/ Benchmark* 

Person Responsible 

Outputs Tutoring sessions  Number of minutes 

spent in one‐on‐one tutoring   

Tutoring logs  Weekly    Tutors 

Stride Academy sessions 

Number of minutes spent on online customized lessons 

Online reports from Stride Academy 

Weekly  60 minutes per week  Education Coordinator 

Addressing barriers to academic achievement 

Number of barriers identified 

Interviews with students, families and teachers. Case management logs 

Monthly    Education Coordinator 

College exposure  Participation in college site visits 

Site visit attendance log  

End of each semester At least one visit each semester 

Field Trips  Participation in off campus field trips 

Field trip attendance log 

Monthly  1 – 2 trips per month   

Mentoring sessions  Minutes spent one‐on‐one with mentor 

Mentoring logs Weekly 60 minutes per week Mentors

Case management  Complete individual development plan 

Case management logs 

Yearly    Education Coordinator 

Short-Term Outcomes Increased school attendance 

Days absent from school 

School attendance records 

Quarterly

Addressed barriers to academic achievement 

Percentage of identified barriers with action plan 

Case management logs 

Quarterly     

Increased study time   Change in minutes using Stride Academy per week, additional reported study time 

Stride academy logs, student surveys 

Quarterly     

44  

 Intermediate Outcomes 

Improved behavior at school 

Number of suspensions, expulsions, detentions 

School records End of each semester

Reduced barriers to academic achievement 

  Case management logs, teacher and family feedback 

End of each semester     

Increased educational aspirations  

Member survey  Mentoring and case management logs 

Yearly     

Improved math and reading proficiency 

Increase evaluation scores from Stride Academy,  Percentage of students above proficient levels on WKCE test, Core GPA 

Stride Academy logs,WKCE test scores, Core GPA 

Stride academy: End of each semester WKCE scores: end of 8th grade Core GPA: Yearly 

Long-Term Outcomes Increased AVID/TOPS acceptance 

Percentage of students accepted into AVID/TOPS 

AVID/TOPS records  Entry into high school     

Improved college readiness  

Number of students who take ACT/SAT, fill out FAFSA, and apply to college 

Follow‐up interviews  End of high school     

Increased educational aspirations 

Number of advanced high school classes (AP or Honors) 

School records, Follow‐up interview 

End of high school

 

45  

Appendix G: Timeline

   

August  September  October  November  December January February March April May June  July

Second semester testing is conducted. 

College Club continues. Data Collected on: 

Case Management, Stride Logs, mentoring logs, attendance logs. 

Prep materials, discuss eval with staff 

College club begins. Initial testing conducted for both college club participants (tx) and BG Club members not in the program (comp).  

Summer programs begin, data collected

End of semester tests conducted. Review first semester data for completeness, not analysis of outcomes. Look at implementation data 

Short Term College Club Evaluation Timeline

College Club continues. Data Collected on: 

Case Management, Stride Logs, mentoring logs, 

End of Semester testing is conducted 

School year data is analyzed for results 

 

Summer programs ends, data analyzed 

46  

2015  2016  2017  2018  2019 2020 2021 2022 2023 2024 2025 2026

3 cohorts in program 

Long Range College Club Evaluation Timeline

2 cohorts in program 

1 cohort in program 

2nd

cohorts graduates HS 

1st cohort graduates HS 

3rd cohort graduates HS 

1st cohort graduate from a 4 year college 

A decade of effects can be analyzed 

2nd cohort graduate from a 4 year college 

3rd cohort graduate from a 4 year college 

47  

Appendix H: College Club Evaluation Budget  

Year Professional Analyst @ $27-$36 an hour

(# hours)

Incentive Budget @ $14 Pizzas (# pizzas)

Total

Year 1 $2160-$2880 (80) $700 (50) $2,860 – $3,580 Year 2 $2160-$2880 (80) $700 (50) $2,860 – $3,580 Year 3 $2160-$2880 (80) $700 (50) $2,860 – $3,580 Year 4 $540 - $720 (20) N/A $540 - $720 (20) Year 5 $540 - $720 (20) N/A $540 - $720 (20) Year 6 $540 - $720 (20) N/A $540 - $720 (20) Year 7 $540 - $720 (20) N/A $540 - $720 (20) Year 8 $540 - $720 (20) N/A $540 - $720 (20) Year 9 $540 - $720 (20) N/A $540 - $720 (20) Year 10 $540 - $720 (20) N/A $540 - $720 (20) 10 Year Total $10,260 - $13,680 $2,100 $12,360 - $15,780

48  

References:  

Afterschool Alliance. 2006. Evaluations backgrounders: A Summary of formal evaluations of the academic impact of afterschool programs. Washington, DC: Afterschool Alliance.

Albouy, David. [2014]. Program Evaluation and the Difference in Difference Estimator.

Econometrics Laboratory. University of California-Berkeley. Accessed December 11. http://eml.berkeley.edu/~webfac/saez/e131_s04/diff.pdf.

Boys & Girls Club of Dane County (BGCDC). 2014. “College Club.”

http://www.bgcdc.org/what-we-do/college-club/. Britsch, Brenda, Nicky Martin, Amy Stucynski, Bethany Tomala and Patti Tucci. 2005. Literacy

in After-School Programs: A Literature Review. Portland: Northwest Regional Educational Laboratory.

Chinn, Menzie. 2014. “Lecture 7: Dummy Variables.” Advanced Statistical Methods for Public

Policy Analysis. Public Affairs 819. Madison: University of Wisconsin. Cosden, Merith, Gale Morrison, Ann Leslie Albanese, and Sandra Macias. 2001. When

homework is not home work: After-school programs for home assistance. Educational Psychologist 36(3): 211 - 221.

Curnan, S., L. LaCava, D. Sharpstee, M. Lelle, and M. Reece. 2004. “WK Kellogg Foundation

Evaluation Handbook.” https://www.wkkf.org/resource-directory/resource/2010/w-k-kellogg-foundation-evaluation-handbook.pdf. 54-57.

Granger, R.C. 2008. After-school programs and academics: Implications for policy, practice and

research. Social Policy Report 22: 3-19. Harvill, Eleanor, Rebecca Maynard, Hoa Nguyen, Claire Robertson-Kraft, and Namrata

Tognatta. 2012. “Effects of college access programs on college readiness and enrollment: A meta-analysis.” Washington, DC: Institute of Education Sciences (IES), US Department of Education (US ED).

Hollister, R. 2003. The Growth in After-school Programs and their Impact. Washington, DC:

The Brookings Institute. McKenzie, David. 2011. “Power Calculations 101: Dealing with Incomplete Take-up.”

Development Impact: News, views, methods, and insights from the world of impact evaluation. http://blogs.worldbank.org/impactevaluations/power-calculations-101-dealing-with-incomplete-take-up.

Power Hour: Making Minutes Count. 2008. After-School Program Outcomes: A Review of the

Literature and Evaluation Data. Branch Associates, Inc.

49  

http://www.bgca.net/programs/eduCareer/Documents/NEW-Power_Hour_microsite/Power_Hour_Literature_Review_FINAL_12-02-08.pdf.

Raudenbush, S.W., J. Spybrook, R. Congdon, X. Liu, A. Martinez, H. Bloom, and C. Hill. 2011.

Optimal Design Software for Multi-level and Longitudinal Research (Version 3.0) [Software]. Available from www.wtgrantfoundation.org.

Shager, Hilary. 2014. “Quasi-Experimental Design Comparison Grid,” Handout from PA 871:

Public Program Evaluation. University of Wisconsin–Madison. Wisconsin Center for the Advancement of Postsecondary Education (WISCAPE). 2014.

AVID/TOPS: Assessing the Outcomes of a Private-Public Pre-College Program. http://www.wiscape.wisc.edu/wiscape/projects/ongoing-projects/rp043.

Wisconsin Council on Children and Families (WCCF). 2013. Race to Equity: A Baseline Report

on the State of Racial Disparities in Dane County. http://racetoequity.net/dev/wp-content/uploads/WCCF-R2E-Report.pdf.

Yancsurak, Lonnie S. 2013. "Effectiveness of a computer-based afterschool intervention to

increase reading comprehension." Walden University.