Evaluation 101: After School Programs
February 1, 2007
Region 3 After School Technical Assistance Center
Conference
Evaluation 101
ASES Evaluation—student data and how to get it
Prop 49 evaluation requirements in SB638
Suggested Timeline
Roles and Responsibilities
ASES Evaluation Annually by October 1 funded programs must
submit to CDE data on participating students Two years data required:
2006/2007 follow up/program year 2005/2006 baseline/prior year
Data submitted to CDE in Excel file Evaluation narrative
Prop 49 Evaluation SB638 Programs must submit individual student data:
Program attendance – participation Regular school day attendance STAR test results
Program effectiveness measures (1 or more from list) STAR test data Positive behavioral change* Homework completion rates* Skill development* Additional measures to be determined by CDE
* Measured by ASP teachers or regular day teachers
Prop 49 Evaluation SB638
State Board of Education will define performance measures by November 2007
Funded programs will comply with state evaluation requirements determined in collaboration with Advisory Committee
Terms ASP: After School Program
CHKS: California Healthy Kids Survey
Participation: after school program attendance
Regular attendance: regular school day attendance
Baseline: prior year 2005/2006
Follow up: program year 2006/2007
Continuing students: those who participated the previous year in After School Program
ASP Evaluation Keys to Success
Four distinctive types of student information needed
Build evaluation database throughout year Clear roles and responsibilities Good ongoing communication Document process and sources
Evaluation data: Four Types Needed on Students
Participation in after school program: number of days attended program in the school year
Regular school attendance: days enrolled and days absent in the school year
California Standards Test proficiency level math and language arts
California Healthy Kids Survey questions at entry and exit from program (4 questions)
Sources of DataData Source
Participation of students in After School Program
ASP Program staff gather and record throughout year
Regular school attendance baseline and follow up
District data system for current and prior years; prior year ASES evaluation for continuing participants
California Standards Test results on math and language arts proficiency
District test director or curriculum director receives CST data annually
California Healthy Kids Survey 4 questions
ASP Program staff gather from students at entry and exit
Sample Excel File in Folder
Who Needs to be involved? ASP director and site staff
District data managers, or evaluation department, or fiscal department with regular school attendance information on students
District testing and curriculum staff who obtain the CST results annually and disseminate them to schools
Staff with expertise in Excel (or database software)
Tips
Create a roster of key people involved in ASP evaluation with their phone numbers, faxes and email addresses.
Do ASP evaluation update every 2-3 months to keep all informed on progress. Will raise ASP evaluation visibility
Immediately thank via email and/or phone call anyone who gives you data
Write it down – for next year!
Participation Data
Participation Data
Prior to program beginning: Decide on how to gather and record program attendance:
after school program software, Excel spread sheets or Access database
Find last year’s evaluation report in Excel format to get baseline data (ongoing programs)
Decide on roles and responsibilities for data entry Who records attendance Who enters data into computer
Participation Data
Record child’s name (first and last), grade, school, Student ID Number and days attended by month
As new children enter, record them in the data system on an ongoing basis
Continuing participants, gather baseline for participation, regular school attendance and CST from prior year’s Excel file
Tips
Keep track of Student ID Numbers – they are the key to obtaining regular school attendance and CST results
Get child’s name as they are registered in school – no nicknames. James Jones not JJ Jones or Jimmie Jones
Don’t record attendance in Word , you will need to be able to add figures and sum them up
Regular School Attendance
Regular School Attendance
Data needed: Days enrolled, days absent current school year
2006/07 and prior school year 2005/06
Where is it kept: District data system Link by Student ID Number and school
Regular School Attendance
What you need to do: Identify where data is kept in district Contact person who can get you access to data
and what you will need: initial information query in fall
Contact in March to obtain baseline prior year attendance for students new to program
Contact in June to obtain current year attendance for all participants
Tips
Don’t be surprised if there is no prior year attendance for new participants – student mobility means students may be new to the school in program year
Check data for new participants in March for reasonableness Days enrolled should be around 170 (150-181) Days absent should be between 1 and 20; rarely
over 20
California Standards Tests
California Standards Tests Results available July or August following end of school
year
Math score and Language Arts Score
Scores are 1 through 5 or Far Below Basic, Below Basic, Basic, Proficient and Advanced
Best source for student demographics: gender, ethnicity, English Language Learner Status
California Standards Tests Data needed
Proficiency level in math and language arts for program year (spring 2007) and baseline year (spring 2006)
Demographics on students Where it is kept
Comes to district testing staff July or August on CD ROM from state testing contractor
For continuing ASP students, baseline data is on prior year evaluation report to CDE
For new ASP students, baseline data is prior year CST results 2005/2006
California Standards Tests What you need to do:
Identify test contact person in district Contact person who can get you access to test data:
initial information query fall Contact in March to obtain baseline prior year
proficiency for students new to program Submit file with id numbers of new students by school with
data request Contact in June to remind them of upcoming
summer request Submit file with id numbers of all participating students in
summer for new test data and new demographics
Tips
• Don’t be surprised if there is no prior year CST results for new participants – student mobility means students may be new to the school in program year
• Keep in touch with testing person for district, sometimes data comes in early
• Don’t wait until the summer, many people are not working in summer months!
California Healthy Kids Survey
California Healthy Kids Survey
Train staff to administer survey when program begins and when students leave the program or near the close of school year
Enter initial (pre) surveys in evaluation database as they come in.
Survey Questions1. In the past 30 days, how often have you wanted to go to school? (Never,
one or two times, About once a week, A few times a week, Everyday) 2. In the past 30 days, how often have you studied for a test? (Never, one or
two times, About once a week, A few times a week, Everyday) 3. In the past 30 days, how often have you felt unsafe at school? (Never, one
or two times, About once a week, A few times a week, Everyday) 4. In the past 30 days, how often have your parents asked you about school?
(Never, one or two times, About once a week, A few times a week, Everyday)
Tips
Be sure to gather pre surveys for students who enter the program during the year.
Train staff to look for red flags if a student reports they don’t feel safe, for example.
Gather the post follow up surveys (same 4 questions as on entry to program) beginning about one month before the program ends.
Enter into database.
Master timeline
Month before program starts
Fall
Decide on database format Set up participation data entry
system Train staff on California Healthy Kids
Survey and attendance procedures Find last year’s evaluation report
Excel files Identify person in district with regular
school attendance data and CST test scores and hold initial meeting
Master timeline, continued
October through March
Check to see that you can generate list of students by school with id numbers and total days attended for first three months or six months
Check that number of initial California Healthy Kids Surveys in database match number of participants
Build baseline for ongoing students using ASP participation, regular school attendance and CST results from previous evaluation Excel file
Master timeline, continued
May of school year
Meet with district attendance data contact; provide list of id numbers of participating students; obtain prior year attendance for new students
Review completeness of database, look for holes on continuing students and fill them
Master timeline, continued
June
July
August
Request current year attendance data for all participants, giving school list of students with ID numbers
Request CST data and demographics from district test contact
Finalize Excel file with Student ID, days of ASP attendance; regular school attendance; CHKS answers
Master timeline, final
September Add test scores from CST to Excel files
Complete narrative Submit to CDE