Supplemental Educational Services (SES) Data Collection

Process: Roles and Responsibilities of LEAs

GaDOE Data Collections ConferenceAugust 17, 2011

Athens, GAJessica Johnson-Operations Analyst

Agenda

• SES Evaluation Overview• SES Data Categories• Parent and Student Surveys• Provider Surveys• District Surveys• SES Student Data Collection• Data Collection Reports

SES Overview• Supplemental Educational Services (SES)

include academic assistance such as tutoring and remediation which are provided outside of the regular school day and designed to increase the academic achievement of students in low-performing schools.

• Title I Schools NI2 or higher, or NI1 schools exercising the Differentiated Accountability option, must offer SES.

SES Evaluation Overview• GaDOE is required by Title I, Part A to:– Develop, implement, and publicly report on

standards and techniques for monitoring the quality and effectiveness of services offered by approved providers.

– Withdrawing approval from providers that fail, for two (2) consecutive years, to contribute to increasing the academic proficiency of students served.

– Title I Section 1116(e)(4)(B)

Purpose of Data Collection• GaDOE collects information from Title I Directors,

parents, and students to evaluate SES providers, the SES program, and Choice implementation.

• Use data for SES provider report cards.• Allows GaDOE to make program modifications based

on survey data and remove SES providers that have not demonstrated effectiveness in increasing student achievement.

SES Data Categories

• Effectiveness• Customer • Service

• Service • Delivery

• Has the • SES provider • contributed • to increasing • student

• achievement?

• Are• stakeholders• pleased • with

• the program?

• Are LEAs• and

• SES providers• implementing• and complying• with SES rules • and regulations?

Service Delivery

• An evaluation of SES provider program implementation and compliance as defined by:– Monitoring score– Survey data

Surveys administered and collected from the following stakeholders:

• LEAs (Title I Directors).• Parent(s) / Legal Guardian(s).• Middle school students.• High school students.• SES providers.

Monitoring Scores provided by: • GaDOE Monitors• LEAs

Customer SatisfactionCustomer Satisfaction measures stakeholder perception of:

• Communication & interaction with the school system.

• Service delivery of a SES provider’s program.

• Satisfaction with a SES provider’s program.

• Perceived impact on student achievement.

• LEA monitoring reports.

Surveys administered and collected from the following stakeholders:

• LEAs (Title I Directors).• Parent(s) / Legal Guardian(s).• Middle school students.• High school students.• SES providers.

8

EffectivenessA matched samples comparison group model* is conducted by an external evaluator, University of Georgia (UGA), for examining SES student test results.

*According to the Center on Innovation and Improvement, this approach is one of the most rigorous methods for determining the effect of SES services on student achievement (Evaluating Supplemental Educational Service Providers: Suggested Strategies for States, 2nd Edition).

Analysis of effectiveness will be based upon results from:

• Criterion Reference Competency Test (CRCT).

• Georgia High School Graduation Test (GHSGT).

• End of Course Tests (EOCT).

9

Categories for SES Customer Service Survey Items

Directors Parents Students

Communication & Interaction with School System (5 Questions)

1.The provider responds to requests to participate in district fairs, town halls, and SES-related parent meetings.6. The provider submits invoices only for services rendered.7. The provider submits invoices for services rendered in a timely manner.8. The provider is easy to contact.9. The provider works collaboratively with the district to resolve any issues that arise.

Compliance/ Service Delivery (14 Questions)

2. The provider begins serving students in a timely manner.3. The provider develops goals for each student receiving services.4. The provider furnishes a written description of how each student’s progress will be measured. 5. The provider submits monthly progress reports for each student.

6. Did the tutor ask your opinion about your child’s learning needs before beginning the tutoring sessions?7. Were you given a copy of your child’s learning plan?8. Did the tutor give you a copy of a progress report for your child’s work at least once a month?9. Did your tutor keep you informed about your child’s progress?10. Were you able to ask the tutor questions about your child’s lessons and progress?

6. Did your tutor give you a test before beginning your after-school lessons?7. Did your tutor share with you a plan for your after-school lessons?8. Did your tutor give you a copy of a progress report on your work at least once a month?9. Did your tutor keep you informed about your progress?16. Did your parent or guardian ever meet with your tutor to discuss your progress?

Categories for SES Customer Service Survey Items

Directors Parents Students

Satisfaction(17 Questions)

10. Overall, this provider offers quality instructional services to students.11. Overall, it is easy for our LEA to work with this provider.12. I would recommend that this provider continue offering SES to students in Georgia.

8a. If you got reports at least once a month, were the reports easy to understand? 10a. If you asked questions, did the tutor answer these questions to your satisfaction?11. Were the sessions the right length of time for your child?10. Would you sign up your child to receive tutoring from this tutor again?12. Overall, were you satisfied with the tutoring your child received?14. If your child missed a tutoring session, was it easy to reschedule? 15. Would you recommend this program to other parents?

11. Were you able to ask your tutor questions about your lessons and progress?11a. If you asked questions, did the tutor answer the questions to your satisfaction? 12. Were the sessions the right length of time for you?13. Would you sign up to receive tutoring from this tutor again?14. Overall, were you satisfied with the tutoring you received?15. If you could, would you like to receive more help from this tutor?17. Would you recommend this program to another student?

Impact(2 Questions)

13. Did your child do better academically in school after working with this tutor?

10. Did your grades in school improve after you started after-school tutoring?

Categories for SES Customer Service Survey Items

Directors Parents Students

Monitoring(9 Questions)

a. The provider’s instruction reinforces the LEA’s instructional program. b. The provider's instructional program is aligned with the Georgia Performance Standards. c. The provider develops a learning plan for each student.d. The provider's instruction is individualized for each student.e. The provider gives positive reinforcement to each

student. f. The provider gives ongoing feedback to each student. g. Provider’s instructional materials are appropriate for student skill levels. h. The provider's instructional program is appropriate for students with limited English proficiency, if applicable. i. The provider offers appropriate SES instruction for students with disabilities (students with an IEP or 504 plan), if applicable.

Parent and Student Surveys• Paper surveys• Distributed by District Title I Directors• All parents of students for which providers billed

hours of service• Middle and High School students for which providers

billed hours of service• English and Spanish versions• Two week window in April• Returned to UGA Program Evaluation Group

Parent and Student Surveys• Parents of SES Students– 2,187 surveys (16.6% return rate)– Responded to questions about 80 SES providers

across 55 school systems– 133 parents (6.1% returned a survey in Spanish)

• Middle and High School SES Students– 3,305 surveys (29.2% return rate)– Responded to questions about 77 different SES

providers across 51 school systems.

Parent and Student SurveysEffective LEA practices:

• Maintain list of students receiving SES throughout the school year

• Develop clearly defined system for distribution and collection that works for your district

• Recruit one person to serve as SES contact for each SES school for collection purposes

• Include name of provider on parent/student surveys before distribution

Provider Surveys

• Providers surveyed about overall experience with LEAs

• SES Providers– 342 surveys from 57 of 92 providers (62% of SES

providers)– 74 school systems represented

Findings from Provider SES Surveys

• According to Providers, Systems do a good job:– Clearly outlining contract obligations– Entering into contracts with providers in a timely

manner– Treating providers fairly– Having clear policies about the use of facilities• 57.6% of providers say they are allowed to use school

facilities

District Surveys• Districts serving SES in one or more schools

complete the following:– Provider Satisfaction Surveys• Questionnaire about overall experience with each

provider

– System survey• Report district level data about SES and Choice

– SES Student Data Collection• Collect individual student level data for students

receiving SES

District Survey Response Rate

• Title I Directors– 465 surveys from 61 school systems– 95% of systems with schools that used SES

providers– 90 different SES providers represented

Provider Satisfaction Surveys

• What information is requested?– Complete a survey for each contracted provider in the

system.

• Why is this information requested?– Provides GaDOE with information on the district

perspective of SES provider compliance and interaction with the district, service delivery, and satisfaction with the SES provider. Results in program modifications and/or follow-up with individual providers.

– Data is used in the SES provider report cards.

Provider Satisfaction Surveys

How are surveys disseminated & collected?• UGA Program Evaluation Group provides Title I

Directors with a website link to access surveys.• Title I Directors complete a survey for each

applicable provider on the website.• Title I Directors submit each completed survey

directly to UGA.

System Survey

What information is requested?• Provide aggregate information on district and

school level eligibility and participation.

Why is this information requested?• Provides GaDOE with information to track

trends as well as information to provide on US Department of Education reports.

System Survey

How are surveys disseminated & collected?• UGA provides Title I Directors with a website

link to access surveys.• Title I Directors complete the survey on the

website.• Title I Directors submit the completed survey

on the website.

System Survey• Number of Title I schools required to offer Choice • Number of students eligible for Choice• Number of students whose parents requested a transfer• Number of students who were offered the opportunity to transfer • Number of students transferring• Number of public and charter schools to which students transferred • Amount spent on transportation for transferred students• Number of Title I schools in your system required to offer SES • Number of students eligible for SES (district and by school)• Number of students whose parents requested SES (district and by school)• Number of students who received SES (district and by school)• Number of SES providers serving students • Amount of Title I funds paid to all SES providers • Per-pupil allotment• Amount of the 20% obligation for SES and Choice• Amount actually set-aside for SES and Choice

SES Student Data Collection

What information is requested?• Information on SES participation for every

participating student.

Why is this information requested?• Allows GaDOE to determine which providers are

effective in increasing student achievement.• After the UGA evaluation, data is used for the SES

provider report cards.

SES Student Data Collection

How are surveys disseminated & collected?• Title I Directors must access the GaDOE Portal and

receive approval to receive the survey.• Once approval is provided, Title I Directors will enter

the required information for each student participating in SES.

• Once all student information is entered, Title I Directors submit the data on the GaDOE Portal.

SES Student Data Collection

The following information is pre-populated by GaDOE:• System name & code• School name & code• Student last & first name • Student GTID• Student grade

The LEA provides the following information:• Provider name• Provider code• Date services began (general date)• Date services ended (general date)• Subjects of service: ELA, reading, math,

science, or social studies• Provider hourly rate• Total expended on the student

What information is requested?

Data Collection Reports

• GaDOE publishes the following reports based on data collected from LEAs:– SES Provider Report Cards– SES Study of Academic Achievement– Survey Results for SES Providers

• http://www.gadoe.org/tss_title_parent.aspx?PageReq=TSSTitleSES

QUESTIONS?

Contact Information

Jessica JohnsonOperations Analyst

Office of School ImprovementGeorgia Department of Education

jejohnson@doe.k12.ga.us(404) 657-9864