SES Data Collection Methods SES Data Collection Methods and Multi-State Resultsand Multi-State Results

Allison PotterAllison PotterSteven M. RossSteven M. Ross

Center for Research in Educational PolicyCenter for Research in Educational PolicyThe University of MemphisThe University of Memphis

Overall implementation efforts and outcomes by state, district, and local school educators

Are providers:

Research FocusResearch Focus

• Raising student achievement in reading/language arts and mathematics?

• Communicating effectively with LEAs, principals, teachers, and parents?

• Aligning tutoring curriculum to standards?

• Developing instructional plans for students?

• Serving the needs of ELL/Special Education students?

Surveys

District/Local Data CollectionDistrict/Local Data Collection

• Online survey system for LEAs, principals/site coordinators, teachers, and providers

• Login information e-mailed to LEAs• Login information mailed to schools with parent

surveys

• Paper-based surveys for parents• Mailed (FedEx) to schools for distribution

Student achievement analysis

District/Local Data CollectionDistrict/Local Data Collection

• Request SES student names, identifiers (SSN), and demographic information from LEAs

• LEAs send confidential student data to SEA

• State department data division adds test scores to SES student files (pretest and current year)

• Comparison group test scores are pulled and added to data file

• Student identifiers are removed (or encrypted) and sent to CREP for analysis

• Selected survey questions:

Data Collection ToolsData Collection Tools

• What was the start date of provider services?

• In which subjects did your students receive services from this provider?

• Are you employed by the provider for which you are completing this survey?

How often does the provider:

• Communicate with you during the school year?

• Meet the obligations for conducting tutoring sessions?

• 2005-2006 School Year Evaluations:

Aggregate ResponsesAggregate Responses

• 4 states (KY, LA, TN, VA)

Respondent groups:

• 148 LEAs

• 209 Principals/Site Coordinators

• 753 Teachers

• 1,388 Parents

• 54 Providers

Data Collection ToolsData Collection ToolsAggregate sample responses:

How often does the provider: Respondent groupFrequently and Occasionally

Talk to me about my child’s progress?

Parents 74.0%

Talk to my child’s teachers about his/her progress?

Parents 52.8%

Send letters or notes home to me about my child’s progress?

Parents 65.6%

Communicate with me during the school year?

TeachersPrincipal/Site CoordinatorDistrict Coordinator

39.5%66.6%49.7%

Communicate with parents during the year?

TeachersPrincipal/Site CoordinatorDistrict Coordinator

37.7%58.4%62.3%

Meet the obligations for conducting tutoring sessions?

TeachersPrincipal/Site CoordinatorDistrict Coordinator

45.6%73.4%90.6%

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

District and Local School Implementation:

• All LEAs notified parents in their district of their rights under NCLB and the SES eligibility status of their child.

• The majority of parents were pleased with the way their LEAs helped them obtain SES for their child and felt they had enough time to make an informed choice on providers.

• Some parents asked for help from school leaders in choosing an SES provider and found it difficult to get information on program/provider details from their local school.

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

Parent perceptions:

• Some parents were pleased with the amount of communication from the provider and with the progress reports sent home with their child.

• A few parents expressed frustration at attempts to sign their child up for services. They told of providers never calling them back and quitting mid-way through tutoring, with no prior notification.

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

Parent perceptions (continued):

• Some parents felt the at-home providers took advantage of their lack of knowledge on parental rights under NCLB and did not fulfill their obligations.

• A few providers acted in an unprofessional manner at the home of the student.

• Some parents were asked by providers to pay for services or the tutoring would be stopped.

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

Teacher perceptions:

• The majority of teachers did not have enough information about providers serving students in their school to respond to questionnaire items.

• A few teachers saw school-based online providers letting children view inappropriate websites during their tutoring services.

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

Teacher perceptions (continued):

• Teachers noted grammatical and spelling errors in many reports they saw from providers.

• A few teachers witnessed providers grouping students in large (20 students) groups with only one tutor per site, with students fighting and running throughout the school.

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

Teacher/Principal perceptions:

• Many teachers and principals were unaware of which providers were working with their students, citing no communication with them during the year.

• Many school leaders described aggressive attempts by providers to gain access to student names and to school sites.

• Selected survey questions:

Data Collection ToolsData Collection Tools

Overall assessment:

• I believe the services offered by this provider positively impacted student achievement

• Overall, I am satisfied with the services of this provider

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

Overall assessment: Respondent group Strongly Agree and Agree

I believe the services offered have helped my child’s/student’s achievement.

ParentsTeachersPrincipal/Site CoordinatorDistrict Coordinator

82.5%50.7%41.6%55.7%

Overall, I am pleased with the services of this provider.

ParentsTeachersPrincipal/Site CoordinatorDistrict Coordinator

82.9%48.3%45.2%59.9%

• Aggregate sample responses:

Data Collection ToolsData Collection Tools

Overall assessment:

• Parents were the most satisfied group of all those surveyed, with the majority expressing gratitude at the efforts tutors made to help their child succeed.

• Many parents, teachers, and principals expressed frustration that provider services did not start sooner in the year, with many students not being served until March or April.

• A few LEAs noted providers were not returning the required paperwork on student attendance but were demanding in their requests for payment.

Data Collection ToolsData Collection ToolsSingle state parent responses: n = 3,523

Why did you enroll your child with this provider? Percent

District told me I had to enroll my child in SES. 13.8

District gave me only one choice of provider. 11.6

This provider offered a gift/bonus to me or my child for enrolling. 5.4

This provider would give rewards to me or my child for attendance. 8.4

This provider would give rewards to me or my child for good academic performance.

8.4

District enrolled my child for me. 10.6

Provider is convenient – no transportation problems. 41.1

Provider’s program fits my child’s needs and learning style. 56.9

Provider name is known to me through advertising. 12.0

School or teacher recommended this provider for my child. 44.9

Another parent recommended this provider for my child. 8.9

• Provider survey selected questions/responses:

Data Collection ToolsData Collection Tools

• Describe the format of your services:• Program duration - about 6 weeks• Setting - school site, online, home• Format - small groups, individual

• Describe qualifications of tutors - most included using certified teachers and background checks

• List information regarding students served, goals achieved, and tutoring sessions attended - Most reported students achieving their goals if they attended the complete program.

Rubric of Overall Evaluation of Provider EffectivenessRubric of Overall Evaluation of Provider Effectiveness

OutcomeInsufficient Information

Below Standards

MarginalQuality

Acceptable Above Standards

1. Student Achievement

There is insufficient information available to determine student achievement outcomes.

Students have not shown gains related to tutoring received from service providers.

About half of the students have made some gain related to tutoring received from service providers.

There has been some gain for the majority (over 60%) of students related to tutoring received from service providers.

The effect size for students in the provider’s program is in the top one-third of all the effect sizes demonstrated by providers meeting standards for student achievement.

2. Communication There is insufficient information available to determine communication outcomes.

Provider has not communicated with the principals, teachers, and parents of students served.

There has been limited communication throughout the year between the provider and at least two of the following: principals, teachers, and parents.

There has been some regular communication throughout the year between the provider and the principals, teachers, and parents of students served.

There is an ongoing and sustained system of communication between the provider and the school-level educators as well as parents of students served.

3. Instructional Plans

There is insufficient information available to determine instructional plans of the provider.

Provider does not plan instruction explicitly geared to student needs or to reinforce their regular academic program.

Provider is in the planning stages of gearing instruction to student needs, and reinforcing the regular academic program.

Provider has made some attempt with the majority of students to plan instruction explicitly geared to student needs and to reinforce the regular academic program.

Provider instructional plans are explicitly geared to the needs of most or all students and reinforce the regular academic program.

4. Local and State Standards

There is insufficient information available to determine alignment with local and state standards.

None of the instructional plans used by the provider are aligned with local and state academic standards for students.

Provider is in the process of aligning instructional plans with local and state academic standards for students.

Some of the instructional plans used by the provider are presently aligned with local and state academic standards for students.

Most or all of the instructional plans are presently aligned with local and state academic standards for students.

5. Special Ed/ELL students

There is insufficient information available to determine special ed/ELL student outcomes.

Provider does not offer accommodations for addressing the needs of special ed or ELL students.

Provider has made limited accommodations for addressing the needs of special ed and ELL students.

Provider has made some accommodations for addressing the needs of special ed and ELL students.

Provider offers appropriate services, if needed, to special education and ELL students.

6. Provider Overall

There is insufficient information available to determine provider overall outcomes.

There is overall dissatisfaction with the provider at the district and school levels.

There is more dissatisfaction than satisfaction with the provider at the district and school levels.

There are mixed but mostly positive reactions about the provider at the school and district levels.

There is overall satisfaction with the provider at the district and school levels.

NO YES

YES

Positive Indeterminable

YES NO

Negative

Positive

Negative

NO YES

NO

YES

NO

YES

Negative

NO

YES

Positive

NO

COMPLIANCE? Serious? Removal

Achievement?

Minor Compliance Violations?

Implementation?

Probation I Probation II

Implementation?

Probation II Last Year?

Removal

Full StandingSatisfactory Standing

Implementation Improved?

Probation II

Achievement Improved?

Probation I

Removal

Probation I Last Year?

Decision Tree for SES ProvidersDecision Tree for SES Providers

Probation I