Evaluating SES Providers

Steven M. RossAllison Potter

Center for Research in Educational PolicyThe University of Memphis

http://www.memphis.edu/crep

Supplemental Educational Services Supplemental Educational Services (SES)(SES)

• Required under No Child Left Behind (NLCB) for Title I Schools that have not made Adequate Yearly Progress (AYP) for three consecutive years.

• Low-income students from identified Title I schools are eligible to receive free tutoring services.

• Students are prioritized by greatest academic need if district funds are limited.

• Potential service providers apply to serve students and may be approved by the State Department of Education.

Service ProvidersService Providers

• Providers contract with Local Educational Agencies (LEAs) to provide tutoring services to students.

• Providers are paid for their services - an amount not to exceed the Title I per pupil allotment.

Effectiveness:

Increased student achievement in reading/language arts or mathematics.

Customer satisfaction:

Positive perceptions by parents of SES students.

Service delivery and compliance:

Positive perceptions by principals, teachers, LEA staff, etc.

Determining Evaluation MeasuresDetermining Evaluation Measures

Effectiveness (Student Achievement)

ServiceDelivery and Compliance

CustomerSatisfaction

ProviderSurvey

District CoordinatorSurvey

Principal/LiaisonSurvey

Teacher Survey

Parent Survey

AdditionalTests

StateTests

Figure 1. Components of a Comprehensive SES/Evaluation Modeling Plan

Overall Provider Assessment

1. Student-level test scores from state-mandated assessments

Effectiveness MeasuresEffectiveness Measures

Considerations:• availability only for certain grades (e.g., 3-higher)?• Lack of pretest scores prevents gains from being

determined

2. Supplementary individualized assessments in reading/language arts or math

Effectiveness MeasuresEffectiveness Measures

Considerations:• Without pretest scores and comparison students, SES

gain cannot be determined• Validity may be suspect if assessments not

administered by trained independent testers

3. Provider-developed assessments in reading/language arts or math

Effectiveness MeasuresEffectiveness Measures

Considerations:• Test results may not be valid or suitable for state’s

evaluation purpose• Tests may favor provider’s strategies

1. Parent and family perceptions

Customer Satisfaction MeasuresCustomer Satisfaction Measures

Considerations:• Parent respondents may not be representative of the

population served by provider• Sample sizes will vary due to provider size• Comparisons limited due to parent familiarity with only

one provider

2. Student perceptions

Customer Satisfaction MeasuresCustomer Satisfaction Measures

Considerations:• Young students may have difficulty judging quality of

services and communicating impressions• Time consuming and may require parent permission to

obtain

1. Records of services provided, student attendance rates, and costs

Service Delivery and Compliance MeasuresService Delivery and Compliance Measures

Considerations:• States may obtain data from a variety of sources,

including providers, teachers, principals, and district staff

• Corroborating data from multiple sources can increase accuracy of evaluation conclusions

2. Feedback from SES customers

Service Delivery and Compliance MeasuresService Delivery and Compliance Measures

Considerations:• First-hand impressions or observations may be

lacking• Translation may be needed to reach parents who do

not speak English• Obtaining representative samples may be difficult

3. Feedback from district staff

Service Delivery and Compliance MeasuresService Delivery and Compliance Measures

Considerations:• Districts may lack firsthand impressions or

observations of tutoring services• Some districts may also be SES providers

4. Feedback from school staff

Service Delivery and Compliance MeasuresService Delivery and Compliance Measures

Considerations:• Teachers may also be SES instructors or lack first-

hand impressions of providers• Teachers may need to provide information on multiple

providers, which may be confusing and time consuming

• Identifying teachers to solicit responses may be difficult

A. Benchmark Comparison

Rating = ++ (Low to Moderate rigor)

Percentage of SES students by provider attaining “proficiency” on state assessment

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

A. Benchmark Comparison

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

Upgrades

• Percentage of SES in all performance categories (“Below Basic”, “Basic”, etc.)

• Comparison of performance relative to prior year and to state norms

• Comparison to a “control” sample

A. Benchmark Comparison

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

Advantages• Inexpensive and less demanding• Easily understood by practitioners and public• Linked directly to NCLB accountability

Disadvantages• Doesn’t control for student characteristics• Doesn’t control for schools• Uses broad achievement indices

B. Multiple Linear Regression Design

Rating = +++ (Moderate rigor)

Compares actual gains to predicted gains for students enrolled in SES, using district data to control for student variables (e.g., income, ethnicity, gender, ELL, special education status, etc.).

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

B. Multiple Linear Regression Design

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

Advantages

• More costly than Benchmark, but relatively economical

• Student characteristics are statistically controlled

Disadvantages

• Doesn’t control for school effects• Less understandable to practitioners and public• Effect sizes may be less stable than for Model C.

C. Matched Samples Design

Rating = ++++ (High Moderate to Strong rigor)

Match and compare SES students to similar students attending same school (or, if not feasible, similar

school)

Use multiple matches if possible

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

C. Matched Samples Design

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

Advantages

• Some control over school effects• Easily understood by practitioners and public• Highest potential rigor of all designs

Disadvantages

• More costly and time consuming• Within-school matches may be difficult to achieve

D. Combination (Hybrid) Design

Evaluation Designs: Student AchievementEvaluation Designs: Student Achievement

• Uses a mixture of three main designs to meet special data situations within the State

• State level analysis may be benchmark for most districts and matched samples for largest district(s)

• Accommodates different student-level data and statistical staff resources

• Surveys for LEAs, principals/site coordinators, teachers, parents, and providers.

Data Collection ToolsData Collection Tools

• Common core set of questions from all groups to permit triangulation.

• Open-ended question, “Additional comments”

Rubric of Overall Evaluation of Provider Effectiveness

OutcomeInsufficient Information

Below Standards

MarginalQuality

Acceptable Above Standards

1. Student Achievement

There is insufficient information available to determine student achievement outcomes.

Students have not shown gains related to tutoring received from service providers.

About half of the students have made some gain related to tutoring received from service providers.

There has been some gain for the majority (over 60%) of students related to tutoring received from service providers.

The effect size for students in the provider’s program is in the top one-third of all the effect sizes demonstrated by providers meeting standards for student achievement.

2. Communication There is insufficient information available to determine communication outcomes.

Provider has not communicated with the principals, teachers, and parents of students served.

There has been limited communication throughout the year between the provider and at least two of the following: principals, teachers, and parents.

There has been some regular communication throughout the year between the provider and the principals, teachers, and parents of students served.

There is an ongoing and sustained system of communication between the provider and the school-level educators as well as parents of students served.

3. Instructional Plans

There is insufficient information available to determine instructional plans of the provider.

Provider does not plan instruction explicitly geared to student needs or to reinforce their regular academic program.

Provider is in the planning stages of gearing instruction to student needs, and reinforcing the regular academic program.

Provider has made some attempt with the majority of students to plan instruction explicitly geared to student needs and to reinforce the regular academic program.

Provider instructional plans are explicitly geared to the needs of most or all students and reinforce the regular academic program.

4. Local and State Standards

There is insufficient information available to determine alignment with local and state standards.

None of the instructional plans used by the provider are aligned with local and state academic standards for students.

Provider is in the process of aligning instructional plans with local and state academic standards for students.

Some of the instructional plans used by the provider are presently aligned with local and state academic standards for students.

Most or all of the instructional plans are presently aligned with local and state academic standards for students.

5. Special Ed/ELL students

There is insufficient information available to determine special ed/ELL student outcomes.

Provider does not offer accommodations for addressing the needs of special ed or ELL students.

Provider has made limited accommodations for addressing the needs of special ed and ELL students.

Provider has made some accommodations for addressing the needs of special ed and ELL students.

Provider offers appropriate services, if needed, to special education and ELL students.

6. Provider Overall

There is insufficient information available to determine provider overall outcomes.

There is overall dissatisfaction with the provider at the district and school levels.

There is more dissatisfaction than satisfaction with the provider at the district and school levels.

There are mixed but mostly positive reactions about the provider at the school and district levels.

There is overall satisfaction with the provider at the district and school levels.

NO YES

YES

Positive Indeterminable

YES NO

Negative

Positive

Negative

NO YES

NO

YES

NO

YES

Negative

NO

YES

Positive

NO

COMPLIANCE? Serious? Removal

Achievement?

Minor Compliance Violations?

Implementation?

Probation I Probation II

Implementation?

Probation II Last Year?

Removal

Full StandingSatisfactory Standing

Implementation Improved?

Probation II

Achievement Improved?

Probation I

Removal

Probation I Last Year?

Decision Tree for SES ProvidersDecision Tree for SES Providers

Probation I

CONCLUSIONCONCLUSION

• Each state should begin its SES evaluation planning process by identifying

a) the specific questions that its SES evaluation needs to answer, and

b) the resources that can be allocated reasonably to support further evaluation planning, data collection, analysis, reporting, and dissemination.

CONCLUSIONCONCLUSION

• Work through the hierarchy of evaluation designs presented here and select the design that allows the highest level of rigor.

• States may wish to engage third-party evaluation experts in helping to plan and conduct these evaluations.