1
School Evaluation ServicesPennsylvania Update
Education Policy and Leadership CenterWestern Pennsylvania Breakfast Series
Jonathan Jacobson, Director
May 23, 2002
2
This Morning’s Presentation
SES Background
Status Update
User Feedback
Enhancements
Pennsylvania Statewide Insights
What’s Next
3
SES Background
4
What is School Evaluation Services?
A powerful diagnostic and decision-making tool for administrators, educators, policymakers, taxpayers, and parents
Annual written report on the strengths, challenges, and risks for each of the state’s school districts
Public website containing these reports and detailed supporting data trends and comparative benchmarks
Annual comprehensive report on the state’s collective circumstances, yearly progress, and policy implications
5
SES converts data to information through its unique analytical framework
Transparency, consistency, accessibility
Objective interpretation
“Return on Resources” – simultaneous examination of spending, results, and context
Multi-year trends measure progress
Each district compared to five benchmarks: state, I.U., county, and demographic peer averages, as well as district itself over time
Website’s interactive display of data allows users to customize comparison groups
6
Five comparison groups to choose from
SES website data menu example – Student Results
Six data subsections to choose from
7
How does SES help Pennsylvania’s education stakeholders?
SES goes beyond reporting test scores – it synthesizes achievement indicators with financial data to measure return and inform resource allocation decisions
There may not be sufficient resources or staff at either the state or the local level to perform data benchmarking and analysis – SES fills this void
SES helps decision-makers at all levels avoid being “data rich but information poor”
SES helps end policymakers’ and stakeholders’ reliance on anecdotal generalizations
8
Pennsylvania Advisory Group
Leslye Abrutyn, Superintendent, Penn-Delco SD
Carolyn Dumaresq, Executive Director, PSEA
Tom Gentzel, Executive Director, PSBA
Jay Himes, Executive Director, PASBO
Sheri Rowe, Office of Educational Initiatives, PDE
Stinson Stroup, Executive Director, PASA
Larry Wittig, State Board of Education, State Workforce Investment Board
Jan Zastawniak, President, PenSPRA
9
SES Status Update
10
Milestones accomplished
SES launched for Michigan – May 2001
SES launched for Pennsylvania – October 2001
Interim data update for Michigan – October 2001
Interim data update for Pennsylvania – November 2001
“Statewide Insights” released for Michigan – December 2001
Round 2 released for Michigan – January 2002
“Statewide Insights” released for Pennsylvania – May 2002
11
Milestones to come
Round 2 (2000 data) release for Pennsylvania – May 2002
Round 3 (2001 data) release for Pennsylvania – 4Q 2002
12
SES website traffic has exceeded expectations
May
2001
Oct
2001
Jan
2002
March
2002
Cumulative
(through 05/04/2002)
Visitor Sessions
24,724 88,740 50,695 71,514 522,307
Page Views 1,620,819 3,890,785 808,288 875,897 13,510,955
13
Parents are the most frequent visitors to the SES website
Parents 39%
Teachers 18%
School Administrators 11%
Taxpayers 10%
School Board Members 3%
Government Officials 2%
Other 17%
Pennsylvania data through 04/30/2002
14
Usage among school personnel is significant
Michigan Pennsylvania
School Districts 554 501
Administrator Log-Ins 18,832 10,746
School Teacher Log-Ins 18,035 18,258
Cumulative data through 04/30/2002
15
While activity peaks predictably during launch periods, visits during non-launch periods are relatively constant
Average Daily “Off-Peak” Logins: 233
PA population is 12 million; MI population is 9.8 million
Daily Site Activity for MI
0
2000
4000
6000
6/2
1
7/2
1
8/2
1
9/2
1
10
/21
11
/21
12
/21
1/2
1
2/2
1
3/2
1
Lo
gin
sDaily Site Activity for PA
0
1000
2000
3000
4000
11
/6
11
/20
12
/4
12
/18
1/1
1/1
5
1/2
9
2/1
2
2/2
6
3/1
2
3/2
6
4/9
Lo
gin
s
Average Daily “Off-Peak” Logins: 331
16
Pennsylvania activity has nearly matched Michigan’s in much less time
Michigan Pennsylvania
Launch Date May 25, 2001
(11 months)
October 3, 2001
(7 months)
Share of Total Log-Ins 52% 48%
Cumulative data through 04/30/2002
17
SES User Feedback
18
Standard & Poor’s seeks, values, and uses customer feedback
E-mail links, state-designated e-mail boxes, and a toll-free telephone number enable easy and frequent communication
All inquiries, comments, and suggestions are logged, identified by type, distributed, and tracked
Responses are timely, thorough, and courteous
More formal research is underway
User feedback is a primary source of SES enhancements
19
Positive commentary about SES repeatedly cited by Pennsylvania school leaders Helpful in strategic planning, budgeting, benchmarking
Comparative data are useful—particularly peers
Effective accountability tool
Useful in communicating to school board, local stakeholders
Information has prompted discussions in community
Helpful for people considering moving into community
Coming from an objective source makes SES more valuable
20
Other positive comments about SES from school administrators
Helped achieve programmatic improvements
Used SES data to renegotiate benefits package—cost savings were converted to teacher salary increases
Potential to use SES data to identify best practices
“Return on resources” concept is innovative and valuable
Explanations, disclaimers, caveats are helpful
Service is needed—grateful that it’s being provided
21
SES Enhancements
22
Many improvements have been implemented since the initial launch of the SES website last year Major enhancements
Intermediate Unit comparison group
Interim statewide data revisions
Display of basic school-level data
“Beating the odds” schools in S&P Observations
Statewide graphics
Navigation menu page
Enhanced district trend data display
23
24
25
Many improvements have been implemented since the initial launch of the SES website last year Other enhancements
Expansion of S&P Observations to include methodology, glossary, and list of peers
“State homepage” – administrator’s toolkit, state education department links, state facts
Navigation path icons – clearer indication of actions and selections
“Contact Us” e-mail links
Homepage content and reconfiguration
Log In page simplification
26
Additional enhancements are in the works
Data display
School-level information
Data sorting tools
Format and organization of S&P Observations
User interface
School/district searching tools
Website tutorial/presentation
Data mapping explanation/crosswalk
27
28
Pennsylvania Statewide Insights
29
PSSA indicators show only slight improvement
Highest district PSSA Composite Score was 1457, while the lowest was 1087
Highest district percent of scores above median was 80.5%, while the lowest was 6.3%
Student Results
30
Across the state, white students’ PSSA scores were significantly higher than those of black, Hispanic, Native American, and multi-ethnic students
Achievement Gaps
31
Return on Resources
PSSA results have increased slightly, but so has spending (net of inflation) – therefore, the state’s return on resources has not improved
Performance Cost Index (PCI) data suggest that reasonable spending increases are not likely to make enough difference by themselves, if the goal is to enable all students to meet state standards
A better rate of return on resources is needed if additional spending is to make a substantial impact
Strategies to improve achievement may need to include, but go well beyond, monetary inputs
A comparison of data for specific school districts leads to the same conclusion…
32
There is no significant correlation between spending and results – at any given spending level, there is a wide range of student achievement Pennsylvania School Districts
1999 Per-Pupil Spending and PSSA Scores
1000
1050
1100
1150
1200
1250
1300
1350
1400
1450
1500
4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000
Operating Expenditures Per Student ($)
PS
SA
Me
an
Co
mp
osi
te S
co
re
Philadelphia: $7,194 PSSA Score = 1134
State Weighted Average Score = 1303
State Weighted Average Spending = $7,367 Per Pupil
Pittsburgh: $9,846 PSSA Score = 1224
Harrisburg: $9,040 PSSA Score = 1106
Return on Resources (continued)
33
Spending “decreases” for Philadelphia, Harrisburg, and Pittsburgh when adjusted for the higher cost of serving students with special needs, and for geographic differences in the cost of education
Pennsylvania School Districts 1999 Adjusted Spending (NCOL & Student Circumstances) and PSSA Scores
1000
1050
1100
1150
1200
1250
1300
1350
1400
1450
1500
4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000 Operating Expenditures Per Student Adjusted for NCOL and Student Circumstances ($)
PS
SA
Mea
n C
om
po
site
Sco
re
Philadelphia: $4,902 PSSA Score = 1134
State Weighted Average Expenditure = $6,828 Per Pupil
State Weighted Average Score = 1303
Pittsburgh: $8,229 PSSA Score = 1224
Harrisburg: $7,323 PSSA Score = 1106
Return on Resources (continued)
34
A number of districts with above-average enrollments of economically disadvantaged students demonstrated a better-than-average return on resources, as measured by lower (more favorable) PCIs.
Pennsylvania School Districts
Performance Cost Index (PSSA) and Low-Income Enrollment
0
10
20
30
40
50
60
70
80
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Enrollment of Low-Income Students (%)
Pe
rfo
rma
nc
e C
os
t In
de
x f
or
PS
SA
Me
an
Co
mp
os
ite
S
co
res
($
)
Weighted Avg. = $22.51
Weighted Avg. = 31.7%
24 Districts(14 districts had PSSA results that were at or above the average)
153 Districts
151 Districts
172 Districts 4 Districts are not shown in this quadrant because they are outliers
Philadelphia
Pittsburgh
Harrisburg
Return on Resources (continued)
35
PSSA scores and socioeconomic status are correlated; however, many schools “beat the odds”
Pennsylvania Elementary SchoolsPSSA Scores and Low-Income Enrollment
800
850
900
950
1000
1050
1100
1150
1200
1250
1300
1350
1400
1450
1500
1550
1600
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Enrollment of Low-Income Students (%)
PS
SA
Me
an
Co
mp
osi
te S
core
Weighted Avg. = 1314
Weighted Avg. = 35.4%
163 Elementary Schools Beat the Odds
481 Schools
707 Schools
153 Schools
Best Practices
36
Many school districts as a whole also “beat the odds”
Pennsylvania School DistrictsPSSA Scores and Low-Income Enrollment
900
950
1000
1050
1100
1150
1200
1250
1300
1350
1400
1450
1500
1550
1600
0 10 20 30 40 50 60 70 80 90 100
Enrollment of Low-Income Students (%)
PS
SA
Me
an
Co
mp
os
ite
Sc
ore
Weighted Avg. = 1303
Weighted Avg. = 31.7%
248 Districts
75 Districts
46 Districts Beat the Odds
10 districts remained in this quadrant for 3 consecutive years
131 Districts
Best Practices (continued)
37
What do the data tell us?
No “silver bullets”
Without improved return, goals of improving student achievement overall and of closing achievement gaps are probably unaffordable
How money is spent is as important as how much money is spent
Need to align policy making with data analysis
Districts and schools that beat the odds – particularly those that do so consistently over time – are potential sources of best practice
38
Next Steps
39
Coming with next release…
Charter school information
School-level data
Peer methodology to include state aid ratio
More detailed trend data
Improved location search features
Website self-tutorial
Data mapping crosswalk
40
Ongoing efforts…
Tutorial and training programs
More detailed school-level information
Data sorting tools
Market research and user outreach
41
Where to find us, how to reach us
Website URL: www.ses.standardandpoors.com
E-mail address: [email protected]