Children FirstFocus on Accountability
Accountability Initiative
July 19, 2006
CONFIDENTIAL
2
• Accountability Initiative
–Vision and Overview
–Achievement Reporting and Innovation
System (ARIS)
• Accountability Initiative Implementation Timeline
CONTENTS
3
THE PROBLEM
Grade 4 ELA - 2001 to 2004 School Average
600.0
620.0
640.0
660.0
680.0
700.0
720.0
0 10 20 30 40 50 60 70 80 90
Mix of Free Lunch / Black or Hispanic /ELL/ Spec. Ed.
Avg
. Tes
t S
core
Number of Schools = 52
Number of Schools = 82
Number of Schools = 113
Number of Schools = 189
Number of Schools = 205
4
THE SOLUTION: A CYCLE OF CONTINUOUS IMPROVEMENT
Leadership / clear goals
Innovation / evaluation
Adjustment
SCHOOLSEnablers / Tools
• Continuous learning and support (e.g. Empowerment Schools Intensive)
• Periodic assessments
• ARIS
Accountabilities / Measures
• Progress Reports
• Quality Reviews
5
INNOVATIONS IN ACCOUNTABILITY
New accountability measurementsOld accountability measurements
• Snapshot of performance
• Movement only across single proficiency drivers- ignores outcomes on high/ low ends
• Only grades schools with challenging populations: SINI, SURR
• Incentives to focus on students least in need
• Progress: what the school adds
• All amounts of improvement in student achievement
• Universal application: All schools are graded: A, B, C, D, F
• Extra credit for gains by students most in need
• School improvement plan and target setting• Leadership change• School restructuring• School closure
4 Year Cycle of Consequences
Scatter Plot of Peer Group Index Scores with Overall School Index Scores with and without Extra Credit
0.000
0.200
0.400
0.600
0.800
1.000
1.200
1.400
0 10 20 30 40 50 60 70 80 90
Peer Group Index Score
Sc
ho
ol I
nd
ex
Sc
ore
SCORETOTNOEC
SCORETOTEC1
Linear (SCORETOTNOEC)
Linear (SCORETOTEC1)
ACCOUNTABILITIES / MEASURES: SCATTER CHART
7
ACCOUNTABILITIES / MEASURES: QUALITY REVIEWS
Gather data
Plan andset goals
Aligninstruction
Build and align capacity
Monitor and revise
Continuous data-driven improvement
• Understand what each student knows and is able to do monitor, and
• Measure the student’s progress over time.
• Set each student’s next learning step.
• Collaborate with student / parents to set goals for improving teaching practice and accelerating student learning
• Align school instructional activity and resources around focused plans for accelerating individual student learning
• Align instructional skills and capacity development around established goals for accelerating individual student learning
• Update documentation / best practices on effectiveness of its improvement plans and flexibility of change processes
8
Identify high leverage points for intervention
to improve student
performance
Align instructional practice and resources,
access external resources to
support change
Assess progress and make mid-
course corrections
Start by analyzing school’s performance data to ID areas that
need improvement
Create urgency
with publicized
results
Select a point of focusAnalyze student-
level data, instructional
practices, and adult learning
needs
ENABLERS / TOOLS: CREATE A CYCLE OF CONTINUOUS LEARNING
Top-down qualitative analysis from Quality Review
Bottom-up quantitative view from Assessments
Publish Progress Reports
Deeplyinformed
interventions
9
ENABLERS / TOOLS: A SET OF DIAGNOSTIC TOOLS-PERIODIC PROGRESS MEASURES
10
ENABLERS / TOOLS: DIAGNOSTIC ASSESSMENT METHODOLOGIES
October 2006 Jessica Holloway
Edward Smith
Raul Fuentes
Summarizing
Main Idea
Fact vs. Opinion
Predicting
Context Clues
Paraphrasing
December 2006 Jessica Holloway
Edward Smith
Raul Fuentes
Summarizing
Main Idea
Fact vs. Opinion
Predicting
Context Clues
Paraphrasing
11
ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION
Periodic assessments
DOE and vendor provided assessments
Default system based on citywide ELA and Math curricula • Math #1• Math #2
• Math #7
• ELA #1
• ELA #5
School customized assessments
School created assessments
Customized version provided to align with different curriculum, sequence or desired outcome (more predictive)
Standards-based alternative assessment system that..• Is developed by
school, network or region in collaboration with a partner
• Ensures equal rigor, measures progress and allows for cross-group comparison
At every grade level, a set of diagnostic assessment tools to help inform interventions and instruction to ensure an individualized instructional plan for every student
12
ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION
DOE and vendor provided assessments
Default system based on citywide ELA and Math curricula • Math #1• Math #2
• Math #7
• ELA #1
• ELA #5
School customized assessments
School created assessments
Customized version provided to align with different curriculum, sequence or desired outcome (more predictive)
Standards-based alternative assessment system that..• Is developed by
school, network or region in collaboration with a partner
• Ensures equal rigor, measures progress and allows for cross-group comparison
ARIS
13
• Accountability Initiative
–Vision and Overview –Achievement Reporting and
Innovation System (ARIS)
• Accountability Initiative Implementation Timeline
CONTENTS
14
OBJECTIVES AND SCOPE OF ARIS
Objectives
Ensure access to the information and tools necessary to enable
• Longitudinal and detailed analysis and reporting of achievement and performance data
• Best-in-class assessment design and implementation processes to extract data from multiple sources, including designing own reports and the capability to slice and dice the data
• Tools to improve learning of students with a range of identified needs;
• Sharing best practices and collaborating across schools to encourage cultures of continuous school improvement and professional learning
Scope of the ARIS
• Provide principals, teachers and parents with online information on student achievement, including periodic assessments
• Support development of longitudinal and detailed analysis
• Develop an integrated portal, including a dashboard to analyze key environment factors and achievement metrics
• Enable real-time prediction of school performance against year-end targets
• Generate standard reports for specific end-users
• Develop scorecards to enable drill-downs by student, assessment, strand and sub-strand analysis
• Provide knowledge management tools that capture teaching and assessment content generated at school level with an ability to approve, publish and share with the networks or the broader district. As well as capturing new quality review data
15
WHAT ARIS IS AND WHAT IT IS NOT
ARIS is… ARIS is not…
• A solution that…– Analyzes and reports– Assesses– Innovates (assessments, responsive
interventions)– Collaborates and personalizes– Shares knowledge (sharing) and
manages content
• A solution that may not be available off-the-shelf
• A solution that may require a partnership or consortium of Bidders to offer, not by a single vendor or a service provider
• A collaborative effort between the Vendor and NYCDOE to innovate, develop and execute
• A system that will replace all of the current applications now in operation at the DOE (e.g. ATS, HSST). Users will still need to update and print reports very much in the same way they do today but the access to all of the key systems will now all be well integrated through a single interface
• An Enterprise Data Warehouse (EDW) that will have all the current available data in DOE (assessment, Financial, HR, etc.). ARIS may be the foundation for an EDW in the long run, but initially the data warehouse will only contain student achievement data and the necessary information to analyze by several key criteria (e.g., age, school, grade, background, etc.)
• An opportunity to replace desktops and/or network infrastructure
16
OVERVIEW OF REQUIRED FUNCTIONALITY
Generate reports and analyze achievement/performance data
Use best-in class assessments to improve student achievement
Innovate the professional learning / share best practices across all schools
Analysis + Reporting• Canned and customized reports• Drill-down, roll-up, filtering• Ad-hoc queries• User-generated databases• Adding new data to central
repository• Data viewing in 3rd party tools
(e.g., Excel)
Advanced analysis (for selected users)• Advanced statistics• Pattern detection for target setting
Knowledge and plan management
• Library of templates for different document types (e.g., lesson plans, curriculums)
• Document creation and upload to repository
• Document tagging
• Collaborative document creation
• Publishing approval
• Document linking to reporting (surveys)
Knowledge and plan management (continued)• Searching documents in the repositories• Document viewing and download• Pattern analysis across text documents
Collaboration• Posting announcements; setting alerts• Online discussions; online messaging• Custom workspaces, integration w/email
Periodic assessments• For school- or center-created
– Library of templates/idea bank for school-created tests (e.g., rubric-based, standardized multiple choice)
– Test creation and upload• For 3rd party-created
– Assessment content upload into online testing, reporting and analysis
• For all– Online testing and scoring– Score data upload into
reporting and analysis
Personalization• Pre-defined home page based on user role• Personalization of home page (HP) by user• Adding new tools to HP from a tool catalog
Auditing (ie.,usage instances, frequency, modification history tracking)
1 2 3
17
KEY TECHNICAL AND SYSTEM REQUIREMENTS
Implications for ARIS RFP / requirementsContext and challenges• Data issues
– Significant archive size– Uncertain data quality– Critical data element gaps
• Legacy applications– Significant number, fragmented
functionality– Diverse but emerging
preferences for platforms– Goal of having a single
Enterprise Data Warehouse
• Infrastructure– Uneven / unclear quality of...
• Desktop standards / availability
• Network access– Stringent performance
requirements– Funding constraints
• Data issuesVendors, based on their experience, to work with DOE on developing a processes and tools for…– Data cleaning and migration– Filling critical data element gaps– Improving data entry quality
• Legacy applicationsBidders to..– Develop new data sources for new
accountability functions (e.g., QR)– Offer a scalable and evolvable data model– Comply with strongly preferred, but not
mandatory standards
• Infrastructure– Bidders have the option to host externally, or
suggest hardware for DOE-hosted solutions– Optimized for mobile/ existing infrastructure– Funding for infrastructure outside of ARIS
contract scope
18
DOE ARIS PREPAREDNESS TASK FORCE FOR KEY TECHNICAL AND SYSTEM REQUIREMENTS
Context
Objectives
• Data Integrity: Create solutions and define processes to ensure and maintain confidence in the quality of data and validity of ARIS outputs
• Infrastructure: Define the infrastructural improvements for maximizing usage and effectiveness of ARIS
• Data Source Inventory: Identify alternate types of approaches/ applications schools use to host data
• Communications: Craft messaging tailored to user experience
NYCDOE has set up a Task Force to set ARIS for success by preparing internal DOE systems and processes to complement vendor solution
The task force will…
• Identify level of cleanliness of existing data1
• Identify key causes for data inaccuracy or incompleteness
• Design and pilot solutions to clean existing source systems feeding ARIS2
• Recommend processes for maintaining data quality going forward
• Concentrate on transactional systems (ATS, HSST, etc.)
19
OVERVIEW OF TRAINING APPROACH
1Includes select admin staff, select teachers2Includes 25 Central Staff/Accountability Office; 10 power users drawn from schools3Includes 150 network/region; 4200 principals and other school achievement leaders4Primarily teachers
• Central/DAA (50)• Network / Regions (280)• Principals (1,400)• Other school achievement leaders1
(5,600)• Teachers (87,000)• Support staff / DIIT (?)
• Parents (~100s K)
• Initial training (role-based, project- and site-based)– One-on-one (for power and advanced)– Train-the-trainer (for standard) - school
achievement leaders train other teachers
• Ongoing need-based training– Phone support service– Online simulation exercises
• Initial training– Train-the-trainer – select
volunteers to train others
• Ongoing need-based training– Phone support service– Online simulation exercises
Phase 1 (Sep. 2007)
Phase 2 (Sep. 2008)
Usersegments
Trainingapproach
Role-based view
Skill-based view
( _ ) Segment size
• Power (352)• Advanced (~4,3503)• Standard (~90,0004)
• Standard (~ 100s K)
RFP requires the vendors to submit plans for measuring training effectiveness
20
• Accountability Initiative
–Vision and Overview
–Achievement Reporting and Innovation
System (ARIS)
• Accountability Initiative Implementation Timeline
CONTENTS
21
ACCOUNTABILITY INITIATIVE IMPLEMENTATION TIMELINE
2006-07 2007-08
Empowerment Schools
Other Schools All Schools
Progress Report (Previous Year) Yes No Yes
Progress Report (Current Year) Yes Yes Yes
School Targets Yes No Yes
School Grade Yes Yes Yes
Quality Review Yes Yes Yes
Rewards/ Consequences Yes No Yes
Data Management System Spring 2007 No Yes
Periodic Assessments Yes No Yes
Support for Continuous Learning Yes No Yes