1 Phase III A. Summary of Phase III, Year 2 In 2014, the Rhode Island Department of Education (RIDE) established the State-identified Measurable Result (SiMR) to improve mathematics achievement (on the statewide assessment) for Hispanic and Black students with specific learning disabilities in grades 3-5 by 4% by fiscal year 2018-2019. To address the SiMR, Rhode Island awarded the American Institutes for Research (AIR) a contract to support with the State Systemic Improvement Plan (SSIP) implementation and evaluation activities (see previous year’s submission for more detailed information regarding the contract award). During this Phase III, Year 2 reporting cycle, AIR engaged in technical assistance activities in 13 schools (see Table 1) from eight districts in Rhode Island. Table 1. Participating Sites by Cohort Cohorts Elementary Sites Middle School Sites* TOTAL Cohort 1 (participation start 2016-2017 school year) 4 2 6 Cohort 2 (participation start 2017-2018 school year) 5 2 7 TOTAL 9 4 13 *Middle school sites in RI often serve students in Grade 5, and many of the students identified in 2014 for the SiMR are now in middle school. The following report will detail implementation and evaluation activities involved in the Intensive Math Intervention Project (i.e., Math Project) since the last reporting period and communicate key findings resulting from the ongoing evaluation of the project. It is important to note that formative student assessment data will not be included in this year’s submission since these data are not available until the end of the 2017-2018 academic year. A detailed description of our approach to collecting these data is discussed in the section titled: Outcomes regarding progress toward short- term and long-term objectives that are necessary steps toward achieving the SiMR. Theory of action or logic model for the SSIP, including the SiMR For this reporting period, RIDE and AIR collaborated with RIMTSS providers (i.e., State Personnel Development Grantee) and the RI Parent Information Network during a cross-initiative alignment meeting to refine the theory of action and logic model to better represent SSIP implementation and help guide progress toward the SiMR. At the onset of the Math Project, both the theory of action and logic model included language suggesting that facilitating a framework of multi-tiered system of
Transcript
1
Phase III
A. Summary of Phase III, Year 2
In 2014, the Rhode Island Department of Education (RIDE) established the State-identified
Measurable Result (SiMR) to improve mathematics achievement (on the statewide assessment) for
Hispanic and Black students with specific learning disabilities in grades 3-5 by 4% by fiscal year
2018-2019. To address the SiMR, Rhode Island awarded the American Institutes for Research (AIR)
a contract to support with the State Systemic Improvement Plan (SSIP) implementation and
evaluation activities (see previous year’s submission for more detailed information regarding the
contract award). During this Phase III, Year 2 reporting cycle, AIR engaged in technical assistance
activities in 13 schools (see Table 1) from eight districts in Rhode Island.
Table 1. Participating Sites by Cohort
Cohorts Elementary Sites Middle School
Sites*
TOTAL
Cohort 1 (participation start
2016-2017 school year)
4 2 6
Cohort 2 (participation start
2017-2018 school year)
5 2 7
TOTAL 9 4 13
*Middle school sites in RI often serve students in Grade 5, and many of the students identified in
2014 for the SiMR are now in middle school.
The following report will detail implementation and evaluation activities involved in the Intensive
Math Intervention Project (i.e., Math Project) since the last reporting period and communicate key
findings resulting from the ongoing evaluation of the project. It is important to note that formative
student assessment data will not be included in this year’s submission since these data are not
available until the end of the 2017-2018 academic year. A detailed description of our approach to
collecting these data is discussed in the section titled: Outcomes regarding progress toward short-
term and long-term objectives that are necessary steps toward achieving the SiMR.
Theory of action or logic model for the SSIP, including the SiMR
For this reporting period, RIDE and AIR collaborated with RIMTSS providers (i.e., State Personnel
Development Grantee) and the RI Parent Information Network during a cross-initiative alignment
meeting to refine the theory of action and logic model to better represent SSIP implementation and
help guide progress toward the SiMR. At the onset of the Math Project, both the theory of action and
logic model included language suggesting that facilitating a framework of multi-tiered system of
2
support (MTSS) was the method to achieve intensive and individualized math instruction. However,
upon reflection, a more accurate representation of school level needs is the facilitation of data-based
individualization (DBI). DBI provides a systematic, iterative process for using data to intensify and
individualize supports for students who are non-responsive to evidence-based math instruction
attempted at earlier tiers of an MTSS structure (National Center on Intensive Intervention (NCII),
2013). In this sense, DBI is more reflective of what constitutes evidence-based practice than the
broader system of MTSS. However, given that MTSS provides an overarching framework for
successful DBI implementation, the Math Project contextualizes DBI within MTSS, with a focus on
data-based decision making to inform math instructional practice.
As a result, the refined theory of action (see figure below) guiding the Math Project contends that if
supports are provided for data-based decision making to inform intensive, individualized instruction
in mathematics throughout the state, there will be a change in adult behavior at the local level, which
will help achieve positive outcomes in mathematics proficiency for Black and Hispanic students with
learning disabilities in Grades 3–5. The logic model was also refined to reflect a focus on DBI,
resulting in changes to the language used in the short-term and intermediate outcomes (i.e., DBI or
intensive intervention instead of MTSS). It is hypothesized that the long-term outcome related to
improved fidelity of school-wide MTSS will still be attainable, even with the narrowed focus on
DBI/intensive intervention, based on previous experiences implementing DBI in RI districts
participating in NCII technical assistance. The refined logic model continues to outline the activities
and outputs that are expected to help RIDE achieve intended outcomes and the SiMR.
Refined Rhode Island SSIP Logic Model State-identified Measurable Result: Improve the mathematics achievement for Hispanic and Black students with specific
learning disabilities in grades 3-5 by 4% by FFY2018 (2018-2019) on the statewide assessment
4
The coherent improvement strategies or principle activities employed during the year, including infrastructure improvement strategies
During this reporting period, RIDE has worked to align other state-level initiatives by identifying
common goals. Specifically, infrastructural initiatives have been leveraged to ensure the Math
Project staff are building on the success of various implementation efforts, including the State
Personnel Development Grant focused on MTSS, the Collaboration for Effective Educator
Development, Accountability, and Reform (CEEDAR) Center, and the National Center on Intensive
Intervention (NCII). The Math Project has made connections across the initiatives to: (a) ensure
consistency in how DBI—as a part of an MTSS model—is communicated, (b) revise implementation
plans based on lessons learned, (c) connect with key personnel from existing RIDE initiatives on a
regular basis, and (d) share ongoing updates with RIDE to facilitate a continuous feedback loop. The
SSIP math focus has also fostered increased collaboration between staff at RIDE’s Office for
Student, Community, and Academic Supports and the Office of Instruction, Assessment, and
Curriculum, on not only the Math Project for the SSIP, but also on general education math initiatives
and statewide curriculum work. For additional information related to status of collaboration, see the
section in this report titled: Description of baseline data for key measures.
Regarding engaging families related to the SSIP implementation and evaluation, RIDE has regular
meetings with the Rhode Island Special Education Advisory Committee (RISEAC) to facilitate their
input and feedback. Staff from the Rhode Island Parent Information Network (RIPIN) are members
of the RISEAC and serve as members of the SSIP Core Team and are integral to informing decisions
about implementation strategies. In addition, RIPIN has a subcontract award on the Math Project to
help achieve the outcomes related to parent and family awareness and understanding of DBI.
The specific evidence-based practices that have been implemented to date
To date, there are two cohorts of schools engaged in the Math Project. Cohort 1 includes six schools
that began participating in the project during the 2016-2017 academic year and have continued to
receive project support through the 2017-2018 academic year. Cohort 2 includes seven schools that
joined the project during the 2017-2018 academic year. Current cohorts will continue to participate
with the Math Project through 2021, focusing on different aspects of implementation (e.g., learning
and implementing DBI, and then scaling and sustaining efforts) in subsequent project years. Before
implementation, the sites identified for the Math Project are engaged in a needs-assessment process
that drives the development of an action plan for the site. During the needs-assessment phase, key
personnel from participating sites are interviewed by Math Project staff using a semi-structured
interview protocol that asks sites to identify their current practices related to (a) tiered instruction in
mathematics (core, targeted, and intensive), (b) their data-based decision making processes (progress
monitoring tools, decision rules, and diagnostic assessments), (c) their approach to parent and family
engagement, and (d) their supports for culturally and linguistically diverse students and students with
disabilities. All schools across cohorts have completed their needs-assessment interviews.
Areas of need revealed through this process include inconsistent procedures for teaming structures in
math to support data-based decision making, a lack of diagnostic tools and processes for struggling
5
learners, gaps in current instructional delivery processes, as well as an overall recognition of a need
to improve the implementation of evidence-based practices (EBPs) in mathematics across the tiers.
To help address the problems, Math Project staff developed site-specific action plans that incorporate
feedback from school personnel. Action plans prioritize two to three goals for the academic year
related not only to increasing knowledge and implementation of common core aligned EBPs in
mathematics across the tiers (see Table 2), but also the structural changes (i.e., teaming processes)
required to achieve results. The action plans also outline the training and coaching activities in which
sites will participate. These goals align to the short-term and intermediate outcomes in the Theory of
Action, as well as to the Logic Model. A summary of goals across sites can be found in Appendix A.
Table 2. Example Evidence-Based Practices across MTSS Tiers*
Examples of EBPs in Mathematics Relevance
at Tier 1
Relevance
at Tier 2
Relevance
at Tier 3
Concrete-Representational-Abstract
(CRA) X X X
Using Manipulatives in Base 10 X X X
Visual Schematic Diagramming (e.g.,
Frayer Model, place value thinking
squares)
X X X
Peer Assisted Learning Strategies
(PALS) in Math X X
Corrective Math X X
Data-based individualization process
(includes evidence-based intensification
strategies)
X
* EBPs may be added to this list as sites identify additional skill deficit areas that require
instruction/intervention
Training in Evidence-Based Practices
Site action plans included goals related to improving knowledge and implementation of EBPs in
math across the tiers. To support sites with meeting these goals, multiple training opportunities were
offered to sites. This submission includes information about trainings that took place between April
2017 and February 2018—to align with the reporting cycle—and to allow for adequate time to
analyze and report on training evaluation data. Additional trainings were held in March 2018, but
those will be reported on in next year’s submission. Two external consultants were identified to
deliver training. Dr. Nancy Butler Wolf, adjunct faculty at California State Polytech University,
Pomona (Pomona, CA) and supervisor of teacher education at the University of California,
Riverside, provided training in Tier 1, evidence-based mathematics instructional strategies that are
common core aligned. At Tier 2, Dr. Sarah Powell, Assistant Professor at the University of Texas,
Austin, provided training to site personnel in Peer-Assisted Learning Strategies (PALS), an evidence-
based, peer tutoring program used to supplement classroom instruction. Six educators from the Segue
Institute for Learning (a Cohort 1 site) participated in a Tier Differentiation training, facilitated by
project staff (Dr. Gena Nelson).
6
The content of Nancy Butler Wolf’s instructional strategies trainings offered to elementary and
middle school educators differed, though all trainings included information about how math skills
align with common core standards and progress across grade levels. Because substitute coverage for
training and professional development opportunities is a challenge in RI, the elementary instructional
strategies training was offered across multiple days to maximize site-level participation. This strategy
allowed sites to send more educators than they would be able to if the training was only offered one
day. The decision for a multi-day training structure also reflects the Math Project team’s
responsiveness to stakeholder feedback at the site-level. Because middle schools have a a different
structure from elementary schools (i.e., content specialist model, rather than grade level teachers
instructing across content areas), the middle school instructional strategy trainings were offered only
one day.
Training Participation
To support the alignment of training activities to the SiMR population (i.e., Black and Hispanic
students with learning disabilities in grades 3-5), Math Project staff encouraged sites to select
educators to participate in trainings from grades 2-5 at the elementary level, and grades 5-8 at the
middle school level. Many sites elected to focus training participation at one grade level, and based
their decision on screening data, suggesting a need for improvement in core instruction. General
education teachers were the primary audience for all trainings. However, many special educators
and/or interventionists working across grade levels participated in training activities to ensure
instructional alignment across MTSS tiers and to ensure short-term and intermediate project
outcomes are achieved.
Elementary School Trainings
Instructional
Strategies 1*
Instructional
Strategies 2*
PALS Math
Date of Training Spring 2017 Fall 2017 Fall 2017
# of Cohort 1
Participants
29 6 12
# of Cohort 2
Participants
N/A 19 NA
*Both Instructional Strategies trainings included the same content with a focus on number sense
and place value
Middle School Trainings
Instructional
Strategies 1*
Instructional
Strategies 2*
PALS Math Tier
Differentiation
Date of
Training
Spring 2017 Fall 2017 Fall 2017 Spring 2017
# of Cohort 1
Participants
17 0 10 6
# of Cohort 2
Participants
N/A 5 N/A N/A
*Both Instructional Strategies trainings included the same content with a focus on patterns and
algebraic thinking
7
Coaching to Build Readiness for Tier 3 (DBI) Implementation
Rather than recruiting and training external personnel to serve as coaches, Math Project staff provide
coaching supports to all participating sites. One site-level coach is a former math interventionist from
RI, who joined AIR as a full-time employee and currently works with 10 sites (five Cohort 1 sites
and five Cohort 2 sites). A second site-level coach, with expertise in MTSS and supporting English
language learners, works with two sites in the same district, one site from Cohort 1 and the other
from Cohort 2. The third coach with expertise in MTSS and DBI is the Project Director who works
with one Cohort 2 site. All Math Project staff meet internally to ensure coaching alignment across
sites, to discuss challenges and solutions, and to identify any additional training or coaching needs
across sites.
Cohort 1 Coaching Activities. During the 2016-2017 academic year, Cohort 1 sites received more
than 64 hours of ongoing coaching support from Math Project staff. Coaching support involved
providing resources on tier differentiation, supporting data meetings, and selecting progress
monitoring measures to support data-based decision making and readiness for DBI implementation.
During the 2017-2018 academic year, Cohort 1 sites continued their progress on goals established in
their action plans from the prior year. For example, Math Project staff conducted observations of
mathematics data team meetings across sites to make recommendations and consult on how to better
refine processes.
Cohort 2 Coaching Activities. After the completion of needs-assessment interviews for Cohort 2
schools (completed between August and October 2017), coaching activities began. Activities
primarily focused on establishing goals for the academic year (based on needs-assessment findings),
as well as helping school teams make connections back to training activities. Cohort 2 schools have
received more than 46 hours of ongoing coaching support from Math Project staff so far this school
year.
Brief overview of the year’s evaluation activities, measures, and outcomes
In this reporting period, RIDE staff, AIR, stakeholders (RIPIN and MTSS Providers), and the
external evaluation team revised the evaluation plan, logic model, and theory of action to better align
with achieving the SiMR. The primary goal of the revision process was to ensure that the appropriate
measures were being collected to assess each outcome, implement a timeline for collecting those
measures, and minimize the risk of potentially burdening school teams and families with evaluation
requests.
To achieve these goals, the Math Project evaluation team and RIDE deliberated whether each data
source could adequately assess the intended outcomes. The team also considered whether each
evaluation task was excessively time intensive for school teams and families. The revised evaluation
plan, which aligns to the Theory of Action, is described in detail in Section C.1.b of this report.
Another significant evaluation activity involved conducting training evaluations during this reporting
period. After each training, participants were given short surveys to assess their professional
development experience. Items on the survey invited participants to rate areas ranging from training
relevance to the likelihood of applying acquired strategies in their daily work. The training evaluation
includes a common set of questions related to (a) training relevance, (b) how well the training
8
improved understanding, (c) whether the training provided something the learner could apply, and (d)
questions related to the quality of the training (e.g., organization, clear and comprehensible
presentation, pace, active participation). In this sense, responses from across trainings, which focused
on different content (i.e., Tier 1 strategies, PALS), can be aggregated for a more comprehensive data
analysis/comparison.
Finally, the other two other major activities that were conducted during this period were the
evaluations of stakeholder engagement and SSIP collaboration across RIDE initiatives. The
evaluation team reviewed Leading by Convening: A blueprint for authentic engagement developed
by the IDEA Partnership and National Association of State Directors of Special Education. The
evaluation team then revised a survey to ensure that questions were contextualized to the project and
could easily assess RIDE’s level of collaboration and stakeholder engagement.
Stakeholder engagement was assessed to determine the degree to which stakeholders were
informed and involved in decision making regarding the project. Peripheral stakeholders—those who
broadly have an interest in/awareness of Rhode Island’s SSIP, but may not work closely with
implementation or evaluation activities—were given a short survey to assess engagement. Relevant
peripheral stakeholders include the state’s Special Education directors and leaders from the Rhode
Island Special Education Advisory Committee (RISEAC). Seventy-six percent of peripheral
stakeholders agreed that RIDE creates opportunities to engage and provide feedback on efforts in the
state related to the SSIP.
As noted above, collaboration across RIDE departments and initiatives was also evaluated with
an adaptation of the Leading by Convening survey. RIDE received survey responses from 18
personnel in the Educator Excellence and Certification Services office; the Instruction, Assessment
and Curriculum office; the College and Career Readiness office; and the Office of Student,
Community and Academic Supports (OSCAS). The majority of these leaders rated that the most
appropriate description of OSCAS’s level of partnership with other departments was Informing—
meaning that OSCAS shares or disseminates information with relevant stakeholders in the state who
care about the SSIP. Over time, RIDE would like to see increases in cross-departmental
collaborations related to SSIP implementation.
Discussion of evaluation data results can be found in section C.1.c. later in this report.
Highlights of changes to implementation and improvement strategies
The Math Project’s implementation plan was continually refined throughout this reporting period to
better align with both the revisions to the Theory of Action and Logic Model. One major highlight is
the development of a process to evaluate family awareness of DBI and intensive intervention,
including how families can support their child. Based on the needs assessment results across Cohorts
1 and 2, the methods of parent and family engagement used by school teams (e.g., sending a letter
notifying that a student is receiving intervention) was not leading to meaningful parent and family
engagement. Many sites indicated that they would like to learn strategies to better engage parents and
families. As a result, AIR worked with the Rhode Island Parent Information Network (RIPIN; SSIP
Core Team member and subcontractor on the project) to develop a plan to address this need.
AIR partnered with external stakeholder—RIPIN—to develop an interview protocol with the families
of students identified to begin intensive math intervention (i.e., DBI “case-study” students) in the
9
schools participating in the Math Project. After the interview, the family will be given a toolkit
developed by RIPIN, leveraging resources available through NCII. Approximately three months after
the initial interview, RIPIN will follow-up to gauge the families’ awareness of DBI, the schools’
approach to supporting their child, and how they can help at home. The responses from the family
interviews will be used to provide feedback to school teams, bolster technical assistance, and
systematize the state’s efforts to engage parents and families in intensive intervention.
Another change to implementation involves the development of a Professional Learning Community
(PLC) for district and building leadership, including interventionists or instructional coaches.
Improving LEA capacity to support, scale, and sustain improvement efforts is a long-term outcome
in the RIDE logic model and directly aligns to the Theory of Action (i.e., change systems and adult
behaviors). The development of a PLC is conceptualized as an additional improvement strategy to
make progress towards this outcome. The PLC will focus on the types of supports leaders need to
ensure that EBPs in math are implemented with fidelity, educators are supported to implement DBI,
as well as additional considerations leaders need to put in place to support tiered math intervention
(e.g., pacing guides, opportunities for peer feedback on instructional practice, scheduling time for
meetings/trainings/intervention). The PLC is in the initial phase of development, with recruitment
activities beginning this spring.
One other area of improvement is the needs assessment process. Alleviating time burden on school
teams remains a primary focus for the Math Project team. The existing needs assessment interview
process initially took approximately two hours to facilitate, with a subsequent meeting with school
teams to discuss findings and prioritize areas of focus. To expedite this process, the Math Project
team has partnered with NCII. NCII has established a small working group tasked with improving
the initial DBI survey sent to school teams, working to set the survey up through an online format.
The survey will incorporate many topics addressed during the former interview process by assessing
the degree to which DBI is implemented in the school. The survey will be coupled with a rubric to
score the survey responses. Finally, the in-person interview will be sharpened to focus on weak areas
of DBI implementation and plans for technical assistance (i.e., school action plans). During the next
reporting period, the needs assessments will follow the refined process and will be scored against the
new rubric.
In the previous submission, the approach of using “readiness” was described to differentiate the
support to sites. At the Math Project’s inception, readiness was measured at the district-level by
scoring a letter of intent that described district-level context (e.g., demographics, processes and
procedures for using data, existing math curriculum and interventions). However, after determining
district readiness for intensive math intervention, and then subsequently beginning the work at the
school-level, the Math Project team noted discrepancies. For example, in the letter of intent, districts
described their curriculum in place. But, when school-level work began, the project staff discovered
that while processes/curricula may have been set by the district, they were not being followed by
school teams, or school teams expressed a need for additional support with how to implement the
district’s processes/curricular sequences. As a result, rather than considering the “readiness” at the
district-level, as previously done, the project team has used the needs-assessment results to identify
the readiness of school-level sites, as opposed to the letters of interest from participating districts.
However, the involvement of the district in this project supports overall school-level implementation,
as the project is seen as more of a priority given the level of involvement. As such, the Math Project
elected to work with existing districts from Cohort 1 at the start of this school year to have them
identify additional school sites. The project also received letters of intent from two new districts
10
(urban ring and suburban), from which three additional sites were added. The approach to how
readiness informs project activities and outcomes—articulated in Section 4 and in the revised logic
model—is responsive to localized contexts to ensure ongoing commitment and buy-in from the
districts and schools participating in the Math Project.
Additionally, Math Project staff—in collaboration with RIDE—are revising the current Cohort
structure to better address overall site-level readiness. When the project was initially conceptualized,
the goal was to have sites begin implementation of the DBI process within the first year of their
participation. However, Math Project staff have discovered Cohorts 1 and 2 have foundational
deficits (e.g., evidence-based math instruction at Tiers 1 and 2, confidence in teaching math,
understanding of data/assessment purposes and use) that need to be addressed through training and
coaching activities prior to focusing on DBI implementation. As a result, a two-year implementation
structure will become the focus of the project for the next Cohort of sites—the first year will provide
support to sites while developing foundational math skills, and the second year will focus on DBI
implementation in math. The Math Project intends to bring on one additional Cohort of sites (number
to be determined). For the third Cohort, recruitment, needs assessment interviews, and action
planning will occur in2018-2019, with the two-year implementation cycle taking place between
2019-2020 and 2020-2021. While the timeline for implementation is shifting from what was initially
conceptualized, the improvement strategies will remain the same and will focus on training,
coaching, and ongoing technical assistance that address site-specific needs; as a result, there are no
anticipated changes to the evaluation plan.
B. Progress in Implementing the SSIP
Description of the state’s SSIP implementation progress
At the end of the last reporting period, the foundation for major SSIP activities including needs
assessments, action plan development, and technical assistance tracking had been established.
Presently, progress on the state’s SSIP implementation involves building on the foundation
established from the prior year by supporting additional sites (Cohort 2), and designing processes to
ensure that all intended project outcomes are achieved.
a. Description of extent to which the state has carried out its planned activities with fidelity—what has been accomplished, what milestones have been met, and whether the intended timeline has been followed
The following table captures the state’s SSIP implementation progress by the primary
implementation areas. Overall, the state carried out its planned activities for fall 2017 with fidelity.
The planned activities for Spring 2018 are well underway.
Key accomplishments and milestones include the following:
Refined theory of action, logic model, and evaluation plan
Held alignment meeting with RIPIN
Finalized Memorandum of Understanding and mini-grant process for new Cohort sites
Scheduled trainings for the academic year
Conducted site observations of school-team meetings
11
Overview of April 2017- March 2018 Implementation Progress
Implementation Area Planned Activities Status of Implementation
Refine theory of action, logic model, and evaluation plan
Complete
Communicate with districts around recruiting sites Cohort 2 cohorts
Complete
Project Planning & Coordination
General Activities necessary for the management of the SSIP
Finalize the MOU and mini-grant process with new Cohort sites
Complete
Collect universal screening data from sites
Anticipated Spring 2018
Collect statewide assessment data Complete
Training
Activities associated with delivering professional development for educators
Schedule trainings for the academic year
Complete
Conduct trainings, as scheduled In Progress
Develop observation tool to support fidelity of implementation
In Progress
Develop coaching guides In Progress
Coaching
Activities associated with technical assistance support
Book Study (site-specific)
In Progress
Conduct site observations and team meetings
Ongoing
Conduct on-site orientation and plan for implementation
Complete
Develop student level plan template (i.e., DBI case-study)
In Progress
Family Engagement
Activities associated with improving family engagement in intensive intervention
Collaborate with RIPIN to develop family engagement protocols
Under Review, to be finalized by Summer of 2018
Develop family engagement timeline for selected case study students
Not Started
April 6, 2017 Special Education Directors Meeting
Complete
September 23, 2017 RISEAC Meeting
Complete
12
Activities involved both peripheral and primary stakeholders
August 15, 2017 Alignment Meeting with Math Project, RI MTSS, RIPIN, and RIDE
Complete
September 28, 2017 Special Education Directors Meeting
Complete
November 16, 2017 Special Educator Directors Meeting
Complete
January 17, 2018 RIDE OSCAS staff meeting (Title I, Title III, 21st Century IDEA, Safe/healthy schools staff)
Complete
January 18, 2018 RISEAC Meeting Complete
Develop and administer stakeholder engagement surveys
Complete
Collaboration Between RIDE initiatives
Activities associated with RIDE collaboration
Develop and administer collaboration surveys
Complete
LEA Capacity to support diverse students in urban settings
Activities associated with increasing LEA capacity
Develop Professional Learning Community
In Progress
* Description of stakeholder engagement activities are further described in Sections: 2.a., 2.b., 3.a., and 3.b. later in this report
b. Intended outputs that have been accomplished as a result of the implementation activities
RIDE has made progress in achieving the intended outputs identified in the logic model. All sites
have developed implementation/action plans used to document progress toward goals moving
forward (see Appendix A). The action plan template was revised from the previous year’s
submission to support reducing the time burden on school teams. The template now includes a
summary of needs-assessment findings, overall strengths, and goals for the work moving forward
(see Appendix B). Additionally, project staff developed a technical assistance tracking template
and coaching logs that will be used throughout the course of the project as training, coaching, and
technical assistance activities occur.
The project has also made progress on improving family engagement. The RIPIN collaboration
described earlier will provide necessary feedback from families to school teams. The feedback
loop between families and schools is intended to build stronger lines of communication, and help
parents gain a better understanding of intensive intervention.
13
Stakeholder involvement in SSIP implementation
a. How stakeholders have been informed of the ongoing implementation of the SSIP
There are two groups of stakeholders associated with SSIP implementation. Primary stakeholders
include school staff and DBI core team members who are involved in the ongoing
implementation efforts. Peripheral stakeholders, including SSIP Core team members, are those
who are not engaged in ongoing implementation efforts, but have a broader interest in statewide
intensive intervention.
Primary stakeholders—school staff from implementation sites—participate in the ongoing
implementation of the SSIP. These stakeholders play a significant role in determining the course
of technical assistance activities by co-developing the final action plans and goals for the
academic year. For example, one Cohort 2 site communicated a need related to examining their
mathematics scope and sequence prior to focusing on data-based decision making. Given the
site’s level of readiness (determined through the needs-assessment process), this goal was
prioritized in their site-level action plan.
Peripheral stakeholders were provided periodic updates from the RIDE Director of the Office of
Student, Community and Academic Supports (OSCAS). The number of schools participating in
the technical assistance, along with district-, school-, and classroom-level data from the Math
Project have been shared. Stakeholders have expressed their support in continuing the state’s
efforts with outreach to families and community members. Additionally, the Director of OSCAS
meets monthly with the Executive Board and presents regularly at the general membership
meetings of the Association of Rhode Island Administrators of Special Education (ARIASE), the
RI Special Education Advisory Council, and statewide special education director meetings. At
these meetings, the Director presents an update regarding the work of the office, which includes
updates on the Math Project. Updates were provided in April, September, November, and
Increased educator beliefs of DBI for mathematics (short term)
To what extent did educator beliefs about math instruction change?
Math Beliefs Survey
Data Driven Instruction Survey
Increased educator application of skills related to DBI for mathematics (intermediate)
To what extent have intensive math intervention and mathematics instructional practice changed adult behavior and practice in participating schools?
Training Evaluation
Observational Tool
End of Year Pulse Check
Training Implementation Survey
Improved formative assessment outcomes for students receiving intensive mathematics interventions (long term)
To what extent have the implementation of intensive math intervention and mathematics instruction practices improved student results?
Universal screening data
Progress Monitoring Data on Student Level Plans
Improved fidelity of school-level implementation of DBI in mathematics (long term)
To what extent did schools implement DBI in mathematics with fidelity?
Needs Assessment
End of Year Pulse Check
Observational Tool
Improved LEA capacity to support, scale, and sustain improvement efforts in urban settings and with diverse populations (long term)
To what extent did LEAs increase their capacity to support, scale, and sustain improvement efforts related to high-quality math instruction?
Professional Learning Community (PLC) capacity survey
Increased parent or family awareness of intensive intervention and how to support their child (short term)
To what extent do families report they are aware of their child's math instruction?
To what extent to families report they understand how to support their child's math instruction?
Needs Assessment
End of Year Pulse Check
Rhode Island Parent Information Network (RIPIN) Family Interviews
Effective communication, coordination, and collaboration among and between RIDE initiatives (short term)
To what extent was communication effective among and between RIDE staff?
Collaboration Survey
16
Logic Model Outcome
Evaluation Question Data/Evidence
Improve the mathematics achievement for Hispanic and Black students with specific learning disabilities in grades 3-5 by 4% by FFY2018
To what extent did the intervention improve the mathematics achievement for Hispanic and Black students with specific learning disabilities in grades 3-5 by 4% FF2018 (schools with target population)
Universal Screening data
State Assessment data
Stakeholder Engagement
(Peripheral)
How have stakeholders been informed and involved in decision making regarding ongoing implementation and evaluation of the project
Stakeholder Engagement Survey
Stakeholder Engagement (Primary)
To what extent do school level stakeholders report feeling engaged in the ongoing implementation and evaluation of the project
End of Year Pulse Check
b. Data sources for each key measure
Data/Evidence Description
Needs Assessment
The needs assessment is completed during the initial interview that sites undergo with project staff at the beginning of technical assistance. Responses on the needs assessment serve as a pre-test to understand the degree to which the site implements math instruction and data-based decision making across the tiers at the onset of participation.
End of Year (EOY) Pulse Check
The EOY Pulse Check is the annual follow-up from the needs assessment. Responses on the EOY Pulse Check serve as a post-test to explore the changes in DBI implementation at the end of each academic year.
Math Beliefs Survey The Math Beliefs survey was adapted from the Teacher Beliefs about Math survey developed by Deborah Stipek and colleagues (2001) and is used to assess teacher beliefs or misconceptions about math instruction. Educators receive a pre/post test each academic year.
Data Driven Instruction Survey
The Data Driven Instruction survey is an internally developed source to assess educator beliefs about using data to inform instruction. A variety of sources were used to develop the survey including Nancy Harris’s (2011) Data-Driven Instruction Survey. Educators receive a pre/post test each academic year.
17
Data/Evidence Description
Training Evaluation Training attendees evaluate each training with a short survey to assess training quality, relevancy, and the potential to influence educator practice.
Training Implementation Protocols (including an observational tool)
As a follow-up to trainings, implementation protocols will be designed to determine the degree to which educators implemented with fidelity the skills attained during training. Implementation protocols will be developed in the next reporting period*
Universal Screening
Data
Screening is conducted to identify students who may be at risk for poor learning outcomes so that early intervention can occur. Screening assessments typically are brief and usually are administered with all students at a grade level. Some schools use a gated screening system, in which universal screening is followed by additional testing or short-term progress monitoring to confirm a student’s risk status before intervention occurs.
Progress Monitoring Data on Student Level Plans
Progress monitoring is used to assess a student’s performance, to quantify his or her rate of improvement or responsiveness to intervention, to adjust the student’s instructional program to make it more effective and suited to the student’s needs, and to evaluate the effectiveness of the intervention
Professional Learning Community (PLC) capacity survey
The PLC survey will be designed to assess LEA capacity to support, scale, and sustain improvement efforts. The PLC survey will be developed in the next reporting period*
Rhode Island Parent Information Network (RIPIN) Parent Interviews
RIPIN will interview families to evaluate the extent to which they understand the intensive intervention process and are aware of strategies to support their child. RIPIN interviews will be analyzed during the next reporting period*
Stakeholder Engagement Survey
Leading by Convening: A blueprint for authentic engagement developed by the IDEA Partnership and National Association of State Directors of Special Education was adapted to assess the engagement of peripheral stakeholders
Coordination and Collaboration Survey
Leading by Convening: A blueprint for authentic engagement developed by the IDEA Partnership and National Association of State Directors of Special Education was adapted to assess the coordination and collaboration across RIDE initiatives and departments.
State Assessment State assessment data is used to monitor progress towards the SiMR.
c. Description of baseline data for key measures
The Math Project team has collected baseline data from site needs assessments, educator beliefs
about math and data driven instruction, training evaluations, stakeholder engagement, and
18
coordination and collaboration across RIDE initiatives. There are no baseline data available for
the end of year pulse checks (to be implemented with all Cohort sites Spring 2018, based on
implementation cycle) or for student-level data (e.g., screening or progress monitoring data—
including DBI case studies and parent interviews). These data are scheduled to be collected at
meaningful intervals for sites (i.e., after spring benchmarking or statewide assessments are
administered).
Needs Assessment
As part of the support and planning to the Cohort sites, Math Project staff conducted a Needs
Assessment with the MTSS teams at each site. The Needs Assessment includes components
related to: (1) Screening and Progress Monitoring, (2) Instruction and Intervention, (3) Special
Education, (4) Data-based Decision Making, (5) Monitoring Fidelity, and (6) Professional
Learning and Development.
At the completion of the needs assessment process, all sites established goals for the academic
year (see Appendix A). Several examples of the goals established include:
Piloting PALS Math;
Developing a consistent procedure with standardized reporting to support data-based
decision making; and
Drafting a math MTSS Implementation guide outlining decision rules and progress
monitoring processes.
Beliefs about Math and Data Driven Instruction
The Mathematics Beliefs and Data Driven Instruction Survey was administered to 84 educators
across Cohorts and prior to attending trainings. Seventy-three educators completed the survey,
and their responses are summarized below. The survey will be resent to individuals periodically
to assess change in beliefs over time.
The Math Beliefs survey includes 39 items designed to assess the level of agreement regarding
the educators' mathematics beliefs using an agreement scale of 1 (strongly disagree) to 6
(strongly agree). The survey was designed based on the research conducted at the UCLA
Graduate School of Education (Stipek, et al. 2011), and includes items in six domain areas:
Math as a set of operations versus a tool for thought,
Correct answers versus understanding as primary goal,
Teacher control versus child autonomy in classroom lessons,
Entity versus incremental view of intellectual ability (i.e., a fixed v. growth mind set),
Confidence in teaching math, and
Enjoyment of math.
Within each of the domains, items varied in terms of whether a positive belief would be reflected
in terms of strong agreement or strong disagreement. For example, within the “enjoyment of
math” domain, the item Math is my favorite subject to teach would be one for which a strong
agreement would indicate positive belief, and for the item I don’t enjoy doing math, strong
disagreement would indicate positive belief. The summary across these domains for 41
responses is described in the following narrative and tables.
19
The items with the least positive belief and most positive belief for each domain based on
average responses are provided in the table below. Overall, the responses—at baseline—suggest
that educators lack confidence in their knowledge of the math content they are teaching, have
more “fixed” mindsets, and believe in more “traditional” approaches to assessing student learning
(e.g., having students complete assigned tasks rather than observing students and listening to how
they arrived at an answer). These responses are consistent with current research and suggest a
need for training and ongoing coaching related to math content and evidence-based instructional
practices to help educators shift from a fixed mindset to a growth mindset and develop the
necessary skills to assess student understanding of mathematical concepts.
Math Belief Survey Item domain
Least Positive Average Belief (among responding educators
across Cohort sites)
Most Positive Average Belief (among responding educators
across Cohort sites)
Math as a set of operations versus a tool for thought
The best way to understand math is to do
a lot of problems.
In math, answers are either right or wrong.
Correct answers versus understanding as primary goal
It doesn’t matter whether students get the right answer as long as they understand the math concepts inherent in a problem.
Students who finish their math work quickly understand the material better than students who take longer.
Teacher control versus child autonomy in classroom lessons
Students cannot be counted upon to evaluate their math work accurately. To assess students' math understanding, it is important to observe them while they are working and to listen to their math conversations.
It's important for students to complete assignments exactly as the teacher planned.
Entity versus incremental view of
intellectual ability (i.e., a fixed v.
growth mind set)
I can improve my math skills but I can't change my basic math ability.
Math ability is something people have a certain amount of and there isn't much they can do to change it.
Confidence in teaching math I feel confident that I understand the math material I teach.
I'm good at communicating math content to students.
Enjoyment of math I enjoy encountering situations in my everyday life (e.g., sewing, carpentry, finances) that require me to use math to solve problems.
I don't enjoy doing math.
The Data-Driven Instruction Survey includes nine items related to data efficacy and data-use. Responses to
the Data-Driven Instruction Survey and attributes related to those skills included a set of items on which
respondents rated their agreement on a scale of 1(strongly disagree) to 6 (strongly agree). The results indicate
a fairly high belief on the part of educators at the Cohort sites, with average scores of "4" and above for each
the items. The average scores for each of the survey items are detailed in the chart below. The responses in
this section are generally more positive than those in the Math Beliefs survey and suggest that educators
believe they are using data to drive their instruction. These baseline results will be compared against results
from the same survey at periodic checkpoints to assess change over time. It may be that these results
decrease as participants learn new techniques for collecting and analyzing data.
20
Based on the results of the Math Beliefs Survey and the Data Driven Instruction Survey, there is need to both
ensure the instrumentation is accurately reflecting the reality of the educators at the Cohort sites and that the
results are meaningful and can be used by the Math Project staff supporting the Cohort sites.
For instance, selected educators form the core team that participates in the Needs Assessment process.
Consequently, all the survey respondents from each site were not always included in the Needs Assessment
process. Therefore, it is difficult to have an accurate, consistent picture of the needs and beliefs at the sites.
Overall, there are areas in need of improvement for both math beliefs and data-based instruction, and the data
specific to each Cohort site will be used to target the needs and supports required by each.
Training Evaluations
Between March and December of 2017, RIDE/AIR conducted seven professional learning opportunities for
general and special educators at the Cohort sites. These included one session on Peer-Assisted Learning
Strategies (PALS) focused for grades 2-6; two multi-day sessions with Nancy Butler Wolf that covered an
array of evidence-based, core math strategies and included targeted support to two of the sites; and two
virtual sessions with one SSIP site that focused on Tier 1 and 2 differentiation and screening vs. progress
monitoring. The professional learning opportunities were tailored for site needs.
For each of the professional learning sessions, a common evaluation form was used to collect data on the
quality and relevance of the session as well as the extent to which participants gained understanding of the
skills addressed in the session and their intent to apply those skills in their daily practices. The survey item:
1.0 2.0 3.0 4.0 5.0 6.0
I use student data to verify my hypotheses about the causes of student
behavior and math performance.
I know what instructional changes to make when data show that students
are not successful in math.
I have clear criteria for determining student success in completing
instructional activities in math.
Overall, I am confident in my ability to interpret student data.
I use assessment results to measure the effectiveness of my math
instruction.
I make changes to my math instruction based on summative assessment
results.
I am confident in my ability to use student data to inform my decisions
about how students are performing.
I am confident in my ability to communicate data related to student
performance to teachers, students, and parents.
I am confident in my ability to use student data to inform instructional
decisions I make in my classroom.
I make changes to my math instruction based on formative assessment
results.
I use assessment data to identify students who are having difficulty
learning math.
I use student data from math assessments to set instructional targets and
goals for students.
Data Driven Instruction Survey
Item Response Averages (n=41)
21
The training provided me with something (e.g., strategy, process, resource) that I can apply in my
work was analyzed to determine the percentage of agreement. Respondents were asked to rate their level of
agreement with the above statement using a scale of Strongly Agree, Agree, Disagree, or Strongly Disagree.
For the purposes of analysis, an overall agreement percentage was calculated by aggregating the item
responses of strongly agree and agree for each of the professional learning sessions; 95.8% of educators
agreed with the statement. The training evaluation template and evaluation summary data of additional items
can be found in Appendix C.
Stakeholder Engagement
Outcome Performance Measure(s)
Stakeholder Engagement (Peripheral)
Stakeholder Engagement (Active)
Increase in Leading by Convening Survey
Scores
End-of-Year Pulse Check
While the End-of-Year Pulse Checks were not conducted based on the current implementation cycle, data to
inform the performance measure regarding peripheral stakeholder engagement was collected via a survey to
assess the extent to which RIDE engages relevant stakeholders—those who broadly have an interest
in/awareness of the SSIP, but may not work closely with implementation/evaluation activities. The survey
was provided to 110 stakeholders in early January 2018 and a total of 29 responses were received, yielding a
response rate of 26.4%. Responses were received from representatives from LEAs, charter schools, state
schools, disability organizations, and staff from TA projects (excluding project staff) and centers.
A rating scale of strongly agree, agree, neutral, disagree, and strongly disagree was used for each of the
survey items. For the analysis, the ratings of strongly agree and agree were combined into an overall
agreement percentage and the same was done for the disagreement responses. As depicted in the graph
below, there was strong agreement that stakeholders were provided opportunities for feedback and
engagement (75.9%) as well as agreement that the process included evolving leadership and facilitation of
understanding diverse perspectives (72.4% and 69.0%, respectively). While there was little disagreement
about the aspects of relevant participation, there were several respondents who indicated neutral, which may
be an area to investigate as RIDE reviews these results with their stakeholder groups. There may be potential
to strategize ways to better or differently provide opportunities for stakeholders to be informed and engage in
SSIP activities.
22
The stakeholders were asked to rate their perception of the level of engagement related to SSIP activities.
The item response options were informing, networking, collaborating, and transforming, and these were
defined for the respondents. The results for this survey item are displayed in the chart below, as is the
definition of each response item.
6
4
13
5
Informing Networking Collaborating Transforming
Perception of Engagement Level by Number of Responses (n=28)
Informing:
RIDE shares or disseminates information with relevant stakeholders in the state who care about the SSIP
Networking:
RIDE asks others what they think about efforts in the state related to the SSIP and listens to what they say
Collaborating:
RIDE engages people in trying to do something of value and working together around efforts in the state related to
the SSIP
Transforming:
RIDE promotes shared leadership and builds consensus across stakeholders in state efforts related to the SSIP,
which leads to cross-stakeholder collaboration to improve efforts
75.9
75.9
72.4
69.0
17.2
20.7
24.1
27.6
6.9
3.4
3.4
3.4
RIDE Provides Opportunities for me to Provide
Feedback on SSIP Efforts
RIDE Creates Opportunities for me to engage in SSIP
Efforts
RIDE develops evolving leadership roles for relevant
stakeholders
RIDE works to facilitate understanding of diverse
perspectives
Ensuring Relevant Participation Responses by Percent
Agreement/Disagreement/Neutral (n=29)
agree neutral disagree
23
The majority of respondents (13) indicated that the engagement level was collaborating, which reflects a
positive perception that RIDE is more meaningfully engaging their stakeholders and moving beyond merely
providing information and asking for input.
Collaboration
Outcome Performance Measure(s)
Effective communication and
coordination among and between RIDE
initiatives (short term)
Effective collaboration and alignment of
RIDE initiatives (long term)
Increase in agreement scores
Increase in perceptions of engagement
In January 2018, a survey was provided to 25 personnel from several departments within RIDE, including
OSCAS, where the SSIP work is housed. Of the 25, 14 staff members completed the survey, yielding a
response rate of 56%. The survey was administered to address the performance measure regarding effective
communication and coordination of SSIP activities and various RIDE initiatives. Details about the
departments or organizations represented by respondents and their general roles are provided in the tables
below.
Respondents by Department Total
Office of College and Career Readiness 2
Instruction, Assessment and Curriculum 2
OSCAS 4
RIPIN 1
Other 4
Not indicated 1
Total Responses 14
Respondents by Role Total
Leadership (i.e., Director, Commissioner) 3
Specialist 7
Other 4
Total Responses 14
The survey also included items addressing the extent to which personnel agreed that they were informed
and engaged in SSIP activities and the extent to which an understanding of diverse perspectives and
evolving leadership were facilitated throughout the process. A rating scale of strongly agree, agree,
neutral, disagree, and strongly disagree was used for each of the survey items. For the purpose of analysis,
the ratings of strongly agree and agree were combined into an overall agreement percentage and the same
was done for the disagreement responses. As depicted in the graph below, most respondents agreed with
these aspects of ensuring relevant participation in the SSIP activities. The highest agreement levels were
related to the opportunities to provide feedback and that the process included an understanding of diverse
perspectives (78.6% and 85.7% respectively).
24
While the majority of respondents (57.1%) agreed that the process included development of evolving
leadership, there were a relatively high number of those who selected the neutral response (42.9%). This
may be due to the stage of implementation for the SSIP activities and perhaps the level of coordination is not
suited to more fluid leadership roles across the departments.
Respondents were also asked to rate their perception of the level of engagement at RIDE regarding the SSIP
activities. The response options were informing, networking, collaborating, and transforming, each of which
was defined for the respondents. The results are displayed on the graph below, as is the definition for each of
the options.
78.6
64.3
85.7
57.1
7.1
7.1
14.3
28.6
14.3
42.9
0 100
Opportunities to Provide Feedback on SSIP Efforts
Opportunities to Engage in Efforts related to SSIP
OSCAS Works to Facilitate Understanding of Diverse
Perspectives
OSCAS Develops Evolving Leadership Roles
Ensuring Relevant Participation Responses by Percent
Agreement/Disagreement/Neutral (n=14)
Agreement Disagreement Neutral
25
Most of the respondents (7) indicated that they perceived the level of engagement as informing, which seems
fitting for the early stages of implementation of the SSIP activities, and may suggest the need for OSCAS to
take more of a lead as they make connections from the SSIP to other RIDE initiatives. Of note, four (4)
respondents perceived the level of engagement as collaboration and three (3) as transforming. This is
positive, but may demonstrate uneven engagement across RIDE, where some departments are more engaged
than others in the SSIP activities. Those who indicated the level of engagement as transforming were from
OSCAS and therefore have more day-to-day involvement in implementation. Whereas those indicating
informing were from state-level departments currently not as closely involved in SSIP implementation such
as the Office of College and Career Readiness. In the coming year, these results may serve as an opportunity
for OSCAS to discuss and strategize with RIDE personnel as they continue to build meaningful collaboration
around RI SSIP activities.
d. Data collection procedures and associated timelines
After finalizing the appropriate data sources to assess logic model outcomes, the project team
established data collection procedures and timelines. AIR leads the effort to collect all data on a
consistent and timely basis. Prior to reporting submissions, the external evaluator supports with
aggregating and analyzing data.
7
0
43
Informing Networking Collaboration Transforming
Perception of Engagement Level by Number of Responses (n=14)
Informing: OSCAS shares or disseminates information with relevant stakeholders in the state who care about the State
Systemic Improvement Plan
Networking: OSCAS asks others what they think about efforts in the state related to the State Systemic Improvement
Plan and listens to what they say
Collaborating: OSCAS engages people in trying to do something of value and working together around efforts in the
state related to the State Systemic Improvement
Transforming: OSCAS promotes shared leadership and builds consensus across stakeholders in state efforts related to
the State Systemic Improvement, which leads to cross-stakeholder collaboration to improve efforts
Data/Evidence Timeline
Needs Assessment
Frequency: Once
Timeline: Fall
End of Year (EOY) Pulse Check Frequency: Annually
Timeline: April-May
26
e. [If applicable] Sampling procedures
Regarding the SiMR target population, no sampling procedures are used. Black and Hispanic
students with learning disabilities represent a small number of students throughout the state,
and the focus on improving their mathematics outcomes remains relevant to RIDE, SSIP
implementation sites, and stakeholders. This school year (2017-2018), RIDE will be moving
from PARCC to RICAS. RICAS is the new statewide assessment in grades 3-8 with different
scale scores and achievement levels, and as such will require a baseline reset. With the shift from
Math Beliefs Survey Frequency: Pre-Assessment Once/Post-Assessment Annually
Timeline: Prior to Coaching or Training/Late Spring
Data Driven Instruction Survey Frequency: Pre-Assessment Once/Post-Assessment Annually
Timeline: Prior to Coaching or Training/Late Spring
Training Evaluation Frequency: After each training
Timeline: Ongoing
Observation/Fidelity Tool Frequency and Timeline to be determined during the next reporting period
Universal screening data Frequency: Annually
Timeline: to be determined during the next reporting period, though anticipated at the end of the school year/early summer
Progress Monitoring Data on Student Level Plans Frequency: Annually
Timeline: to be determined during the next reporting period, though anticipated at the end of the school year/early summer
Professional Learning Community (PLC) capacity survey
Frequency and Timeline to be determined during the next reporting period
Rhode Island Parent Information Network (RIPIN) Parent Interviews
Frequency: At least twice a year per selected student undergoing intensive intervention
Timeline: Varies
Stakeholder Engagement Survey
Frequency: Annually
Timeline: Winter
Coordination and Collaboration Survey
Frequency: Annually
Timeline: Fall
State Assessment data Frequency: Annually
Timeline: Late Spring
27
PARCC to RICAS, RIDE does not anticipate any changes to the SiMR population, and is
encouraging sites to focus their efforts on positively influencing math achievement for Black and
Hispanic students with learning disabilities. At the site level, where the most focused
implementation is occurring, Math Project Staff are working with school personnel to identify
“DBI Case-Study” students as they begin to learn the DBI process. As possible, sites are
encouraged to identify case-study students who reflect the SiMR population to collect additional
formative assessment data while the shift in the statewide assessment occurs.
f. [If appropriate] Planned data comparison
At the site level, longitudinal comparison of cohort performance over time will provide for data
comparison from the 2015 cohort to the 2016 cohort to the 2017 cohort. Student-level
performance on the PARCC assessment will provide for planned data comparison in two ways.
First, assessment scores from the SiMR identified in Phases I and II will be compared over time;
student performance will allow RIDE to examine if modifications should be made to the target
population.
Second, the assessment scores from students at each of the cohort sites will be compared
annually; scores on both formative (i.e., screening/benchmarking measures) and summative (i.e.,
PARCC) assessments will be compared over the course of the project as a way for RIDE to
assess the effectiveness of the SSIP implementation activities. Data on individual students who
are tracked through the case-study approach using the DBI process will be compared over time to
determine if students are making progress toward intervention goals. Case-study students are
currently being identified. Since data from the 2017 administration of PARCC provides 3 years
of continuous test data, those comparisons are currently underway and will be available to report
in next year’s SSIP submission.
g. How data management and data analysis procedures allow for assessment of progress toward achieving intended improvements
As the data are collected and analyzed, the regular structure of SSIP Core Team meetings will
support the review of the results and decision-making needs to continue effective implementation
of SSIP activities. Student level assessment data are matched with enrollment and IEP census
demographics using the SASID – state assigned student identifier – unique ID numbers assigned
to all RI public school students. Data analysis begins in the Office of Data and Technology
Services in consultation with IDEA staff for creation of data files consistent with those produced
in prior years. Data are reviewed and further analyzed by the SSIP Core Team and shared at
OSCAS staff meetings and RIDE leadership meetings.
How the state has demonstrated progress and made modifications to the SSIP, as necessary
a-e. How the State has reviewed key data that provide evidence regarding progress toward achieving intended improvements to infrastructure and the SiMR
The Indicator 17 submission each year is completed prior to the end of the academic year. Therefore,
the State has not completed reviewing key data from the current academic school year to provide
evidence regarding progress toward achieving intended improvements to infrastructure and the
28
SiMR. However, the state is scheduled to review universal screening data at the end of the current
academic year (2017-18). Data will be used to inform changes, next steps, and modifications to the
SSIP, as needed. Since the available PARCC state assessment collection period does not align with
the reporting period of the SSIP, scores from the spring 2017 administration of PARCC math are
examined. As previously mentioned, the state will be implementing a new statewide assessment
during the current academic school year, which may confound future comparative analyses and will
require yet another baseline reset. RIDE will be able to report a comparison of student performance
from 2015 to 2016 to 2017 administrations of PARCC using a SASID match from the original group
of grades 3-5 Hispanic or Black students with Learning Disabilities. RIDE will then use another
SASID match to examine the performance of students in grades 3-5 for the 2017 PARCC with the
same students on the new RICAS 2018. The PARCC to RICAS cohort comparison will be reported
in future submissions.
Currently, the target population is not as well represented in the sites participating in the project as
planned. While two urban core districts are part of cohort 1, both are small districts. Two additional
urban ring districts are participating and the remaining participants are suburban. The largest urban
core district in the state was recruited for the project but declined to participate. The State and Math
Project Staff are initiating additional plans to recruit an additional Cohort with sites that have are
larger percentage of Black or Hispanic students with learning disabilities in Grades 3-5. Statewide,
there are fewer than 800 students with LD who are Black or Hispanic. At the site level, Math Project
staff are encouraging sites to select DBI case-study students who match the target population. DBI
case-studies will provide an additional level of data to report on progress toward outcomes—
especially formative data that help inform the short-term and long-term outcomes of this project.
Current data analysis of PARCC mathematics from the 2017 spring administration show that 12.4%
of Grades 3–5 Black or Hispanic students with learning disabilities (LD) scored an “approaching
proficiency (3),” “proficient (4),” or “exceeds expectations (5)” on PARCC mathematics assessments
in 2016–17 compared to 7.8% in 2014–15 and 11.5% in 2015-16. A closer examination of the data
reveals that a racial gap persists for that subset of students and parallels the racial gap present for
students who do not have IEPs in those grades. The performance of Black and Hispanic students with
LD in Grades 3–5—when compared to students with LD in other racial categories—remains lower
(see the following figures) with 25% of Grades 3–5 White or Asian students with LD and 15.5% of
students of all other races with LD in Grades 3-5 scoring a 3, 4, or 5 on PARCC math in 2016–17.
Fifty-seven percent of Black or Hispanic general education (non-IEP) students in Grades 3–5 scored
a 3, 4, or 5 on the 2015–16 PARCC mathematics assessment compared to 81% of White or Asian
general education (non-IEP) students and 65% of general education (non-IEP) students of all other
races.
29
30
414
426
68
254
337
29
83
200
13
1361 5
0 2 0
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Black + Hispanic Asian + White All other
PARCC 2017 Math Proficiency by Race/Ethnicity for students SLD grades 3-5
Did not Meet Partially Proficient Approaching Proficiency Proficient Exceeds Expectations
31
Stakeholder involvement in the SSIP evaluation
a. How stakeholders have been informed of the ongoing evaluation of the SSIP
Primary stakeholders—district and school staff from implementation sites—are informed of the
ongoing evaluation of the SSIP. At the onset of site-level participation with the Math Project, school
personnel are informed of the project’s short- and long-term outcomes, including the goal of
improving math achievement for the SiMR target population. Training evaluation results are also
discussed with school personnel, including leadership who may not be present for
training/professional development. At many sites, leaders have offered anecdotal evidence
confirming the positive training evaluation data gathered thus far.
As noted earlier, peripheral stakeholders (individuals who have a broad interest in state intensive
intervention efforts, but do not have regular engagement in the SSIP) were engaged in the SSIP
activities during the November 16 Special Educator Directors’ meeting. At this meeting, the Special
Educator Directors (some of whom are also primary stakeholders) were asked about implementation
and plans for evaluation efforts. The directors praised the project’s use of student-level progress
monitoring data in mathematics. The directors noted that the data positively focus on the learning of
students and supports the targeted instruction for specific areas of needs. Additionally, in reviewing
the feedback with RIDE, project staff determined that presently, the project evaluation was closely
aligned to expectations of local special education directors. Regular project updates, which include
information on the Evaluation plan have been provided to additional peripheral stakeholder groups
such as cross office RIDE teams and the RI Special Education Advisory Council. The RISEAC is
invited to contribute feedback at least twice a year at their September annual retreat and the winter
APR update.
32
b. How Stakeholders have had a voice and been involved in decision-making regarding the ongoing evaluation of the SSIP
Primary stakeholders are actively engaged in evaluation activities from the onset of their
participation. School personnel are involved throughout the needs-assessment and action planning
processes, and are encouraged to discuss any feedback on summaries and goals initially drafted by
Project staff. As DBI case-study students are identified, Math Project staff will encourage sites to
consider the SiMR population when selecting students, but school personnel will have the final “say”
in who is selected. Additionally, school personnel from Cohort 1 expressed that the needs-assessment
process was time consuming, so Math Project staff refined the protocol (previously discussed in this
report) to expedite the process for future Cohorts (including Cohort 2). All sites will also have an
opportunity to provide feedback that may lead to revisions on the Project, their implementation, and
how they are being assessed at the end of year pulse check interview (i.e., essentially a re-
administration of the needs-assessment interview to measure progress over time).
Stakeholder feedback during the Special Educator Directors’ meeting affirmed the current evaluation
plan. RIDE and Project staff plan to share implementation data across primary and peripheral
stakeholder groups to ensure they have a voice in how the data can be used to inform
implementation, as well as preferred methods for providing input on the ongoing evaluation.
D. Data Quality Issues: Data limitation that affected reports of progress in implementing the SSIP and achieving the SIMR
Concern or limitations related to the quality or quantity of the data used to report progress or results
The SSIP aims to utilize local assessments to provide a more in-depth understanding of student progress.
One major area of concern is that sites use different local assessments and tools to collect universal
screening and ongoing progress monitoring data. Our project has experienced ongoing challenges while
supporting sites to collect the appropriate local assessment data for evaluation purposes. We have found
that prior to evaluating the data, sites need a great deal of additional training on data entry and
systematizing data collection processes.
Another area of concern is changing state assessments from PARCC to RICAS. RICAS is the new
statewide assessment in grades 3-8 with different scale scores and achievement levels and as such will
require a baseline reset. SASID matches will be used to examine student data across the two measures.
Implications for assessing progress or results
While the response rate to the training survey was high, it is critical to continue efforts to encourage the
training participants complete the evaluation. These data are essential to the project’s continuous
improvement, and ensuring trainings are relevant and useful to school personnel.
Reviewing progress on the SiMR from Phase I through the February 2019 submission will be
challenging with two state assessment changes and two baseline resets. Examination of local data,
33
implementation data, and other evaluation measures as described in the previous sections will be vital to
understanding progress in improving outcomes for the target population.
Plans for improving data quality
As described in the sections above, completing the Needs Assessment Scoring Rubric and supporting
sites with appropriately collecting and analyzing data from multiple sources (i.e., screening, ongoing
progress monitoring, and statewide assessments) are areas on which to focus for improving data quality.
Training school level participants to extract universal screening data by disability category and race will
improve future outcome measures. Adding the case study approach to examine progress monitoring data
for specific disabilities and races will also strengthen data quality in the evaluation.
E. Progress Toward Achieving Intended Improvements
Infrastructure changes that support SSIP initiatives, including how system changes support achievement of the SiMR, sustainability, and scale-up
At RIDE, there are now cross-division (Educator Quality; Instruction and Assessment, School
Improvement, and OSCAS) meetings occurring, with plans to ensure they happen more
frequently in the future. The overarching goal of these meetings is to align practices and
initiatives at the state-level to reduce confusion for LEAs around potentially competing initiatives
from across divisions. This approach to changing RIDE’s infrastructure has potential to reduce
barriers related to initiative overload on LEAs, thus resulting in more sustainable, scalable
efforts.
To produce greater cross-office collaboration, OSCAS staff have been included in curriculum
work at RIDE. There is also a focus on more active collaboration instead of information sharing.
For example, Math specialists have opened core math training preparation to OSCAS staff and
project partners for feedback and input. Additionally, OSCAS staff have participated in new
Curriculum team work in the department with the outcome that districts will be supported tools
for choosing quality core math curriculum beginning spring 2018. RIDE personnel including
math specialists, curriculum specialists, and assessment specialists work alongside one another in
the same office space; and those specialists are now in the same Teaching and Learning Division
as OSCAS staff. Overall, leadership has been focused on ensuring infrastructural changes to
support collaboration across RIDE initiatives.
RIDE continues to align projects to support continuous improvement in DBI and tiered systems
of support. Elements of DBI are included not only in the SSIP Math Project, but in CEEDAR and
MTSS work around the state. Additionally, RI was recently selected to participate as an
“Advanced Implementation” state with NCII (extending previous TA efforts). NCII’s support to
RI is likely to support with scaling up DBI practices across initiatives to support sustainability,
considering the frequency with which LEA staff move around the state. The goal of this work is
to ensure that momentum need not be lost should there be changes within personnel/leadership.
Furthermore, infusion of DBI in pre-service preparation programs through CEEDAR will support
sustainability and scale up of those practices.
34
Evidence that SSIP’s evidence-based practices are being carried out with fidelity and having the desired effects
The training activities this reporting period have focused on developing participants’ knowledge
of evidence-based, core math instructional strategies and PALS-Math that are aligned with
common core standards. The project team is currently developing and piloting—in collaboration
with the trainer and site-level personnel—an observational tool that can be used to support with
monitoring the fidelity of implementation of learned strategies. While that tool is under
development, project staff are supporting school teams with analyzing their current scope and
sequence in math to identify where EBPs in math can be included throughout the school year.
PALS-Math has fidelity monitoring tools included with the teacher handbooks. Project staff
follow-up with school teams to determine their use of the fidelity tools, as well as to introduce
any fidelity tools associated with additional EBPs/interventions that sites select for future
implementation. Moving forward, project staff will continue to coordinate with school-level
personnel to address any challenges related to implementation fidelity. Ensuring that school-level
personnel take ownership of monitoring fidelity is critical to sustained implementation.
Additionally, as sites begin to identify students for DBI case-studies, fidelity to student-level
plans (e.g., implementation logs), and to the DBI process more generally (e.g., end of year pulse
check) will be included as another measure.
Outcomes regarding progress toward short-term and long-term objectives that are necessary steps toward achieving the SiMR
Implementation data from needs assessments, training evaluations, collaboration surveys, and
stakeholder engagement surveys were reported earlier. These implementation data will be compared
to future administrations of data collection to assess progress toward short-term and long-term
objectives that are necessary steps toward achieving the SiMR.
The project team will also collect universal screening data and progress monitoring data for case-
study students at each site. Universal screening data are not uniformly collected at the district level.
Therefore, the project team will discuss data sharing processes with administrators to provide data
reports at the end of each academic year. The processes will include data for all students, and exclude
personally identifiable information. Data sharing was incorporated into the memoranda of
understanding signed by all sites.
Data from the Math Project will be evaluated, according to the plans outlined in this submission,
after the final data collection at the end of the school year with analysis over the summer months to
be ready for next year’s reporting cycle. Rhode Island has already witnessed there is need to push the
needle forward for the target population of students with learning disabilities who are Black and
Hispanic.
Measurable improvements in the SiMR in relation to targets
Since the three years of PARCC data collection do not match the implementation timeline of the
SSIP, direct causation to the current math project is not feasible. SSIP math project implementation
began January 2017 after a fall 2016 recruitment and needs assessment process. PARCC data
collected spring of 2017 likely do not reflect those initial implementation efforts, but may reflect
35
prior math pilot work, MTSS, and NCII project work. RICAS 2018 is not yet administered. RICAS
2019 administration will be more likely to reflect SSIP implementation efforts. RIDE intends to
examine PARCC performance of students with LD who are Black or Hispanic from participating
districts compared to nonparticipating districts over the three years of PARCC administration, but
that work is in progress. Very small n sizes will make meaningful comparisons unlikely.
Nonetheless, trends in the data will be explored.
F. Plans for Next Year
Additional activities to be implemented next year, with timeline
The table below provides an overview of the additional activities to be implemented next year, with
the timeline delineated by project activity. Additionally, the previous section 1. Infrastructure
changes that support SSIP initiatives, including how system changes support achievement of the
SiMR, sustainability, and scale-up provides additional detail on additional activities that will be
implemented in the coming year.
Planned evaluation activities including data collection, measures, and expected outcomes
As the training, coaching, and TA are implemented, the Math Project team will continue to put into
action data collection instruments to gather data on quality, knowledge gain, and fidelity of
implementation. These tools will include a standard end-of-training survey; a needs assessment, and
a beliefs assessment; protocols for reviewing action plans and other documentation to assess fidelity
of implementation; and protocols for interviews and focus groups with SSIP participants and
stakeholders.
Anticipated barriers and steps to address those barriers
As the Math Project continues to move forward, sites will be required to demonstrate their progress
toward their implementation/action plans. These plans delineate training and coaching activities that
sites are expected to be a part of on an ongoing basis. Sites, because of when the project started
working with them, often were committed to participation in activities with other projects (i.e.,
coaching from the MTSS initiative). Math Project staff will work with district- and site-level
administrators to ensure this project is aligned to other state-level initiatives, so they understand the
connections across the efforts to support their outcomes. In that way, scheduling barriers may be
remediated. In addition to the above barriers, we have experienced a significant amount of leadership
turnover at both the district and school levels (principal turnover at one site, and district level
leadership turnover in two districts). The PLC previously described is a strategy the Math Project
will use to address barriers at the leadership level.
Additional Support and/or Technical Assistance Needed
Currently, RIDE and the state Core team will continue to participate in the NCSI Math Cross-State
Learning Collaborative. To date, this has been a very effective resource for the state in the
development of the design decisions for the Intensive Math Intervention Project, examination of
evidence-based research, and support for implementation challenges. It is expected that this
36
collaborative will continue to serve as a helpful tool for the SSIP. Additionally, RIDE will continue
to leverage CEEDAR, NCII, and IDEA Data Center (IDC) technical assistance to continue
development and implementation of the SSIP.
37
Implementation Plan and Timeline
Project Implementation Areas
Completed Activities Planned Activities Timeline for Implementation
Work with current districts to identify additional sites for Cohort 2.
Work with current districts to identify sites for Cohort 3 and conduct a targeted outreach to districts with better alignment to the SiMR (i.e., two urban districts)
Spring/Summer 2018
Conduct informational meeting/kickoff with Cohort 2 sites.
Conduct informational meeting/kickoff with Cohort 3 sites.
August/September 2018
Draft and finalize the MOU and mini-grant process with Cohorts 1 and 2 sites.
Draft and finalize the MOU and mini-grant process with Cohort 3 school sites.
September 2018
Implement action plans with Cohort 1 and 2 sites.
Implement action plans with Cohort 1 and 2 sites. Ongoing
Complete needs-assessments with Cohort 2 sites.
Complete needs-assessments with Cohort 3 sites. October 2018
Have Cohort 2 sites prioritize needs-assessment results and develop action plans.
Have Cohort 3 sites prioritize needs-assessment results and develop action plans.
October 2018
Identify objectives and targets for school year.
Identify objectives and targets for school year. Early Fall 2018
Draft and review training content for Year 2 trainings for Cohort 1 that includes cultural and linguistic responsiveness, data-based individualization, and family engagement strategies, as well as assesses barriers to implementation in urban settings.
Draft and review training content for Year 3 trainings for Cohorts 1 and 2 that includes cultural and linguistic responsiveness, data-based individualization, and family engagement strategies, as well as assesses barriers to implementation in urban settings.
Summer 2018
Adapt Cohort 1’s trainings for Cohort 2’s specific needs.
Nothing additional planned. N/A
Schedule and implement trainings for Cohort 1.
Schedule and implement trainings for Cohort 2. Fall 2018–Spring 2019
Schedule and implement trainings for Cohort 2.
Schedule and implement trainings for Cohort 3. Fall 2018–Spring 2019
38
Implementation Plan and Timeline
Project Implementation Areas
Completed Activities Planned Activities Timeline for Implementation
Administer evaluation protocols and instruments, including fidelity assessments (evaluation methods vary by cohort).
Administer evaluation protocols and instruments, including fidelity assessments (evaluation methods vary by cohort).
Ongoing
Conduct site observations, including data team meetings.
Conduct site observations, including data team meetings.
Ongoing
Review site improvement plan with Cohort 1 schools.
Review site improvement plan with Cohort 2 schools.
Early Fall 2018
Conduct intervention inventory with Cohort 2 schools.
Conduct intervention inventory with Cohort 3 schools.
Fall 2018
Support teams with selecting DBI case studies.
Support teams with selecting DBI case studies. Ongoing
Model EBPs with schools. Model EBPs with schools. Ongoing, as needed
39
Appendix A. Needs-Assessment/Action plan Goals School Site Goals
Urban Elementary
Pilot PALS Math for a tier 1.5/2 support
Developing a structure to support sustained implementation of core instructional math strategies (i.e., number sense, place value)
Urban ring elementary
Evaluate the current scope and sequence to ensure that educators in the building have an understanding of math progressions to ensure skills are introduced, revisited, and supported throughout the curricular sequence (and across grade levels)
Development of a consistent procedure with standardized reporting to support data-based decision making.
Evaluating effectiveness of current Tier 2 interventions and determining if they should continue, be modified (fidelity? Standardized?) or changed. Tier 2 interventions are matched to student skill deficit areas (as evidenced by diagnostic data)
Urban ring elementary
Development of a consistent procedure with standardized reporting to support data-based decision making.
Evaluating effectiveness of current Tier 2 interventions and determining if they should continue, be modified (fidelity? Standardized?) or changed. Tier 2 interventions are matched to student skill deficit areas (as evidenced by diagnostic data)
Urban Middle Train the team on the progress monitoring process
Support team with sustainable implementation of PALS and fidelity of intervention
Draft a math RTI Implementation Guide with decision rules, PM process, etc.
Suburban Middle
Developing a structure to support sustained implementation of core instructional math strategies (i.e., number sense)
Implement a tier 2 Prevention Program
Monitoring Fidelity
Support the school with implementation of the district plan for teaming structures.
Suburban elementary
Build special education teachers’ knowledge and capacity to support core and Tier 2 instruction
Training using the DBI Modules for tier 3 on Progress monitoring.
On site coaching related to the implementation of core instructional strategies.
Suburban elementary
Develop a structure to support sustained implementation of core instructional math strategies (i.e. number sense, place value)
Evaluate effectiveness of current Tier 2 interventions and determine if they should continue or be modified
40
School Site Goals
Urban Elementary
Develop a consistent procedure for teaming structures in math to support decision making
Evaluate the diagnostic tools/processes for struggling learners and pilot the DBI process with at least one student
Urban ring middle
Build a teaming structure and define decision rules
Develop teachers’ knowledge on implementing core instructional strategies
Urban ring elementary
Evaluate the current scope and sequence to ensure that educators in the building have an understanding of math progressions to ensure skills are introduced, revisited, and supported throughout the curricular sequence (and across grade levels)
Develop a structure to support sustained implementation of core instructional math strategies (i.e., number sense, place value)
Urban ring elementary
Develop a structure to support sustained implementation of core instructional math strategies (i.e., number sense)
Develop a consistent procedure for teaming structures in math to support data-based decision making.
Urban ring middle
Evaluate the efficiency of current teaming structures and define math decision rules
Develop teachers’ knowledge on implementing core math instructional strategies
Suburban elementary
Develop a structure to support sustained implementation of core instructional math strategies (i.e., number sense, place value)
Evaluate effectiveness of current Tier 2 interventions and determining if they should continue, be modified (fidelity? Standardized?) or changed
41
Appendix B. Needs Assessment Action Plan Template
Needs-Assessment Results and Priorities
School Site:
Overall Strengths:
Action Plan:
Goals Evidence (How will we
know that we met our
goal)
Action Item/Next Step Person Responsible Timeline
Summary of Findings: Needs Assessment (date)
Needs-
Assessment
Domain
Summary of Findings (From Needs
Assessment)
Recommendations Priority
Universal
Screening
☐Low
☐Medium
☐High
Progress
Monitoring Tools
☐Low
☐Medium
☐High
42
Progress
Monitoring
Process (i.e.
frequency,
fidelity)
☐Low
☐Medium
☐High
Core Instruction
(including
differentiation of
core)
☐Low
☐Medium
☐High
Tier 2 Prevention (including how
this is different
from core)
☐Low
☐Medium
☐High
Tier 3 Prevention
(including how
this is different
from Tier 2)
☐Low
☐Medium
☐High
Barriers to
Implementation
☐Low
☐Medium
☐High
Decision Rules
(i.e., movement
between tiers)
☐Low
☐Medium
☐High
Special
Education
☐Low
☐Medium
43
☐High
Parent/Family
Engagement
☐Low
☐Medium
☐High
Teaming
Structures
☐Low
☐Medium
☐High
Monitoring
Fidelity
☐Low
☐Medium
☐High
Cultural and
Linguistic
Responsiveness
☐Low
☐Medium
☐High
Other team
comments worth
noting
☐Low
☐Medium
☐High
44
Appendix C. Training Evaluation Template and Summary Data
Rhode Island Intensive Math Intervention Project
Name of Training
Date
Thank you for attending one of the training sessions with [Training Name] this week. We would like to know if the training met its
objectives and aligned with your work. Your honest feedback will help us provide more effective support. Your participation in this
survey is completely voluntary. However, we would very much appreciate your input regarding the training. Your responses will be
kept confidential and only aggregated results will be reported. Thank you in advance for your participation!
1. Please select the job title that most accurately reflects your primary and current role (choose 1 option):
a. General education teacher
b. Special education teacher
c. Instructional Coach
d. Interventionist
e. District administrator
f. School administrator
g. Other (please specify):
2. Please rate the level of relevance of the content described below:
Not at
all relevant
Slightly relevant
Relevant Very relevant
How relevant was this training to your current need in
enhancing daily core math instruction or providing math
intervention?
1
2
3
4
45
How relevant was this training to supporting your need to
learn how to instruct students through a variety of
mathematical strategies?
1
2
3
4
How relevant was this training to your current need in
selecting specific math strategies/manipulatives to support
your struggling learners or students with disabilities?
1 2 3 4
3. To what extent did the training improve your understanding of:
Not at all Minimal
extent Somewhat Moderate
extent
Strategies for teaching patterns and algebraic
thinking using manipulatives.
1
2
3
4
Strategies for teaching proportions and
proportional reasoning.
1
2
3
4
Strategies for teaching integer operations.
1
2
3
4
Progressions of mathematical skills that build
upon one another.
1 2 3 4
46
4. Please rate your level of agreement with the following statements regarding the content of the training:
Strongly
disagree Disagree Agree Strongly
Agree
The training provided me with something (e.g.,
strategy, process, resource) that I can apply in
my work.
1
2
3
4
After participating in this training, I have a
better understanding of the strategy, process,
and/or resource that was the focus of this
training.
1
2
3
4
5. Please rate your level of agreement with the following statements regarding the format of the training:
Strongly
Disagree Disagree Agree Strongly
Agree
The training was well organized.
1
2
3
4
Information was presented in a clear and comprehensible
manner.
1
2
3
4
47
The training was appropriately paced.
1 2 3 4
Overall, the training was of high quality.
1 2 3 4
The training allowed me to actively participate in learning
the content.
1 2 3 4
6. Considering the amount of time allotted for this training, the information presented today was:
a. Too basic
b. Just right
c. Too advanced
7. Are there any topics discussed in today’s meeting that are still unclear to you? If so, please elaborate in the space provided.
8. Please share any other comments or suggestions that you may have.
48
Note. The data are derived from attendee responses to items related to training relevance on evaluations from six professional
development sessions offered to sites.
68%
23%
8% 0%
Very Relevant Relevant Slightly Relevant Not at all relevant
Evaluations of Training Relevance
49
Note. The data are derived from attendee responses to items related to improved understanding on evaluations from six professional
development sessions offered to sites.
61%
31%
6% 2%
Moderate Somewhat Minimal Not At All
Evaluations of Improved
Understanding
50
Note. The data are derived from attendee responses to items related to the quality and comprehensiveness of training content on
evaluations from six professional development sessions offered to sites.
