Date post: | 20-Jan-2016 |
Category: |
Documents |
Upload: | jeremy-newman |
View: | 217 times |
Download: | 0 times |
The Indiana Acuity Efficacy Study: Year 2 Results and Implications
Terry SpradlinBridget SchleichJune 21, 2010
CCSSO National Conference on Student Assessment
2
About the Center for Evaluation & Education Policy
•The Center for Evaluation & Education Policy (CEEP) is a client-focused, self-funded research center associated with the School of Education at Indiana University
•CEEP promotes and supports rigorous program evaluation and nonpartisan policy research primarily, but not exclusively, for education, human service and non-profit organizations
•In the area of K-12 education policy, CEEP’s mission is to help inform, influence and shape sound policy through effective, nonpartisan research and analysiswww.ceep.indiana.edu
Contents
I. Indiana’s Comprehensive Assessment Plan
II. 2009-10 Testing Schedule (formative and summative)
III. Objectives of Efficacy Study in Indiana
IV. Study Design and Methods
V. Findings for Each Qualitative Analysis
VI. Overall Summary of Qualitative Findings
3
I. Indiana’s Comprehensive Assessment Plan
• Adopted by the Indiana State Board of Education on November 1, 2006
• Plan called for moving the Indiana Statewide Testing for Educational Progress-Plus (ISTEP+) from Fall to Spring and the implementation of formative/diagnostic assessments
• Features implemented during the 2008-09 school year:
o Wireless Generation’s mClass Reading 3D and Math (Grades K-2 formative)o CTB/McGraw-Hill’s Acuity Assessment Program (Grades 3-8 formative)o Phase-out of the Graduation Qualifying Exam (GQE)
• Class of 2011 last to be required to pass GQE• To be replaced with end-of-course assessments in core subject areas
o Moved ISTEP+ from Fall to Spring• (Students in grades 3-10 were tested twice during the 2008-09 school
year)
4
II. 2009-10 Fall Testing Schedule (Formative and Summative)
5
Assessment Testing Window Grade
mCLASS: Reading 3D 8/24-9/18 K-2
mCLASS: Math 9/21-10/16 K-2
Acuity Predictive A ELA/Math 9/28-10/9 3-8
Acuity Diagnostic 1 10/19-10/30 3-8
Acuity Predictive B ELA/Math 11/30-12/11 3-8
Acuity Predictive B Science 12/7-12/18 4, 6
Acuity Predictive B Social Studies 12/7-12/18 5, 7
II. 2009-10 Spring Testing Schedule (Formative and Summative)
6
Assessment Testing Window Grade
Acuity Diagnostic 2 1/4-1/15 3-8
mCLASS: Reading 3D 1/11-1/29 K-2
mCLASS: Math 2/1-2/19 K-2
Acuity Predictive C Science 2/8-2/19 4, 6
Acuity Predictive C Social Studies 2/8-2/19 5, 7
Acuity Predictive C ELA/Math 2/11-2/26 3-8
ISTEP+ Applied Skills 3/1-3/10 3-8
Acuity Diagnostic 3 3/15-3/26 3-8
ISTEP+ Multiple Choice 4/26-5/5 3-8
mCLASS: Reading 3D 4/26-5/21 K-2
mCLASS: Math 5/3-5/28 K-2
Acuity Diagnostic 4 5/10-5/21 3-8
III. Objectives of Efficacy Study in Indiana
Objectives of CEEP Study:
• Evaluate the effects of CTB/McGraw-Hill’s Acuity Assessment Program, a formative assessment system, on instructional practice and student achievement, particularly ISTEP+, for Acuity schools
• Information intended to inform CTB and the IDOE about the kind of support needed to make the implementation of Acuity most effective during subsequent school years
7
IV. Study Design and Methods
Qualitative and Quantitative Methods
1. Spring Statewide Online Survey of Acuity Schools2. Intensive Case Study3. Completion of a Comparison-Group Study
8
Study Design and Methods
Spring Statewide Online Survey of Acuity Schools
• Measure attitudes and perceptions of teachers and administrators on the benefits and outcomes of their participation in the Acuity Assessment Program during the 2009-10 school year
• Survey window was open from April 12 to April 30, 2010
• 1029 respondents included 655 (64%) teachers, 198 (19%) principals, 70 (7%) testing coordinators, and 106 (10%) other school personnel
9
Study Design and Methods
Spring Statewide Survey (cont.)
Questions probed:
• Educator opinions regarding Acuity Assessment Program content, technology/user experience, professional development, and customer support after use of the system for a full school year or more were assessed
• In addition, a primary objective of the survey to obtain suggestions for improvement of the program and to gauge views regarding the impact of the program on classroom instruction, general student achievement, and student achievement on ISTEP+
10
Study Design and Methods
11
Spring Statewide Survey (cont.)
2 Surveys were available :Schools in Year 1 : 16-question surveySchools in Year 2+: 18-question survey
Questions were identical on both surveys; however, Year 2+ schools were additionally asked:
1) The frequency with which they are using the Acuity tools (e.g. not using the tools; beginning to use the tools; using the tools extensively, etc.)
2) The extent to which they are using the Acuity data (e.g. not using the data; using Roster Reports; using Assessment Reports, etc.) to alter or inform instruction
Study Design and Methods
Qualitative Methods
Intensive case study• Determine what factors make a difference in the effective
implementation of Acuity and use of Acuity data• Extent to which schools have implemented the Acuity Assessment
Program and identify obstacles encountered• Extent to which Acuity has altered or informed classroom instruction,
and impacted general student achievement as well as ISTEP+ performance
12
Study Design and Methods Intensive Case Study (cont.)
• The Project Team identified 11 schools (9 elementary schools; 6 middle schools) for the case study that were reflective of Indiana’s student population based on prior ISTEP+ performance, free and reduced-price meal program eligibility data, school size, and local type (urban, suburban, and rural)
• Additionally, the Project Team identified four schools with significant scale score improvement from fall to spring ISTEP+ during the 2008-09 school year
• Qualitative procedures included one-on-one, face-to-face interviews with 34 principals, testing coordinators, and Acuity trainers as well as focus groups with six to 10 teachers in all 15 schools (109 teachers total); conducted 2/03/10 through 4/06/10
13
School Location Scale Score Change
Forest Glen Elementary School Indianapolis, IN 71.5
Dupont Elementary School Dupont, IN 37
Blackhawk Middle School Fort Wayne, IN 30
Blair Pointe Upper Elementary School Peru, IN 24.5
Study Design and MethodsQuantitative MethodsCompletion of a Comparison-Group Study • Using matching, regression and other statistical techniques to analyze quantitative
data collected from the Acuity Predictive and Diagnostic assessments as well as from Indiana’s ISTEP+ summative assessmento These analyses will be used to assess the degree to which use of the
predictive and diagnostic assessments are associated with increased achievement on the state-required summative assessments in mathematics and English/language arts
o Comparison schools will be matched with Acuity schools using prior ISTEP+ performance, free and reduced-price meal program data, race/ethnicity, school size, and local type (urban, suburban, and rural)
o CEEP will subsequently conduct a longitudinal study adding 2009-10 data to the aforementioned analyses to monitor performance differences from one school year to the next.
14
V. Findings for Each Qualitative Analysis
A. Spring survey
B. Case Study Site Visitso Positive Educator Feedbacko Pressing Issueso Educator Recommendations and Considerations
15
A. Spring Survey Results: Findings, Opinions, and Suggestions
16
Frequency of Formative Assessment Use
• Respondents most commonly used the program only during administration windows.
17
Percent of Respondents from Year 1
Schools
Percent of Respondents from Year 2+
Schools
Once/week14.9 13.5
Multiple times/week15.9 15.3
Once/month 20.0 20.0
Only during administration windows
36.5 39.8
Other (Varied)13.5 11.3
Total Number of Respondents 416 600
Other Frequency of Use Issues• Most Acuity schools (78% for Year 1 schools; 74%, Year 2+) used the
online tools exclusively in their administration of the Acuity Assessment Program – 21% of Year 1 and 20% of Year 2+respondents use both online and
paper/pencil tests.
• Educators are not using all components of Acuity as often as necessary to maximize the use and benefits of the system – e.g., only 45% of Year 1 respondents and 47% of Year 2+ respondents
indicated using the Instructional Resources; only 18% of Year 1 and 26% of Year 2+ respondents indicated using the Item Bank
18
Perceived Impact on Classroom Instruction
• A majority of respondents (63.5% in Year 1 schools; 65.6%, Year 2+) indicated that participation in the Acuity Assessment Program helped to somewhat improve classroom instruction
• 19.4%/22.7% felt participation had no impact on instruction• 6.3%/5.5% indicated that participation led to a decreased quality of instruction (due to the
multiple assessments scheduled during the school year, both formative and summative, limiting instructional time)
• When comparing last year’s spring survey findings to schools in Year 2+ this school year, the percent of respondents who felt Acuity had at least somewhat improved classroom instruction increased from 51% to 66%
19
Percent of Respondents from Year 1 Schools
Percent of Respondents from Year 2+ Schools
Greatly decreased quality of instruction 1.0 1.3Somewhat decreased quality of instruction 5.3 4.2
No impact on instruction 19.4 22.7Somewhat improved quality of instruction 63.5 65.6
Greatly improved quality of instruction 10.8 6.2
Total Number of Respondents 417 598
Perceived Impact on Student Achievement
• The majority (63.5% in Year 1 schools; 61.9%, Year 2+) of respondents indicated that participation in the Acuity Assessment Program during the 2009-10 school year led to somewhat improved student achievement outcomes
• Opinions of the impact of Acuity on student achievement were very similar in schools using Acuity for the first time and schools in year 2+
• From last spring’s statewide survey of Acuity during the 2008-09 school year, 45% of respondents felt Acuity somewhat improved (43%) or greatly improved (2%) student achievement. Over the course of one year, respondents’ opinions improved 20 percentage points overall regarding the influence of Acuity on student achievement.
20
Percent of Respondents from Year 1 Schools
Percent of Respondents from Year 2+ Schools
Greatly decreased achievement 0.5 0.5
Somewhat decreased achievement 2.2 4.2
No impact on achievement 29.1 30.1Somewhat improved achievement 63.5 61.9
Greatly improved achievement 4.4 3.3
Total Number of Respondents 409 575
Perceived Impact on Student Performance on ISTEP+
• A majority (70.2% in Year 1 schools; 58.6%, Year 2+) of respondents indicated that they anticipated participation in Acuity will lead to somewhat improved student performance on the Spring 2010 ISTEP+
• 23.6%/36.0% stated that participation would have no impact on student ISTEP+ scores• Year 1 schools were more optimistic about Acuity’s impact on ISTEP+ scores• In the 2008-09 Spring Survey, 49% of respondents believed Acuity somewhat improved
ISTEP+ performance as compared to 59% of Year 2+ respondents this 2009-10 school year. This indicates an increase of 10 percentage points in respondents positive perception regarding the impact of Acuity on student ISTEP+ performance.
21
Percent of Respondents from Year 1 Schools
Percent of Respondents from Year 2+ Schools
Greatly decreased performance 0 0.5Somewhat decreased performance 1.2 3.0No impact on performance 23.6 36.0Somewhat improved performance 70.2 58.6Greatly improved performance 4.9 1.9
Total Number of Respondents 406 575
Impact on Instruction and/or Student Achievement: Year 1 Group
(Q 11) Why do you think classroom instruction and/or student achievement declined, improved, or did not change? Of the 296 written responses:
• For those who believed Acuity led to improvement in classroom instruction and/or student achievement, the largest number of respondents (44) cited that Acuity shows student strengths, weaknesses, and/or areas for improvement.
• For those who believed Acuity led to no change in classroom instruction and/or student achievement, the largest number of respondents (27) cited that the test’s impact could not yet be determined.
• For those who believed Acuity led to declined classroom instruction and/or student achievement, the largest number of respondents (25) indicated students were being tested on material they didn’t know yet.
22
23
(Q 13) Why do you think classroom instruction and/or student achievement declined, improved, or did not change? Of the 424 written responses:
• For those who believed Acuity led to improvement in classroom instruction and/or student achievement, the largest number of respondents (93) cited that teachers are better able to target teaching and instructional materials based on demonstrated student needs.
• For those who believed Acuity led to no change in classroom instruction and/or student achievement, the largest number of respondents (64) reported educators are not using all of the Acuity resources available.
• For those who believed Acuity led to declined classroom instruction and/or student achievement, the largest number of respondents (37) indicated students were not taking Acuity seriously because of boredom, frustration, or because they knew they were not being graded.
Impact on Instruction and/or Student Achievement: Year 2 Group
(Q 13) What was the most helpful component of the Acuity Assessment Program? (Year 1)
• Nearly half of respondents from Year 1 schools indicated Data and Instructional Resources were the most helpful component
• Teachers especially appreciated the Matrix Reports and having data available immediately
24
Most Helpful Program Components
Frequency PercentData and Instructional Resources 129 48.9
User friendliness for teachers 95 36.0
Reports 22 8.3Custom tests 14 5.3Predictive assessment 4 1.5
Total Number of Respondents 264
(Q 15) What was the most helpful component of the Acuity Assessment Program? (Year 2+)
• Acuity reports were frequently cited as the most helpful component (34.8%)
• 71 of 77 respondents who cited data mentioned using data to drive instruction
25
Most Helpful Program Components
Frequency Percent
Reports 139 34.8
Data 77 19.3
Instructional Resources 50 12.5User-friendly/aided teacher instruction 45 11.3
Preparation for ISTEP+ 37 9.3
Assessments 14 3.5
Alignment with Standards 9 2.3Breaking questions into standards; Blueprints 6 1.5
Measuring Student Progress
3
1.0
Total Number of Respondents 399
(Q 14) How can CTB/McGraw-Hill enhance the Acuity Assessment Program for future assessment
administrations? (Year 1)• The largest number of
respondents (34.3%) cited the need for further alignment of Acuity, ISTEP+ and curriculum pacing
• Content concerns included need for more questions per standard (especially in science and social studies) and less repetition of material
26
Suggested Program Improvements
Frequency Percent
Improve Instructional Resources 83 30.6
Improve content 42 15.5
Alignment with ISTEP+ and instruction 93 34.3
Better/more training and Professional Development 30 11.1
Make Acuity more student-friendly 18 6.6
Make Acuity more teacher-friendly 5 1.8
Total Number of Respondents
271
(Q 16) How can CTB/McGraw-Hill enhance the Acuity Assessment Program for future assessment
administrations? (Year 2+)• The largest number of
respondents (93 out of 426) indicated they would like improvements to Instructional Resources, especially ease of assigning and automatic assignment
• 15.3% of respondents were concerned with the difficulty of use for students, including technological concerns like scrolling in a window, aesthetics, and special needs accommodations
27
Suggested Program Improvements
Frequency PercentImprove Instructional Resources 93 21.8More student-friendly 65 15.3Content 44 10.3More teacher-friendly 37 8.7Improve reporting features 34 8.0Align Acuity with instruction and state standards 30 7.0
Improve the Instructional Resources 27 6.5
More/better training and Professional Development
26 6.1
Scheduling 26 6.1Alignment with ISTEP+ 18 4.2Improve Custom Tests 15 3.5Overload of testing 5 1.2General positive feedback 5 1.2No comment 23 5.4
Total Number of Respondents
426
B. Case Study Results: Site Visit Findings, Suggestions, and
Implications
28
Positive Educator Feedback
The program helps teachers identify skills and standards with which students struggle; many use it to help guide instruction
Teachers found the step-by-step approach for student use of the Instructional Resources extremely helpful
Teachers liked Acuity’s relation to ISTEP+, including comparability of questions and providing clear information to the student and teacher alike if the child is on target for passing ISTEP+
Teachers using the Diagnostic test appreciate aligning instruction with the test
Students are navigating the Predictive assessment system more easily now that they are used to it
29
Positive Educator Feedback (cont.)
30
Teachers reported that giving the Predictive assessment is fairly straightforward
Teachers who understand how to create and utilize reports are doing so, particularly batch reports, class rosters and Matrix reports
Educators are pleased with the response from the Help Desk
Teachers and students liked the clickers and said their use decreases cheating
The majority would continue using Acuity if the decision was theirs alone
Pressing IssuesTiming and Scheduling Acuity Predictive C is too close to the ISTEP+ window to be useful Time constraints impede feasibility of accessing reports and assigning Instructional
Resources, lab time, etc. Material students are being tested on is not aligned with the curriculum causing
frustration for both teachers and students
Instructional Resources Instructional Resources are complicated to assign and monitor Instructional Resources need more questions for each standard and variety Assigning Instructional Resources is time consuming to the point that negative
experiences are diminishing teacher use of the system
Professional Development Teachers would appreciate hands-on, differentiated training
31
Pressing Issues (cont.)
Student Experiences Acuity is not engaging for all students Vocabulary used is too difficult Constructed Response is too time consuming and seems pointless to
many since the score is not calculated into the final grade Acuity is not adaptable or appropriate for lower-ability students
Reports and Data Parent reports are needed that are easy to understand and readily
accessible
32
Educator Recommendations & Considerations
• Improving Student Experiences with Acuityo Teachers recommended that CTB/McGraw-Hill makes Acuity more visually appealing when students
receive their results; this may be a gateway to providing motivation to improve.o Students should be required to review items they answered incorrectly, and retests should be an
alternate form which tests the same content.o One suggestion for dealing with difficult vocabulary was to provide an easy definition of a word
which appears when a cursor scrolls over it.• Teaching Strategies
o Teachers would like to be able to personally arrange questions to more closely align with their schools’ curriculum pacing.
o The vast majority of teachers would like an auto assignment feature added to the system to assign Instructional Resources to students below a certain threshold on either the Diagnostic or Predictive assessments.
• Tips From Educatorso Educators would like an easily accessible parent report that can be pulled up during meetings. The
report should indicate areas in need of attention; should be easy for all parents to understand; and should help teachers indicate whether a child is expected to pass the ISTEP+.
33
V. Overall Summary of Qualitative Findings
The program helps teachers identify skills and standards with which students struggle allowing for instruction to better meet the needs of students
Teachers desire more training on using student and class data to inform and improve instruction
Educators expressed high levels of satisfaction with customer service and attentiveness by CTB
Teachers and students alike enjoy the clickers (student response devices)
Educators offered many suggestions for modification to Acuity; CTB appears to be listening
34
Summary of Qualitative Findings (cont.)
Some teachers expressed concern regarding the alignment between Acuity and the curriculum
Educators would like to see the initial assignment and reassignment of Instructional Resources made easier. They would also like to see an automatic tracking component of assigned Instructional Resources
Questions on the test as well as in the Instructional Resources need to be improved. This can be done by limiting repetition of material and ensuring adequate content is available for addressing each standard
The full impact of Acuity is yet to be determined, but more educators are becoming optimistic about the impact of Acuity on classroom instruction, student achievement in general, and on ISTEP+ in particular.
35
CEEP Contact Information
Terry E. Spradlin, MPAAssociate Director for Education [email protected]
Bridget SchleichGraduate Research [email protected]
1900 East Tenth StreetBloomington, Indiana 47406-7512812-855-4438Fax: 812-856-5890http://ceep.indiana.edu
36