Date post: | 18-Dec-2015 |
Category: |
Documents |
Upload: | allen-fowler |
View: | 213 times |
Download: | 0 times |
C R E S S T / U C L A
UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation
National Center for Research on Evaluation, Standards, and Student Testing
Evaluating Creative Teaching Practices and Collaborative Learning Environments
Ann Mastergeorge and Ingrid Roberson
Annual AERA ConventionApril 1, 2002 New Orleans, LA
C R E S S T / U C L A
Overview
The Construct of Engagement in School Reform
Methodological Issues in Measuring Engagement in School Reform
Discussion and Implications of Engagement for Classroom Practice
C R E S S T / U C L A
Introductionto Creative Learning
Communities
Supports school reform in schools across the country serving at-risk student populations
Promotes change centered on creative teaching practices and collaborative learning environments
Emphasizes the enhancement in students’ joy of learning in engaged learning environments
C R E S S T / U C L A
Procedural Engagement
Focuses on student on-task behavior in learning activities
Emphasizes student concentration and completion of learning activities
Substantive Engagement
Focuses on the emotional quality of student’s involvement in learning activities
Emphasizes student’s enjoyment of learning activities since it is self-motivating and self-reinforcing
The Construct of Engagementin School Reform
C R E S S T / U C L A
Developing A Dynamic Model of Engagement
Substantive Engagement depends on both student and teacher engagement in classroom learning activities
Includes the social dimension of learning
Requires a high degree of reciprocity
Learning Environments and Teaching Practices construct opportunities for students to experience engagement
Communicates support and care
Allows for student choice and control
Models enthusiasm for learning
C R E S S T / U C L A
Student Engagement
Teacher Engagement
Creative Learning
Community
Learning links
Teaching links
Shifts in attitude
Shifts in attitude
Classroom Collaboration
Learning Communities
Teaching Communities
C R E S S T / U C L A
Methodological Issues in Measuring Engagement in School Reform
Multiple Methods of Evaluation
Utilizes quantitative and qualitative approaches
Implementation Process of School Reform
Impact of School Reform on Engagement
Employs a theory of action
Provides focal points for the evaluation
Suggests a sequence in which various effects may be expected
C R E S S T / U C L A
Anchor incurriculum, instruction
& assessment
Attend to cultural and structural change
Providematerials
Creative Teaching Practice
Professional Development &Accountability
School Environment
Leadership & Goals
Facilitate change
Mobilize change
Build a shared vision
Connect to student learning
Provide Financial Support
ProvideTechnical Support Embed in
school culture
Build a community
Attend to changingattitudes, beliefs, & values
Promote and provide time for on-going professional inquiry
Embed in classroom practice
Provide opportunities for teacher collaboration
Align withproject goals
Provide the type and timely information for
project adjustmentsDevelop measures
aligned with creative teaching practices
Align withstandards
Link to curriculum
Align with assessments (standardized and performance)
Shift teacher attitudes, beliefs, and
expectationsIdentify
innovative practices Embed in school cultureEngaged Student
LearningPromote the “joy of learning” through exploration, collaboration, and interpretation
Improve student attitudes towards learning
C R E S S T / U C L A
Measuring the Implementation Process
Features Five Dimensions of School Reform Framework
Goals
Leadership
School Environment (Culture and Structures)
Professional Development
Accountability
Focuses on the implementation of creative teaching practices
Teacher Survey
Teacher Interview
Classroom Practice Observation Protocol
C R E S S T / U C L A
Measuring the Impact on Engagement
Features both Procedural and Substantive Engagement Theory
Focuses on Student Engagement
Teacher Survey
Teacher Interview
Student Survey
Classroom Practice Observation Protocol
C R E S S T / U C L A
Classroom Practice Observation Protocol
Sensitive to initial changes in teaching practices
Appropriate for innovative and creative teaching practices
Captures students’ attention to and enjoyment of learning activities
C R E S S T / U C L A
Dimensions of Classroom Practice
Challenge of Lesson Activity
Implementation of Lesson Activity
Procedural Engagement
Substantive Engagement
Quality of Instructional Discussion
Quality of Instructional Feedback
C R E S S T / U C L A
Implementation of the Learning Activities
Notappli-cableto thisactivity
Challenge of the Lesson Activities
1 2 3 4 Learning activitiesinvolve students intasks that do notrequire any degreeof complex thinkingand do not engagestudents withsubstantive contentmaterial.
(Little or nocomplex thinking)
Example:Using encyclopedias,students record thename of medicinalplants, its scientificname, where it ismost commonlygrown, how tall itcan grow, and how itcan be usedmedicinally onto aworksheet.
Learning activitiesinvolve students intasks that requiresome moderatelycomplex thinking.Student’sengagement withthe material doesnot promotelearning in arigorous way.
(Some complexthinking)
Example:Students recordinformation fromencyclopedias to aworksheet and at theend, answer thefollowing question,"What was the mostsurprising thing youlearned aboutmedicinal plants?”
At least some of thelearning activities requirestrongly complex thinkingas a major focus of thelesson. This level requiresthe use of higher ordercognitive functions, takingstudents beyond recall,recognition, andreproduction ofinformation to evaluation;analysis; synthesis; andproduction of arguments,ideas, and performances.Students may be asked tosynthesize ideas; analyzecause and effect; identifya problem and posereasonable solutions;hypothesize, speculategiving details orjustification; defendopinions or argue aposition with evidence;evaluate; analyze; ordetermine bias, values,intent.
(Mostly high orderthinking)
Example:Students research amedicinal plant, choosingon their own what theythink is importantinformation about theplant. They then createan informationalpamphlet about the plantusing that information.
Much or all of the learningactivities require stronglycomplex thinking as a majorfocus of the lesson. Studentsalso engage in substantivecontent material. Studentsmay be asked to analyzecause and effect, identify aproblem and posereasonable solutions,speculate giving details orjustification, defendopinions or argue a positionwith evidence to a greatextent.
(Almost all complexthinking)
Example:Students research severalmedicinal plants and writea report selecting a fewplants that may be useful totheir families, using specificdetails about the plant tosupport thier choices. Thereport is used to make apresentation to the classabout why the plants theychose are useful.
NA
C R E S S T / U C L A
Challenge of Lesson Activity Exemplar
Teacher: Let’s think about all of the different conditions.
Student: My experiment needs more water.
Teacher: I heard someone else say too cold… so cold or warm.
Student: Ours is darker and theirs is lighter [amount of light] .
Teacher: Can you think of anything else?
Student: I think we should put less water and less worms.. with theworms they will just dry, will shrivel and do nothing and the water will just get everything damp.
Teacher: What do you think is the most important thing to make theleaves decompose?
Student: Dry, the leaves crumble up. It’s like they turn into dirt.
Teacher: Maybe one thing we can do this week, half put water and the other half put no water and keep everything else the same. (CO.10.08.YR2)
C R E S S T / U C L A
Implementation of Lesson Activity Exemplar
After reviewing both the proof reading process and scoring rubric (both clearly written on chart paper), teacher runs through following example.
Dear Curator:I really love the ring of the bell. I also thought the water wheel was cool.Finally, the machines were interesting to see.Thank You, Ms. X
Teacher: Alright, so this letter… Based on that, where would you putthem if you were the teacher?
Student: 2, though it does have an introduction.
Student: 2, it doesn’t give them a sense and is supposed to be in paragraphs.
Teacher: Anybody agree or disagree and why?
Student: It looks like a letter but it doesn’t sound like a letter (CO.01.03.YR2)
C R E S S T / U C L A
1 2 3 4 10-29% of thestudents appearto be procedurally engaged.
Example:Few studentsconduct their lightbulb experimentswhile the rest of thestudents wanderaround the room,chatting with otherstudents aboutunrelated topics orplay with eachother.
NA
Notappli-cableto thisactivity
30-59% of thestudents appearto be procedurally engaged.
Example:Some studentsconduct their scienceexperiment but abouthalf of the classwanders around theroom or sits at theirdesks chatting withother students aboutunrelated topics orstares out into spaceday dreaming,
60-79% of thestudents appearto be procedurally engaged.
Example:Most students areon-task, conductingtheir experiment.They write theirscience notes,detailing their stepsand results. Moststudents complete theexperiment and mostcomplete or nearlycomplete theirscience notes.
80-100%of thestudents appearto be procedurally engaged.
Example:Almost all students areon-task, trying to get thelight bulb to work, usingdifferent metal wires.They note theirobservations in theirscience journal. Moststudents complete both theexperiment and theirwriting-up of the science
experiment in theirjournal.
Approximately what percent of procedurally engaged?
Procedural Engagement
C R E S S T / U C L A
Procedural Engagement Exemplar
All students are on-task, attentive to the teacher. Some students raise their hands to answer teacher questions.
Teacher: Is this a sentence?
Student: Yes.
Teacher: So what does it have to have at the beginning?
Student: A capital letter.
Teacher: What does it have to have at the end?
Student: A question mark.
Teacher: What do I have to do to show these words came out of my mouth? [Three students raise their hand]...come up to the board X.
Student goes to board and puts quotation marks around the sentence.(CO.02.01.YR2)
C R E S S T / U C L A
NA
Notappli-cableto thisactivity
10-29% of thestudents appearto be substantively engaged.
Example:Most students areeither not doing theexperiment, playingor talking with otherstudents or ifconducting theexperiment, they arerushing through it,being careless,paying more attentionto unrelated talk.
30-59% of thestudents appearto be substantively engaged.
Example:About half of thestudent are notactively engaged,even if they areconducting theexperiment, theirattention is focusedelsewhere, listening ortalking to otherstudents, playing withtheir pencils, lookingoutside the window.
60-79% of thestudents appearto be substantivelyengaged.
Example:Most student are activelyengaged. Some studentstalk excitedly with theirneighbors saying, "Didyou see that?, That ispretty neat; Did yours dothat?" Other students,smile to themselves astheir light bulbs lights up,while others still tryingscrunch up their faces indeep concentration.
80-100% of thestudents appearto be substantively engaged.
Example:Almost all student are activelyengaged. Many students shout,"It works!, I did it! or Its alive!"when their light bulb works. Somestudents smile, their eyeswidening, as their light bulb goeson. Other students still tryingreview their notes or ask anotherstudent, “Hey, can you help me?”Near the end, students can beheard talking with each otherabout which metal worked better
as a conducter.
Approximately what percent of students are substantively engaged?
Substantive Engagement
1 2 3 4
C R E S S T / U C L A
Substantive Engagement Exemplar
The students are all listening. They are riveted, looking at the teacher and the book intently. The teacher engages them in a discussion with 90% of the students participating. Students’ hands shoot up in the air waving back and forth frantically with wiggling fingers, going "oh, oh”.
Teacher: What woke him up?
Student: The sun.
Teacher: X, what wakes you up?
Student: My mom.
Student: Sometimes my dog comes in and licks me.
Student: Sometimes my brother.
Student: My mom’s boyfriend hits me with a pillow.
Student: My dad gets me up.
Teacher: So our caterpillar comes out of the egg and how did hefeel? He was what?
Students: HUNGRY!!! (CO.02.03.YR2)
C R E S S T / U C L A
NA
Notappli-cableto thisactivity
1 2 3 4 A discussion doesnot take place at allor interaction betweenteacher and students ispredominantlyrecitation style, withteacher mediating allquestions and answers.The teacher’s questionsare close-ended,known-answerquestions.
(No discussion/Teacher guided, Q&A.)
Example:Teacher asks, “16divided by 4 is?” “4,”answers studentscollectively.
Teacher makes someattempt to engagestudents in truediscussion with unevenresults. Some of theteacher’s questions areopen-ended. Theremay be some attemptto have studentsrespond to otherstudents or invitationsto comment about abook. Teacher’sattempt at engagingstudents may fail.
(Teacher guided Q&Awith few open-endedquestions.)
Example:Teacher asks, " I have16 apples I want toshare with my fourfriends. Do I haveenough?" "Yes", thestudents respondcollectively. “Howmany would eachfriend get?” “4,”answer the studentstogether.
Most of teacher’squestions are of highquality. Adequate timeis available for studentsto respond and teacheractivity solicits studentinput (e.g., "Tell mewhy you think that.""Can you say a littlemore about that?").Teacher builds onstudent contributions.There is some student tostudent discussionregarding thesubject/classroomactivity.
(High qualityopen-ended questionswith few Q & As.)
Example:Teacher says, "I have 16apples that I want toshare with four friends,how many would eachfriend get?" Teacherwaits a few minutes andthen says, "Who can tellme the answer and howyou figured it out?Student responds, "Idrew four circles andthen passed out anapple to each circleuntil they were allpassed out."
Quality of Instructional Discussion
Classroom interactionrepresents true discussion.Students initiate topics andmake unsolicited, on-topiccontributions. Studentsformulate many questions.Teacher’s questions areuniformly high quality withadequate time for students torespond. Teacher builds onstudents’ contributions, andstudents build on each other’scontributions. Student tostudent discussion about thesubject/classroom activity issignificant.
(Discussion betweenstudents and teachersand/or among students.)
Example:The student called uponresponds, "I drew four circlesand then passed out an appleto each circle until they wereall passed out." Anotherstudent calls out, "I did itanother way". Teacher buildson this comment by saying,“Let’s discuss differentstrategies used."Another student says, “I knew4 plus 4 is 8 and 8 plus 8 is16, so I counted by fours.There were 4 fours.“
C R E S S T / U C L A
Quality of Instructional Discussion Exemplar
At one table, there are three rather lively boys and as each of them works on drawing two parallel lines the following discussion ensues:
Student 2: “That’s parallel,” referring to the littlest one’s two parallel
lines.
Student 4: “Can I make it parallel?”
Student 2: “That is parallel. It doesn’t matter the size of the other
line, if they don’t intersect, then they’re parallel.”
Student 1: “If they don’t intersect, they’re parallel.”
The little one is still unconvinced by the two boys, looking at his board with a confused expression. [Student 2] continues to teach the other student the concept of parallel lines, emphasizing the word intersect and then upon seeing the continued state of confusion lets out a loud sigh and starts again. He explains that the length of the line segment or how close they are doesn’t matter as long as they don’t intersect.
(CO.05.03.YR2)
C R E S S T / U C L A
Feedback is uniformly highquality. Provision is madefor students to usefeedback in their learning.Feedback fully supports theattainment of theinstructional goals.Feedback is provided foralmost all studentsdemonstratingneed.
(Uniformly high quality)
Example:While the teacher rovesaround the room helpingalmost all students, shesays, "John, you used agreat adjective becauseyou described how a pillbug looks. Now can youuse an adjective thatdescribes how it feels?"
Feedback is mostlyhigh quality.Feedback mostlysupports theinstructional goals.It is provided eitherconsistentlythroughout theobservation periodor in a focused wayduring a portion ofthe period.Feedback isprovided for moststudentsdemonstratingneed.
(Mostly highquality)
Example:Teacher says, “I likeyour sentence.Bumpy is a gooddescribing word.”Helping moststudent with theirwriting, heoccasionally says,"good job or needsmore work.”
1 2 3 4 Feedback is eithernot provided or is ofuniformly poorquality. Feedbackmay beinappropriate.Feedback does notsupport instructionalgoals. Feedback isprovided only for afew studentsdemonstrating need.
(Not provided ormostly poor quality)
Example:First grade studentsare working on awriting assignmentwith the sentencestarter "A pill bugis…" Teacher sits athis desk whilestudents attempt todo the assignment.
Feedback isinconsistent inquality. Elements ofhigh quality may bepresent during asmall portion of theobservation orminimallyinformative feedbackthat only somewhatsupports theinstructional goalsmay be giventhroughout theobservation.Feedback is providedfor some studentsdemonstrating need.
(Inconsistent inquality)
Example:Teacher rovesaround the roomsaying to somestudents "good job,nice work, or youneed to start writing,this is not anadjective".
NANotappli-cableto thisactivity
Quality of Instructional Feedback
C R E S S T / U C L A
Quality of Instructional Feedback Exemplar
Teacher: “Did you measure first? If you’re going to do me, put my name in, and then you’ll have to do an estimate.”
Both students quickly go to their papers to write down their estimates on a small piece of paper.
Teacher: “Now this is in meters,” upon seeing one student write down 5.
The student nods knowingly but continues to write down the number 5.
Teacher: “Do you think I’m 5 meters high?”
He looks at her questioningly, not understanding. So she takes the yardstick and starts to raise it slowly.
Teacher: “One meter.” Raises it a meter. “Two meters.” Raises it again. “Three meters.”
Student: “Oh, two!” He vigorously erases the number five and writes down the number two. (CO.06.01.YR2)
C R E S S T / U C L A
Comparison of All and Matched Mean Ratings
Domain 2000All Scores
2001All Scores
AllMeanChange
2000MatchedScores
2001MatchedScores
MatchedMeanChange
Challenge ofLesson Activity
2.42 2.46 .02 2.21 2.43 .22
Implementationof LessonActivity
2.88 2.86 -.02 2.54 2.91 .37
ProceduralEngagement
3.20 3.45 .25 2.96 3.52 .56
SubstantiveEngagement
2.32 2.86 .54 2.13 2.91 .78
Quality ofInstructionalDiscussion
2.20 2.49 .29 2.17 2.22 .05
Quality ofInstructionalFeedback
2.46 2.44 -.02 2.33 2.43 .10
C R E S S T / U C L A
Comparison of Year 1 and Year 2 mean ratings by domain across all case study schools
C R E S S T / U C L A
Comparison of Year 1 and Year 2 matched mean ratings by domain across all case study schools
C R E S S T / U C L A
Discussion and Implications for Classroom Practice
Documenting Shifts in Classroom Practice from Year 1 to Year 2
Issues in Accountability and Sustainability of School Reform
Re-focusing, Re-thinking, and Re-tooling Measures of Classroom Practice
C R E S S T / U C L A
Documenting Shifts in Classroom Practice
Overall, small effect on Teacher Practice (Effect Size of .33) and Student Engagement (Effect Size of .201)
Between Schools, difference in degree of implementation of creative practices
Professional development linked to promoted teaching practices
Assessments to monitor progress
C R E S S T / U C L A
Issues in Accountability and Sustainability in School Reform
Combining and creating assessments that are accessible for progress monitoring
Improvements
Inquiry
Creating and sustaining supportive infrastructures to sustain school reform
Leadership
Community Culture
C R E S S T / U C L A
Re-focusing, Re-thinking, and Re-tooling
Measures of Classroom Practice
Re-focus efforts to include realistic expectations for reform implementation
Re-think the product and outcome model of classroom measures: measures should reflect process in action
Re-tool measures to reflect conceptual and procedural realities of classroom practice
C R E S S T / U C L A
Paper will be posted on the UCLA CSE/CRESST website:
www.cse.ucla.edu
Ann Mastergeorge
Phone 530-754-9629
E-mail [email protected]
Ingrid Roberson
Phone 310-794-4404
E-mail [email protected]