Shanghai van aalst 040614

INTEGRATING INSTRUCTION & ASSESSMENT IN HIGHER EDUCATION

Dr. Jan van Aalst, [email protected]

Associate Professor

Assistant Dean (Research Centers & Proposals)

Faculty of Education

The University of Hong Kong

mailto:[email protected]

Overview• Problems and strategies• Outcomes-based instruction• Conclusions

2Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk

PROBLEMS AND STRATEGIES

Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk 3

10 problems with assessment1. Assessments not aligned with current learning theories

2. Assessments focus on isolated bits of content, not structured knowledge of the domain and thinking in the domain

3. Assessments not aligned with 21st century goals

4. Students do not know what they are expected to learn

5. Feedback does not help students to improve

6. Assessments of group projects confuse learning and effort and are unable to measure individual learning

7. Difficult to know if it really is the student’s work

8. Assessments of creative work lacks objectivity

9. Designing and evaluating valid assessment tasks is difficult

10. Assessing is stressful and time-consuming


Changing views of assessment

Knowledge Building @ HKU - kbc2.edu.hku.hk 5

Laurie Shepard (2000). The role of assessment in a learning culture. Educational Researcher, 29 (No. 7), pp. 4-14.

Wednesday, 4 June, 14

(Problem 1)

Shepard 2000Old paradigm


(Problem 1)

Shepard 2000New paradigm


(Problem 1)

Bits of contentWhy Undergraduates Leave the Sciences• Cumulative—limited

opportunity to experience success

• Lack of opportunity to see the big picture


(Problem 2)

Tobias, S. (1990). They’re not dumb, they’re different. Stalking the second tier. Arizona Research Corporation.

Seymour, E., & Hewitt, N. M. (1997). Talking about leaving: Why undergraduates leave the sciences. Boulder, CO: Westview Press.

SOLO taxonomyJohn Biggs• Structured Observed

Learning Outcome• For complex

performances such as writing essays

• Goes beyond isolated facts to integrating the elements


(Problem 2)

HKU educational aims

1. Pursuit of academic/professional excellence, critical intellectual inquiry, and lifelong learning

2. Tackling novel situations and ill-defined problems

3. Critical self-reflection, greater understanding of others, and upholding personal and professional ethics

4. Intercultural understanding and global citizenship

5. Communication and collaboration

6. Leadership and advocacy for the improvement of the human condition


(Problem 3)

Understanding by design

• Primary goal of education is to develop student understanding

• Students reveal their understanding most effectively when they are provided complex, authentic opportunities to explain, interpret, apply, and self-assess

• Effective curriculum design is backward process—long-term goals and assessment are considered before activities and instruction

• Student performance is achieved through regular review of results followed by targeted adjustments to curriculum and instruction. Instructors become most effective when they seek feedback from students and their peers and use that feedback to adjust approaches to design and teaching

• Grant Wiggins & Jay McTighe (1998)

• Backward planning framework


(Problem 4)

Constructive alignmentJohn Biggs• Start with the

intended learning outcomes

• Align teaching and assessment to them


(Problem 4)

Constructive alignmentJohn Biggs• Start with the

intended learning outcomes

• Align teaching and assessment to them


(Problem 4)

Formative assessment• The use of feedback

to improve the learning process, while learning is still ongoing

• Depends on formative feedback, which is based on the gap between current performance and the desired performance

• One of the largest effects on achievement in educational research—0.9 standard deviation (meta-analysis)


Effect size: 31% more students would pass (40 instead of 25 would have a score > 50%)

Problems with feedback• Students don’t

understand it• Students don’t use it

to improve performance

• Critical feedback attached to a grade can be demotivating

How can we give better feedback?

(Problems 4-5)

Theory of formative assessment• Three aspects: (a) determine where students are, (b)

where they need to go, and (c) how they can get there• Five strategies

• Clarifying and sharing learning intentions and conditions for success

• Engineering effective classroom discussions and other activities that elicit evidence of student understanding

• Provide feedback that moves students forward• Activating students as instructional resources for one another• Activating students as owners of their own learning

15Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk

Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5-31.

(Problems 4-5)

Feedback in formative assessment• Avoid quantifying the

feedback• Say what’s been

done well• Say what can be

improved & explain how this can be done

• Say what you don’t understand

• Suggest ways to develop it


Assessment conversations Construct shared understanding of the performance, the feedback, and how to make use of the feedback

(Problems 4-5)

Formative assessment in English language learning

Experimental course on business communication

• Not traditional course on college English (grammar, etc.)

• But course on business communication in English (letters, memos, etc.)

• Problem-based instruction

Dr. Ke Zhou CoCo (in progress)


(Problem 5)

Assess, assess …• Students’ perception of over-assessing

• “We had two major tests in the college English course: midterm and final test, both of which were mainly on vocabulary, grammar, listening, reading and writing. No need to prepare as my English is good enough. … However, it now becomes troublesome with so many assessed tasks. Besides, it is hardly to imagine how linguistic knowledge and language skills are assessed in form of case-study writing”

• Raises interesting issues• Who decides whether a student needs formative assessment?

Should the students initiate the process?• Formative assessment still stands apart from the learning process,

but it should be part of the learning process—it’s one of the things we do to learn.


(Problem 5)

Making sense of rubrics• Rubric-based feedback only becomes meaningful when

students try to make sense of it together in an attempt to revise their work• “Surprisingly, in this course the tutor had provided us with the

rubrics before we started writing. At first, I glanced it over before completing the first draft. However, my understanding of rubrics was only substantially deepened in revising process when I have to address the problems identified in teacher’s written feedback.”

• “Although our tutor explained the rubrics, I never read it in first draft writing. Why? Probably because I may not fully understand the abstract points. Out understanding of the rubrics develops when we assessed others’ writing and reflected on our own.”


(Problem 5)

Knowledge Building @ HKU - kbc2.edu.hku.hk 20

Peer Instruction• Students study

relevant material from text before class

• Mini-lectures based on difficult material (~10 min)

• Conceptual questions that students answer and then ask with peers, during the lecture

Wednesday, 4 June, 14

Eric Mazur et al.Harvard University

(Problem 5)

Assessment triangleNational Research Council (2001). Knowing what Students Know.


Observation Interpretation

Cognition

(Problems 6-8)

Assessment triangleNational Research Council (2001). Knowing what Students Know.


Observation Interpretation

• What is cognitively involved in desired performance?

• Also metacognition• Need to understand the

process by which students learn it

• What does desired performance look like?

• How can we measure it?

• What kinds of inferences can be made from what has been observed?

• The right data for the right inference?

Cognition

(Problems 6-8)

Assessing online discussions• Discussions are

shared (collective) achievements

• But they require everyone to contribute

• Studies show that students who participate more also achieve more

• But participation ≠ learning

• Conceptually problematic to evaluate quantity & quality of individual notes


(Problems 6-8)

Assessing online discussions• Major problems with

unequal participation• Many instructors

give marks for # notes created and read by a student, or on quality

• Heavy workload for instructor

• Better approach: evaluate the discussion as a whole—does it achieve its goals?


A B C D E0

100

200

300

400

500

1st quartile

2nd quartile

3rd quartile

4th quartile

Class

Med

ian

No

tes

Rea

d p

er S

tud

ent

(Problems 6-8)

Discussion as shared achievement


Knowledge Connections Analyzer

Discussion portfolios• Principles guiding students’ review

• Working at the cutting edge• Collaborative effort• Constructive use of authoritative sources• Progressive problem solving• Identifying high points

• Initial assessment design• Students review the database to evaluate evidence of the principles• They do this alone or in teams, but review the notes of the whole

class• They write a portfolio note: narrative + links to source notes

• Results from high school classes (Lee et al. 2006)• Discussion < discussion + portfolios < discussion + principles +

portfolios on essays and conceptual questions (Geography)


(Problems 6-8)

Assessing online discussions • Integrate content

across contributions, even across discussions

• Reflect on the community’s performance

• Set goals for further development

• Metacognitive -> deeper insight into domain

• Instructor evaluates portfolios rather than individual contributions


(Problems 2, 6-8)

Discussion portfolio

van Aalst, J., & Chan, C. K. K. (2007). Student-directed assessment of knowledge building using electronic portfolios. Journal of the Learning Sciences, 16, 175-220.

Sample portfolioFrom Dr. Larry Lei Chunlin

Shanghai International University of Business and Economics• Introduction to

Business course (n = 60)

• Several group-based portfolios during the course


(Problems 2, 6-8)

Performance levels

Low Students might think it is fine to reproduce what others think and draw some notes together to finish the task.

Medium Students might want to understand what other people say and perhaps to join in the discussion and enrich their personal minds.

High Students demonstrate higher order abilities to synthesize, evaluate and assume collectives responsibility; the discourse is geared to address collective knowledge gaps and dedicated to something new and meaningful for the community.


(Problems 2, 6-8)

Dr. Lei’s Explanation

“By composing e-portfolios, students self-evaluated their KF notes and selected best clusters of notes to exemplify the knowledge growth in their learning community. E-portfolio played a duel role of both charactering and scaffolding collaborative knowledge. The qualitative and quantitative analysis of students’ reflective strategies in conducting e-portfolios suggested that we need not only to encourage students to work more on KF, but more importantly and specifically, we need to facilitate students in more meaningful discourse moves such as constructive use of information, meta-cognition, meta-discourse, emergent question and theory building.”


(Problems 2, 6-8)

Group projectsThere’s no doubt—people working together accomplish more!


(Problems 6-8)

Problems with group projects• Differences in

interest, goals• Differences in prior

knowledge• Differences in group

dynamics• Lack of skills to deal

with differences


(Problems 6-8)

Assessing group projects

Presentation of a case study (team of 4 students)

• Assess the group’s overall achievement

• Assess achievement of individual student during project

• Assess learning appropriated by each student


• Project management (Student A)• Studying the case report (all students)• Reviewing and commenting on all presentation

materials (all students)• Presenting the background (Student A)• Presenting the case study methods (Student B)• Presenting the main findings (Student C)• Presenting the conclusions (Student D)• Reflections on the case (all students)• Facilitating the discussion (all students)

How do we assess it? (All students learn different things.)Easier to do if based on many experiences, not just one.

(Problems 6-8)

Assessing group projects

Collaborative research project

• Assess the group’s overall achievement

• Assess achievement of individual student during project

• Assess learning appropriated by each student


• Project management (Student A)• Initial design concept and discussion)• Updating shared design concept (Student B)• Planning of work (all students)• Assignment of tasks (Student A)• Execution of various tasks (all students)• Peer review of tasks (one other student per task)• Preparation of report (all students)• Improvement of report (all students)• Reflection on project (all students)

How do we assess it? (All students learn different things.)• Keep a trace of how the project develops• Peer assessment of important contributions

(Problems 6-8)

OUTCOMES-BASED INSTRUCTION


Designing a program• A program must

achieve at least some university aims

• Course outcomes must be relevant to at least some program outcomes

• Taken together, the courses must achieve all program outcomes

• Sequential courses could achieve the same program outcome at more advanced levels (learning progression)


Program-level Outcomes

UNIVERSITY AIMS

Course 1 outcomes

Course 2 outcomes

Course 3 outcomes

37

DPLE curriculum

Program outcomes

1. Understanding the learning sciences as a field and a foundation for educational design

2. Locate and evaluate research relevant to a decision; develop a design proposal

3. Awareness of trends and tools in educational technologies; skill in using them; skill in carrying out lessons that use them

4. Ability to conceive, plan, carry out, and evaluate educational designs

University aims

• Pursuit of academic/professional excellence, critical intellectual inquiry, and lifelong learning

• Tackling novel situations and ill-defined problems

• Critical self-reflection, greater understanding of others, and upholding personal and professional ethics

• Intercultural understanding and global citizenship

• Communication and collaboration• Leadership and advocacy for the

improvement of the human condition

Wednesday, 4 June, 14 Knowledge Building @ HKU - kbc2.edu.hku.hk

38

DPLE curriculum

Program outcomes

1. Understanding the learning sciences as a field and a foundation for educational design

2. Locate and evaluate research relevant to a decision; develop a design proposal

3. Awareness of trends and tools in educational technologies; skill in using them; skill in carrying out lessons that use them

4. Ability to conceive, plan, carry out, and evaluate educational designs

Core courses

• The Learning Sciences• Research Methods• Learning in the Disciplines• Knowledge Building• Teaching & Learning with

Technology• Project or Dissertation• Electives


Course outcomes

Research Methods outcomes

• Ability to identify, locate, and interpret research in relation to a question

• Ability to design a research or design study

Learning Sciences outcomes

• Understanding the learning sciences as a field of inquiry

• Familiarity with the most common instructional approaches generated by the learning sciences

• Ability to implement and evaluate learning strategies in classrooms


Instructional approach• Technology rich

environment• Strong sense of

community• Little lecturing on

materials students are studying

• Short teacher presentations

• Frequent online and offline discussions

• Frequent peer presentations

• Emphasis on design• Program portfolio


41

PBL and English writing – new immigrants

• Background• Hong Kong has many immigrant children who need to learn English to

function in school. But courses provided for such students concentrate on grammar and involve little writing

• The project• This teacher has developed an after-school program that uses PBL

methods. Authentic problems from school life provide the problems that students discuss and write about. The course consists of 8 sessions of 3 hours each. Students developed their ability to express their ideas in English

• Learning sciences & ICT• Implements a method from the learning sciences, but does not use ICT.

It could be enhanced by using tools for collaborative writing

• Impact• Addresses an important need and provides a prototypical curriculum


DPLE portfolio rubric


43

DPLE portfolio• Using Google apps• Reflection and self-

assessment of 4 areas of performance

• Just getting started with them


DPLE course design


DPLE teaching


Key differences• All results very good but Research Methods substantially

higher on• Expectation to achieve learning outcomes• Course helped to achieve learning outcomes• Assessment methods relevant to learning outcomes• Assessment standards were clear


Key differences• All results very good but Research Methods substantially

higher on• Expectation to achieve learning outcomes• Course helped to achieve learning outcomes• Assessment methods relevant to learning outcomes• Assessment standards were clear


What were the differences in assessment?• Research methods: Research proposal, and several

formative assessments before that (problem statement, different analysis tasks)

• Learning sciences: More flexible—literature review or paper that conceptualizes goals for the rest of program, a formative assessment to check understanding of key concepts

Improving alignment to outcomes• Introduce what students will need to achieve early in the

course rather than at the end (recall: understanding by design)

• Provide milestones along the way that establish progress toward the outcome

• The learning sciences course was interesting and inspired students to learn, but it was less clear what they needed to achieve


CONCLUSIONS


Overall approach

1. Focus on program-level outcomes, articulated with university aims

2. Progressively more sophisticated performances on program-level outcomes as students advance through program (attainment of course outcomes)

3. Focus on program outcomes -> more coherent instruction and shared mission between instructors

4. Program-level portfolio to track and guide progress of each student within and across courses

5. Much more emphasis on informal formative assessments that integrate assessment with learning—assessment becomes part of learning process

6. Much more student agency in informal assessments


Some benefits of informal assessments

• Less stressful for instructor, more about teaching• Many process skills can only be assessed informally and

over long periods• Note reading and writing• Social structure of networks• Management skills, collaboration skills, presentation skills

• Learning environment becomes less stressful for students, active engagement because it’s more fun to spend time that way

• Opportunities to build onto informal assessments: students’ insight into their development and the domain

• More in line with the “educational data” movement (uses much more evidence of learning and development)



Thank You!

http://web.hku.hk/~vanaalst

Van Aalst, J. (2013). Assessment in collaborative learning. In C. E. Hmelo-Silver et al. (Eds.), International handbook of collaborative learning (pp. 280-296). New York: Routledge.

http://web.hku.hk/~vanaalst

Date post:	27-May-2015
Category:	Education
Upload:	jan-van-aalst
View:	108 times
Download:	1 times