STREAMLINING PROGRAM LEARNING OBJECTIVE ASSESSMENT WITH TARGETED ACTIVITIES AND DESCRIPTIVE RUBRICS Carrie Menke, Ph.D. Physics, LPSOE University of California, Merced

Abstract •  The undergraduate physics program has five program learning

objectives (PLOs): (1) physical principles, (2) mathematical expertise, (3) experimental technique, (4) communication and teamwork, and (5) research proficiency. With one PLO assessed each year, we have just completed our first cycle. Our approach strives to maximize the ease and applicability of our assessment practices while maintaining faculty's flexibility in course design and delivery. Objectives are mapped onto the core curriculum and identified coursework is collected as direct evidence. We have found that descriptive rubrics lend themselves to more efficient assessment, higher inter-rater reliability, and can be applied to course and program-level assessment. The presentation will outline our progress, success with a descriptive presentation rubric, and our ongoing work with rubrics applied to more abstract PLOs that utilize students' written work.

2 Streamlining Program Assessment Research Week 2014

Outline • Assessment Cycle • Physics PLOs • Curriculum Matrix & Signature Assignments • Rubrics

•  Developing: Types & Sources •  Refining: Inter-rater reliability •  Applying: Course and Program Assessment

• Successes • Challenges

3 Streamlining Program Assessment Research Week 2014

Assessment Cycle

4

Hybrid of Suskie, CIRTL Network, Wiggins & McTighe

Laura Martin, Introduction to Assessment Presentation, CGS Workshop, January 2013

Streamlining Program Assessment Research Week 2014

Physics PLOs 1.  Physical Principles. Students will be able to apply basic physical principles—including

classical mechanics, electricity and magnetism, quantum mechanics, and statistical mechanics—to explain, analyze, and predict a variety of natural phenomena.

2.  Mathematical Expertise. Students will be able to translate physical concepts into

mathematical language. Furthermore students will be able to apply advanced mathematical techniques (e.g., calculus, linear algebra, probability, and statistics) in their explanations, analyses, and predictions of physical phenomena.

3.  Experimental Techniques. Students will be able to take physical measurements in an

experimental laboratory setting and analyze these results to draw conclusions about the physical system under investigation, including whether their data supports or refutes a given physical model.

4.  Communication and Teamwork Skills. Students will be able to clearly explain their

mathematical and physical reasoning, both orally and in writing, and will be able to communicate and work effectively in groups on a common project.

5.  Research Proficiency. Students will be able to formulate personal research questions

that expand their knowledge of physics. Students will be able to apply sound scientific research methods to address these questions, either by researching the current literature or developing independent results.

5 Streamlining Program Assessment Research Week 2014

Curriculum Matrix: Courses & PLOs

6

Physics  Core  Courses  

Year   Course  Title  

Program  Learning  Objec9ves  Signature  Assignment  or    Indirect  Evidence  Support  

1   2   3   4   5  Physical    Principles  

Mathema9cal  Exper9se  

Experimental  Techniques  

Communica9on  &  Teamwork  

Research  Proficiency  

1   Introductory  I   I   I   I   W,  T:  I   I  1   Introductory  II   I   I   I   W,  T:  I   I  2   Introductory  III   I   I   R   W,  T:  R   R  

2   Classical  Mechanics   R   R,  A   W:  R,  A  O:  I   R,  A   PLO  2:  Final  exam:  quanBtaBve  

PLO  4,  5:  Literature  review  2  –  3   Thermodynamics   R,  A   R   PLO  1:  Final  exam:  conceptual  3  –  4   Electrodynamics   R   R,  A   PLO2:  Final  exam:  quanBtaBve  3  –  4   Modern  Physics  Lab   R   R,  A   W,  O,  T:  R   R   PLO  3:  Technical  report  

3  –  4   Quantum  Mechanics   R,  A   R/M   O,  T:  R,  A   R   PLO  1:  Final  exam:  conceptual  

PLO  4:  Group  video    4   Senior  Research   M   M   (M,  A)   (T:  M)   (M,  A)   PLO  3,  5:  Advisor  feedback  4   Senior  Thesis   M,  A   M,  A   M,  A   W,  O:  M,  A   M,  A   Senior  Thesis  &  PresentaBon  All   Indirect  Evidence   A   A   A   A   A   Campus:  CRTE-­‐led  Focus  Group  All   Indirect  Evidence   A   A   A   A   A   Campus:  IPA  Senior  Exit  Survey  

Streamlining Program Assessment Research Week 2014

I = Introduce, R = Reinforce, M = Master

Curriculum Matrix: Development

7

Physics  Core  Courses  

Year   Course  Title  

Program  Learning  Objec9ves  Signature  Assignment  or    Indirect  Evidence  Support  

1   2   3   4   5  Physical    Principles  

Mathema9cal  Exper9se  

Experimental  Techniques  

Communica9on  &  Teamwork  

Research  Proficiency  

1   Introductory  I   I   I   I   W,  T:  I   I  1   Introductory  II   I   I   I   W,  T:  I   I  2   Introductory  III   I   I   R   W,  T:  R   R  

2   Classical  Mechanics   R   R,  A   W:  R,  A  O:  I   R,  A   PLO  2:  Final  exam:  quanBtaBve  

PLO  4,  5:  Literature  review  2  –  3   Thermodynamics   R,  A   R   PLO  1:  Final  exam:  conceptual  3  –  4   Electrodynamics   R   R,  A   PLO2:  Final  exam:  quanBtaBve  3  –  4   Modern  Physics  Lab   R   R/M,  A   W,  O,  T:  R   R   PLO  3:  Technical  report  

3  –  4   Quantum  Mechanics   R,  A   R/M   O,  T:  R,  A   R   PLO  1:  Final  exam:  conceptual  

PLO  4:  Group  video    4   Senior  Research   M   M   (M,  A)   (T:  M)   (M,  A)   PLO  3,  5:  Advisor  feedback  4   Senior  Thesis   M,  A   M,  A   M,  A   W,  O:  M,  A   M,  A   Senior  Thesis  &  PresentaBon  All   Indirect  Evidence   A   A   A   A   A   Campus:  CRTE-­‐led  Focus  Group  All   Indirect  Evidence   A   A   A   A   A   Campus:  IPA  Senior  Exit  Survey  

Streamlining Program Assessment Research Week 2014

I = Introduce, R = Reinforce, M = Master

Curriculum Matrix: Assessments

8

Physics  Core  Courses  

Year   Course  Title  

Program  Learning  Objec9ves  Signature  Assignment  or    Indirect  Evidence  Support  

1   2   3   4   5  Physical    Principles  

Mathema9cal  Exper9se  

Experimental  Techniques  

Communica9on  &  Teamwork  

Research  Proficiency  

1   Introductory  I   I   I   I   W,  T:  I   I  1   Introductory  II   I   I   I   W,  T:  I   I  2   Introductory  III   I   I   R   W,  T:  R   R  

2   Classical  Mechanics   R   R,  A   W:  R,  A  O:  I   R,  A   PLO  2:  Final  exam:  quanBtaBve  

PLO  4,  5:  Literature  review  2  –  3   Thermodynamics   R,  A   R   PLO  1:  Final  exam:  conceptual  3  –  4   Electrodynamics   R   R,  A   PLO2:  Final  exam:  quanBtaBve  3  –  4   Modern  Physics  Lab   R   R/M,  A   W,  O,  T:  R   R   PLO  3:  Technical  report  

3  –  4   Quantum  Mechanics   R,  A   R/M   O,  T:  R,  A   R   PLO  1:  Final  exam:  conceptual  

PLO  4:  Group  video    4   Senior  Research   M   M   (M,  A)   (T:  M)   (M,  A)   PLO  3,  5:  Advisor  feedback  4   Senior  Thesis   M,  A   M,  A   M,  A   W,  O:  M,  A   M,  A   Senior  Thesis  &  PresentaBon  All   Indirect  Evidence   A   A   A   A   A   Campus:  CRTE-­‐led  Focus  Group  All   Indirect  Evidence   A   A   A   A   A   Campus:  IPA  Senior  Exit  Survey  

Streamlining Program Assessment Research Week 2014

I = Introduce, R = Reinforce, M = Master, A = Assessment

Curriculum Matrix: Signature Assignments

9

Physics  Core  Courses  

Year   Course  Title  

Program  Learning  Objec9ves  Signature  Assignment  or    Indirect  Evidence  Support  

1   2   3   4   5  Physical    Principles  

Mathema9cal  Exper9se  

Experimental  Techniques  

Communica9on  &  Teamwork  

Research  Proficiency  

1   Introductory  I   I   I   I   W,  T:  I   I  1   Introductory  II   I   I   I   W,  T:  I   I  2   Introductory  III   I   I   R   W,  T:  R   R  

2   Classical  Mechanics   R   R,  A   W:  R,  A  O:  I   R,  A   PLO  2:  Final  exam:  quanBtaBve  

PLO  4,  5:  Literature  review  2  –  3   Thermodynamics   R,  A   R   PLO  1:  Final  exam:  conceptual  3  –  4   Electrodynamics   R   R,  A   PLO2:  Final  exam:  quanBtaBve  3  –  4   Modern  Physics  Lab   R   R/M,  A   W,  O,  T:  R   R   PLO  3:  Technical  report  

3  –  4   Quantum  Mechanics   R,  A   R/M   O,  T:  R,  A   R   PLO  1:  Final  exam:  conceptual  

PLO  4:  Group  video    4   Senior  Research   M   M   (M,  A)   (T:  M)   (M,  A)   PLO  3,  5:  Advisor  feedback  4   Senior  Thesis   M,  A   M,  A   M,  A   W,  O:  M,  A   M,  A   Senior  Thesis  &  PresentaBon  All   Indirect  Evidence   A   A   A   A   A   Campus:  CRTE-­‐led  Focus  Group  All   Indirect  Evidence   A   A   A   A   A   Campus:  IPA  Senior  Exit  Survey  

Streamlining Program Assessment Research Week 2014

I = Introduce, R = Reinforce, M = Master, A = Assessment

Curriculum Matrix: Indirect Evidence

10

Physics  Core  Courses  

Year   Course  Title  

Program  Learning  Objec9ves  Signature  Assignment  or    Indirect  Evidence  Support  

1   2   3   4   5  Physical    Principles  

Mathema9cal  Exper9se  

Experimental  Techniques  

Communica9on  &  Teamwork  

Research  Proficiency  

1   Introductory  I   I   I   I   W,  T:  I   I  1   Introductory  II   I   I   I   W,  T:  I   I  2   Introductory  III   I   I   R   W,  T:  R   R  

2   Classical  Mechanics   R   R,  A   W:  R,  A  O:  I   R,  A   PLO  2:  Final  exam:  quanBtaBve  

PLO  4,  5:  Literature  review  2  –  3   Thermodynamics   R,  A   R   PLO  1:  Final  exam:  conceptual  3  –  4   Electrodynamics   R   R,  A   PLO2:  Final  exam:  quanBtaBve  3  –  4   Modern  Physics  Lab   R   R/M,  A   W,  O,  T:  R   R   PLO  3:  Technical  report  

3  –  4   Quantum  Mechanics   R,  A   R/M   O,  T:  R,  A   R   PLO  1:  Final  exam:  conceptual  

PLO  4:  Group  video    4   Senior  Research   M   M   (M,  A)   (T:  M)   (M,  A)   PLO  3,  5:  Advisor  feedback  4   Senior  Thesis   M,  A   M,  A   M,  A   W,  O:  M,  A   M,  A   Senior  Thesis  &  PresentaBon  All   Indirect  Evidence   A   A   A   A   A   Campus:  CRTE-­‐led  Focus  Group  All   Indirect  Evidence   A   A   A   A   A   Campus:  IPA  Senior  Exit  Survey  

Streamlining Program Assessment Research Week 2014

I = Introduce, R = Reinforce, M = Master, A = Assessment

Experimental Skills Development

Streamlining Program Assessment Research Week 2014 11

Revised  

Year   Course  Title  PLO  3  

Expt. Techniques 1   Introductory  I   I  1   Introductory  II   I  2   Introductory  III   R  2   Classical  Mechanics  

2  –  3   Thermodynamics  3  –  4   Electrodynamics  3  –  4   Modern  Physics  Lab   R/M  3  –  4   Quantum  Mechanics  4   Senior  Research   (M)  4   Senior  Thesis   M  

I = Introduce, R = Reinforce, M = Master

Original  

Year   Course  Title  PLO  3  

Expt. Techniques 1   Introductory  I   I  1   Introductory  II   I  2   Introductory  III   I  2   Classical  Mechanics  

2  –  3   Thermodynamics  3  –  4   Electrodynamics  3  –  4   Modern  Physics  Lab   R/M  3  –  4   Quantum  Mechanics  4   Senior  Research   (M)  4   Senior  Thesis  

Teamwork Development

Streamlining Program Assessment Research Week 2014 12

Revised  

Year   Course  Title  PLO  4  

Teamwork 1   Introductory  I   I  1   Introductory  II   I  2   Introductory  III   I  2   Classical  Mechanics  

2  –  3   Thermodynamics  3  –  4   Electrodynamics  3  –  4   Modern  Physics  Lab   I  3  –  4   Quantum  Mechanics   R  4   Senior  Research   (M)  4   Senior  Thesis  

I = Introduce, R = Reinforce, M = Master

Original  

Year   Course  Title  PLO  4  

Teamwork 1   Introductory  I   I  1   Introductory  II   I  2   Introductory  III   I  2   Classical  Mechanics  

2  –  3   Thermodynamics  3  –  4   Electrodynamics  3  –  4   Modern  Physics  Lab   I  3  –  4   Quantum  Mechanics  4   Senior  Research   (M)  4   Senior  Thesis  

Descriptive Rubrics

13 Streamlining Program Assessment Research Week 2014

!"#$%&#'()*+,)+-).,(

!"#$%&$'!()$)*+,+%-*'./0(%&'

/0&1&2%&0(3$4&'(5&$0(6(7&04'( 78%9&(:;(7$9<((

1(%+)(%,' 1,2$+-*)' 3%4)$+-*)' 5)*&"6,(7' !--('8' 9' :' ;'

<(=,*%>,+%-*',*?'!4,**%*='

=(>&99?#&;82&#(1%0@A%@0&(81(82A9@#&#(>8%B(A:2C&2%8:2$9(&9&4&2%1(D1%$%&4&2%(:;("0:E9&4F(E$A<G0:@2#F(4&%B:#1F(0&1@9%1($2#(A:2A9@18:2HI(J%@#&2%(4:C&1(&$189K(E&%>&&2(198#&1F($2#(4$82%$821($(A:2181%&2%9K(14::%B("$A&(%B0:@GB:@%("0&1&2%$%8:2I(

L&2&0$99K(>&99?1%0@A%@0&#(%$9<(>8%B(4:1%(&9&4&2%1("0&1&2%I(J%@#&2%(4:C&1(G&2&0$99K(>&99(E&%>&&2(198#&1F(4$82%$821($(14::%B("$A&(%B0:@GB(4:1%(:;(%B&("0&1&2%$%8:2I(

J:4&(1%0@A%@0$9(&9&4&2%1("0&1&2%F(1"&$<&0(1:4&%84&1($""&$01(@2;$4898$0(>8%B(198#&(A:2%&2%($2#(1&M@&2A&I(7B&(@2&C&2("$A&("&08:#8A$99K(#&%0$A%1(;0:4(%B&("0&1&2%$%8:2I(

N$A<(:;($(A9&$0(1%0@A%@0&(82(%B&(%$9<I(J98#&1(1&&4(#81:0G$28O&#($2#+:0(1"&$<&0(81(@2;$4898$0(>8%B(%B&80(A:2%&2%I(7B&(@2&C&2("$A&(#&%0$A%1(;0:4(%B&("0&1&2%$%8:2($2#(4&11$G&(D8I&I(:C&09K(9:2G(82%0:($2#(0@1B&1(%B0:@GB(A:2A9@18:21HI(

!()$)*+,+%-*'@+#4)'

J"&$<&0(81(A9&$0($2#(A:2;8#&2%I(L8C&1($("0:;&118:2$9(84"0&118:2I((

P9&$0(1"&&ABF($2#(M@8A<9K(:C&0A:4&1(:AA$18:2$9(9$"1&1(82(A:2;8#&2A&(:0(B&18%$%8:2I(

J:4&>B$%(2&0C:@1(:0(B&18%$2%(1%K9&F(E@%(G&%1(%B&(4&11$G&($A0:11I(J:4&(;9$>1(8I&I($C:8#1(&K&(A:2%$A%F(9::<82G($(;9::0+1A0&&2(:0(4@4E982GI(

Q&0K(2&0C:@1F(B&18%$2%(:0(#81R:82%&#(1%K9&F(>B8AB(82%&0;&0&1(>8%B($E898%K(%:(A:44@28A$%&(82;:04$%8:2(%:($@#8&2A&I(

A$)'-B'C,*=/,=)'

!1&1(#&1A08"%8C&F(1A8&2%8;8A(9$2G@$G&(%B$%(81(2:%(:C&0%9K(SR$0G:2KIT(P:2A&"%1($0&(A9&$0($2#("0:;&118:2$99K(&U"9$82&#I(

!1&1(4:1%9K(#&1A08"%8C&F(1A8&2%8;8A(9$2G@$G&F($2#(&U"9$2$%8:21($0&(4:1%9K("0:;&118:2$99K($2#(A9&$09K(&U"9$82&#I(

V$18A(9$2G@$G&(AB:8A&1F($""0:$AB82G("0:;&118:2$9(&U"9$2$%8:21F(E@%(4&11$G&(81(1%899(A9&$0I((W8284$9(;899&01(DS@4THI(

N$A<1(&U"&A%&#(1A8&2%8;8A(C:A$E@9$0KI((W$K(@1&(4$2K(;899&01(DS@4THF(184"981%8A+(R@C&289&(9$2G@$G&F(9&$#82G(%:(@2A9&$0(1%$%&4&2%1I(

D%$/,4'E%?$'' =8#1($0&(A9&$0F(>&99(:0G$28O&#($2#(&2B$2A&(%B&("0&1&2%$%8:2(18G28;8A$2%9KI(

=8#1(&2B$2A&(%B&("0&1&2%$%8:2(E@%(>8%B(1:4&(;9$>1(D8I&I(;:2%(18O&1F(A:2;@182G(9$K:@%1H(

=8#1($0&($#&M@$%&(E@%(2:%(>&99(982<&#(%:(%B&("0:R&A%($2#(A:2%$82(1&C&0$9(;9$>1I(P$2(E&(#81%0$A%82GI(

=8#1($0&(#81:0G$28O&#F("::09K(AB:1&2F(#&%0$A%(;0:4(%B&("0&1&2%$%8:2($2#(4&11$G&I(

1)*+(,4'3)$$,=)'

/@0":1&(:;(0&1&$0AB(81(A9&$09K(1%$%&#($2#(4&%B:#1(R@1%8;8&#(A9&$09K($2#(A:2A81&9K(

/@0":1&(:;(0&1&$0AB(81(1%$%&#($2#(982<&#(%:(4&%B:#1(E@%(1:4&%84&1("::09K(

L&2&0$9(%B&4&(:;(0&1&$0AB(81(82#8A$%&#I(3:(R@1%8;8A$%8:2(:;(4&%B:#1I(

/@0":1&(:;(0&1&$0AB("::09K(&U"9$82&#(:0(2:%($0%8A@9$%&#I(

E??%+%-*,4'1-66)*+$F'

Sources for Rubrics

14 Streamlining Program Assessment Research Week 2014

!"#$%&!''()*&#+*!)%,#$(-%"(."*& !"#$%"#&$'(!"#%)*'"(+$,-&).&$/"(*)/*$0)-1&2))/13"#4

$

/012324253% Oral communication is a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners' attitudes, values, beliefs, or behaviors.

50)-1)*"#.$)#&$&(/"1#)4&6$*"$)..'4($)$7&#"$*"$)(8$9"#:$.)%,-&$"#$/"--&/*'"($"! $9"#:$*;)*$6"&.$("*$%&&*$<&(/;%)#:$=/&--$"(&>$-&0&-$,&#!"#%)(/&3$

% &6784530%4

'290845308%3 2

.03:;<6=>%1

!=?632@64253% Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is clearly and consistently observable and is skillful and makes the content of the presentation cohesive.

Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is clearly and consistently observable within the presentation.

Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is intermittently observable within the presentation.

Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is not observable within the presentation.

$63?A6?0% Language choices are imaginative, memorable, and compelling, and enhance the effectiveness of the presentation. Language in presentation is appropriate to audience.

Language choices are thoughtful and generally support the effectiveness of the presentation. Language in presentation is appropriate to audience.

Language choices are mundane and commonplace and partially support the effectiveness of the presentation. Language in presentation is appropriate to audience.

Language choices are unclear and minimally support the effectiveness of the presentation. Language in presentation is not appropriate to audience.

/092B0=C% Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation compelling, and speaker appears polished and confident.

Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation interesting, and speaker appears comfortable.

Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation understandable, and speaker appears tentative.

Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) detract from the understandability of the presentation, and speaker appears uncomfortable.

DA775=423?%'640=269% A variety of types of supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make appropriate reference to information or analysis that significantly supports the presentation or establishes the presenter's credibility/authority on the topic.

Supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make appropriate reference to information or analysis that generally supports the presentation or establishes the presenter's credibility/authority on the topic.

Supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make appropriate reference to information or analysis that partially supports the presentation or establishes the presenter's credibility/authority on the topic.

Insufficient supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make reference to information or analysis that minimally supports the presentation or establishes the presenter's credibility/authority on the topic.

&034=69%'0886?0% Central message is compelling (precisely stated, appropriately repeated, memorable, and strongly supported.)

Central message is clear and consistent with the supporting material.

Central message is basically understandable but is not often repeated and is not memorable.

Central message can be deduced, but is not explicitly stated in the presentation.

VALUE Rubrics. Assessing Outcomes & Improving Achievement: Tips & Tools for Using Rubrics, T. L. Rhodes ed., Association of American Colleges & Universities, 2010.

Rubrics: Research Skills

15 Streamlining Program Assessment Research Week 2014

Level 1

Students research at the level of a closed inquiry* and require a high degree of structure/guidance

Level 2

Students research at the level of a closed inquiry* and require some structure/guidance

Level 3

Students research independently at the level of a closed inquiry*

Level 4

Students research at the level of an open inquiry* within structured guidelines

Level 5

Students research at the level of an open inquiry* within self-determined guidelines in accordance with the discipline

A. Students embark on inquiry and so determine a need for knowledge/understanding

Respond to questions/tasks arising

explicitly from a closed inquiry.

Respond to questions/tasks

required by and implicit in a closed

inquiry.

Respond to questions/tasks

generated from a closed inquiry.

Generate questions/aims/

hypotheses framed within structured

guidelines.

Generate questions/aims/hypotheses

based on experience, expertise and

literature.

B. Students needed information/data using appropriate methodology

Collect and record required

information/data using a prescribed

methodology from a prescribed

source in which the information/

data is clearly evident.

Collect and record required

information/data using a prescribed

methodology from prescribed

source/s in which the information/

data is not clearly evident.

Collect and record required

information/data from self-selected

sources using one of several

prescribed methodologies.

Collect and record self-determined

information/data from self-selected

sources, choosing an appropriate

methodology based on structured

guidelines.

Collect and record self-determined

information/data from self-selected

sources, choosing or devising an

appropriate methodology with self-

structured guidelines.

C. Students critically evaluate information/data and the process

information/data

Evaluate information/data and

the inquiry process using simple

prescribed criteria.

Evaluate information/data and the

inquiry process using prescribed

criteria.

Evaluate information/data and the

inquiry process using criteria related

to the aims of the inquiry.

Evaluate information/data and the

inquiry process comprehensively

using self-determined criteria

developed within structured

guidelines.

Evaluate information/data and the

inquiry process rigorously using

self-generated criteria based on

experience, expertise and literature.

D. Students organise information collected/generated and manage the research process

Organise information/data and

manage the research process

according to a simple prescribed

structure.

Organise information/data and

manage the research process

according to prescribed structures.

Organise information/data and

manage the research process by

adapting provided structures.

Organise information/data and

manage the research process using

provided guidelines.

Organise information/data and

manage the research process

using self-determined protocols in

accordance with the discipline.

E. Students synthesise and analyse and apply new knowledge

Synthesise and analyse information/

data to reproduce existing

knowledge in prescribed formats.

curiosity.

Synthesise and analyse information/

data to reorganise existing

knowledge in standard formats.

Ask relevant, researchable

questions.

Synthesise and analyse information/

data to construct emergent

knowledge.

Ask rigorous, researchable

questions based on new

understandings.

Synthesise, analyse and apply

knowledge gaps.

Synthesise, analyse and apply

gaps or extend knowledge.

F. Students communicate knowledge and the processes used to generate it, with an awareness of ethical, social and cultural issues

Use mainly lay language and

prescribed genre to demonstrate

required knowledge and

understanding for lecturer/teacher

as the audience.

language and prescribed genre

to demonstrate self-selected

knowledge and understanding

from a stated perspective and for a

language and appropriate genre

to demonstrate knowledge and

a scholarly perspective and for a

Use the language of the discipline

and appropriate genre to address

knowledge and understanding gaps

from several perspectives for a self-

selected audience.

Use the language of the discipline,

choosing appropriate genre to extend

knowledge and understanding, from

diverse perspectives for a range of

audiences.

Research Skill Development Framework

www.adelaide.edu.au/clpd/rsd - john.willison@adelaide.edu.au

A conceptual model to make explicit the incremental and cyclic development of student research skills

LEVEL OF STUDENT AUTONOMY

FAC

ET

OF

INQ

UIR

Y

Concept by John Willison and Kerry O'Regan, design by Peter Murdoch and Nik Cornish, Centre for Learning and Professional Development. Facets derived from ANZIIL(2004) standards, reworked using the Bloom, et al (1956) Taxonomy and implementation trials by Eleanor Peirce and Mario Ricci. © The University of Adelaide, October 2006.

* closed = lecturer speci!ed. open = student initiated. Lecturers and teachers determine scope of inquiry and standard required; student achievement determines the Level their research actually attains. For example, the provision of an open inquiry within structured guidelines (Level 4) in the First Year University context will see some students providing evidence of Level 1 attainment for a speci!c facet, with others demonstrating Level 2, Level 3 or Level 4, depending on their degree of rigour.

Persuasiv

e

Creativ

e

Organised

Critical

Determ

ined Curio

us

J. Willison, K. Regan, The Research Skills Development Framework, www.adelaide.edu.au/rsd/framework, 2006.

Level 1

Students research at the level of a closed inquiry* and require a high degree of structure/guidance

Level 2

Students research at the level of a closed inquiry* and require some structure/guidance

Level 3

Students research independently at the level of a closed inquiry*

Level 4

Students research at the level of an open inquiry* within structured guidelines

Level 5

Students research at the level of an open inquiry* within self-determined guidelines in accordance with the discipline

A. Students embark on inquiry and so determine a need for knowledge/understanding

Respond to questions/tasks arising

explicitly from a closed inquiry.

Respond to questions/tasks

required by and implicit in a closed

inquiry.

Respond to questions/tasks

generated from a closed inquiry.

Generate questions/aims/

hypotheses framed within structured

guidelines.

Generate questions/aims/hypotheses

based on experience, expertise and

literature.

B. Students needed information/data using appropriate methodology

Collect and record required

information/data using a prescribed

methodology from a prescribed

source in which the information/

data is clearly evident.

Collect and record required

information/data using a prescribed

methodology from prescribed

source/s in which the information/

data is not clearly evident.

Collect and record required

information/data from self-selected

sources using one of several

prescribed methodologies.

Collect and record self-determined

information/data from self-selected

sources, choosing an appropriate

methodology based on structured

guidelines.

Collect and record self-determined

information/data from self-selected

sources, choosing or devising an

appropriate methodology with self-

structured guidelines.

C. Students critically evaluate information/data and the process

information/data

Evaluate information/data and

the inquiry process using simple

prescribed criteria.

Evaluate information/data and the

inquiry process using prescribed

criteria.

Evaluate information/data and the

inquiry process using criteria related

to the aims of the inquiry.

Evaluate information/data and the

inquiry process comprehensively

using self-determined criteria

developed within structured

guidelines.

Evaluate information/data and the

inquiry process rigorously using

self-generated criteria based on

experience, expertise and literature.

D. Students organise information collected/generated and manage the research process

Organise information/data and

manage the research process

according to a simple prescribed

structure.

Organise information/data and

manage the research process

according to prescribed structures.

Organise information/data and

manage the research process by

adapting provided structures.

Organise information/data and

manage the research process using

provided guidelines.

Organise information/data and

manage the research process

using self-determined protocols in

accordance with the discipline.

E. Students synthesise and analyse and apply new knowledge

Synthesise and analyse information/

data to reproduce existing

knowledge in prescribed formats.

curiosity.

Synthesise and analyse information/

data to reorganise existing

knowledge in standard formats.

Ask relevant, researchable

questions.

Synthesise and analyse information/

data to construct emergent

knowledge.

Ask rigorous, researchable

questions based on new

understandings.

Synthesise, analyse and apply

knowledge gaps.

Synthesise, analyse and apply

gaps or extend knowledge.

F. Students communicate knowledge and the processes used to generate it, with an awareness of ethical, social and cultural issues

Use mainly lay language and

prescribed genre to demonstrate

required knowledge and

understanding for lecturer/teacher

as the audience.

language and prescribed genre

to demonstrate self-selected

knowledge and understanding

from a stated perspective and for a

language and appropriate genre

to demonstrate knowledge and

a scholarly perspective and for a

Use the language of the discipline

and appropriate genre to address

knowledge and understanding gaps

from several perspectives for a self-

selected audience.

Use the language of the discipline,

choosing appropriate genre to extend

knowledge and understanding, from

diverse perspectives for a range of

audiences.

Research Skill Development Framework

www.adelaide.edu.au/clpd/rsd - john.willison@adelaide.edu.au

A conceptual model to make explicit the incremental and cyclic development of student research skills

LEVEL OF STUDENT AUTONOMY

FAC

ET

OF

INQ

UIR

Y

Concept by John Willison and Kerry O'Regan, design by Peter Murdoch and Nik Cornish, Centre for Learning and Professional Development. Facets derived from ANZIIL(2004) standards, reworked using the Bloom, et al (1956) Taxonomy and implementation trials by Eleanor Peirce and Mario Ricci. © The University of Adelaide, October 2006.

* closed = lecturer speci!ed. open = student initiated. Lecturers and teachers determine scope of inquiry and standard required; student achievement determines the Level their research actually attains. For example, the provision of an open inquiry within structured guidelines (Level 4) in the First Year University context will see some students providing evidence of Level 1 attainment for a speci!c facet, with others demonstrating Level 2, Level 3 or Level 4, depending on their degree of rigour.

Persuasiv

e

Creativ

e

Organised

Critical

Determ

ined Curio

us

Rubrics: Research Skills

16 Streamlining Program Assessment Research Week 2014

Based on J. Willison, K. Regan, The Research Skills Development Framework, www.adelaide.edu.au/rsd/framework, 2006.

Mentor Questionnaire for PLO#5: Research Proficiency 3

Attributes The student with research skill!

Level I Students research at the level of a closed inquiry* and require a high degree of structure/guidance

Level II Students research at the level of a closed inquiry* and require some structure/guidance

Level III Students research independently at the level of a closed inquiry*

Level IV Students research at the level of an open inquiry* within structured guidelines

Level V Students research at the level of an open inquiry* within self-determined guidelines in accordance with the discipline

“!research questions that expand their knowledge of physics.”

Synthesize and analyze information/data to reproduce existing knowledge in prescribed formats.

Synthesize and analyze information/ data to reorganize existing knowledge in standard formats.

Synthesize and analyze information/ data to construct emergent knowledge.

Synthesize and analyze information/ data to fill recognized knowledge gaps.

Synthesize, analyze, and apply information/ data to fill self-identified gaps or extend knowledge.

1. Characteristics of Mentee:

a. How does the mentee exhibit the characteristics for the level you highlighted above?

b. What would the mentee need to do to reach the next level? And how is that higher level expected, needed, or otherwise appropriate for the mentee’s education and/or career goals?

2. Opportunities within their project/working in your research group: a. How have you gauged the mentee’s expansion of their knowledge?

3. Other examples: a. Describe any other instances (if any) where the mentee has formulated personal research questions.

Inter-rater reliability • PLO: Physical Principles • Distribution Matrix • Correlation coefficient: 0.78

Streamlining Program Assessment Research Week 2014 17

Mitchell E A U

Men

ke E 6 1 0

A 2 15 3 U 0 1 6

Unacceptable Acceptable Excellent • Knowledge of basic physical

principles is missing. • Knowledge of basic physical

principles evident, but o Application is missing. o Significant errors exist in their

application. o Example: student can write down

Maxwell’s equations, but cannot calculate the magnetic field around a wire.

• Knowledge and/or application of two or more physical principles are confused.

• Knowledge of basic physical principles is evident.

• Those principles are applied correctly, o although some errors exist. • Misconception in knowledge or

application of more subtle feature of principle may exist.

• Knowledge of basic physical principles is evident.

• Those principles are applied correctly. o although minimal errors

may be present. • Evidence that more subtle

aspects of physical principles known and correctly applied.

Course and Program

Streamlining Program Assessment Research Week 2014 18

Conclusions & Actions of Assessments •  Program Learning Objectives: Important discussion on what constitutes research

proficiency (PLO 5), led to focus on information literacy in early years and encouraging students to engage in undergraduate research early.

•  Curriculum Matrix & Evidence: •  Allowed us to identify gaps in learning objective development opportunities with Experimental

Techniques (PLO 3) and Teamwork (PLO 4). •  Evidence is consistently collected and spread out over all semester-long core courses.

•  Pedagogical Validation: Existing writing assignments and presentations effectively support Communication skills (PLO 4).

•  Pedagogical Adoptions: •  Conceptual questions explicitly integrated into final exams to separate principles (PLO 1) from

mathematics (PLO 2). •  Video project adopted in Quantum Mechanics to provide in-class teamwork opportunities (PLO 4). •  Stronger emphasis on data analysis in Intro. III provide reinforcement for experimental techniques (PLO

3).

•  Applying Rubrics to Course and Program: •  Robust program rubrics are applicable to course-level work and allowing for quicker program

assessment when taken in aggregate. •  Inter-rater reliability checks are needed to ensure program applicability.

Streamlining Program Assessment Research Week 2014 19

Challenges • Organization • Student writing • Robust rubrics • Participation

•  Faculty •  Students

Streamlining Program Assessment Research Week 2014 20

References 1.  Laura Martin, Introduction to Assessment Presentation, CGS Workshop,

http://crtecertificatespring2013.files.wordpress.com/2012/12/introduction-to-assessment-1-14-2013-final.pdf, January 2013

2.  L. A. Suskie, Assessing Student Learning: A Common Sense Guide (Jossey-Bass, San Francisco, CA 2009).

3.  Center for the Integration of Research, Teaching, and Learning (CIRTL

Network), Teaching-as-Research (TAR): Developmental Framework, www.cirtl.net/CoreIdeas/teaching_as_research, 2013.

4.  G. Wiggins, J. McTighe, Backward Design in Understanding by Design (Assn.

for Supervision & Curriculum Development, Alexandria, 2005). 5.  VALUE Rubrics. Assessing Outcomes & Improving Achievement: Tips & Tools

for Using Rubrics, T. L. Rhodes ed., Association of American Colleges & Universities, 2010.

6.  J. Willison, K. Regan, The Research Skills Development Framework,

www.adelaide.edu.au/rsd/framework, 2006.

Streamlining Program Assessment Research Week 2014 21