Post on 20-Dec-2015
transcript
Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Analyzing CS Competenciesusing The SOLO Taxonomy
Claus Brabrand((( brabrand@itu.dk )))((( http://www.itu.dk/people/brabrand/ )))
Associate Professor,IT University of Copenhagen Denmark
ITiCSE'09 – Keynote
[ 2 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Outline
1) Introduction Constructive Alignment The SOLO Taxonomy
2) From Content to Competence Advocate a shift in perspective Elaborate The SOLO Taxonomy
3) Analyzing CS Competencies …using The SOLO Taxonomy Compare: CS vs NAT vs MAT
[ 3 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
“Teaching for Quality Learning at University
- What the student does”
“Teaching for Quality Learning at University
- What the student does”
Constructive Alignment & SOLO Taxonomy:
Introduction to…:
“Teaching Teaching & Understanding Understanding”
“Teaching Teaching & Understanding Understanding”19 min award-winning short-film on Constructive Alignment(available on DVD in 7 languages, epilogue by John Biggs)
John Biggs’ popular and heavily cited book:
Note: 3rd Edition now available [J.Biggs & C.Tang, 2009]
[ 4 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
What are the ‘main messages’of the film (which did YOU findparticularly relevant, …if any)?
Activation Exercise
Discuss with your neighbour:
T
[ 5 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Outline
1) Introduction Constructive Alignment The SOLO Taxonomy
2) From Content to Competence Advocate a shift in perspective Elaborate The SOLO Taxonomy
3) Analyzing CS Competencies …using The SOLO Taxonomy Compare: CS vs NAT vs MAT
[ 6 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
From Content to Competence
My old course descriptions (Concurrency 2004): Given in terms of a 'content description':
Essentially:
This is a bad ideafor two reasons...!
Goal is…:
To understand: deadlock interference synchronization ...
[ 7 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Goal is…:
To understand: deadlock interference synchronization ...
Problem 1 !
Problem with 'content' as goals ! analyze ...theorize ...
define deadlockdescribe solutions
name solutionsrecite conditons
Stud. C
Stud. A
Stud. B
analyze systemsexplain causes
Censor
Teacher
P.S.: even if it werepossible to agree, we know that the
exam will dictate thelearning anyway.
agreem
ent
analyze systemsexplain causes
tacit kno
wled
ge fro
m a
research-b
ased trad
ition
no
t kno
wn
by stu
den
t
[ 8 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Problem 2 !
Problem with 'understanding' as goals !
The answer is simple:
'concept of deadlock' ?!
Goal is…:
To understand: deadlock interference synchronization ...
It cannot be measured !
[ 9 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Competence !
'Competence' as goals !
Have the student do something;and then "measure" the product and/or process
'SOLO' = Structure of the Observed Learning Outcome
Note': inherently operational (~ verbs)
Objective !
To learn how to: analyze systems for... explain cause/effects... prove properties of... compare methods of... ...
Note: 'understanding' is of course
pre-requisitional !
Competence := knowledge + capacity to act upon it
[ 10 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO Advantages
Advantages of The SOLO Taxonomy: Linear hierarchical structure Aimed at evaluating student learning Converges on research (at SOLO 5)
Research:Production ofnew knowledge
[ 11 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO (elaborated)Note: the list is non-exhaustive
SOLO 2”uni-structural”
SOLO 3“multi-structural”
SOLO 4“relational”
SOLO 5“extended abstract”
theorize generalize hypothesize predict judge reflect transfer theory
(to new domain) …
analyze compare contrast integrate relate explain causes apply theory
(to its domain) …
combine structure describe classify enumerate list do algorithm apply method …
define identify count name recite paraphrase follow (simple)
instructions …
Graphic Legend
problem / question / cue known related issue - given! hypothetical related issue - not given! student response
Q
R
QUANTITATIVE QUALITATIVE
R
R'Q
RQRQRQ
[ 12 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Using SOLO in Practice
Intended Learning Outcomes [Algorithms 101]
After the course, the students are expected to be able to:
identify and formulate algorithmic problems ;
classify and compare algorithms ;
construct and analyze algorithms using standard paradigms;
implement algorithms for simple problems.
2) List sub-goals as 'bullets': Clearer than text
3) Use 'Verb + Noun' formulation:
What the student is expected to
do with a given matterV N
N
N
V
V V N
V V
V V
N
4) Avoid 'understanding-goals':
"To understand X", "Be familiar with Y", "Have a notion of Z", ...!
Recommendations on course descriptions:
1) Use 'standard formulation':
a) puts learning focus on the student
b) competence formulation: "to be able to"
[ 13 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Activation Exercise
Which do you predict are key CS competences ?
T
Concurrency:
analyze systemscompare models
Concurrency:
analyze systemscompare models
[ 14 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Outline
1) Introduction Constructive Alignment The SOLO Taxonomy
2) From Content to Competence Advocate a shift in perspective Elaborate The SOLO Taxonomy
3) Analyzing CS Competencies …using The SOLO Taxonomy Compare: CS vs NAT vs MAT
Joint work with Bettina Dahl at Aarhus University
[ 15 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Grade Scales
ECTSSCALE
A, B, C, D, E, Fx, F
...
4 steps
8 steps
10 steps
21 steps
...
4 steps
8 steps
10 steps
21 steps
7 steps:
...
... ...
Conversion (between EU countries):
All Universities:Explicit ILO's
The SOLO Taxonomy!
Grade := Degree of realization
of course objectives!
[ 16 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Massive DATA set
Unique Opportunity…:
Systematically formulated ILO's for all courses Quantifiable (analyzable) via The SOLO Taxonomy
5,608
734
institutesTWO universities
competenciescourses
21
[ 17 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO Mapping
Mapped by: B. Dahl & C. Brabrand
With help from: 3 Educational research
colleagues (medicine) J. Biggs & C. Tang
[ 18 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Top 10 Competencies
Top 10 Competencies:
" " := { Physics, Chemistry, Biology, Molecular Biology }
Natural Sciences
[ 19 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Histogram of Top Competencies
If we look closer (comparative visualization)...:
Legend:
Computer Science Natural Science Mathematics
%
CS
CS
CS
MAT
MAT
MAT MAT MAT
MAT
More than 3x More than 3x
CS
…also: program construct structure
NAT
NAT
CS: 15 %NAT: 1.0 %MAT: 0.3 %
CS: 4.5 %NAT: 4.4 %MAT: 40 %
CS: 14 %NAT: 14 %MAT: 60 %
More than 2x
with apply
[ 20 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO Distribution
SOLO distribution:
The 15% "programming competences" (all at SOLO 4): { implement, program, design, construct, structure }
SOLO 2 SOLO 3 SOLO 4 SOLO 5Legend:
15% E[X] = 3.7
E[X] = 3.4
E[X] = 3.1
[ 21 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Assumptions
SOLO is an appropriate competence measure (we refer to [J.Biggs & K.F.Collis, 1982] )
Context independence of SOLO mapping (for each competence we inspected several goals)
Subject independence of SOLO mapping (we limit ourselves to a 'science context')
Equal weight assumptions (Competences in a goal & goals in a course have equal weight)
Outcomes: intended formulated achieved (we “analyze” formulated, but “reason about” achieved)
Assumptions:
[Biggs’ studies]
[approximation]
[approximation]
[approximation]
[implicational]
[ 22 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Conclusions Most frequent CS Competences are:
describe (13%), explain (10%), apply method (9%), implement (7%), analyze (6%), …
"Programming-related" skills: 15% of CS-curriculum
The "Essence of Math" is: reproducing, formulating,
proving, solving, argueing,(and applying)
SOLO-levels of subjects: CS >SOLO NAT >SOLO MAT
15%
[ 23 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
1) Introduction Constructive Alignment The SOLO Taxonomy
2) From Content to Competence Advocate a shift in perspective Elaborate The SOLO Taxonomy
3) Analyzing CS Competencies …using The SOLO Taxonomy Compare: CS vs NAT vs MAT
Outline
[ 24 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Keynote Points
Constructive Alignment …addresses many teaching / learning problems; e.g.:
Esp. student motivational issues (learning incentives) ...and student performance issues (learning support)
The SOLO Taxonomy …is good for reasoning about competencies:
Esp. for designing courses and curricula
DATA Study, analyze, and reflect on teaching / learning
…using (objective) DATA!
[ 25 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Questions...
"What is good teaching?"
The Short-Film
Cognitive processes
Association
new ~ old
The Book
John Biggs
"understanding"
content competence
Student activation
Susan & RobertTeacher models
levels 1 - 2 - 3
Course descriptions
Constructive AlignmentTop 10 Competences
15% programming
CS v. NAT v. MAT
Students at University
My researchand teaching
'TLA'Teaching / Learning
Activities
Tips'n'Tricks
?
recitegeneralize
R
R'
Q
The SOLO Taxonomy
Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Thank You!
((( http://www.daimi.au.dk/~brabrand/short-film/ )))
Film's homepage:
[ 27 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Related References ”Teaching for Quality Learning at University (what the student does)”
John Biggs & Catherine TangSociety for Research into Higher Education, 2007. McGraw-Hill.
”Evaluating the Quality of Learning: The SOLO Taxonomy”John Biggs & Kevin F. CollisLondon: Academic Press, 1982
”Teaching Teaching & Understanding Understanding”Claus Brabrand & Jacob Andersen19 minute award-winning short-film (DVD)Aarhus University Press, Aarhus University, 2006
”Using the SOLO Taxonomy to Analyze Competence Progression of University Science Curricula”Claus Brabrand & Bettina DahlHigher Education, 2009
"Constructive Alignment & The SOLO Taxonomy: a Comparative Study of University Competencies in Computer Science vs. Mathematics"Claus Brabrand & Bettina DahlCRPIT, Vol. 88, ACS 3-17, R. Lister & Simon, Eds., 2007
[ 28 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Implementing Alignment
Alignment Implementation Process:
1) Think carefully about: overall goal of course (what are the stud. to learn?)
2) Operationalize these goals and formulate them as SOLO intended learning outcomes
3) Choose carefully the form(s) of assessment (~ intended learning outcomes)
4) Choose carefully the form(s) of teaching (~ intended learning outcomes)
alignmentlearning incentive learning support
Think of teachingactivities as
”training for exam”
[ 29 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO Progression
SOLO Progression: Computer Science vs. Mathematics vs. …
[ 30 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Conclusion (Progression)
What have we really shown?!?
A) SOLO has "proved" that progression exists in curricula (since we "believe" in SOLO as a measure)
B) SOLO has "been proven" to be a good tool for analyzing competence progression (since we "believe" in the existence of progression)
xor
[ 31 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Progression Assumptions
Numeric quantification of SOLO [assumption] (i.e., numeric step from 2-3 is comparable to 3-4 and 4-5)
Progression manifests itself as competences [assumption] (i.e., in 'verb'-, not 'noun'-dimension)
Extra assumptions wrt. Progression:
[ 32 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO Calculation Method
Calculation Example (for a course):
"SOLO average": [ (2+3)/2 + (3+4)/2 + (4+4)/2 + 4 ] / 4 = 3.50
"SOLO distribution":
identify (2) and formulate (3) algorithmic problems;
classify (3) and compare (4) algorithms;
construct (4) and analyze (4) algorithms using standard paradigms;
implement (4) algorithms for simple problems.
"double weight averaging"
[ 33 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Neighbour Discussion
Discuss with neighbour:"does this make sense ?!?"
(content competence)
T
E.g.: ("Learning about programming" vs. "Learning to program" )
[ 34 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Discuss what you predict wewould find in the DATA set ?
Activation Exercise III
Discuss with your neighbour:
T
Questions: a) most frequent CS competences? b) percentage of "programming-related" competences? c) CS v. NAT v. MAT (wrt. SOLO levels)?
[ 35 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Post-It exercise
Write down 1-2 key competences (i.e., verbs)
(for your course)
T
Concurrency:
analyze systems for deadlock
compare models wrt. behavior
Concurrency:
analyze systems for deadlock
compare models wrt. behavior
[ 36 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Tips'n'Tricks (activation)
Neighbour discussions:
Frequent breaks:
Post-It exercise: focus: zoom in anonymous (!) swap'able everyone will engage empathetic control shared knowledge pool
pu
lse
re
ad
er
me
asu
rem
en
ts:
more questions (students dare ask them)
better questions (students had a chance to discuss)
1-2 min timeout [Phil Race]
Form variation:
lecturing blended with in-class activation exercises
[ 37 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Tips'n'Tricks (cont'd)
"Less-is-more":
Use many examples:(build on student pre-knowledge)
Explicit structure:
analyze compare relate
common deadlock, uncommon deadlock, A-synchronization, B-synchronization, hand-shake, multi-party synchronization, multi-party hand-shake, binary semaphores, generalized semaphores, blocking semaphores, recursive locks, ...
vs.
Emphasize depth over breadth (coverage)
NEWOLD
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
self evident to you [ teacher ] not to a learner [ student ] (esp. during learning process)
Student 'recap' at end:
after 1 dayafter 1 week
after 3 weeks
after 2 weeks
now
[ 38 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Now, please: "3-minute recap"
Please spend 3' on thinking about and writing down the most important points from the talk – now!:
After 1 dayAfter 1 week
After 3 weeksAfter 2 weeks
Immediately
[ 39 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
E.g. course: ”Databases” (at RUC/Roskilde):
Note: almost entirely non-operational(!)
i.e. measure how?!
obtain knowledge about the structure of database systems; be familiar with design of databases by use of special notations like E/R and analysis through normalization; get an overview of the most important database models and a detailed knowledge about the most important model - the relational model as well as the language SQL; get an overview of database indexing and query processing; obtain knowledge about application programming for DB systems.
Problematic Courses
Familiar with ?!
Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
BONUS SLIDES
[ 41 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Based on John Biggs' Theories
2nd edition
(3rd edition expected this fall)
"Teaching for Quality Learning at University", John Biggs
[ 42 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Teacher’sintention
Student’sactivity
Exam’sassessment
e.g.- explain- relate- prove- apply
e.g.- memorize- describe
UNALIGNED COURSE
e.g.- memorize- describe
"Dealing with the test"
[ 43 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Teacher’sintention
Student’sactivity
Exam’sassessment
e.g.- explain- relate- prove- apply
ALIGNED COURSE
e.g.- explain- relate- prove- apply
e.g.- explain- relate- prove- apply
e.g.- explain- relate- prove- apply
e.g.- explain- relate- prove- apply
[ 44 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Top 10 Competencies
Top 10 Competencies:
" " := { Physics, Chemistry, Biology, Molecular Biology }
Natural Sciences
[ 45 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO (elaborated)Note: the list is non-exhaustive
SOLO 2”uni-structural”
SOLO 3“multi-structural”
SOLO 4“relational”
SOLO 5“extended abstract”
theorize generalize hypothesize predict judge reflect transfer theory
(to new domain) …
analyze compare contrast integrate relate explain causes apply theory
(to its domain) …
combine structure describe classify enumerate list do algorithm apply method …
define identify count name recite paraphrase follow (simple)
instructions …
Graphic Legend
problem / question / cue known related issue - given! hypothetical related issue - not given! student response
Q
R
QUANTITATIVE QUALITATIVE
R
R'Q
RQR2
R3
R1
Q
RQ
RQ
[ 46 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Exercise
Buzz Session:
1) Discuss w/ neighbour:
2) Write it on a Post-It3) Swap Post-Its…
T
Just KeepSwapping…
"which film messages did you find particularly relevant?"
[ 47 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Student Motivation
Susan: (”intrinsic motivation”) - wants to…: learn ! Robert: (”extrinsic motivation”) - to…: pass exams !
[ 48 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Constructivism
”Transmission is Dead…” : (lectures = ) Knowledge is… Actively Constructed !
active teacher &passive students
!
risk
[ 49 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
SOLO Taxonomy
Hierarchy for Competences:
Deep learning (not surface) !
5: generalize, theorize, predict, …4: explain, analyze, compare, …3: describe, combine, classify, …2: recite, identify, calculate, …
[ 50 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Stud Learning Focus
Focus on Student Learning ! (instead of ”what teacher does” & labelling students: ’good/bad’) Student activitation learning
[ 51 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Alignment
Make explicit ILO’s (Intended Learning Outcomes):
(…and tell this to students)
Exam = ILO’s = Teaching
[ 52 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
The Role of the Exam
Alignment: A theory of planning (over the course of a course) A theory of motivation (and incentive)
The exam as a...:"Necessary evil"
Motivational and learning-guidingpedagogical tool for the teacher(!)
applicationof alignment
"The exam does not come after, but before the course!"
[ 53 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Di-Transitive Verbs
Mono-Transitive verbs:
Di-Transitive verbs:
[ 54 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
Data Set (XML and XQuery)(: Extracts all mathematics courses w/ maximum 1 goal and 2 competencies :)
xquery version "1.0";
<result>{ for $course in fn:doc("data-au.xml") //institute[@name = "MAT"]//course let $goals := $course/goal where (fn:count($goals) le 1) and (fn:count($goals/competence) eq 2) order by $course/@name return $course }</result>
Data set:
[ http://www.itu.dk/people/brabrand/solo.xml ][ http://www.itu.dk/people/brabrand/data-au.xml ][ http://www.itu.dk/people/brabrand/data-sdu.xml ]
XML
XQuery
[ 55 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
The BLOOM Taxonomy (1956)
The BLOOM Taxonomy:
Knowledge
Comprehension
Application
Analysis EvaluationSynthesis
Qualitative
Quantitative
SO
LO 4
+5
SO
LO 2
+3
”[…] really intended to guide the selection of items for a test rather than to evaluate the quality of a student’s response to a particular item”
-- (Biggs & Collis, 1982)”
[ 56 ]Claus Brabrand ITiCSE 2009 – Keynote Paris, France (July 06, 2009)
CS vs Math Distributions
Computer Science:
Mathematics:( = 3.06, = 0.24)
( = 3.68, = 0.39)