1Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
QUALITY ASSURANCEIN LARGE SCALE E-ASSESSMENTS
Prof. Dr. Heinz-Werner Wollersheim
2Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
Content
1) Quality assurance: more than a technical problem
2) Aspects of quality assurance in MCA
3) Shortcut: important terms
4) Quality assurance process in MCA
5) Prospects
3Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
1) Quality assurance: more than a technical problem
4Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
1) More than a technical problem
qualitymanagement
qualityassurance
analysiscurrent state
implementingimprovementsdocumentation
measuringeffects
1. detecting & making awareprocesses
2. detecting & fixing frictions
5Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
1) Quality assurance: more than a technical problem
optimizationcircle
ofquality
development
qualityassurance
analysiscurrent state
implementingimprovementsdocumentation
measuringeffects
6Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
1) Quality assurance: more than a technical problem
qualitymanagement
Goals to achieve:
Doing the right things. (effectiveness)
Doing the things in the right manner. (efficiency)
Doing the things at the right time. (efficiency)
„Our mission is to do the right things right at the right time.“
7Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
1) Quality assurance: more than a technical problem
Beyond technical “controlmania“
§ technical control is working just in addition to social control (not asa substitute)
§ reaching high acceptance: university students, colleagues
§ needs lots of communication: The right things done right at the right time?
§ But: Assessing verifiable objectives is indicating a shift from holistic„education“ to specific „training“
8Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
1) Quality assurance: more than a technical problem
Doing the right things: validity
kinds of validity relevance determining/ indicators
content validity best way to operationalize contents ratings by experts
construct validity Does the test measure the intendedlearning outcomes and competences?
a) convergent validity(data of tests designed to measuresimilar competences are highly correlating)
b) divergent validity(data of tests designed to measuredifferent competences are lowly correlating)
criterion validity correlation between measurementinstrument and empirical criteria
a) diagnostic validity
a) prognostic validity
9Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
1) Quality assurance: more than a technical problem
Academic teaching: typical issues
§ common practice at the high schools and universities:modules and courses are designed at most
§ challenge:developing quality assured assessments in this context
§ validity reduced to content validity
§ Modified Constructive Alignment (MCA)
10Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
2) Aspects of quality assurance in MCA
11Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
2) Aspects of quality assurance in MCA on
§ objectivity: implementing objectivity
objectivity of analysis
§ validity: content validity à ratings by expertsà assessement plan (blueprint)
§ reliability: internal consistency à consistency analysis
12Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
2) Aspects of quality assurance in MCA on
analysis of items:
§ empirical difficulty: index of difficulty 20 ≤ Pi ≤ 80
§ discrimination: coefficient of discrimination r ≥ .20
13Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
2) Aspects of quality assurance in MCA
§ objectivity: objectivity of implementation
objectivity of analysis
§ validity: content validity à ratings by experts, à assessement plan (blueprint)
§ reliability: internal consistency à consistency analysis
§ empirical difficulty: index of difficulty à 20 ≤ Pi ≤ 80
§ discrimination: coefficient of discrimination à r ≥ .20
14Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
15Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
§ selected response
– selected response: multiple possible answers, choosing the right one(s)
– legal term (Germany): Antwort-Wahl-Verfahren
– terms that describes the wayto give an answer: multiple-choice question type (MCQ),
matching, sequence, hotspot
– assessment question: item
16Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
§ item
“The smallest separately identified question or task within an assessment plus, its associated information (for example mark scheme, curriculum reference, media content, performance information etc), usually a single objective question. Distinguished from a ‘question’, which may be a longer and less-objective task but often used synonymously.”
(Qualifications and Curriculum Authority 2007, 107)
17Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
§ selected response
“The selected-response item format is the best choicefor test developers interested in efficient, effective measurement of cognitive achievement or ability.”
(Downing 2006, 287)
18Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
§ item structureDas Bild kann derzeit nicht angezeigt werden.high quality structure
item stem
vignette and question
options
correct answerand
several incorrect answer(distractors)
more detailed stem
short stem
(A) short options (A) long options
avoid
(B)
(C)
(D)
(E)
(B)
(C)
(D)
(E)
19Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
§ constructive alignment: the german view
Ø „Beim 'Constructive-Alignment'-Konzept geht es im Kern darum, dassdie intendierten Outcomes des Lernprozesses klar definiert und denStudierenden explizit verdeutlicht werden und die Prüfungs- undLernaktivitäten stringent auf die Learning Outcomes abgestimmtwerden.“
(Schaper 2012, 62)
à In nuce C.A. means coherence betweenlearning outcomes,assessment andlearning process
20Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
§ constructive alignment
“In constructive alignment, we start with the outcomes we intend students to learn, and alignteaching and assessment to those outcomes. The outcome statements contain a learning activity, a verb, that students need to perform to best achieve the outcome, such as “apply expectancy-value theory of motivation”, or “explain the concept of … “. That verb says what the relevant learning activities are that the students need to undertake in order to attain the intended learning outcome. Learning is constructed by what activities the students carry out; learning is about what they do, not about what we teachers do. Likewise, assessment is about how well they achieve the intended outcomes, not about how well they report back to us what we have told them. […] Constructive alignment can be used for individual courses, for degree programmes, and at the institutional level, for aligning all teaching to graduate attributes.”
Source: http://www.johnbiggs.com.au/academic/constructive-alignment/
21Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
3) Shortcut: important terms
§ learning outcomes
Ø “Learning outcomes describe what a learner is expected to know,understand and be able to do after successful completion of a processof learning.”
(ECTS Users‘ Guide, Europäische Gemeinschaft 2009, 11)
22Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
23Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
§ focus:
24Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
2 dimensions of quality assurance
pla
nnin
g m
odel
for
did
act
ics
in h
ighe
red
uca
tion
constructive alignment
analysis evaluation
process stages
development implementation
25Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
Planning stages according to constructive alignment
learning outcomes§ defining intended learning outcomes
before starting the course§ align teaching and assessment to those
outcomes
designing tests§ developing assessment according to
intended learning outcomes § adapting those outcomes if necessary
designing learning process§ designing learning process according to
intended learning outcomes after developing assessment
1
2
3
learningoutcomes
learning process and
learning activities
assessment
1
2
3
26Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
Workflow: Creating e-assessements according to constructive alignment
Das Bild kann derzeit nicht angezeigt werden.
plan
ning
leve
ls
modulcourse (seperate course unit)
modul
workload/ ECTS
contentstrucutredesign
of the course
topi
c of
a c
ours
e
indexing of topics and
contents
previous knowledge
competence descriptions
DQR/ HQR
issues to be considered
lear
ning
out
com
es th
atar
ere
lava
ntfo
ras
sess
men
ts(if
nece
ssar
y)
form
atan
d m
etho
dof
asse
ssm
ent
asse
ssm
ent p
lan
(blu
eprin
t) of
targ
etst
ate
(con
tent
s, p
erfo
rman
cele
vels
, ty
pes
ofte
stite
ms)
blue
prin
tof
cur
rent
stat
e
peer
-revi
ew p
roce
ss
crea
ting
item
s
crea
ting
test
item
poo
ls
designing tests compiling tests
crea
ting
test
s
plan
ning
stag
es
topi
car
eas
spec
ific
cont
ents
lear
ning
out
com
es
27Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
2 dimensions of quality assurance
pla
nnin
g m
odel
for
did
act
ics
in h
ighe
red
uca
tion
constructive alignment
analysis evaluation
process stages
development implementation
specific implementation?
28Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
simplified process model for e-assessment
developingan e-exam
implementingan e-exam
adaptingan e-exam
analysing and evaluatingan e-exam
29Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
developingan e-exam
implementingan e-exam
adaptingan e-exam
analysing and evaluatingan e-exam
§ constructive alignment (Biggs and Tang 2007)§ taxonomy for learning, teaching, and assessing
(Anderson and Krathwohl 2001)§ learning outcomes § assessment plan (blueprint)§ scoring model§ content and form of test items§ peer-review
analysis of tests and items§ empirical difficulty§ discrimination§ reliability
renormalization
§ practice e-exam session§ instructions for working
with test items§ risk management
2 levels:§ content criteria for organizing the
course and developing the tests§ ideas for innovating the electronical
assessments system
30Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
§ knowledge of constructive alignment (Biggs and Tang 2007) and the taxonomy for learning, teaching, and assessing (Anderson and Krathwohl 2001)
§ knowledge of the form and content of correct test items
§ different work assistance tools to organize the developing process§ defining learning outcomes for courses§ designing the assessment plan (blueprint):
number of items, performance levels, forms of knowledge, topics; ensuring one-dimensionality
§ calculating guessing probability
§ organizing the item developing (work assistance tools)§ organizing the peer review process § creating tests:
selecting items according to the assessment plan (blueprint)equal opportunities in case of more than one test in course
Entwicklung der E-Klausur
developingan e-exam
31Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
implementingan e-exam
§ training version of the e-exam session: university students get to know about testprocedure (e.g. navigation) and kinds of item types and designs to become familiar withundertaking e-exams
§ instructions for working with test items
§ transparency of scoring model, e.g. getting marks according to all-or-nothing-principle orfor each part of the right answer
§ risk management: controling examination server, back-up server, log files, bug documentation
32Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
analysing andevaluating an e-Exam
analysis of items and test:
item level:§ empirical difficulty: index of difficulty 20 ≤ ≤ 80 § selectivity (corrected item-total discrimination):
coefficient of item discrimination r ≥ .20
test level:§ reliability: internal consistency à consistency analysis .80 ≤ KR 20 ≤ .90
renormalization
33Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
4) Quality assurance process in MCA
simplified process model for e-assessment
developingan e-exam
implementingan e-exam
adaptingan e-exam
analysing and evaluatingan e-exam
34Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
5) Prospects
35Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
5) Prospects
objectivity
validity
reliability
empirical difficulty
selectivity / item-total discrimination
security against miscalculation
scaling
dealing with guessing of university students
dealing with different valuate modes in item analysis
quality criterion/ quality feature actuallysolved
implementedin
ILIAS 4.3.7
36Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
5) Prospects
§ descriptive statistics + inferential statistics are necessary toverification the one-dimensionality of the items
„Die einzig angemessene Umgehensweise mit dem Problem des Ratens ist der Einsatz ganz spezieller Methoden der IRT, die bei der Schätzung des jeweils gesuchten Fähigkeitsparameters einer Person aufgabenspezifisch das faktische Erfolgsausmaß beim Versuch des Lösungerratens mit ein-kalkulieren (d.s. insbesondere das 3-PL Modell und das Difficulty plus Guessing PL Modell; vgl. wieder Kubinger, 2009).“
(Kubinger 2014, 170)
§ Beispiel: Verrechnungsregel im Rahmen der Skalierung(z.B. Anzahl gelöster Aufgaben)à sollte mit Hilfe der Item-Response-Theorie auf empirische Angemessenheit geprüft werden
37Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
5) Prospects
dealing with guessing – proposal for solution I:
§ guessing probability included in the calculation of scoring model
à issue: different behaviour of university studentsin relation to guessing
38Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
5) Prospects
dealing with guessing – proposal for solution II:
§ increasing the number of distracors§ “1 of 6“, “1 of 7“, “1 of 8“§ “x of 5“, scoring model with all-or-nothing-principle
à issues:1. valid assessment required appropriate distracors
(e.g. plausible, homogeneous)à analysing distractor quality
2. scoring model with all-or-nothing-principle (applied to “x of 5“) assumes that partly knowledge is not enough
(cf. Kubinger 2014)
39Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
5) Prospects
minimum goals to reach validity
creating workflow tomanaging the working process, observing the standards,improving the usability
40Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
literature reference
Amtsblatt der Europäische Union (AblEU) Nr. 2008/C 111/01 v. 6.5.2008.Source: http://eur-lex.europa.eu/LexUriServ/LexUri-Serv.do?uri=OJ:C:2008:111:0001:0007:DE:PDF
Downing, Steven M. (2006): Selected-Response Item Formats in Test Development. In: Downing, Steven M. / Haladyna, Thomas M.: Handbook of Test Development. Mahwah, N.J., S. 287-301.
Europäische Gemeinschaft (2009): ECTS-Leitfaden. Source: http://ec.europa.eu/education/tools/docs/ects-guide_de.pdf
Fisseni, H.-J. (1990): Lehrbuch der psychologischen Diagnostik. Göttingen: Hogrefe.
Kubinger, K. D. (2009): Psychologische Diagnostik - Theorie und Praxis psychologischen Diagnostizierens. Göttingen: Hogrefe.
Kubinger, K. D. (2014): Kubinger, Klaus D. (2014): Gutachten zur Erstellung „gerichtsfester” Multiple-Choice-Prüfungsaufgaben. In: Psychologische Rundschau 65 (3), S. 169–178.
Lienert, G. A. & Raatz, U. (1998): Testaufbau und Testanalyse. Weinheim: Beltz PVU.
Schaper, N. (2012): Fachgutachten zur Kompetenzorientierung in Studium und Lehre. Source: http://www.hrk-nexus.de/fi leadmin/redaktion/hrk-nexus/07-Downloads/07-02-Publikationen/fachgutachten_kompetenzorientierung.pdf
Qualifications and Curriculum Authority (2007): e-Assessment. Guide to effective practice.Soruce: http://www.e-assessment.com/wp-content/uploads/2014/08/e-assessment_-_guide_to_effective_practice_full_version.pdf
41Prof. Dr. H.-W. Wollersheim
Quality assurancein Large Scale E-Assessments
Tempus Workshop Laubusch 2016
THANK YOU FOR LISTENING.
Prof. Dr. Heinz-Werner WollersheimLeipzig University
Institute of Educational SciencesChair for General Pedagogy