Revisiting e-Assessment: Revisiting e Assessment: Leveraging online platforms for outcomes assessment in the general education
Jennifer Hill, Ed.D.
Matt Serra Ph DMatt Serra, Ph.D.
Office of Assessment, Trinity College
Duke University
assessment.aas.duke.edu
1
We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning
i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g
[ and ]
Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and
l i l l b ti f l iexplain local observations of learning.
4
We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning
i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g
[ and ]
Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and
l i l l b ti f l i
How do we do this at Duke?explain local observations of learning. Duke?
5
Gen Ed assessment initiative
Our purposes are
( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and
(2) to follow the development of student competencies, broadly defined, over time.
6
Gen Ed assessment initiative
Our purposes are
( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and
(2) to follow the development of student competencies, broadly defined, over time.
Project design allows (a) wide engagement among incoming and exiting students
7
Gen Ed assessment initiative
Our purposes are
( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and
(2) to follow the development of student competencies, broadly defined, over time.
Project design allows (a) wide engagement among incoming and exiting students, (b) collection of data from three separate nationally-normed instruments, three separate nationally normed instruments,
8
Gen Ed assessment initiative
Our purposes are
( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and
(2) to follow the development of student competencies, broadly defined, over time.
Project design allows (a) wide engagement among incoming and exiting students, (b) collection of data from three separate nationally-normed instruments, and (c) three separate nationally normed instruments, and (c) efficiencies of administration, data analysis, and reporting.
9
Gen Ed assessment initiative (the front end)
What
To whom
When
How
10
Gen Ed assessment initiative (the front end)
What: Administration of several assessment instruments i t t d t ’ t i l ti t D k E h t t prior to students’ matriculation at Duke. Each targets a
general education student learning outcome.
To whom To whom
When
H How
11
Gen Ed assessment initiative (the front end)
What
To whom: Voluntary sample of incoming first year (non-transfer) students, A&S and Engineering
When
How
12
Gen Ed assessment initiative (the front end)
What
To whom
When: Project started in summer 2010 and continues jannually. Early communications start in April, with periodic reminders in July and August. Data collection ends at the start of first year student orientation.
How
13
Gen Ed assessment initiative (the front end)
What
To whom
When
How: Deploy three online, externally hosted instruments via a central project homepage. Data are returned to us b tby our partners.
14
Gen Ed assessment initiative (the front end)
15
Gen Ed assessment initiative (the front end)
16
Gen Ed assessment initiative
Defining Issues Test (DIT-2)ethicaldevelopment.ua.edu
Global Perspectives Inventory (GPI)gpi.central.edu
Reasoning about Current Issues Test (RCI)reflectivejudgment.org
Critical thinking Assessment Test (CAT)tntech.edu/cat/home
17
Gen Ed assessment initiative
Publicity: Blurb in “Blue Book”. Email from Director of Academic Advising Center Class Facebook page Academic Advising Center. Class Facebook page posting. Direct email from our office.
Incentives
Consent
Return rates
Challenges
Findings Findings
Next steps18
Gen Ed assessment initiative
Publicity
Incentives: Drawing among eligible students. Awards and award values vary. “Hierarchy” of awards.
Consent
Return rates
Challenges
Findingsg
Next steps19
Gen Ed assessment initiative
Publicity
Incentives
Consent: Consent statement is embedded on initial project homepage, and on each instrument’s website.
Return rates
Challenges
Findingsg
Next steps20
Gen Ed assessment initiative
Publicity
Incentives
Consent
Return rates: Surprisingly high in year 1, fluctuating slightly over time. Lower among graduating cohort.
Challenges
Findingsg
Next steps21
Gen Ed assessment initiative
Publicity
Incentives
Consent
Return rates
Challenges: Monitoring completions on externally hosted b R t d i i t ti t i di t web apps. Remote administration prevents immediate
distribution of incentives. Maintaining high return rates.
Findings
Next steps
22
Gen Ed assessment initiative
Selected findings from pre-tests (matriculates 2010-2014):
Our voluntary samples of study participants are large and tend to be representative of their respective incoming classes classes.
Results at the pre-test tend to be stable across incoming classes. classes.
Individual students’ results and Duke’s overall results are emailed to them as early as possible. Content of the mailing is confidential, secure, contextualized, and written in accessible language.
23
Gen Ed assessment initiativeResponse rates by instrument, and by matriculating cohort
Overall59% Overall70%
80%90%
100%
d st
uden
ts
Overall48% Overall
42% Overall40% Overall
36%
30%40%50%60%
nt o
f inv
ited
DIT-260%
DIT-246%
DIT-243%
DIT-241%
DIT-238%
GPI66%
GPI54%
GPI47%
GPI42%
GPI33%
RCI52%
RCI45%
RCI37%
RCI38%
RCI36%
0%10%20%
Incoming 2010 Incoming 2011 Incoming 2012 Incoming 2013 Incoming 2014
Perc
en
24
g g g g g
DIT-2 GPI RCI Overall
GPIGPIGlobal Perspectives InventoryGlobal Perspectives Inventory
25
Gen Ed assessment initiative: GPI
1.0
Observed movement in GPI results (pre-test to post-test)Repeat measures cohort (N=119), graduating class of 2014
0.50.60.70.80.91.0
0.00.10.20.30.4
* * * * * al al al y n
Ove
rall G
PI
Cog
nitiv
e
og. K
now
ing
. Kno
wle
dge
ntra
pers
onal
Intra
pers
ona
Iden
tity
Intra
pers
ona
Affe
ct*
Inte
rper
son a
erpe
rson
alRe
spon
sibilit
y
terp
erso
nal
al In
tera
ctio
n
Co
Cog
. In
Inte
Soci
al
Int
Soci
a
26
* Represents significance at p < 0.001
Gen Ed assessment initiative: GPIst
Observed movement in GPI results (pre-test to post-test)Repeat measures cohort (N=119), graduating class of 2014
By duration of foreign language coursework
0.5
0.6
0.7
est t
o po
st-te
s By duration of foreign language coursework
0.3
0.4
ovem
ent,
pre-
te
Exactly twoclasses (N=15)*
Exactly one class (N=22)*
0.0
0.1
0.2
3 50 3 60 3 70 3 80 3 90 4 00gnitu
de o
f mo
Non-completion of FL requirement (N=21)
classes (N=15) class (N=22)
Exactly threeclasses (N=61)*
27
3.50 3.60 3.70 3.80 3.90 4.00
Mag
Mean overall GPI score at pre-test* Represents significance at p < 0.01
Gen Ed assessment initiative: GPI
0.7test
Observed movement in GPI results (pre-test to post-test)Repeat measures cohort (N=119), graduating class of 2014
By curricular and co-curricular experiences
0.5
0.6
e-te
st to
pos
t-
Participation in exactly
0 2
0.3
0.4
mov
emen
t, pr
e p y2 SL courses (N=10)*
Participation in study abroad (N=68)*
0.0
0.1
0.2
Mag
nitu
de o
f m Participation in 1 or more DukeEngage experiences (N=36)*
Participation in exactly 1 SL course (N=43)*
28
3.50 3.60 3.70 3.80 3.90 4.00M
Mean overall GPI score at pre-test* Represents significance at p < 0.01
Gen Ed assessment initiative: GPIte
st
Observed movement in GPI results (pre-test to post-test)Repeat measures cohort (N=119), graduating class of 2014
By division of the students' first major
0.5
0.6
0.7
e-te
st to
pos
t-t
S i l S i (N 41)*
0 2
0.3
0.4
mov
emen
t, pr
e Social Sciences (N=41)*
Humanities (N=21)* Other (N=3)
0.0
0.1
0.2
3.50 3.60 3.70 3.80 3.90 4.00Mag
nitu
de o
f m
Engineering (N=14)Natural Sciences (N=40)
29
3.50 3.60 3.70 3.80 3.90 4.00M
Mean overall GPI score at pre-test* Represents significance at p < 0.05
DIT 2DIT-2Defining Issues TestDefining Issues Test
30
Gen Ed assessment initiative: DIT
3 0nd. s
core
,
Observed movement in DIT-2 results (pre-test to post-test)Repeat measures cohort (N=101), graduating class of 2014
By curricular and co-curricular experiences
1.52.02.53.0
nge
in T
ype
Into
Pos
t-tes
t
0.00.51.0
01)*
FL
27)*
FL
=9)*
L 49)*
udy
* ctly
)* ct
ly
0)itude
of c
han
Pre-
test
ticip
ants
(N=1
0
Com
plet
ion
of
requ
irem
ent
with
1 c
lass
(N=2
Com
plet
ion
of F
requ
irem
ent
th 2
cla
sses
(N=
ompl
etio
n of
Fre
quire
men
t h
3 cl
asse
s (N
=4
ticip
atio
n in
Stu
Abr
oad
(N=5
5)*
cipa
tion
in e
xac
L co
urse
(N=3
1)
cipa
tion
in e
xac
L co
urse
s (N
=10
Mag
ni
All
par C w C
wit C
wit h Pa
rt A
Parti
c1
SL
Parti
c2
SL
* Represents significance at p < 0.05
32
Gen Ed assessment initiative: DIT
e,
Observed movement in DIT-2 results (pre-test to post-test)Repeat measures cohort (N=101), graduating class of 2014
By division of first major
2 0
2.5
3.0
pe In
d. s
cor
-test
y j
1.0
1.5
2.0
chan
ge in
Ty
-test
to P
ost-
S i l S i (N 31)*E i i (N 18)
0.0
0.5
1 0 2 0 3 0 4 0 5 0 6 0 7 0Mag
nitu
de o
f cPr
e- Social Sciences (N=31)*
Humanities (N=16)
Engineering (N=18)
Natural Sciences (N=34)*
33
1.0 2.0 3.0 4.0 5.0 6.0 7.0M
Mean overall DIT-2 Type Indicator score at pre-test* Represents significance at p < 0.05
RCIRCIReasoning about Current Issues testReasoning about Current Issues test
34
Gen Ed assessment initiative
Next steps:
Disseminate program-level findings to stakeholders; facilitate informed discussion and improvement of pedagogy and curricula.
Continue reviewing pre-test/baseline data as cohorts matriculate.
Longitudinal and additional cross sectional analyses continue Longitudinal and additional cross-sectional analyses continue. Review study findings in the context of other assessment endeavors underway at Duke.
Continue to explore other instruments (e g quantitative Continue to explore other instruments (e.g., quantitative reasoning), rotating instruments as assessment needs evolve.
37