Date post: | 23-Dec-2015 |
Category: |
Documents |
Upload: | emery-garrett |
View: | 215 times |
Download: | 1 times |
Meaningful Assessment and the Impact of Technology
Meaningful Assessment and the Impact of Technology
Catherine Kelley, Ph.D.
Senior Faculty Consultant
Assistant Professor
Meaningful Assessment and the Impact of Technology
Assessment of Mobile Computing at Seton Hall
Catherine Kelley, Ph.D.
Assistant Professor, Psychology
Senior Faculty Consultant, TLTC
Meaningful Assessment and the Impact of Technology
Why assess?
• Track progress being made on your project
• Greater understanding of teaching & learning
• Ensure high-quality experience for your students
• Communicate successes / challenges to colleagues
Meaningful Assessment and the Impact of Technology
Traditional (Flawed) theoretical model:
• Teaching => Learning
• Teaching + Computers => Improved Learning
• What is wrong with this model?
Meaningful Assessment and the Impact of Technology
Traditional (Flawed) Research Design:
• Control group - no computer
• Experimental group - computer
• Measure exam scores
• What is wrong with this design?
Meaningful Assessment and the Impact of Technology
Some problems with traditional model
• Hawthorne effects• Multivariate nature of change means that an
experiment would be hopelessly confounded• Your outcome variables may (will probably!)
change with new teaching techniques– You may not know yet what the really important
outcome variables are!
Meaningful Assessment and the Impact of Technology
What about a multivariate model?
• So Many Variables!
• Each So Difficult to Measure!
• Huge n required to do any meaningful multivariate study
• Not a bad idea, but was outside of the intended scope of our project
Meaningful Assessment and the Impact of Technology
So what do you do?
• What about educational quality variables?
• Theoretical models of educational quality: e.g. Chickering & Gamson, “Powerful Partnerships,” etc.
• Common themes emerge
Meaningful Assessment and the Impact of Technology
What is “educational quality”?
• Student - professor contact
• Student - student contact
• Prompt feedback on performance
• Explore course materials in many ways
• Take responsibility for learning
• Etc.
Meaningful Assessment and the Impact of Technology
Seton Hall Instruments
• Two Versions: “Standard Form”, “New Form”
• Standard Form has been used for several years
• New Form has been piloted with considerable success
Meaningful Assessment and the Impact of Technology
Standard Form
• “Compare your mobile courses, or those incorporating information technology, to your non-mobile courses, or those not incorporating information technology. Indicate how the use of information technology impacts each of the following items.”
Meaningful Assessment and the Impact of Technology
Standard Form (examples)
• Have contact with my professors on course content
• Spend time with other students on course related materials
• Discuss, write and think about course material in depth
• Get prompt feedback on my performance from my professors ….
• 29 of these questions
Meaningful Assessment and the Impact of Technology
New Form
• Same questions, but students answer only about one course
• And then indicate how much IT is used in that class
• But also, how many traditional pedagogical techniques are used in that class
Meaningful Assessment and the Impact of Technology
Results:Old Form
• Mobile courses look better on educational quality - Sum across all measures
Meaningful Assessment and the Impact of Technology
Results: Old Form
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Very Negative Negative Neutral Positive Very Positive
Pe
rce
nt
Re
sp
on
din
g
Meaningful Assessment and the Impact of Technology
Results: New Form
• The more IT, the higher the educational quality score
Meaningful Assessment and the Impact of Technology
Results: New form
-80
-60
-40
-20
0
20
40
60
80
0 5 10 15 20 25
Number Computer Techniques
Ed
uc
ati
on
al Q
ua
lity
Meaningful Assessment and the Impact of Technology
Results: New Form
• The more traditional techniques, the higher the educational quality score – but effect is very weak
Meaningful Assessment and the Impact of Technology
Results: New Form
-80
-60
-40
-20
0
20
40
60
80
0 10 20 30 40
Number Traditional Techniques
Ed
uc
ati
on
al Q
ua
lity
Meaningful Assessment and the Impact of Technology
Summary of findings
• Students perceive that educational variables are higher in Mobile classes
• Student satisfaction with Mobile remains high
Meaningful Assessment and the Impact of Technology
The future
• Institutionalizing the assessment process– Risks: over-surveying students, poor
response rate– Asking some tough questions: Retention,
student skill, negative impact of technology
• Surveys of faculty, administrators