Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | frederick-barker |
View: | 214 times |
Download: | 0 times |
Moving into the mainstream: Researching the institutional introduction of EVS, from the realm of the enthusiasts to supporting the later adopters
www.herts.ac.uk
Amanda Jefferies School of Computer Science
Marija CubricBusiness School
Contents
Background
Literature review
Methodology
Preliminary findingsFocus on staff views on technology adoption and use of EVS
Conclusions
Background
Major institutional investment by the university in 2010-11 (Assessment and Feedback project) • 3,845 students form 8 academic schools issued with a personal handset from Jan 2011
JISC EEVS project (stand B) , alongside iTEAM Strand A , started in September 2011• EEVS = Evaluating Electronic Voting Systems for Enhancing Student Experience
EEVS Objectives• To provide an up-to-date view of the student and staff experiences of EVS • To identify a set of critical success factors for introducing and maintaining the use of EVS in
support of an institutional assessment and feedback strategy.• Focus on large-scale deployment and differences across multiple disciplines.
Literature Review: Use of EVS in HE
Survey of literature on the introduction of EVS in HE
• 300+ papers on published between 1998 and 2011• A wide variety of writers :
– Brown, Davis, Draper, d’Inverno, Kennedy, Nichol, Oliver, White et al – Our own early adopters :- Lorimer, Hilliard, Thornton and Willis (University of
Hertfordshire)
Most authors reported on local use in classes: • to engage students, • to encourage interactivity, • to support constructivist approaches to teaching and learning in line with e.g. Chickering and
Gamson’s guidelines.
Lack of literature on institutional deployment, with the exception of:• Twetten et al (2007) Successful Clicker Standardisation, Educause Quarterly 4
– Experience from EVS standardisation in four North American universities
Methodology: Data
Exploratory pragmatic approach:• Gaining understanding of the meanings from various stakeholders • Focus on practical applied research, integrating different perspectives to help interpreting data
(Sounders et al 2009)
Mixed-method data collection:• Qualitative data: students’ blogs, staff interviews, staff reports • Quantitative data: students’ questionnaire, staff questionnaire
Time horizon: Mainly cross-sectional (a ‘snapshot’ taken in 2011/12) with some elements of longitudinal study (students’ blogs over a period of 2-4 weeks).
Sampling: • Self-selection (voluntary student participants for questionnaires)• Self-selection & purposive (voluntary student participants for blogs; further selection made to enable
balanced representation across different schools)• Purposive (heterogeneous): staff involved with or leading EVS adoption in different Schools
Methodology: Students’ survey design
Survey launched in February 2012
Includes 27 questions spread across 9 sections
Total number of responses N= 590 (across 11 different Schools)
Response rate: 14.4%. (based on 4093 EVS handsets registered in June 2012 )
Section Source
Background Information
Standard demographic questions
Usage (Venkatesh et al, 2003 Ease of use Bangor et al , 2008Impact on Learning Draper & Brown 2002Performance Expectancy Venkatesh et al, 2003
Engagement JISC Evaluation guidelinesSatisfaction JISC Evaluation guidelinesOperational procedures and funding iTEAM questions
Free text comments
Methodology: Staff survey design
Survey launched in June 2012
Includes 18 questions spread across 5 sections
Total number of responses N= 88 (across 13 different Schools)
Response rate: 5.9%. (based on 1500 academic staff registered on staffq list in June 2012)
Section Source
Background Information Standard demographic questions
Technology adoption (Venkatesh et al, 2003)
Usage (Venkatesh et al, 2003) + iTEAM questions
Effect on teaching practice JISC Evaluation guidelines
Impact on Learning Draper & Brown 2002Operational procedures and funding
iTEAM questions
Free text comments
Methodology: Students’ blogs: Participants
Student blogs were completed in a variety of ways according to individual preferences : • Word Files (46), online blog entries (14), Short video clips (11), Audio files (1) and• submitted via a restricted access group area on StudyNet (UH MLE)
65 students registered for the web/ written blog of whom 27 were chosen to ensure a spread of gender and programme. One failed to complete their blog
7 Academic Schools: Business (6), Computer Science (7), Humanities (6), Law (4), Education (1), Sports management (1), Mathematics (1)
Equal male and female numbers applied but ratios varied according to subjects 15M:11F
All the bloggers had previously used EVS since at least January 2011
The reflection period varied between 2 and 4 weeks in the period Nov 11- Feb 12
Methodology: Student blogs:Original and Emerging Categories and Themes
All Categories Sub-categories (Themes) Total
Pedagogy
Examples of use ,Interesting lectures, Subject suitability, Subject-specific EVS use, Improvements in use, Potential use, Learning
292Formative use, Inadequate use, Misaligned expectations, Variety of assessment, Edutainment and more …
Usage Previous experience with EVS ,Use by modules and levels ,Frequency of use 78
TechnologyType of handset ,Technical issues ,EVS functions ,
99Changing channels
Institut’l benefits Attendance ,Institutional motivation 47
Expectations Initial thoughts ,Institutional use of data 37
Effort Expectancy Ease of use, Accessibility 40
Facilitating conditions Training ,Handsets distributionTechnical competence of tutors, Tutor's experience with EVS use/question writing 56
Institutional deployment
Handset ownership32
Handset cost, Inconsistent application, Too ambitious implementation
Social aspects Anonymity 28
Student experience Satisfaction and Dissatisfaction 21
Personal emotions Anxiety, and extra pressure, privacy 7
Methodology: Staff interviews: Participants and Developing Categories and Themes Original
CategoriesSub-categories (Themes)
Technology Adoption
Drivers for change, Institutional Support for technology adoption, Positive factors of EVSNegative factors of technology adoptionNegative factors of EVS usageEmbedding EVS successfully
Personal Competence
Training & Support at institutional levelSupport at local levelChampions for EVS Benefits of Training
Changing styles of Pedagogy
Use of MCQs for learning and teachingDeveloping Question BanksBenefits of EVS for Student support
Semi-structured interviews with12 staff members incl:• Local School
Champions (2)• Associate Heads
and Heads of School (3)
• Lecturers (6)• The iTEAM Project
Director (1)
Preliminary Findings : Summary of students’ views (survey and blogs) Usability or ease of use of the EVS was rated very highly by students
Very positive impact on students’ learning and satisfaction:• Responding to questions makes me think more about the
course material (80%+)• EVS provides them with an immediate check of their
understanding (80%+)• Viewing responses give them an idea of how they are doing in
relation to the rest of the class (80%+)
Less positive impact on perceived performance and engagement :• use of EVS will increase their chances of getting a good grade
(49.9%)• Spend more time discussing the subject with colleagues
(47.5%)• Thinks about the subject outside of the classroom (34%)• Ask more questions in classes that use EVS (21%)
Time given for answering questions is not always sufficient
Formative use of EVS has been seen as adding a value to students’ learning experience, however summative use has in some cases created “an unfortunate tension between the administrators, the tutors and the students” (MC12)
All students find the EVS very easy to use, with little or no training; however, accessibility might be an issue for students with sight or dexterity impairments. Majority agree that EVS is a great tool for answering QUESTIONS, but inadequate as a TEST tool
Preliminary Findings : Staff Survey
There is an overall (UH-wide) highly positive attitude to technology (EVS) acceptance with 83% of surveyed staff intending to use EVS in the next 6-12 months.
• A more positive attitude in schools from Social Sciences and Arts subject area than in other subject groups
There is an overall (UH-wide) high agreement on handing out EVS for free upon registration and disagreement that the use of EVS has led to immediate operational efficiencies
Preliminary Findings : Staff interviewsThe Head of School
‘Our biggest challenge was getting staff on board with it. I would think when I started, 25% of staff were very enthusiastic and 75% were not, now we’re probably running at 50/50. Everybody who’s used EVS and got it to work comes back and says it’s great and they’re going to use it again… the technology has got to be easy so that people can get in the swing of it’ Head of School interview June 2012
Preliminary Findings : Staff interviewsThe Lecturers
‘Once it’s embedded in the workload for staff then that should be greatly reduced in terms of marking, writing assignments, it’s really refreshing the bank of multiple choice questions [they’ve created.]’
…it was so obviously simple to generate questions and I think that has always been the main barrier in people’s minds, that because it’s something new and that it’s a technology that it has to be difficult as well
Conclusions
• High student enthusiasm for using EVS in class especially for formative and low-stakes summative assessments
• Plenty of top-down support and leadership and local support is important to embed the technology
• Academics using EVS are not yet seeing all the expected efficiencies in time saved that they had hoped for.
• BUT 93% of those responding to the survey and using it for the first time in 2011 said they plan to use it in the next 6 -12 months
• Patterns of previous technology adoption at UH are mirrored –– The example of the MLE saw early student enthusiasm but
academics took more time to feel fully at ease with the change in practice and pedagogy.
References• Bangor, A., Kortum, P., & Miller, J.A. (2008). The System Usability Scale (SUS): An Empirical Evaluation, International Journal of Human-Computer
Interaction, 24(6).
• Chickering and Gamson http://www.uis.edu/liberalstudies/students/documents/sevenprinciples.pdf
• D’Inverno, R., Davis, H., & White S. (2003), Using a personal response system for promoting student interaction, Teaching Mathematics and its Applications, Vol. 22 (No. 4): 163-169.
• Draper, S. W., & Brown, M. I.(2002).Use of the PRS handsets at Glasgow University, Interim Evaluation Report: March 2002 http://www.psy.gla.ac.uk/~steve/evs/interim.html accessed April 2012
• Draper, S. W., & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20(2), 81-94.
• JISC (2004) http://www.jisc.ac.uk/media/documents/publications/effectivepracticeelearning.pdf
• Kanter, R.M., Stein, B.A. & Jick, T.D. (1992) The Challenge of Organizational Change. Free Press. New York
• Kennedy, G. E., & Cutts, Q. I. (2005). The association between students' use of an electronic voting system and their learning outcomes. Journal of Computer Assisted Learning, 21(4), 260-268
• Lorimer, J., & Hilliard, A. (2009). Use of a Electronic Voting System (EVS) to Facilitate Teaching and Assessment of Decision Making Skills in Undergraduate Radiography Education. Paper presented at the Proceedings of 8th European Conference on e-Learning, Bari, Italy
• Moore, G. A. (1991). Crossing the Chasm. New York: Harper Business.
• Nicol, D., & Draper, S. (June 2009). A blueprint for transformational organisational change in higher education: REAP as a case study. In J. T. Mayes (Ed.), Transforming Higher Education through Technology-Enhanced Learning.
• Oliver, M. (2006). New pedagogies for e-learning? ALT-J: Research in Learning Technology, 14(2), 133 - 134.
• Robins, K., (2011) EVS in the Business School, University of Hertfordshire Internal Report
• Thornton, H. A. (2009). Undergraduate Physiotherapy students’ choice and use of technology in undertaking collaborative tasks. Open University, UK, Milton Keynes
• Twetten, J., Smith, M.K., Julius, J. and Murphy-Boyer, L. (2007) Successful Clicker Standardization EDUCAUSE QUARTERLY • Number 4 2007
• Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. B. (2003). User acceptance of information technology: toward a unified view. MIS Quarterly, 27(3), 425–478.
• Willis, J. (2009) Using EVS in the School of Life Sciences, University of Hertfordshire Internal Report