Ensuring Success: Assessing the ePortfolio I f t t f D L iInfrastructure for Deep Learning
Kathryne D. McConnell, C. Edward Watson, Peter Doolittle, M A L i & M Z ldiMary Ann Lewis, & Marc ZaldivarJanuary 28, 2012 – AAC&U ePortfolio Symposium
OverviewOverviewOverviewOverview
• Who we are why we are here• Who we are, why we are here• 8 issues of implementation• Lessons learned• Lessons learned• Future directions
2
Overview of Virginia TechOverview of Virginia TechOverview of Virginia TechOverview of Virginia Tech
• Comprehensive Research Institution• Comprehensive, Research Institution• Land Grant – Nine colleges and grad school• 65 bachelor's degree programs• 65 bachelor s degree programs• 145 master's and doctoral degree programs
31 000 t t l FTE• ~31,000 total FTE• ~1,700 full-time
t hi f ltteaching faculty
Who We Are, How We Work on FYEWho We Are, How We Work on FYEWho We Are, How We Work on FYEWho We Are, How We Work on FYE“facilitate and promote a climate of continuous
Assessment
“facilitate and promote a climate of continuous improvement of student learning”
(Office of Assessment & Evaluation)
“design, develop and
“provide a
TechnologyPedagogy
(Center for Instructional Development &
develop, and implement disciplinary and interdisciplinary learner
learning infrastructure that meets modern needs
(Learning Technologies)Development & Educational Research
CIDER)
learner-centered instruction”
for integrating technology”
Copyright © 2011 by Kathryne Drezek McConnell, Virginia Tech
What Started Us on This What Started Us on This Particular Journey?Particular Journey?
SACSSouthern Association of Colleges and SchoolsSouthern Association of Colleges and SchoolsQEPQuality Enhancement PlanQuality Enhancement PlanFYEFi t Y E i P th t SFirst Year Experience – Pathways to Success
5
Essential OutcomesEssential Outcomes
Problem Solving
Lifelong Learning
Integration ofI i
Learning
Integration of LearningInquiry
6
Essential ElementsEssential ElementsEssential ElementsEssential Elements• Ground in AAC&U Essential Learning
OutcomesOutcomes• Develop Student Centered Paradigms• Design and Use Multi faceted Assessment• Design and Use Multi-faceted Assessment
Strategies• Base in Disciplines• Base in Disciplines• Include Existing and Proposed Institutional
CommitmentsCommitments• Incorporate Technology to Enhance Learning
and Assessment -> ePortfoliosand Assessment ePortfolios
7
8 Issues Central to Implementation 8 Issues Central to Implementation (Chen & Light, 2010)(Chen & Light, 2010)
5 Including multiple1 Defining learning 5. Including multiple forms of evidence
6. Using rubrics to
1. Defining learning outcomes
2. Understanding your evaluate ePortfolios
7. Anticipating external uses of evidence
learners3. Identifying
stakeholders uses of evidence8. Evaluating the impact
of ePortfolios
stakeholders4. Designing learning
activities
8
Which of the 8 issues are Which of the 8 issues are YOUR issues? Take a look at your listYOUR issues? Take a look at your listyour list…your list…• Which of the 8 are strengths, under control,
and/or you have a good handle on at your institution?
• Which of the 8 are challenges, problems, d/ k t i tit ti ?and/or weaknesses at your institution?
9
@ Virginia Tech@ Virginia Tech• Going through these issues at the time of
implementation, we would have:Identified a fe strengths and challenges• Identified a few strengths and challenges
• “Divided and conquered” issues by Office responsibilityp y
• Big lesson learned – needed to revisit ALL 8 ISSUES with our faculty partnersy p
• Faculty are NOT just a stakeholder!
10
Learning OutcomesLearning Outcomes• Lessons Learned
• Can be “sea change”N d t h l f lt “fit” ithi th i ( )• Need to help faculty see “fit” within their course(s)
• Cautionary TalesBeware the “nods and smiles”• Beware the nods and smiles
• Assumptions that certain outcomes “just happen automatically” and do not require instructiony q
11
Understanding Your LearnersUnderstanding Your LearnersUnderstanding Your LearnersUnderstanding Your Learners• Lessons LearnedLessons Learned
• The myth of the “Digital Native”• Survey studentsy
• Cautionary Tales (Questions to Ask)• What is their prior knowledge of the content area?• What is their attitude toward content/ePortfolio?• What technology skills do they actually possess?• Do they have a learning preference?
Identifying StakeholdersIdentifying StakeholdersIdentifying StakeholdersIdentifying Stakeholders• Lessons LearnedLessons Learned
• ePortfolio is only multi-purpose when purposefully designed to be so
• Faculty not just stakeholders• Cautionary Tales
• Overselling the power of ePortfolio to leadership can set you back significantly
Designing Learning ActivitiesDesigning Learning ActivitiesDesigning Learning ActivitiesDesigning Learning Activities• Lessons LearnedLessons Learned
• Course-embedded instructional activities first, ePortfolio reflection and synthesis second
• Embed ePortfolios in pedagogy, not as an add-on • Cautionary Tales
• Move beyond the ePortfolio as simply a container• Understanding ePortfolios for instruction takes time
Multiple Forms of EvidenceMultiple Forms of EvidenceMultiple Forms of EvidenceMultiple Forms of Evidence
• Lesson Learned• Scaffolding for faculty and for students important here
D ’t ll t d t f d t ’ k• Don’t collect data for data’s sake• Cautionary Tales
Be clear about what specifically you are collecting• Be clear about what, specifically, you are collecting evidence for –
• To evaluate a specific learning outcome?• To evaluate reflection (outcome and/or process)?• To evaluate the overall portfolio (product or process)?
Using RubricsUsing RubricsUsing RubricsUsing Rubrics
• Lesson Learned• Another sea change for some
D t d t t ll d l d b i h l l• Do not adopt any externally developed rubric wholesale• Cautionary Tales
Be honest rubrics (at least initially) more work but• Be honest – rubrics (at least initially) more work, but more meaningful work
• Importance of alignmentp g
External UseExternal UseExternal UseExternal Use
• Lesson Learned• Again, don’t oversell
F d t i t/ t bilit• Focus on pedagogy, not requirement/accountability• Cautionary Tales
Mixed Metaphor Time: Requirement only brings folks to• Mixed Metaphor Time: Requirement only brings folks to the ePortfolio water, does not make them drink the Koolaid
• Consider smaller, “first steps”
Evaluating the ImpactEvaluating the ImpactEvaluating the ImpactEvaluating the Impact
• Lesson Learned• Iterative process
St t ith f lt• Start with faculty• Redefine infrastructure = “Technology +”• Deep learning only possible when tools work faculty• Deep learning only possible when tools work, faculty
know how to work them• Cautionary Talesy
• Any evaluation must account for all subtleties/differences in users’ level of adoption
Future DirectionsFuture DirectionsFuture DirectionsFuture Directions
• Instructors’ Perceptions of ePortfolio Survey• Instructors Perceptions of ePortfolio Survey• FYE – Connect-to-Learning• ePortfolio Innovation Committee (EPIC)• ePortfolio Innovation Committee (EPIC)
19
Questions?Questions?Questions?Questions?
• Some useful URLs:• Some useful URLs:• First-Year Programs, http://www.fye.vt.edu• ePortfolio Initiatives, http://eportfolio.vt.eduePortfolio Initiatives, http://eportfolio.vt.edu• Center for Instructional Development and
Educational Resources (CIDER), htt // id t dhttp://www.cider.vt.edu
• Office of Assessment & Evaluation,http://www aap vt eduhttp://www.aap.vt.edu
20
