Date post: | 13-Jan-2016 |
Category: |
Documents |
Upload: | marian-blake |
View: | 216 times |
Download: | 1 times |
Assessment and Technology Research
Maj Marie Revak
Director of Academic Assessment
Center for Educational Excellence
US Air Force Academy
Context of the Project
• Project: USAFA Faculty Notebook Computer Study
• Nov 98 – Dec 99 • 85 faculty members turned in desktop
computers for notebooks• 100 faculty members received new desktop
computers• Groups equal demographically
Purpose of the Study
• Are notebooks suitable replacements for desktops?• How do notebook computers affect teaching,
research, and service?• Which features and software packages are used?• Should we conduct a similar study with cadets?
Under what criteria?• What relevant information can we provide to
others: Air Force, DoD, and Higher Education
Multiple Uses for Assessment Data
• For identifying needs
• For evaluating programs
• For improvement
• For accountability
Methods and Measurements
• Goal was to collect data from multiple sources– 6 surveys: 1 initial, 4 intermediate, 1 exit– Maintenance logs, network logs, training rosters– Anecdotal data
• Data– Quantitative and qualitative– Process and product
Findings and ResultsThe Notebook computer group:
– Reported higher satisfaction with their overall computer experience, computer speed, response time, and ergonomic design
– Provided a higher proportion of positive comments on the surveys
– Spent more time (per person) dialing in– Took their computers home an average of 2-3 times per week– Used their notebook computers 93% of the time– Reported more required repairs– Added more hardware and software– Suffered no losses due to theft or accidents
Findings and Results• The Desktop Computer Group:
– Used their desktop computers 80-85% of the time (they relied on other computers more)
• Both Groups:– Reported ergonomic problems– Used computers in class about one-third of the time
Findings and Results• Biggest Positives for Notebooks:
– Work ubiquity– Increased productivity
• Biggest Negatives for Notebooks:– Mouse– Keyboard
• Two computers not necessary
Decisions
• For faculty, notebook computers are a valuable addition to the computing “mix” at USAFA
• Proceed with a cadet study
Lessons Learned
• Use rewards (or hammers) to encourage participation• Don’t rely too much on technology (to assess
technology use)• Think about data analysis and reporting from the start• Don’t collect more data than you can use• Report results quickly• Must be willing to provide support while collecting
data• Anecdotal data is powerful!
Assessment of the Impact of Ubiquitous Computing on
Learning
Ross A. GriffithWake Forest University
Ubiquitous Computing Conference
Seton Hall UniversitySouth Orange, NJ
January 4-6, 2001
Wake Forest UniversityFall 2000
Programs Enrollment (HC) Undergraduate 3944 Graduate 584 Divinity 50 Law 476 MBA 628 Medicine 440 Allied Health 136 Total
6258
Overview of Presentation
• Major Elements of the Strategic Plan
• Assessment Structure
• The Computing Initiative
• Changes by Students and Faculty
• Academic Outcomes
• Summary
Major Elements of the Strategic Plan
• IBM laptop computers provided to all entering freshmen and faculty effective with the fall semester 1996
• A new first-year seminar, ensuring each freshman an in-depth intellectual encounter effective with the fall semester 1996
• 40 new tenure-track faculty members
representing a 15 percent increase
• Scholarships for 175 students to study
abroad
• Fellowships for 150 students to perform
joint research with faculty members
Major Elements of the Strategic Plan (continued)
Assessment Structure
Evaluation Committee consisting of faculty and administration formed to evaluate effectiveness of the strategic plan.
Evaluation Committee sanctioned:
• College Student Experiences Questionnaire (CSEQ)• Cooperative Institutional Research Program (CIRP)
Freshman Survey
• Freshman Essay
Evaluation Committee sanctioned: (continued)
• Higher Education Data Sharing (HEDS) Consortium Alumni/ae Survey
• HEDS Senior Survey
• Higher Education Research Institute (HERI) Faculty Survey
• In-house Faculty Computer Survey
• In-house Faculty Survey
• In-house Student Computer Survey
• Wake Forest Fact Book
• Wake Forest Key Measures of Quality
Evaluation Committee
• Reviews Fact Book and Key Measures
• Analyzes survey data
• Communicates results
• Conducts follow-up
Evaluation Committee sanctioned: (continued)
The Computing Initiative
• IBM laptop computers provided to all entering freshmen and faculty
• Standing Faculty Committee on Information Technology formed
• Entire campus wired
• Created new position of Academic Computing Specialist (ACS)
The Computing Initiative(continued)
• Information Systems Support Center (ISSC) revamped
• Student and faculty training provided by library
• Computer-Enhanced Learning Initiative (CELI) formed by faculty
• Student Technology AdvisoRS (STARS) created
Changes by Students and Faculty College Student Experiences Questionnaire
(CSEQ)
• The survey directed by George Kuh of the University of Indiana was administered to a random sample of one-half of the freshmen, sophomores and juniors in March of 1996, 1997 and 1998 and all freshmen, sophomores and juniors on the web in spring 2000.
• The main purpose of the survey is to determine the activities of students with respect to how they spend their time.
Items for 2000 CSEQ with Significantly Higher Mean Scores than 1998 CSEQ
Time spent:• Used computer to prepare reports or papers• Used e-mail to communicate with instructor/other students• Used computer tutorial to learn material for a course• Searched the WWW or Internet for course material• Used computer to retrieve material from outside library• Used computer to produce visual displays• Used computer to analyze data• Developed a Web page/multimedia presentation
Items for 2000 CSEQ with Significantly Higher Mean Scores than 1998 CSEQ (continued)
Institutional Emphasis:• Information literacy skills (using computers)
Made gains:• Use Computers and other information technology
• 28 other “intellectually developing” items
Items for 2000 CSEQ with Significantly Lower Mean Scores than
1998 CSEQ
Time spent:• Participate in class discussion via electronic medium• Use dictionary or thesaurus• Use campus learning lab to improve study skills• Use campus recreational facilities• Had discussions w/students of different valuesOpinion:• Overall opinion of college• Attend same institutional again
CSEQ - Quality of Effort: Computer and Information Technology Scale
Sum of Means for Nine Computer Items
23.2922.73
21.87
20.67
21.99
20.73
25.15
181920212223242526
2000 WF 1998 WF RU DU CCU SLA GLA
HEDS Senior Survey
• The HEDS Senior Survey was administered to the Classes of 1993 through 1998 and the Class of 2000 in the spring of their senior year.
• The HEDS Senior Survey is an excellent survey for indicating the degree of educational enhancement and satisfaction in a number of areas.
• The Class of 2000 is the first class to graduate under the Plan for the Class of 2000 while the Senior Survey results are compared to the Class of 1998 as well as a College Group and a University Group of peer institutions.
HEDS Senior SurveyUse of Quantitative Tools - Enhancement
Mean
2.94
2.722.66
2.76
2.5
2.6
2.7
2.8
2.9
3
WFU 2000 WFU 1998 College Peer Group 2000
University PeerGroup 2000
HEDS Senior SurveyUse of Technology – Enhancement
Mean
3.40
2.923.02
2.50
2.70
2.90
3.10
3.30
3.50
WFU 2000 College Peer Group 2000 University Peer Group 2000
HEDS Senior Survey - 2000 Wake Forest Items Significantly Above & Below 1998 Wake Forest,
2000 College and 2000 University
AboveEnhancement:• Use Quantitative Tools• Read or Speak Foreign Language
BelowSatisfaction:• Relive college experience at same institution• Social life on campus• Ethnic/Racial diversity• Climate for minority students on campus
In-house Faculty Survey
• The Faculty Survey was developed by the evaluation committee to assess the effectiveness of all portions of the Wake Forest strategic plan.
• The survey was administered to the undergraduate faculty in December 1995 and February 1998.
In-house Faculty Survey Results Mean Scores Significantly Higher:
1998 vs. 1995
• Computers in teaching
• Computers in communication
• Computers in individual instruction
• Computers for presentations
• Computers with information gathering
• Computers for modeling/simulation
• Computer skill
• Computer training & assistance
In-house Faculty Survey Results Mean Scores Significantly Higher
1998 vs. 1995 (continued)
• Students proficient with computers
• Technology changed effectiveness of teaching
• Effect of computers on communication
• Effect of computers on resource material
• Effect of computers on presentations
• Use of technology in teaching
• Intellectual climate among students
• Religious development of students
• Prepare students for graduate/advanced education
In-house Faculty Survey Results Mean Scores Significantly Higher:
1998 vs. 1995 (continued)
• Prepare students for employment after college
• End of course student evaluations
• Enroll more graduate students
In-house Faculty Survey Results Mean Scores Significantly Lower:
1998 vs. 1995
• Number of papers or compositions published since previous fall
• Number of professional meetings attended since previous fall
1998 HERI Faculty Survey
• Administered by UCLA in Fall 1998 to faculty nationally
• Several questions contained items regarding computer use and opinions
• Wake Forest full-time undergraduate faculty results compared with peer group of nine private institutions
Use of Computers1998 HERI Faculty Survey
62
93
22
36
48
97
71
40
31
12
39
98
91
75
59
41
0 20 40 60 80 100
Participate in on-line discussion groups
Conduct data analysis
Create presentations
Conduct research on internet
Work from home
Conduct scholarly research
Write memos/letters
Communication via e-mail
Percent of faculty using computers at least twice a week
Wake ForestPeer Group
Sources of Stress1998 HERI Faculty Survey
7 12 17 22 27 32
Colleagues
Institutional procedures and "red tape"
Committee work
Teaching load
Keeping up with info technology
Personal finances
Review/Promotion process
Research/Publishing demands
Percent of faculty indicating "extensive"Wake ForestPeer Group
Opinion in General1998 HERI Faculty Survey
55
36
22 20
37
1913
28 31
59
8
18
28
38
48
58
68
Diverse studentbody enhances
education
Tenure attractsbest to
academe
Computersenhance
student learning
Encouragestudents to do
communityservice
Western Civfoundation of
undergradcurriculum
Percent of faculty indicating "agree strongly"
Wake Forest Peer Group
Academic Outcomes First-time Freshmen Retention
Following Year
Year Entered
94.3%
91.0%
93.4%
94.5%
92.0%
89
90
91
92
93
94
95
1995 1996 1997 1998 1999
Average GPAEnd of Freshman Year
2.87
2.83
2.81
2.89
2.77
2.7
2.8
2.9
1995 1996 1997 1998 1999
Year Entered
Summary
• Student & faculty computer usage has increased significantly since implementation of ubiquitous computing at Wake Forest
• Student & faculty computer usage is higher than peer group institutions
• Students are more engaged intellectually but are less satisfied with their overall college experience
• Freshmen retention rate and average GPA have increased slightly overall
Evaluating (and planning and implementing) Ubiquitous
Computing:
Hypothetical Example of
Flashlight Methods
Stephen C. Ehrmann, Ph.D.
Thanks
• 150+ institutional subscribers to TLT Group services
• TLT Group Founding Sponsors
– Blackboard, Compaq, Microsoft, SCT, WebCT
• TLT Group Program Funders– FIPSE, Mellon Foundation, National Science
Foundation
Apology
• For past examples of real studies, see http://www.tltgroup.org
• Click on "resources"
• Click on Flashlight Case Studies
• Or subscribe to F-LIGHT (free) (directions on the Web site)
The Challenge• Your institution is about to make an
expensive upgrade in computers, connectivity
• Goals of the study: – Document whether the IT helps improve
educational outcomes– Increase those educational gains – Control costs, reduce stress
An Important Digression
• Three ways of thinking about technology, two of which are usually wrong
Monadic Thinking
• Monad = one thing: just think about technology• “Computers for all” = GREAT (or AWFUL). It’s
obvious. Don’t waste money on evaluation.
Technology!
Dyadic Thinking
• “Computers for all: kids will learn calculus 20% better!”
Technology! Outcome
For example, if we invest in ubiquitous computing but our study shows calculus scores did not improve,Then either we need better machines or the investment was a mistake.
Triads
• Triad: an activity using a technology to help produce an outcome
•You need to consider at least these three elements in order to predict or evaluate whether technology can foster a learning outcome
Technology! OutcomeActivity
Explaining the Picture
• Activity: what users do with technology – using computers to study together (e-mail; chat; file transfer…)
• Outcome: mastery of calculus.• Dotted yellow arrows: choices about how to use the
computers.
Technology! OutcomeActivity
The Red Arrow
• Factors (other than technology availability) influence the activity (studying together) and thereby influence benefits and costs of technology use.
Technology! OutcomeActivity
Yeast and Bread
• Bread = Better learning outcomes
• Yeast = computers• Doubling the yeast
doesn't double the bread if you don't supply the other parts of the recipe, too.
The Crux of the Gist
• Which educational activities are going to be improved in the most important ways (you hope) by this injection of IT?
• Other than computers, what are the other missing ingredients for those activities?
• Boost the other ingredients first (computers depreciate fastest so get them last)
Step 1. Identify Activities• Which educational activities at your
institution are:– Crucial for important outcomes (who graduates,
what they can do)– Important for most or all courses– Likely to benefit most from use of the new
technology– Time consuming, expensive– Somewhat out of sight (no one really knows for
sure what's going on)
Examples of Activities• Collaboration & community
• Information literacy (paper, electronic)
• Learning by designing, composing, creative work
• Activities that bridge cultural divides and take advantage of learner diversity
2. Baseline Study• Study 1-3 such activities that represent the
heart of your hopes:– Current levels of the activities?– Current barriers? – Current incentives?– When computers are available today, are they
used to do this better/differently? Why/not?– Baseline level of most important outcomes of
those activities?
3. Lower Barriers…
• Lower barriers and, if possible, increase the incentives for these activities
• Study other institutions to see what unexpected things happen when more computing made available for these activities
Examples of Barriers
• Some students believe that studying together is a waste of time?
• Some faculty worried about cheating?• Some faculty aren't sure how to grade work when
students work in teams?• Computer screens are difficult for a group to see?• Most people don't use threaded conferencing very
well?
4. Startup
• After creating a better environment for the activities, take the big step in ubiquitous computing $$
5. Track
• After 6-12 months, repeat initial study of the activities, paying attention to IT use: – Is each activity improving? Thanks to computer
availability?– Unexpected barriers? – Use data on early successes, failures help guide
next steps of improving the activity (and implementing ubiquitous computing)
6. Study "Costs"
• Which elements of the activity are so costly/exhausting that (once early enthusiasm wanes) they could cause burnout, cost over-runs?
• With that insight, can you discover alternative ways to do those tasks that are less stressful on time, budgets, good humor?
7. Outcomes Improved?• Repeat the study annually:
– Are the learning outcomes improving (might take 2-3 years after the implementation)
– Costs and other stresses under control?– Unanticipated problems, opportunities for
improving or transforming the activities?
Technology! OutcomeActivity
The Point
• If you spend all your money and attention on "yeast," don't expect much bread! (Rapture of the technology)
• Use your studies to focus everyone's attention on the whole recipe and to make sure all the ingredients are there.
How Flashlight Can Help
• Evaluation tool kits and methods
• Training in how to do studies
• Do studies with or for you
• Help link you with others doing similar studies
• Free resources
• For all that see www.tltgroup.org
Free Resources; Info
• Information about Flashlight at
http://www.tltgroup.org– You can find the article on which this talk is
based; – Article was published in AAHE Bulletin
November 2000.
What Do You Think?
• When you go home, will you recommend consideration of this sequence of study and action? Why/not?