Evaluate if virtualization and simulated labs in Computer Hardware Technology course meet their stated objectives
A Report Submitted to American National University
By
Shobhana Ganapathi
7/30/2013
2
Table of Contents
Executive Summary ………………………………………………………………….. 3
Description of the course Computer Hardware Technology with simulated labs ……. 4
Evaluation Method……………………………………………………………………. .5
Evaluation Methods & Tools…………………………………………………….....5, 6
Results……………………………………………………………………………. 7, 8, 9
Assessment Performance…………………………………………………………… .7
Learner Completion Rate and Average Final Performance Scores……………….... .8
Learner perceptions and Learner satisfaction of Computer Hardware Technology
Course with Labsims………………………………………………………………… 9
SME Feedback……………………………………………………………………… .9
Discussion……………………………………………………………………….... .10, 11
Project Cost………………………………………………………………………….. .12
APPENDIX
Appendix A: EDP……………………………………………………………….13 to 15
Appendix B: Learner Perception and Satisfaction Questionnaire……………… 16 to 19
Appendix C: SME Questionnaire………………………………………………… .20, 21
Appendix D: Student interview & Evaluator Observations Report………………… 22
3
Executive Summary
This report details the results of evaluating ‘if virtualization and simulated labs meet their
stated objectives in the Computer Hardware Technology Course’. Data was collected and
analyzed to find the effectiveness and efficiency of the Computer Hardware Technology course
with the simulated labs and it was compared to the performance in face-to-face labs. The main
objective of the evaluation was to determine if replacing f-t-f labs with simulations
compromised student learning. The evaluation results are being provided to the program
directors of American National University (ANU) to enable them to make a decision to migrate
the course from a face-to-face setting to a fully online platform.
The individual components that were evaluated were the content and the instructional
support materials in the simulated labs. The technology components of graphics, video and
audio and all the exercises, discussion activities and assessments were examined in both
versions of the course. Quantitative data was collected from both versions of course offerings
for performance in tests, projects and assessments and qualitative data was collected from
questionnaires and interviews for learner perceptions, satisfaction and ease of use of technology
components.
An analysis of the results indicate that there is no significant difference in performance
scores in the Computer Hardware Technology course with simulated labs vs. the version with
face-to-face labs. Student perceptions and attitudes indicate they prefer to work with online
simulations as it gives them the flexibility of working from anywhere, anytime convenient and
more importantly the ability to repeat the labs any number of times was cited as a plus point in
its favor. (Appendix D for excerpt of student interview)
Computer Hardware Technology course with simulated labs meets all of its stated
objectives and is recommended as a viable option to the course with face-to-face labs. The
stakeholders can rest assured that student learning is not compromised and they can move
forward in migrating the course online.
4
Description of the course Computer Hardware Technology with
simulated labs
CHT 102- Computer Hardware technology is an Associates level computer science
course that is required of all Information Systems Engineering students at ANU. Enrolments are
typically between 10 to 15 students. The students who are enrolling for CHT should have
successfully completed the pre-requisite of ISE 100- Introduction to Computers course. The
course is broken into 5 modules. Each module has two distinct parts theory and practicals done
in labsims. Module 1 to Module 4 is of 11 hours duration and module 5 is of 6 hours. The total
duration of the course including the introduction and conclusion is about 60 hours.
Each module comprises of some instructional support materials and simulated labs for
practice. The student work in each module involves completing exercises, discussion activities
and assessments. The students navigate through graphics, video and audio clips to complete the
activities in each module.
The main objectives of this evaluation were: (Appendix A for details)
To check if Goal statement is clear
To check Goal & objective alignment
Is the content in the virtual lab presented in a clear and concise manner?
Is the content appropriate for Computer Hardware Technology?
Is the content up to date covering current hardware technologies?
Do the technology applications like embedded videos, graphics and lab simulations function
properly?
Is organization and structure of the message logical and coherent?
5
Evaluation Method
The evaluation was done from two orientations, participant orientation and objective
orientation. It was deemed important to evaluate the course with integrated labsims from a dual-
perspective as both the participants and the objectives were critical to the success of this course.
The participants were students enrolled in the Computer Hardware Tehnology course
who have successfully completed the pre-requisite of ISE100- introduction to computers. Other
participants were the SME (HOD computer science department), the instructor and the
evaluator.
The decision making methodology was used for this evaluation as the evaluation results
were going to directly impact the decision being made by program directors to migrate the face-
to-face course to online instruction.
The results of the evaluation are being presented in two parts. The reports detail both
short term and long term outcomes. The current report focuses on short term outcomes like
student learning, knowledge, attitude, performance, skill and understanding.
Long term outcomes will be examined to see if the online course produces a ROI by
increasing enrollments. Since it is beyond the scope of the current evaluation report this will be
presented in another report after collecting and analyzing data from four course offerings of labs
with simulations.
Evaluation Methods & Tools
Both qualitative and quantitative methods were used to collect data
• Qualitative: Valuable information was collected from interviews, observations,
surveys and responses provided via e-mail, messages and forum posts to explore the
context of participants' experiences of the course using simulated labs. (Appendix B)
6
• Quantitative: Data was collected through user survey's and questionnaires and
assessment data was collected from performance in lab sims, tests, quizzes and
project weight age mentioned in project rubrics. (Appendix B & C)
Specifically, the average performance scores from four course offerings were obtained, two from
the course with face-to-face labs and two from the course with simulated labs. This was
compared to find the difference in completion rates and average performances. Student interview
(Appendix D), SME feedback via questionnaire (Appendix C) & email and student questionnaire
(Appendix B) results were used to report learner satisfaction.
7
Results
Assessment Performance
All the scores from past four course offerings of tests quizzes and project work
based on assessment rubrics were obtained and tabulated to arrive at the average scores
and median value. It was determined that average performance scores were about the
same, in the 80% to 85% range in both versions of the course offerings as shown in the
table and graphs below.
It was determined that student completion rate was 100% in Term1 (Fall 2012
with face-t-face labs) and 80% in Term2 (Winter 2012 with face-t-face labs) and 84% in
Term3 (Spring 2013 with simulated labs) and 91.6% in Term 4 (Summer 2013 with
simulated labs). The face–to-face course offering had an average completion rating of
90% and the version with simulated labs had an average completion rate of 89.55%
Learner Completion Rate and Average Final Performance Scores
Course Offering
Number of Learners
Number who completed the course
Completion Rate
Average Final Scores
Lowest Score (including Fail)
Highest Score
Median Value of scores
Term 1 ( f-t-f labs)
8 8 100% 85.6% 61 98 91.5
Term 2 ( f-t-f labs)
10 8 80% 79.12% 74 100 74.5
Term 3 (Simulated labs)
8 7 87.5% 84% 57 97 93
Term 4 (Simulated labs)
12 11 91.6% 82.54% 47 98 85
8
8 100%
10
80% 8 87.5%
12 91.6%
0
2
4
6
8
10
12
14
No. of Students No. Who Completed the course
Learner completion Rate
Term 1 (f-t-f labs) Term 2 (f-t-f labs) Term 3 (simulated labs) Term 4 (Simulted labs)
85.60%
79.12% 84.00% 82.54%
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
85.00%
90.00%
95.00%
100.00%
Average scores
Final Performance Scores
Term 1 (f-t-f labs) Term 2 (f-t-f labs) Term 3 (simulated labs) Term 4 (Simulted labs)
9
Learner Perceptions and Learner Satisfaction of Computer Hardware
Technology Course with Labsims
The survey was taken by 9 out of the 12 students enrolled for the summer course. 6 participants out of 9 agreed they liked working with simulated labs 2 were neutral and only 1 disagreed. 6 participants were happy with the clarity of the content, 2 were neutral and 1 disagreed.7 of nine participants agreed the content covered latest technologies and embedded graphics and simulations worked properly and 2 were neutral in their views. There was no disagreement in this regard.
QUESTIONNAIRE ITEM
STRONGLY AGREE N%
AGREE N%
NEUTRAL N%
DISAGREE N%
STRONGLY DISAGREE N%
I liked working with simulated labs in the course
1 (11.11%) 5 (55.56%) 2 (22.22%) 1 (11.11%) 0 (0%)
The content in the virtual lab was presented in a clear and concise manner
2 (22.22%) 4 (44.44%) 2 (22.22%) 0 (0%) 1 (11.11%)
The embedded videos graphics and simulations functioned properly
3 (33.33%) 4 (44.44%) 0 (0%) 2 (22.22%) 0 (0%)
The content was up to date and covered latest technologies
4 (44.44%) 3 (33.33%) 2 (22.22%) 0 (0%) 0 (0%)
N= 9 learners
SME Feedback
A questionnaire was given to the SME to give his opinion and feedback about the CHT course. The questions covered the goal, content, technology and message design to evaluate efficiency, effectiveness and appeal of the content and simulated labs in the course. (Appendix C)
The SME feedback via questionnaire and email indicated that the content was appropriate and aligned to the objectives of the Computer Hardware technology Course.
10
Discussion
Overall scores, averages and median values were used to investigate how (and if) scores
improved after using the simulated labs. The demographics of students enrolled in both
versions of the course were the same. They were students who had passed the pre-
requisite of introduction to computers course. The only variable was that students
enrolled in Term 1 and Term 2 did the labs in a face-to-face setting and the students
enrolled in Term 3 and Term 4 did the labs in an online simulated setting.
Learner completion rate and average performance scores in the course offering with face-
to-face labs were 90% and 82% respectively.
Learner completion rate and average performance scores in the course offering with
simulated labs were 85.5% and 83% respectively.
Since the scores are about the same it is determined that student learning is not
compromised while using simulated labs.
The analysis of Learner satisfaction data compiled from the questionnaire ( Appendix B)
shown in the table under ‘results’ as frequency and percentages indicate 66% of the
students indicated they liked working with the simulations and were happy with the
clarity of the content in the virtual labs. 22% were neutral in their views and only 11%
disagreed with this. 78% of the students were happy with the quality of the simulations
and alignment of content to latest technologies.
Since the survey was taken only by a small number of students student perceptions were
also gathered by interviewing two students who had worked with simulations in a
previous course offering. The SME feedback was collected via questionnaire (Appendix
C) and feedback via email.
11
Students had mixed perceptions about watching the video and learning before working
with simulated labs. See excerpts of interview below. (Details of interview available in
Appendix D)
Student 1: “I found the material more challenging to comprehend when I tried to read the textbook. But when I watched the video and practiced the lab I understood it better and did not have to read the chapter a second time.” Student 2: “Sometimes watching the video becomes tedious especially with these boring guys in the video”
But they seemed to be unanimous in their preference for working online with virtual labs. An excerpt from their conversation is given below.
Student 1: “I would prefer to do it online because I can work from home when I get time.” Student 2: “I prefer online labs because I can repeat it many times till I feel I am ready. This will help me prepare for my A+ certification”
After the analysis of survey results, student interview, SME questionnaire and feedback
via email it is determined that a majority of students were satisfied with using the
simulated labs (78%) and would prefer this mode of instruction (100%).
12
Project Cost
The following is the cost of the evaluation including travel, materials, and SME and PA fees.
Professional Salaries
Main Evaluator (PM) 15days @$500/day $7500
SME 5 modules @ $300/module $1500
Personal Assistant (PA)
10 days @$120/day $1200
$10,200
Travel to ANU
Estimated mileage 500 miles @ $0.30/mile $150
5 site visits @ estimated average of $150 travel & per diem per visit
$750
Supplies, Materials, Communication $1200 $2100
Total
$12,300
13
APPENDIX
Appendix A: EDP
Evaluation Questions
Program Objectives
Project Activities to Observe
Data Source Population/Sample
Data Collection Design
Data Analysis
Whose Responsibility
Goals: Is the
purpose stated
concisely?
To check if Goal statement is clear
Request SME to read and compare with checklists of F-T-F course materials
Reference checklists of F-T-F course materials
HOD Computer science Department
Prepared checklists of Objectives
Compare the two checklists
Evaluator
Do goal and lesson objectives align with each other?
To check Goal & objective alignment
Students read the goal and all the objectives
End-user survey
All students enrolled in Computer Hardware Technology
Administer end-user survey through webmonkey
Plot graphs
evaluator
Content: Is the content in the virtual lab presented in a clear and concise manner?
Subject Matter Expert (HOD Computer Science) browses the virtual lab with checklists of F-T-F course
Feedback via email & messages
SME (HOD) Prepare Checklist of course materials
Compare the two checklists
evaluator
14
materials ,
Is the content appropriate for Computer Hardware Technology?
HOD & Instructor browse the virtual labs and complete simulated lab activity
Interview report from SME (HOD Computer science) & instructor. Assessment data will be collected through lab sim and online quiz activity in the portal and discrepancies in check list of project rubrics.
SME & Instructor Report Look for negative remarks
evaluator
Is the content up to date covering current hardware technologies?
HOD & Instructor browse the virtual labs and complete simulated lab activity
Interview report from SME (HOD Computer science) & instructor
SME & Instructor Report Look for negative remarks
evaluator
Technology
Do the technology applications like embedded videos, graphics and lab simulations
Expert & end-user browse and review the online course. Data is
Expert review report & End-user survey
Expert & All students enrolled in Computer Hardware Technology
Administer end-user survey through webmonkey
Plot graphs
evaluator
15
function properly?
collected from portal to find time spent on each task.
Message Design
Is organization and structure of the message logical and coherent?
Expert & end-user browse and review the online course
Expert review report & End-user survey
Expert & All students enrolled in Computer Hardware Technology
Administer end-user survey through webmonkey
Plot graphs
evaluator
16
Appendix B: Learner Perception and Satisfaction Questionnaire
Student Opinions about Computer Course with Simulated Labs:
Your response to this questionnaire will be helpful and is greatly appreciated. Your input will
help us modify the course as necessary. Thank you.
1. Please fill out the following demographic data: Gender
Please fill out the following demographic data: Gender Male
Female
2. What is your age?
What is your age? 18 to 24
25 to 34
35 to 44
45 to 54
55 to 64
65 to 74
75 or older
3. Please select the option that best matches your response: I have a great deal of computer experience.
Please select the option that best matches your response: I have a great deal of computer experience.
Strongly Agree
17
Agree
Neutral
Disagree
Strongly Disagree
4. I liked working with simulated labs in this course.
I liked working with simulated labs in this course. Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
5. The content in the virtual lab was presented in a clear and concise manner.
The content in the virtual lab was presented in a clear and concise manner. Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
6. The content was up to date and covered latest technologies
The content was up to date and covered latest technologies Strongly Agree
18
Agree
Neutral
Disagree
Strongly Disagree
7. The embedded videos, graphics and simulations functioned properly.
The embedded videos, graphics and simulations functioned properly. Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
8. The organization and structure of the message was logical and coherent.
The organization and structure of the message was logical and coherent. Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
9. I think I learned as much from the simulations as I would have done in the physical labs.
I think I learned as much from the simulations as I would have done in the physical labs. Strongly Agree
19
Agree
Neutral
Disagree
Strongly Disagree
10. It prepared me well to take the A+ Certification
It prepared me well to take the A+ Certification Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Next
Powered by SurveyMonkey
20
Appendix C: SME Questionnaire
Evaluation Criteria
Specific Evaluation Questions to Evaluate Computer Hardware Technology (CHT) with simulated labs
Effectiveness
Goals Is the information accurate?
Are the goals and objectives clear?
Are the goals and objectives achievable?
Are the goals and objectives appropriate for classroom instruction?
Content Does the information cover the content properly?
Does the content match the objectives?
Are activities and final project congruent?
Do activities promote learning?
Technology Are the videos and simulations working properly?
Are the simulations appropriate to learn computer hardware concepts?
Do they sufficiently meet learning objectives?
Message Design Are messages an integrated whole?
Do videos and simulations provide sufficient instruction leading to successful lab completion?
Do learners understand the instruction?
Are there design features which are distracting?
Are directions clear?
Is time allotted for the course appropriate?
Does the text stand alone if the graphics are unavailable?
21
Efficiency Is the purpose stated concisely?
Goals Do goal and lesson objectives align with each other?
Is the content presented in a clear and concise manner?
Content Is the content appropriate for Computer Hardware Technology?
Is the content up to date covering current hardware technologies and collaboration efforts?
Do the technology applications like embedded videos and links work properly?
Technology Does the firewall prevent students from accessing links in their search results?
Is organization and structure of the message logical and coherent?
Message Design Are titles and subtitles used while organizing the content?
Appeal
Goals Are the goals relevant to learners?
Content Is the content interesting?
Is the content in the virtual labs presented in a way that can be understood by students who have just completed only ISE 100?
Is the content challenging at the same time being enjoyable?
Technology Are instructions logical and easy?
Is it easy for students to access materials?
Are the videos presented well?
How about the audio?
Is the language level and tone suitable for adult learners?
Message Design Do graphics and video enhance the instruction?
General Comments
Based on content and quality do they qualify to replace face to face labs? Please provide your comments on this
22
Appendix D: Student interview & Evaluator Observations Report
Excerpts of Interview Conducted by Evaluator Comments Evaluator: Did you like watching the videos before working with the simulations?
Students have mixed perceptions about watching the video and learning
Student 1: I found the material more challenging to comprehend when I tried to read the textbook. But when I watched the video and practiced the lab I understood it better and did not have to read the chapter a second time. Student 2: Sometimes watching the video becomes tedious especially with these boring guys in the video Evaluator: Do you like to take tests and exams in the virtual labs? How does it compare to tests in face-to-face setting?
Students generally seem to be inclined towards taking tests which demonstrates their practical ability to do something rather than paper and pencil multiple choice tests. Student 1: After our virtual lab tests we were
able to view our scores immediately. This proved to us that we knew something. After our multiple choice test we were just glad that it was over! Student 2: I like the virtual labs because the exam tests us on what we can do and not what we can memorize. Evaluator: If you were given the choice to come to school to do your labs, would you prefer that to doing labs online?
Students seem to be unanimous in their preference for working online with virtual labs
Student 1: I would prefer to do it online because I can work from home when I get time. Student 2: I prefer online labs because I can repeat it many times till I feel I am ready. This will help me prepare for my A+ certification