Interactive Multimedia Courseware and the Hands-on Learning Experience:An Assessment Study
Melissa Regan,Stanford University
Sheri Sheppard,Stanford University
Abstract
An Experimental Bicycle Dissection Exercise was designed in which students perform an
abbreviated and slightly altered version of the "Bicycle Dissection Exercise" offered in the
Stanford course, "Mechanical Dissection." Like the formal course version of the Exercise,
student groups were supplied with a bicycle, tools, a manual, and an assigned set of questions;
unlike the formal course Exercise, groups in the Experimental Exercise were also supplied with a
multimedia stack that explained (through graphics, text, sound and movies) various aspects of the
bicycle. This multimedia stack was created to enrich the dissection experience by providing
information and background on the mechanics of bicycles. Three groups of students were
videotaped performing this Experimental Bicycle Dissection Exercise. The videotapes were then
evaluated by a multidisciplinary group of reviewers using a technique called Video Interaction
Analysis (VIA). This paper presents a summary of the experimental assessment process and our
findings regarding the educational value of the multimedia courseware in exercises with a hands-
on component.
I. Introduction
The use of multimedia-based curriculum in engineering education is rapidly gaining
popularity; one need only look at a recent issue of Syllabus magazine to get a feeling for the level
of courseware production that is going on nationally and internationally in engineering
education. Engineering education-related multimedia curriculum, where multimedia means a
package that combines video, text, photos, simulations, animation and/or graphics in a hypertext
environment, can be classified into six broad categories:
1. case studies, where the aim is to illustrate the design, development and/or failure of engineered
devices/structures/systems (e.g., references (1)-(6)) and/or to show the connections between
devices and the underlying design issues and physics (e.g., references (7), (8));
2. tutorials to develop specific skills (e.g., computer-based multimedia exercises that are aimed
at helping students develop visual thinking skills such as Blockstacker (9) and Blockbuilder
(10));
3. laboratory "mentors" where the multimedia courseware serves as guide, stepping students
through the various aspects (e.g., theory, physical set-up, data reduction procedures) of
performing a physical experiment (e.g., references (11)-(14)) or a simulated experiment (e.g.,
(15));
4. supplement to lectures or a resource necessary to complete assigned homework (e.g.,
references (16)-(21));
5. a major vehicle for dissemination of lectures, course materials and/or homework (e.g.,
references (8), (22)-(24));
6. or as a platform for student report creation (e.g. reference (22)).
The courseware referenced above can also be interpreted in terms of Simon's three modes of
"Computer Aided Instruction (reference (25)). Category 1. above is similar to his simulation-
mode courseware, category 2. to his drill & practice-mode courseware, and categories 3,4 and 5
to his tutorial mode courseware (although this mappiing is far from perfect, as a single piece of
courseware may contain all three modes).
In contrast to the growing volume of engineering multimedia courseware, there is generally
a dearth of formal studies to assess whether multimedia facilitates enhanced/improved student
learning. There are some notable exceptions of engineering education-related multimedia
courseware that has undergone some formal assessment; the approaches taken can be broadly
classified as "student-input centered" or "student-observational centered."
Examples of "student-input centered" approaches include student surveys that ask students
for comments related to their use of multimedia (e.g., references (24), (26)); student interviews
(e.g., references (27), (28)); and student GPAs or test performance (e.g., reference (23), (29).
Davis' and Humphrey's work (30) discusses the advantages and disadvantages of student surveys
and interviews as assessment approaches. Another example of a "student-input centered"
approach to assessment is the collection (and subsequent analysis) of student comments,
performance and/or reflections throughout a course or "event" as facilitated by the Multimedia
Forum Kiosk (MFK) (31), or by the multimedia assessment-engine proposed in reference (32).
MFK was used to assess engineering courseware in, for example, reference (11).
Whereas "student-input centered" approaches use either students' comments or summative
performance as the basis for assessment, "student-observational centered" approaches involve
"observing and capturing" the students engaged in activities, then interpreting what was captured.
One example is tracking (i.e., the actual keystrokes and sections visited by students using the
multimedia courseware are recorded) so that questions such as "Which sections of the
August 11, 1996 (Regan, Sheppard)
courseware did students enter," "How much time did they spend there," "What order did they
access pieces of the courseware in," "Did they play each movie?" can be answered, as used in
references (27), (33), and (34). Another example is ethnography, an observational technique
with a human observer that has been used extensively in anthropology; Mazur in reference (28)
used this general approach to assess a new multimedia-based course created at Cornell
University called "Engineering in Context." A third "student-observational centered" approach is
Video Interaction Analysis (VIA), which has only very recently been used to assess engineering
education related multimedia (e.g., reference (35)). VIA has its roots in ethnography but uses a
video record of students working as the basis for subsequent discussions. VIA will be presented
in greater detail later as it is the assessment approach taken in the current work.
The successful application of multimedia in a number of engineering domains, along with
intuition and blind optimism about the potential of multimedia-based technology to enhance
education all strongly suggest that multimedia courseware may enrich formalized hands-on
learning experiences. The goal of our work was to explore this potential as it applies to hands-on
mechanical dissection exercises.
II. Background And Expectations
A. Mechanical Dissection Exercises
The multimedia courseware that is the topic of this paper was created to support a course
called "Mechanical Dissection" at Stanford University. The course was initiated in the fall of
1990 for freshman- and sophomore-level undergraduate engineering students to help them
become familiar with the machines/mechanisms that surround them, as described in references
(36) and (37). It is built around a series of mechanical dissection exercises (here, "dissection"
refers to disassembling and reassembling a mechanical artifact). Students participate, both
individually and in groups, in these in-depth dissection exercises so that "Experience (may be)
the mother of knowledge" (after Cervantes). Examples of devices studied in the class include
bicycles, electric drills, wind-up toys, sewing machines, engines, and computer printers. An
important aspect of each of the exercises is for the students to become "users" of the device,
identifying all aspects of the external functionality. Tasks related to these exercises include
recording form and function of the device in a personal log book, mapping external-to-internal
functionality, answering specific questions related to assembly or maintenance of the device, and
participating in formal and informal presentations.
While the course is tremendously popular with students (and has been adapted by several
other Universities), the teaching staff continues to struggle with how to present content material
(e.g., types of bearings, power considerations, gear trains) in a timely, connected manner amid
August 11, 1996 (Regan, Sheppard)
the hands-on exercises that students are engaged with; it is partially out of this struggle that our
exploration of the potential of multimedia arose.
B. Multimedia Courseware
While actually having students "roll up their sleeves and get their hands dirty" with the
hardware is an extremely rich experience for them, we felt the dissection experiences could
benefit from multimedia courseware in several ways. In order to explore this potential benefit,
the Bicycle Dissection Multimedia Courseware was created. More specifically, we created the
courseware to:
1) help students to discover the basic concepts and principles of engineering and physics that are
relevant to the physical device;
2) illustrate details of "how it works" that may not be clear from mere inspection, or specifics of
parts and systems that are not readily accessible for inspection (i.e., hidden from view);
3) convey a sense of the design rationale behind individual parts of the device, and the system as
a whole.
The Bicycle Dissection Multimedia Courseware (henceforth to be referred to as the
"Courseware") is a SupercardTM application running on the MacintoshTM platform. We
designed it so that students could obtain a certain level of understanding of the various
subsystems that make-up the bicycle by viewing and interacting with the Courseware, but it was
assumed that this understanding would be less grounded than that obtained by using multiple
resources. The development required approximately 200 graduate student hours, with periodic
reviews by peers and faculty. The development process included formulating the project scope,
shooting photographs and video, as well as program and interface design.
The Courseware was in no way meant to replace the hands-on experience, the instructor, the
teaching assistants/coaches, or student interaction. Nor was it meant to replace individual
student thought, rational deduction and creativity. Rather, it was meant to complement all of
these components. The major consideration in the design of the Courseware was to ensure its
usability and effectiveness by people with different learning styles (as described in reference
(38)).
The Courseware also contains elements of Simons' three modes of "Computer Aided
Instruction" (CAI) (25): tutorial, simulation, and drill and practice:
• Tutorial: This provision of content of instruction is the most pervasive mode of the
Courseware. Content is provided in many forms: text, illustrations/drawings, color
photographs, animated drawings, videos, and audio. Additionally, two or more of these
forms are often presented together. For example, a "Force Analysis" presentation offers a
narrated textual description of the transmission of forces through the drive train, while a
August 11, 1996 (Regan, Sheppard)
visual symbolic description "builds itself" on screen. This presentation form supports Mayer
and Anderson's instructional design principle (39), the "contiguity principle," which states
that "the effectiveness of multimedia instruction increases when words and pictures are
presented contiguously (rather than isolated from one another) in time or space."
Additionally, while a significant amount of content is presented in the Courseware, the
navigation controls make it accessible in a relatively unstructured and non-linear form. After
a brief introduction, the user is free to access the information in any order, and as many times
as desired.
• Simulation: Two simulations are included in the Courseware, namely, an animation of the
freewheel mechanism and an interactive "Design Calculator." The freewheel animation
provides a visual (with accompanying text) description of the operation of the ratchet
mechanism of the freewheel. The "Design Calculator" allows the user to input the physical
configuration of the drive train, as well as rider input conditions, and immediately calculates
the bicycle output parameters. The "Calculator" was intended to help develop an intuitive
sense of how input parameters affect output parameters, and show students the supporting
calculations/analysis.
• Drill and Practice: The "Bottom bracket" assembly puzzle is the sole example of this mode.
The puzzle provides students the opportunity to test their knowledge of the part names and
assembly configuration of the bottom bracket, and provides feedback as to the number of
successful and unsuccessful attempts.
Finally, the Courseware contributes strongly to three of the five facets of a learning
environment defined by Perkins (40). The role of Perkins' Information Banks (defined as any
resource that serves as a source of explicit information about the topic) is partially played by the
Courseware ; Perkins' Microworlds (an area for the specific purpose of presenting phenomena
and making them accessible to scrutiny and manipulation) are present in the Courseware's
"Design Calculator." The "Bottom bracket" puzzle in the Courseware serves as Perkins'
Construction kit (a fund of prefabricated parts and process for the execution of experiments).
The Courseware only mildly serves as Perkins' Task Manager ("entity" that sets tasks to be
undertaken in the course of learning, guides and sometimes helps with those tasks, and provides
feedback regarding process and/or product), and does not provide a Symbol Pad (surface for the
construction and manipulation of symbols).
August 11, 1996 (Regan, Sheppard)
III. The Experiment
This Courseware was designed to complement the Bicycle Dissection Exercise offered in the
Stanford Mechanical Dissection course. We performed preliminary assessment of the
Courseware in conjunction with the formal bicycle dissection exercise by adding a number of
multimedia-based questions to the regular written task. We found that students were inclined to
seek out only those sections of the Courseware that would help answer the task question at hand,
and that they waited until the very end of the task period to answer all of the multimedia
questions "in bulk." This was partially due to constraints on time and computer accessibility;
there was a single MAC II available in the classroom to ten dissection student groups.
Additional details about the formal classroom Bicycle Dissection Exercise with multimedia are
given in Table 1, Column (A). We also found that extensive assessment of the Courseware
during the regular course was limited by the time and logistical constraints of performing
detailed observations in the midst of our delivering the course. We decided therefore, that a
controlled experimental environment would far better inform the design of this new-media
curriculum.
TABLE 1.Comparison of Formal Course and Experimental Bicycle Dissection Exercises
(A)Full Course Dissection BikeExercise
(B)Experimental Bike DissectionExercise
Time Period twenty hours 70 minutes + 2 interviewsTeamwork work in groups of two or three work in groups of two or threeStudent Body freshmen/sophomores
apparent interest in engineeringminimal experience with analysiscoursesthree weeks of prior dissectionexperience (3 labs)
freshmen/sophomoresapparent interest in engineeringminimal experience with analysiscoursesno formal dissection experience
ResourceAvailability
show computer shared with 9other groupsone bike per team to be dissectedrepair manuals, toolsprofessor, teaching assistants
dedicated high speed computertwo bikes, one available to bedissectedrepair manuals, toolsno outside human contact
PerformanceRequirements
fifty detailed, specific questions,to be answered on paper, withsketches and textural explanations
several general "thoughtprovoking" questions, to bediscussed and recorded on paper ifdeemed important for laterdiscussion.
For the experiment, paid "volunteers" were recruited from a sophomore level "Strength of
Materials" class. The experiment consisted of three stages: A. "pre-event" (which involved team
August 11, 1996 (Regan, Sheppard)
formation and a preliminary interview to assess basic understanding of bicycle mechanics); B.
"dissection activity" (in which the teams engaged in the actual dissection exercise); and C. "post-
event" (which consisted of a final interview to assess growth in basic understanding of
mechanics). These three stages are discussed in greater detail below.
A. Pre-event Activities
Pre-event Assessment and Grouping: Prior to students' participation in the actual dissection
event, each participant completed a preliminary questionnaire which sought to determine
previous experience with and attitude towards bicycles and other mechanical systems. The
results of the questionnaire were used both as criteria for group formation and as background
information while studying individual and group behavior observed during the dissection
activity. Students with similar experiences/attitudes with riding and maintaining bicycles were
grouped together. The students were divided into three groups: Group 1 was comprised of two
female students; Group 2, two male students; and Group 3, one female and two male students.
Preliminary Group Interview: Each group was interviewed to establish, on videotape, the
group's collective understanding of bicycles and mechanical systems. As part of this interview,
each group was instructed to sketch, from memory, the drive train of a bicycle, label its parts,
and discuss with the interviewer the functionality of the system. This information was used,
along with the preliminary questionnaire, to establish a point of reference for understanding
behavior observed during the dissection exercise.
B. Dissection Activity
Students were supplied with the following resources in order to engage in the actual
dissection activity:
1) multimedia Courseware running on a Macintosh "Quadra 900" with a twenty-inch color
monitor
2) dissection bicycle (functioning 10-speed bicycle, not on a bike stand, available to be
dissected)
3) tools necessary for dissection
4) reference bicycle (functioning 10-speed bicycle, on a bike stand, not available to be
dissected)
5) bicycle repair manuals (soft cover books, one copy each of two different books)
6) large sheets of white paper and colored markers
7) the assignment
Students were instructed to spend the first ten minutes getting familiar with the first six
resources (they had not seen the assignment yet), The main purposes of this ten minute "free
August 11, 1996 (Regan, Sheppard)
exploration period" was to observe which resources students gravitate towards, how they use
these resources given no specific instructions, and to encourage them to introduce themselves to
resources that they might otherwise ignore once a specific task is provided.
Students were then given the assignment and allowed one hour to figure out how a bike
works and why it is designed the way it is. The assignment consisted of a list of seven questions
which was posted on the wall to act as a catalyst for inquiry (see Table 2). Students were not
required to record the answers to the questions, but were encouraged to record whatever they felt
they might need in order to engage in conversation about the issue at a later time (e.g.,
definitions, sketches). Students were also asked to interactively assess their own learning
experience by recording factors that hindered their learning, and suggestions for improvement of
the resources and/or learning environment. The main differences between the Experimental
Exercise and the formal Classroom Dissection Exercise are described in Table 1.
TABLE 21: Questions posed to students in the Experimental Exercise (where "it" is the elements of the drive train)1. What does it do?2. How does it do it?3. What are its most important parts?4. Why is it needed?5. Why is it designed the way it is?6. How do the dimensions of the different bike parts affect the performance of the
bike?7. How are force, speed and power related?
1.Questions 1 through 5 apply to the individual bike parts as well as the bike as a whole
The task, consisting of the questions in Table 2 and the six additional resources listed above
(e.g., Courseware, dissection bicycle, etc.) make up the total learning environment in our
experiment. All of these resources can be interpreted in terms of Perkins' five facets of a
learning environment (40) that were discussed previously. For example, the role of Perkins'
"Information Banks" is played by not only the Courseware, but also by the manuals, and the
reference and dissection bicycles. Perkins' "Microworlds" are present in the Courseware's
"Design Calculator", and in the reference and dissection bicycles. The paper and colored
markers serve as "Symbol Pads." Finally, primarily the student groups, and secondarily the task
serve as the "Task Managers." It is interesting that having the students play the primary role of
task managers is in contrast to the approach taken in much of engineering education.
C. Post Event Activities
As in the preliminary interview, each group was again asked to create a labeled drawing of
the drive train (from memory), and discuss the functionality of its parts as part of an exit
interview. The interviewer posed one or more questions requiring application of the group's new
August 11, 1996 (Regan, Sheppard)
knowledge. Finally, students were asked to discuss those aspects of the experience which they
found to be an aid or hindrance to their learning.
IV. Assessment Method
Our objective in assessing the Courseware was to determine whether or not the Courseware,
enriched hands-on learning. In order to answer this question, we wanted to explore four main
issues:
[1] the relationship between the seven resources;
[2] effectiveness of the Courseware in functioning as an independent learning tool (e.g., as
students searched for something specific, was the information there?);
[3] effectiveness of the Courseware in supporting teamwork;
[4] effectiveness of the Courseware in supporting the given task;
These four issues are best explored by watching students working in the learning environment for
they represent the process of learning, and not purely the outcome (the traditional "test"
mentality, as discussed in reference (41)). Therefore, we chose a "student-observational
centered" assessment technique, namely, Video Interaction Analysis (VIA).
A. Background
Video Interaction Analysis (VIA) was the method of assessment used to monitor and
evaluate the learning process. This method assumes that learning is largely an ongoing social
process, and that the process(es) can be accurately observed. VIA has its roots in ethnography
and other methods of the social sciences. The method involves videotaping classroom activities [
a) in Figure 1]. Then the data (videotapes and results of questionnaires and interviews) are
reviewed by an individual with a primary interest in and understanding of the project, its goals
and history. A collection of annotated video clips is then produced from raw data [ b) in Figure
1]; these clips are then evaluated by an interdisciplinary group of researchers and educators [ c)
in Figure 1] in order to gain perspective on the viewed activities [ d) in Figure 1]. There are
limitations on the assertions that may be made by the reviewers in that all assertions must be
grounded in empirical evidence, i.e., observable in the videotape. It is completed described in
reference (42).
August 11, 1996 (Regan, Sheppard)
perspective/framework ofunderstanding
viewvideosegments
events,segments,processes,postures
discuss /reflect
VideoInteraction Analysis Cycle
a)
b)
c)
d)
Figure 1. VIA Cycle Used inAssessment
This method, as applied to the study of engineering learning, has evolved through use by a
variety of researchers loosely associated with the "Interaction Analysis Laboratory" (IAL), a
joint venture between Xerox Palo Alto Research Center (Xerox PARC) and the Institute for
Research on Learning (IRL) in Palo Alto, California. Researchers with varied backgrounds have
been using this method for "the study of human-machine interaction, collaborative design
practice, and the situated nature of skill and knowledge acquisition", reference (42). More
recently this core group of researchers has been building an understanding of engineering
learning by assessing tapes of innovative engineering curricular activities, and has brought
sensitivity and understanding to the analysis of the "engineering issue oriented" tapes. For
example, a few researchers (e.g., references (35), (43-45)) have used VIA to illustrate the issues
present in integrated learned situations when students participate in "Synalysis exercises"
(exercises that integrate activities in problem formulation, analysis and synthesis), and in
mechanical dissection exercises.
B. Implementation
Measures such as the preliminary questionnaires and the pre- and post- interviews provided
context and background for the VIA. The clips chosen in this work were selected for their
illustrative power relative to two main issues: [1] the relationship between the seven main
resources, and [2] the effectiveness of the Courseware in facilitating learning. The importance of
the other two issues, namely, the effectiveness of the Courseware supporting [3] teamwork and
August 11, 1996 (Regan, Sheppard)
[4] the task, became apparent during the assessment process. Approximately forty hours were
required to view, edit, and provide preliminary evaluation of six hours of videotape [b) in Figure
1].
The video clips were then presented to an interdisciplinary group of researchers and
educators for evaluation [c) in Figure 1]. For this particular experiment, a total of three clips
were viewed by the VIA evaluation team. This included one clip from each group, each clip
approximately ten minutes in length. The evaluation of these clips was spread out over two 90-
minute VIA sessions, held two weeks apart. There were approximately ten participants in each
session, with approximately five of the individuals attending both sessions. Backgrounds of the
participants included anthropology, psychology, linguistics, mathematics and mechanical
engineering.
The two most significant potential shortcomings of VIA are the intensive time requirement,
and the lack of quantitative results. Performing VIA effectively requires a substantial time
investment for the primary evaluator(s) and the multidisciplinary evaluation team. Additionally,
because the technique analyzes the process of learning, instead of product, VIA does not yield
hard and fast quantitative results. We determined, however, that because of the newness and
complexity of the issues we were attempting to define and examine, simple quantitative data
would not suffice [see reference (46) for a complete discussion]. Qualitative data about the use
of all resources in concert was required to inform future design of the curriculum, and therefore
we concluded that the time investment was justified. Jordan and Henderson in reference (42)
have identified additional assets and liabilities of the VIA method.
V. Assessment Observations And Their Implications
Again, the fundamental objective of using VIA was to determine whether or not the
Courseware, in fact, enriched hands-on learning. The following sections discuss observations
from VIA as related to the four observational issues. In addition, the implications of these results
for improving the learning environment are discussed. Specific examples are given in italics.
A. Relationship Among Resources
The relationships between the Courseware and the resources were the main issue we set out
to investigate with this assessment study. Our goal was to discover what role each resource
played in the learning process and ultimately how the entire learning environment could be
improved.
Defining relationships between resources is key to creating criteria by which the Courseware
should be evaluated. We defined three significant questions that needed to be answered: is there
August 11, 1996 (Regan, Sheppard)
a "typical" usage cycle?; how is each resource used?; and, what motivates the transition from one
resource to another?. With answers to these questions, we could define more specifically the
role(s) played by the Courseware.
Below is the typical work pattern observed with VIA of Groups 1 and 2 utilizing the
resources:
1. The cycle begins with a new topic of inquiry. Topics are introduced from the given list of
questions, from a question generated in a student's mind, or a question generated from some
unclear or interesting aspect of the Courseware or a bicycle. Examples of topics are: the
freewheel (and how it achieves its various modes of relative rotation), (Groups 1 and 2); the
bottom bracket (Group 2).
2. The student group looks to Courseware for answers. At this point, one of two things tended
to happen: no satisfactory explanation would be found, and students would look to the repair
manual as the next resource expected to hold "the answer;" or, the Courseware would provide
a theoretical understanding of the topic, and leave the students lacking an internalized
understanding of what is happening. An example of the latter is: while looking at the
assembly drawing of the rear hub-freewheel assembly in the Courseware, one member
remarked "there's still something connected to the freewheel when you are riding your bike,
but I don't know". The group then went over to the reference bike to experiment with the
freewheel, frequently cross-referencing the assembly drawing on the computer screen (Group
1).
3. The student group locates the part(s) on the bicycle. Here, students try to learn from the
bicycle by examination (e.g., "what are the parts on the rear derailleur?"). Often bike
examination was accompanied by inspection of a repair manual, either for disassembly
instructions, or labeled part drawings and brief explanations of function.
4. The student group returns to the Courseware for another "dose of theory." One member does
a calculation using elementary equations or a simple sketch of the component/subsystem
under study using the paper and pen.
5. The group reinforces understanding by working with the bicycle to run experiments (e.g.,
"how is speed affected when I change to a smaller rear cog?") and to dissect.
6. Either return to step 1 or 2.
In general with Groups 1 and 2 all of the resources were accessed fluidly and when needed.
That is, when a question arose that could not be answered by the resource in current use,
attention was easily refocused to another that was thought to contain an answer. These two
groups appeared to be not particularly motivated by the assignment as represented by the seven
questions posted on the wall, but rather by a deeper internal motivation and curiosity. They used
the Courseware to provide a theoretical understanding, while the hardware was used to
August 11, 1996 (Regan, Sheppard)
experiment with the theory, thereby clarifying, reinforcing and solidifying understanding. Group
3 focused almost exclusively on the Courseware and the assignment, the only exception being
one member using the paper and pens to write down answers. Additional details regarding use of
the resources are available in reference (47). Note that the seventh resource, the assignment, is
not included in the discussion above; its impact and relationship to the other responses will be
discussed in detail below.
It is important to point out that despite the "new-technology" excitement of the multimedia
Courseware, the resource with the highest excitement value was the dissection bicycle. Upon
dissecting the rear derailleur, a member of Group 1 exclaimed, "this is the most fun part."
Similarly, both members of Group 2 bypassed the bottom bracket puzzle in the Courseware
because they were so deeply involved in exploring the actual bottom bracket -- despite the
attention grabbing music included in the bottom bracket puzzle.
The members of Group 2 pointed out in the closing interview that the Courseware did not
seem to be designed to explicitly take advantage of the hardware. They suggested that more
direct and explicit references between the resources be made. That is, placing "tags" of a sort
within the Courseware, directing the students' attention to a particular part or experiment to be
performed on the bike. It was observed by one of the student groups that " ...we used the
Courseware to get the theory, and the bike to do the experiment... it would have been nice to be
told, 'go to the bike and try this experiment.'" Dr. Ted Kahn, a member of the VIA review team,
suggested the opposite arrangement; place physical "tags" on the bike, pointing to a place in the
Courseware for further exploration. Taking this even one step further, it could become a
scavenger hunt, in which the students follow a trail of notes or clues.
B. Effectiveness of the Courseware as an Independent Learning Tool
The following effects of the Courseware on learning were observed [organized based upon
Simons' three modes of CAI in reference (25)]:
• Tutorial Mode: It was observed in Groups 1 and 2 that the content served as a starting point
for discussions and experimentation, and the Courseware's non-linear access mode
encouraged its use as a content source for randomly occurring questions. For these two
groups, even the most passive parts of the Courseware became an active part of their learning
because they were fully engaged in trying to interpret the contents and challenge their
understandings. Comments such as "I am still curious about the spring thing," "Is this what
you mean?" and "This is still kind of sketchy," were common in the work of Groups 1 and 2.
Group 3, however, used the content as a source from which to copy answers to our questions,
and did very little questioning on their own.
August 11, 1996 (Regan, Sheppard)
In general, the content seemed to suffer from a lack of navigational clues that would
allow for fine-grain searching; the Courseware contains a simple tree diagram with hyper-
links to major topic headings, but no detailed index is provided.
• Simulation Mode: This mode of the Courseware proved useful for all three groups. The
freewheel animation was used by all three groups to understand the functionality of that
mechanism which is hidden from view on the actual bicycle. The "Design Calculator,"
which provides immediate feedback by calculating drive train performance according to user
defined parameters (both bicycle hardware specifications and rider input power/force/speed)
was used by all groups. The "Calculator" proved to be the resource most useful to Group 3,
who did not experiment with the actual bicycles, but gleaned their "hands-on" trial and error
experimentation using the software.
• Drill and Practice Mode: Groups 1 and 3 both used and seemed to enjoy the bottom bracket
puzzle. Group 2, however, stumbled across the puzzle at a time when they already had a
focused goal in mind (dissection of the actual bike) and were not encouraged in any way to
return to and solve the puzzle, and so never did. The lesson of the puzzle did not become a
goal of Group 2, and was not used as a resource.
These observations plus direct student feedback suggest the following improvements to the
Courseware itself:
1. Integrate a more detailed index to increase the Courseware's usefulness as an "Information
Bank". With the current design, the Courseware works well as an environment for free-
flowing exploration, but is weak in its support of specific questions and issues. Such a
detailed index may help the Courseware to be more responsive to immediate, seemingly
random questions. It is interesting to note that one of the bicycle manuals (book) did not
contain an index. Upon inspection of the book, this fact was immediately discovered by a
student, and the book was immediately put down and not referenced again for the duration of
the exercise.
2. Inclusion of more experiences encouraging interaction between resources. This could
include more design exercises, "quizzes" and puzzles that require, for example, actual
measurements on the reference bike to proceed.
3. Increase student motivation by including more scenarios (or "Microworlds," in the vernacular
of Perkins) where performance is evaluated. For example, it may be worthwhile to integrate a
set of design criteria into the "Design Calculator," and prompt students to evaluate several
different designs relative to the criteria.
4. Take greater advantage of the visual simulation power of the multimedia Courseware by
offering views not possible in the natural world. One possibility would be to offer an "x-ray
August 11, 1996 (Regan, Sheppard)
view" into the internal operation of a mechanism, while simultaneously observing the external
functionality of that same mechanism. For example, seeing the rear wheel move while
watching the ratchet mechanism and bearings in the freewheel. Another example would be to
superimpose lines representing force vectors on a mechanism shown in movie form.
C. Effectiveness of the Courseware in Supporting Teamwork
The set-up in this experiment consists of the seven physical and software-based resources. These
seven resources comprise the learning environment and should co-exist and co-function as a
system. As such, it is important to comment on how effectively not only the Courseware but all
resources supported and encouraged teamwork. Based upon our VIA observations, the following
comments can be made:
1. Enough different resources were provided such that both (all) team members could
simultaneously play active roles in the learning; (for example, the members of Group 1
worked with the Courseware together, discussing the images that they were seeing on the
screen. They then both got up to explore the reference and dissection bicycles. Group 2 used
separate resources in tandem to answer a common question. For example, one member of
Group 2 worked with the Courseware, while the other member carried out experiments with
the reference bicycle; there was, however, significant discussion between the two and they
would observe the resource that the other member was "controlling." Group 3 (with three
members) worked exclusively with the Courseware for the entire exercise and one individual
"drove" the stack. Another member took on the role of "recorder," working with the pens
and paper. The third member, situated between the other two, directed much of the
discussion and stack navigation, but did not actually handle any of the resources).
2. The resources supported different individual learning styles and individual needs, allowing
the individuals to work effectively as a team; (for example, on Group 1's initial trip to the
reference bicycle, one member took on the prime role of suggesting various gear-ratio-
shifting experiments to run (a more reflective role), while the other member physically
performed the experiment (a more active role). Both observed the outcomes.)
3. All team members could at least observe each resource at the same time. In fact, utilizing
some of the resources required the cooperation of two people (the individuals of both Groups
1 and 2 collaborated when observing the derailleurs in action and experimenting with
power/force/speed/gear ratio issues—several hands were needed for pedaling and changing
gears and making meaningful observations).
One caveat with the Courseware is that while all team members can observe the screen,
only one student at a time can control movement through the stack. Our findings were
August 11, 1996 (Regan, Sheppard)
similar when student groups work at dissecting small mechanical artifacts (e.g., a fishing
reel); only one student at time can physically manipulate the device.
These three observations are to be contrasted with the more structured dissection exercises
used in the formal Dissection Course. Rochille [see references (35), (43)] found that when
written answers are required, one student usually assumes the role of the "recorder." The
Experimental Exercise did not require a written task, so no particular student was focused on
being the "recorder" in two of the three groups.
D. Effectiveness of the Courseware in Supporting the Assignment
The assignment given to the student groups (as shown in Table 2) represents a loosely
defined assigment. Its most positive characteristic was that it appears to encourage freer
exploration. The pressure of performance was reduced due to the lack of a "hard-point" measure
of success or failure. Freedom to explore personal interests has two main benefits; namely, that
students set their own goals, which may lead towards truer understanding instead of just
retention; and, there is the possibility for random/unexpected learning and exploration of ideas
they may not have otherwise approached. The VIA data seem to indicate that this sort of task
definition worked with Groups 1 and 2. They undertook the task as an opportunity to explore,
and to ask and answer questions.
Like the task definition, the Courseware did not impose any stringent external goals, acted as
an "Information Bank" resource to help students in their own discovery process, and often
presented concepts that generated questioning. In addition, the easy accessibility of various
topics facilitates browsing and allowed for exposure to major topics with which students were
previously unfamiliar.
This positive aspect of the lack of a specific task definition has the potential to be a negative
aspect as well. In addition, students may miss important connections and relationships because
the assignment itself does not prompt them to look at the information in any specific way. There
is the danger of students lacking goals, and therefore establishing no personal investment or
accountability in the exercise—this was generally observed in the actions of Group 3. The
Courseware did nothing to counter the student group lacking learning goals but did provide a
systematic way of viewing and learning about the vocabulary and functionality of the various
subsystems of the bicycle once a student or group had "committed" to studying a particular
subsystem. As with any learning resource, there is always the possibility of students using it to
merely "copy the answers from the book." Even with the open-endedness of the Exercise Task,
Group 3 is quoted as saying,
F: "We don't even need to look at the bike, we can just copy... are you for that?"
August 11, 1996 (Regan, Sheppard)
M: "Yeah, we can just copy."
As previously mentioned, all dissection exercises in the full Mechanical Dissection Course
include an initial step in which the team has a common in-class experience using the device (i.e.,
they become "users"). This step was left out of the Experimental Exercise; the results from VIA
only serve to underscore the importance of this step. The step serves to "bond the team" and also
gives each team member a more or less equal foundation on which to do the dissection. The
Courseware is not able to infuse teams with this physically-based experience.
VI. Longer Term Implications Of Assessment Observations And Conclusions
Our VIA observations and student feedback have made us acutely aware of the importance of
task definition on the success or failure of using multimedia Courseware concurrently with a
hands-on dissection exercise. This importance was greatly underestimated by the authors. It
became apparent after assessment of the exercise that an appropriately "revised" version of the
"Bicycle Exercise" used in the Mechanical Dissection course needs to be designed to take full
advantage of the Courseware, and that the Courseware should be redesigned with the "new"
Bicycle Exercise in mind. This redesign is really part of good engineering design practice, where
design iterations is accepted practice. Brereton in references (44) and (45), and Minneman in
reference (48) discuss at length the role of iteration in the design of educational exercises and
design environments, respectively.
It is imperative that the exact scenarios in which the Courseware is to be used be clearly "played
out" in the minds of the developers. Development of Courseware should proceed in parallel with
development of a lesson plan (where the lesson plan is the instructor's complete script for
teaching and assessing a particular body of material). Basic and very important strategic
decisions that need to made at the onset of creating the details of the lesson plan are related to the
roles that each resource might (or should) play in the overall exercise. Possible design strategies
include embedding the task/assignment within the Courseware (e.g., the Courseware could be
redesigned to be engaged in a more "linear fashion" with "points of departure" from which more
information/insight could be acquired from one ore more additional resources), embedding the
task within the physical hardware, or explicitly leveraging students' personal interests and
internal motivation (e.g., students explicitly defining their own learning goals, then signing a
contract of commitment to achieving these goals). These three approaches are discussed in
greater detail in reference (47).
We summarize by reiterating that past results in using multimedia Courseware in engineering
education, as well as unresolved challenges that we faced in the Stanford "Mechanical
August 11, 1996 (Regan, Sheppard)
Dissection" course led us to investigate the potential of multimedia-based technology to enhance
hands-on mechanical dissection exercises. The effectiveness of the Courseware in enhancing
learning was assessed using Video Interaction Analysis (VIA). Major conclusions that can be
drawn from this assessment study are:
1. For a motivated group, the Courseware played a positive role in assisting learning, often
providing a central focus for discussion, and a "point-of-departure" to the other resources. It
also provided the group with vocabulary, and a systematic and organized framework for
exploration of the various subsystems. We observed that students seem to prefer a
combination of multimedia-based and hardware-based learning.
2. Students access the Courseware in a variety of ways (e.g., the navigational strategy they use,
the time spent viewing videos or reading, the time spent with the various resources). The
Courseware should be able to accommodate these differences.
3. The Courseware did not eliminate the need for hands-on device experience (either "user"
experience, or as a vehicle on which to run experiments) and the Courseware should be able
to support these experiences. This conclusion is similar to the findings of Hsi and Agogino
(1); they found that the combination of Multimedia and hands-on exercises in the visual skills
domain can result in a powerful learning experience.
4. There need to be explicit "points-of-departure" between the various resources to provide
better integration of the entire learning environment. In addition, the Courseware needs to
give feedback to students (e.g., quizzes) to help them calibrate their understanding, and
needs a larger navigational framework that not only helps students move through the stack,
but also helps them see connections between various parts.
5. VIA provides an effective way to view how students use the Courseware. This assessment
technique is labor- and time-intensive but helped us to define and understand many of the
underlying issues related to student learning in the "resource-rich" environment that we
created in the Experimental Bicycle Dissection Exercise. Our understanding of the issues
involved with students working in this environment has matured using VIA to the point that
we are in the position to develop a student survey as another assessment tool to supplement
VIA data; this would allow us to get information from a greater number of students.
6. Our findings underscore that explicit selection of a lesson plan strategy should occur before
the development of any Courseware module. Content materials need to be placed within a
curriculum framework for the design of a learning environment is in fact a systems problem.
All of the elements of the learning environment (including the role of the teaching team,
which was not addressed in this study) need to be considered in concert. Flori discussed
something similar in references (15,17), but his conclusion was in terms of a framework for
August 11, 1996 (Regan, Sheppard)
the navigation within the Courseware—we advocate that Courseware be designed in the
context of the more global learning experience/environment within a course.
VII. Acknowledgments
We wish to thank those who have supported this work financially (The Synthesis Coalition
funded by the National Science Foundation, and The Powell Foundation), and intellectually (Ed
Carryer, Charlotte Linde and Jeramy Rochelle at IRL, Mitzi Renee Vernon, Margot Brereton,
Larry Leifer, Dao-liang Chou, Alice Agogino, Sherry Hsi, Bill Wood, Ted Kahn, Homer
Sheppard, and the student group: Sherri Taylor, Joseph Kosinski, Damien Crochet, Anna Bonner,
Jennifer Wu, Haresh Kamath).
REFERENCES
1. Hsi, S. and Agogino, A.M. (1994) "The Impact and Instructional Benefit of Using
Multimedia Case Studies to Teach Engineering Design," Journal of Educational
Multimedia and Hypermedia, 3 (3/4), 351-376.
2. Agogino, A.M., "The Proprinter Case Study," available on the NEEDS Database, as
presented at the session on The Role of Multimedia in Design Research and Education at
the ASME Design Theory and Methodology 1994 Conference, Minneapolis, MN, Sept. 11-
14, 1994. (WWW URL :http://www.synthesis.org/)
3. Deierlein, G.G., Valenzuela, M.L., White, R.N. (1993), "Multimedia and Case Studies in
Structural Engineering," Computer Applications in Engineering Education, J. Wiley, Vol.
1, No. 2, pp. 159-171.
4. Yaghi, H., Ali-Ahmad, W., "PC-Based Multimedia Educational Technology to Present and
Perceive Geographic Information with an Application on Water Resources of Lebanon,"
Proceedings from the 1994 ASEE Annual Conference, pp. 1564-1575.
5. Lipman, A., Lieu, D.K., "A Multimedia Case Study of an Engineering Failure for New
Engineering Students," Proceedings from the 1994 ASEE Annual Conference, pp. 2173-
2180.
6. Tan, S., Hong, J., Sheppard, S.D., "The Multimedia Drill Stack," available on the NEEDS
Database, as presented at the session on The Role of Multimedia in Design Research and
Education at the ASME Design Theory and Methodology 1994 Conference, Sept. 11-14,
1994, Minneapolis, MN. (WWW URL :http://www.synthesis.org/)
7. Huston, J.C., Gillette, J.C., Hiemcke, C., Johnson, R.M., Eversden, M.R., Pletka, R.J.,
"Prototype Educational Software for Integrating the Engineering Curriculum," Session
August 11, 1996 (Regan, Sheppard)
6B4, Proceedings Frontiers in Education, 24th Annual Conference, Nov. 2-6, 1994, San
Jose, CA, pp. 633-637.
8. Huston, J.C., Barrance, P.J., Heimcke, C., Multimdia Coursware in the Instruction of
Undergraduate Dynamics, Proceedings of the 22nd Annual Frontiers in Education
Conference, 1992, pp. 162-167.
9. Osborn, J.R., Agogino, A.M., "An Interface for Interactive Spatial Reasoning and
Visualization," in Proceedings of CHI'92 (Conference on Human Factors in Computing
Systems), Monterey, California, May 3-7, 1992, ACE, New York, NY, 1992, pp. 75-82.
10. Curtis, G., Faste, R., "Blockbuilder; A Project Sponsored by the Synthesis Coalition,"
available on the NEEDS database. (WWW URL :http://www.synthesis.org/)
11. Eibeck, P.A., 1993, "Multimedia Courseware Support for Instruction of Engineering
Experimentation," Proceedings of Frontiers in Education 23rd Annual Conference, (ed.
L.D. Grayson), Nov 6-9, 1993, Washington, DC.
12. Carlson, L., "Multimedia Applications in the Integrated Teaching and Learning
Laboratory," a working paper from the ITL at the University of Colorado at Boulder.
13. Eaton, J.K. and Eaton, L.I.J., LabTutor: A Friendly Guide to Laboratory Computer
Interfacing and LabView Programming, Oxford U. Press, 1995.
14. Kriz, R., "Data Visualization and its Role in Mulitmedia-Based Design Education," as
presented at the session on The Role of Multimedia in Design Research and Education at
the ASME Design Theory and Methodology 1994 Conference, Sept. 11-14, 1994,
Minneapolis, MN,
15. Flori, R.E. Jr., "Computer-Aided Instruction in Dynamics: Does it Improve Learning?"
Session 3D1 77-81, Proceedings Frontiers in Education, 24th Annual Conference, Nov. 2-
6, 1994, San Jose, CA.
16. Miller, G., "Multimedia Education in Structural Design," as presented at the session on The
Role of Multimedia in Design Research and Education at the ASME Design Theory and
Methodology 1994 Conference, Sept. 11-14, 1994, Minneapolis, MN.
17. Flori, R.E., Koen, M.A., Oblesby, D.B., "Classroom Experiences with 'BEST' Dynamics
('Basic Engineering Software for Teaching' Dynamics)," Proceedings from the 1994 ASEE
Annual Conference, pp. 959-966.
18. Darnell, J.H. , Stern, H.P., "Creating a Smooth Transition Between the Classroom and
Laboratory via Computer Simulation," Session 3D1, Proceedings Frontiers in Education,
24th Annual Conference, Nov. 2-6, 1994, San Jose, CA, pp. 93-97.
19. Martin, P.T., "An Overview of Multimedia for the Teaching of Engineering
Fundamentals," Proceedings from the 1994 ASEE Annual Conference, pp. 988-991.
August 11, 1996 (Regan, Sheppard)
20. Abbanat, R., Gramoll, K., Craig, J., "Use of Multimedia Development Software for
Engineering Courseware," Proceedings from the 1994 ASEE Annual Conference, pp. 1217-
1222.
21. Carver, C.A., Biehler, M.A., "Incorporating Multimedia and Hypertext Documents in an
Undergraduate Curriculum," Session 3D1, Proceedings Frontiers in Education 24th
Annual Conference, Nov. 2-6, 1994, San Jose, CA, pp. 87-92.
22. Oloufa, A.A. "Bringing the Real World to the Classroom with Multimedia," ASEE Annual
Conference Proceedings, Edmonton, Canada, June 26-29, 1994, pp. 2742-2745.
23. Meyer, D.G., Krzyzkowski, R.A., "Experience Using the VideoJockey System for
Instructional Multimedia Delivery," Session 4C4, Proceedings Frontiers in Education, 24th
Annual Conference, Nov. 2-6, 1994, San Jose, CA, pp. 262-266.
24. Cobourne, W.G., Lindauer, G.C., "A Flexible Multimedia Instructional Module for
Introductory Thermodynamics," J. of Engineering Education, July 1994, pp. 271-277.
25. Simons, J.M., "Computer-Assisted Instruction Programs as a Productivity Tool in
Engineering Education," Proceedings ASEE/IEEE Frontiers in Education Conference,
IEEE, New York, 274, 1989.
26. Hsi, S., Agogino A.M., "Use of Multimedia Technology in Teaching Engineering Design,"
Proceedings of HCI International 1993, 5th International Conference on Human-Computer
Interaction, pp. 778-783.
27. Hsi, S., Agogino A.M., "Navigational Issues in Multimedia Case Studies of Engineering
Design," Proceedings of HCI International 1993, 5th International Conference on Human-
Computer Interaction, pp. 764-769.
28. Mazur, J., "Engineering in Context: the Evaluation Report," Interactive Multimedia Group,
Cornell University, May 5, 1993.
29. Crismond, D., Wilson, D.G., "Design an Evaluation of an Interactive Multimedia Program:
Assess MIT's EDICS Program," 1992 Frontiers in Education Conference, Session 14B2,
656-661.
30. Davis, B. G., Humpreys, S., "Evaluation Counts: A guide to evaluating math and science
programs for women," (based on work supported by NSF grant number SED-8016785),
1983, Part II: Tools of Evaluation.
31. Hoadley, C.M., Hsi, S., "A Multimedia Interface for Knowledge Building and
Collaborative Learning," Proceedings of InterCHI'93.
32. Paul, J., Lewis, S.B., Supon, P.A., "Improving Education Through Improved Assessment,"
Session 3C1 17-21, Proceedings Frontiers in Education, 24th Annual Conference, Nov. 2-
6, 1994, San Jose, CA.
August 11, 1996 (Regan, Sheppard)
33. Gay, G., Mazur, J., Lentini, M, (1994) "The Use of Hypermedia Data to Enhance Design,"
Computer Graphics, SIGGRAPH, 28(1), pp. 34-37.
34. Gay, G., Mazur, J, "The Utility of Computer Tracking Tools for User-center Design,"
Educational Technology, 34(3), pp. 45-49.
35. Rorchelle, J., Charlotte, L., "Drill Dissection: Formative In-Depth Assessment Report,"
February, 1994, Institute for Research on Learning, Palo Alto, CA.
36. Sheppard, S.D., “Mechanical Dissection: An experience in how things work,” Proceedings
of the Engineering Education: Curriculum Innovation & Integration, Jan. 6–10, 1992,
Santa Barbara, CA.
37. Agogino, A.M., Sheppard, S.D., Oladipupo, A. “Making Connections to Engineering
During the First Two Years,” IEEE Proceedings of the Frontiers in Education Conference,
Nov. 13, 1992, Nashville, TN, pp 563-569.
38. Felder, R.M., Silverman, L.K., "Learning and Teaching Styles in Engineering Education",
Engineering Education, 674-681, April, 1988.
39. Mayer, R.E, Anderson, R.B., "The Instructive Animation: Helping Students Build
Connections Between Words and Pictures in Multimedia Learning," Journal of
Educational Psychology 1992, Vol. 84 No. 4, 444-452.
40. Perkins, D.N., "Technology Meets Constructivism: Do They Make a Marriage?"
Constructivism and the Technology of Instruction: A Conversation, edited by T. M. Duffy
and David Jonassen, LEA, Inc., 1992, pp. 45-55.
41. Frederiksen, J.R., Collins, A., "A Systems Approach to Educational Testing," Educational
Researcher, Dec. 1989, 27-32.
42. Jordan, G., Henderson, A., "Interaction Analysis: Foundations and Practice," a report for
Xerox Palo Alto Research Center (PARC) and Institute for Research on Learning (IRL),
Palo Alto, CA, 1992, accepted for publication in The Journal of the Learning Sciences.
43. Rorchelle, J., Charlotte, L., "Toy Dissection: Formative In-Depth Assessment Report,"
December 1993, Institute for Research on Learning (IRL), Palo Alto, CA.
44. Brereton, M.F., Greeno, J., Lewis, J., Linde, C., Leifer, L., "An Exploration of Engineering
Learning," Proceedings of the 5th International Conference on Design Theory and
Methodology, ASME Design Technical Conferences, Albuquerque, NM, Sept 19-22, 1993.
45. Brereton, M.F., Greeno, J., Leifer. L.J., Lewis, J., Linde, C., "Innovative Assessment of
Innovative Curricula: Video Interaction Analysis of Synthesis Exercises," ASEE Annual
Conference Proceedings, 1993.
46. Hawkins, J., Frederiksen, J., Colins, A., Bennett, D., Collins, E., "Assessment and
Technology," Communications, 74-76, May 1993, Vol. 36, No. 5.
August 11, 1996 (Regan, Sheppard)
47. Sheppard, S., Regan, Sheppard, "Bicycle Multimedia Courseware: Formative In-Depth
Assessment Report," December, 1995, Center for Design Research Internal Report,
Stanford University.
48. Minneman, S.L, The Social Construction of a Technical Reality: Empirical Studies of
Group Engineering Design Practice, Ph.D. Dissertation, Stanford University, Mechanical
Engineering, August, 1991.
August 11, 1996 (Regan, Sheppard)