Humanoid Robot Instructors for Industrial Assembly Tasks
Thomas Quitter
SemVox GmbH
Saarbrücken, Germany
[email protected]
Ahmed Mostafa
University of Calgary
Calgary, Canada
[email protected]
D'Arcy Norman
University of Calgary
Calgary, Canada
[email protected]
André Miede
Saarland University of Applied
Sciences
Saarbrücken, Germany
[email protected]
Ehud Sharlin
University of Calgary
Calgary, Canada
[email protected]
Patrick Finn
University of Calgary
Calgary, Canada
[email protected]
ABSTRACT
We are interested in the interactive aspects of deploying
humanoid robots as instructors for industrial assembly
tasks. Training for industrial assembly requires workers to
become familiar with all steps of the assembly process,
including learning and reproducing new tasks, before they
can be employed in a production line. The derived
challenges in current practice are limited availability of
skilled instructors, and the need for attention to specific
workers’ training needs. In this paper, we propose the use
of humanoid robots in teaching assembly tasks to workers
while also providing a quality learning experience. We
offer an assembly robotic instructor prototype based on a
Baxter humanoid, and the results of a study conducted with
the prototype teaching the assembly of a simple gearbox.
Author Keywords
Baxter humanoid robot; Godspeed questionnaire; human
computer interaction; human robot interaction; humanoid
robots; industrial assembly; industry; innovation;
instruction; learning; learning technology; performance;
performance technology; robotic instructor prototype;
robotics; teaching system; training; workplace health and
safety
ACM Classification Keywords
Experimentation; Design; Human Factors; Performance;
Standardization
INTRODUCTION
In industrial working environments, especially in the
assembly industry, it is critical for workers to learn how to
carry out a new task and reproduce it within the context of
an assembly line. Before workers can be employed in a
production line, they have to become familiar with all the
steps of the assembly process. For example, in the case of a
gearbox manufactured in a production line, workers have to
train in performing the complex assembly process several
times in order to ensure a flaw- and frictionless assembly.
There is a widespread need for supporting methods that
help workers learn new assembly skills. Currently,
“overseers” - observers or mentors - instruct, guide and
supervise workers during the learning process. However,
many challenges arise when relying on overseers. Drawing
on our combined experience, and collaboration with the
assembly industry, we present two examples of challenges
we hope to address: (1) there may be difficulties in serving
individual learning needs of different workers; and, (2)
there are a limited number of skilled overseers qualified to
teach specific components of a complex assembly processes
within the relevant context of the assembly process.
Our research aims to tackle these challenges by exploring
new ways to design and improve the learning process using
robots. Our goal is to understand how humanoid robots can
teach new assembly tasks to a worker, freeing up scarce
resources of available human overseers, while providing a
quality learning experience, which includes learning the
tasks required and experiencing the learning process in a
positive way.
Following strict research ethics and safety guidelines, we
developed an assembly teaching prototype using a Rethink
Robotics Baxter humanoid as an assembly task instructor.
We then conducted an evaluation study of our approach to
verify if it is possible to train an inexperienced worker to
successfully assemble a mechanical gearbox. Our goal was
to explore how humans interact in this learning situation,
and to measure user experience (UX), reflecting on
acceptability, intimidation, and other factors related to
collocated interactions with robots [1]. The results of our
study demonstrate that humanoid robots can become
effective assembly line instructors, and have the potential to
provide alternatives to visual-only learning systems such as
print, video, or augmented reality.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to
post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from [email protected]. HAI '17, October 17–20, 2017, Bielefeld, Germany
© 2017 Association for Computing Machinery.
ACM ISBN 978-1-4503-5113-3/17/10...$15.00
https://doi.org/10.1145/3125739.3125760
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
295
The contributions of this paper are as follows:
A novel design and implementation of an
assembly humanoid-based teaching system;
A study providing results that explore why and
how humanoid robots can be instructors for
industrial assembly tasks.
The remainder of the paper starts by highlighting previous
work in the area, followed by the design rationale for our
prototype. We then describe the technical system
components, and detail the study conducted. We conclude
by presenting our results, and discussing implications for
future work.
BACKGROUND
In industrial training and instruction, instructors are often
drawn from the ranks of personnel who are expert in a
given trade [6]. This approach can lead to shortages of
available instructors capable of training large numbers of
new workers, leading to a reliance on distance learning,
print, and electronic resources, to extend individual
instruction [16].
Robotics in Education
The use of robotics in training and education may follow
patterns of adoption in other areas of work. Research on
technologies that enable the integration of new forms of
media interaction, such as personal computers [4],
intelligent tutoring systems [5], social robots and virtual
agents [11] in education and physical rehabilitation [13] are
instructive.
Technology-enabled, competence-based, training has been
successfully used to train doctors to perform psychomotor
tasks, such as those used in laparoscopy (e.g., [9]). These
initiatives were designed using a model of simulated task
performance using virtual and augmented reality, and
mechanical bench training activities [14]. Simulated
representations of a patient, or mechanical system, were
used, with students learning and performing tasks within a
simplified, artificial environment.
Human-Computer Interaction
Virtual and augmented reality are employed in the training
of mechanical assembly tasks (e.g., [12, 14, 22, 24]).
Students demonstrate more rapid adoption of tasks when
interacting with fidelity simulated control interfaces, and
when manipulating tangible physical mechanical objects as
part of the learning process [3].
Virtual reality has been enhanced with mixed reality;
simultaneous modeling of a virtual world with real-world
counterparts for interaction [19]. This approach places a
layer of hardware and software between the learner and the
objects they interact with, because participants are required
to wear specialized equipment, such as head-mounted
displays.
Our research attempts to increase tangible direct
manipulation while decreasing the abstraction of interface
between instructor and student. Using this new approach,
students will learn to assemble mechanical components in-
situ, guided by gestural, verbal, and graphic instructions
and feedback provided by a Baxter robot, with the goal of
creating a fully, and correctly, assembled device.
Human-Robot Interaction
Robots have been shown to help learners overcome
language barriers by combining the use of gestures with
spoken commands [8,10], and can be effective as tutors
providing social gestural cues to students [21]. Social gaze
in robot interaction improves spatial management functions
involved in mechanical assembly operations by directing
attention, and movement, with nonverbal cues that
supplement or augment verbal, and written, instructions[7].
In typical human-robot instruction scenarios, humans
interact with robots using human movement to train
trajectories and movement for the robot [2]. In this work,
we explore the other side of the interaction. The robot leads
the interaction, and guides the human worker’s movement
during the assembly process.
PROTOTYPE HUMANOID TEACHING SYSTEM
Our approach integrates a humanoid robot into a teaching
system, where the robot takes on the task of overseer or
instructor teaching the industrial process of mechanical
assembly. We argue that while assembly tasks might be
automated completely by other robots, current assembly
industrial processes are still dependent on humans that
cannot to substituted by robots. Therefore, we focus on
addressing the current lack of skilled experts to train
“novice” new workers.
The robot trains workers in the assembly process by giving
instructions for each step, providing relevant information
for critical complex contexts, and offering assistance for
potential errors. The teaching system is not limited by time,
capacity or language constraints [25] as in the human,
overseer approach. In other words, the proposed system
allows any person regardless of linguistic differences to be
trained by Baxter, whose instructions are encoded and can
be localized as needed. These efficiencies lead us to believe
our approach will reduce training costs, and enable
inexperienced learners, as well as those requiring additional
experience, or who face language barriers, to learn and
work on an industrial production line.
Early Explorations
Beginning our research, we conducted pilot sessions to
study how people teach and learn technical skills, how
technology can support such learning, and how the Baxter
humanoid robot, can support or take on the role of
instructor.
We first identified basic elements of instructions that a
human uses while teaching (Figure 1). These elements
include expressive gesturing, specific pointing, speaking,
and demonstrating with tools such as images and video. Our
exploration simplified apprenticeship-based learning
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
296
demonstrated that humanoid robots have the potential to
substitute many basic elements involved in teaching a new
mechanical assembly task.
Figure 1. Example of one of the pilot sessions
A key design exploration during our pilot studies focused
on understanding the capabilities of Baxter as a humanoid
instructor. Baxter has: a rotating, face-like display, a
camera, and movable arms with flexible joints (Figure 2).
We explored variations of Baxter’s speed of motion, range
of movement, and the ability to display media. We found
that Baxter is able to successfully perform complex gestural
movements with its arms and grippers, but was not as fast
as human motion when moving between task-specific
locations. When playing multimedia, Baxter is capable of
providing supplemental media (e.g., text, images and
animation) that support the teaching of critical contextual
aspects of assembly tasks. By implementing speech
synthesis software, Baxter can also speak. This speech can
operate in any language, accommodating learners not
fluent in the original language of instruction.
Figure 2. Baxter, our humanoid robot instructor
Another design criterion requires generic system
development supporting usability with any type of
(humanoid) robot. Although our work involved Baxter, the
software, and methodology, can be generalized for
adaptation to other robots and approaches. This
transferability is achieved because of a design architecture
that abstracts robot actions into simple commands (e.g.,
move to a specific location, show a specific rotation
gesture, etc.).
Techniques to Support Teaching
Many of existing learning systems use visual techniques to
teach tasks. Examples include, the use of virtual,
augmented, and mixed reality, and computerized
simulation. While there are advantages to these approaches,
such as low associated cost, portability, and ease of
deployment, there are many disadvantages. Clearly, such
visual-only techniques lack the physicality, spatiality, and
personal aspects (e.g., facial expression and movement) a
human instructor provides. These interaction cues are
essential to provide effective teaching and learning. Further,
occlusion of physical objects or interface elements often
result when augmenting physical task components with in-
situ visualization, thus hiding some task operations, or
making them unclear, which hinder an efficient learning
process.
Our Approach
We designed, developed, and evaluated, an assembly
teaching system (ATS) utilizing the humanoid robot Baxter,
which is capable of basic hand gestures, movement, as well
as displaying media that simulates facial expression. While
we acknowledge the importance of robotic gaze [7] we
decided, mainly for experimental simplification, to support
Baxter with basic head movements while displaying a
simplified human-like face. By utilizing a humanoid robot,
our proposed approach aims to provide a moderate solution
between the low cost visual-only approaches, and the
expensive option of making a human instructor available
for all learning interactions.
The design of our prototype provides step-by-step
instructions to train workers on the assembly of a simple
gear box. This task includes 23 assembly steps. The
humanoid robot explains each step in succession using hand
gestures, visual diagrams, and speech. Our generic design,
utilizing XML, supports any kind of stationary assembly
process, and any kind of humanoid robot. In particular,
mapping the structure of any assembly task through our
XML format is achievable using the primitive learning
steps we identified (e.g., robot movement, gestures, etc.).
We added support to encode patterns for repeating task
components, which simplifies the creation of assembly
plans. Using XML for assembly tasks, our approach allows
for the embedding of specialized tags to address user
mistakes (e.g., branching to assist a trainee when he or she
makes a mistake during the teaching process). This
approach allows for future work, task flow, instructions,
and potential errors that can be automatically generated by
an artificial intelligence component, thereby simplifying the
creation of project-specific XML content.
We conducted an evaluative study to verify whether it is
possible to train an inexperienced person in a specific
assembly task. In addition, we explored human interaction
in the learning process with the robot. Our UX measures
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
297
include reflection on acceptability, and other factors
relating to interactions with robots [1].
Since our focus is on the UX, we designed our Baxter
prototype to operate through a high-level Wizard-of-Oz
approach [15] where, unknown to the trainee, a human
operator or “Wizard” oversees the robot’s actions. This
manual operation mode simulates the output of an artificial
intelligence algorithm, which could be implemented in
future designs. While this approach was crucial to our
study, it is important to note that our Baxter prototype
already includes completely functional gesture, and media
modules, for step-by-step assembly instructions. The
Wizard controls only the instructional flow: e.g. selecting
the next step for the robot to perform, or to repeat a step.
SYSTEM DESCRIPTION
The following section describes the assembly teaching
system prototype, used to evaluate whether, and how,
humanoid robots can teach industrial assembly tasks.
In this work, we used Rethink Robotics’ industrial
humanoid robot Baxter. Baxter is dual-armed, and 1 meter
tall providing seven degrees of freedom. Pedestal mounting
enables stationary usage in a single, fixed location.
Architecture
As described in the previous section, the robot should
impart instructions to the user, in a human style, using arm
gestures, speech, and facial expressions. Thus, our design
divides the prototype system into two discrete units; the
robot acting as the presentation medium, and the Assembly
Teaching System as a back-end application that assumes all
information processing tasks (Figure 3).
Baxter can be operated using the Robot Operating System
(ROS) open-source framework [17], which allows a simple
application creation using the Python API provided. The
framework allows the implementation of actions such as
moving each of Baxter’s joints to a specific position,
controlling of grippers (hands), or setting the content to be
displayed by the head (screen). These actions are published
by ROS to the robot for execution.
Figure 3. Structure of Assembly Teaching System Components
Using ROS we developed an application that serves as an
interface for the Assembly Teaching System. By making
this interface available using socket connections, the
Assembly Teaching System is able to send actions in a
predefined format to the interface, and thereby control the
robot. This client-server architectural approach isolates the
control unit of the robot from the operator application,
which provides robot-instructions. This encapsulation
makes the system generic, enabling it to operate with
different kinds of robots without the need to revise the
whole system whenever one of its components changes.
Task Representation
The Assembly Teaching System contains all information
required for the assembly training. In lay terms, an
assembly-training task needs to be simplified as a set of
steps, which can be sent to the robot one-by-one for
execution. Depending on the user’s response to a given
instruction (e.g., the user correctly performed the step, or
made a mistake), the software decides which subsequent
step appropriate. We implemented a flowchart-utilizing
state machine to represent the sequence of events for the
assembly process of the gearbox object used in our
prototype (Figure 4). Each state of this flowchart represents
an instruction given to the user. The transition between
states is determined by the outcome, or user action.
It took several iterations to identify and optimize the
individual assembly steps for the gearbox object, as
described in the pilot study below. Our final description
involved 23 concrete assembly steps, and 22 state
transitions required to represent the necessary information
for successful assembly (Figure 5).
A challenge for this type of state-representation is that the
generated steps only cover the ideal flow of the process,
and exclude potential user mistakes. After incorporating the
most common, and predictable, mistakes a user might make
during the process, and then matching them with an
appropriate solution, the flowchart contained more than 100
steps, and 200 different transitions, which created the need
for a more precise representation.
Figure 4. Gearbox object (top) and its discrete parts (bottom)
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
298
Figure 5. Example of a 3-step flow diagram without failures
Analyzing the flowchart, we isolated several repetitive
instructions. Our results indicated that for these types of
instructions, predicted failures, and subsequent resolution
steps were always the same. One example is the instruction,
“Pick Object A and place it at Position B in Orientation C”.
Predictable mistakes in this case include the user picking
the wrong object, or placing it at the wrong
position/orientation. These findings allowed us to introduce
“patterns” to represent repeating procedures in an efficient
manner. As a result, our methodology provides a significant
simplification while sequencing the assembly process.
Implementation Details
We implemented a Java-based, state-machine component
that maps the XML structure mentioned earlier. This
method streamlines the move between the instructions’
steps based on transitioning conditions. By representing
steps in a data packet structure, networked communication
offers an efficient means to engage the robot interface.
To provide audible information, in addition to the visual
movements, we used the SemVox Ltd. ODP S3 Speech
Dialogue Platform [20]. This platform enables not only
speech synthesis of the information defined in the assembly
instruction files, but also the ability to talk and control the
system by voice input. By placing speakers and microphone
near the robot, the user gets the impression that the robot is
talking and understanding during the interaction.
To support evaluating the system using the “Wizard of Oz”
approach, we developed a graphical user interface (GUI)
that allows the experimenter to control the robot indirectly
through the Assembly Teaching System. As shown in
Figure 6, the GUI provides the functionality to load and
navigate through an assembly plan.
Figure 6. Main interface of the Assembly Teaching System
As soon as the experimenter starts an assembly teaching
process using the Wizard GUI, the robot executes the
instructions for the first step. The operator can then choose
how to proceed. If the user follows the instructions given by
Baxter, the operator can choose the function “OK” and the
system proceeds to the next step. If the user makes a
mistake, or needs guidance, the operator can choose the
appropriate function from the list of mistake protocols. This
action initiates a step providing a solution for the problem
along with detailed, corrective information.
It is difficult to design for all user mistakes, so the Wizard
GUI contains several functions to provide general help
given as user feedback. The operator can choose from of a
list of 12 sentences to be spoken aloud by the system.
Examples of spoken feedback include “This is the wrong
position for this object”, “Please return the tool to the
toolbox”, and “Yes, this is correct; well done”.
For complex assembly steps, or situations in which a user
has difficulty following an instruction, a picture can be
displayed on Baxter’s screen, clarifying the instruction
steps. Currently, Baxter only gestures to the correct location
or indicates the action to be carried out. This behavior may
be extended allowing Baxter to physically grasp or
manipulate objects during the training to demonstrate
complex assembly steps where the trainee has failed after
multiple attempts. It is worth noting that this approach can
still be limited in supporting particular assembly steps that
require flexibility beyond what can be provided by a robot
(e.g., complex rotation or fast interaction). Our research
shows that the robot as physical entity best supports
learning, and thus refrain from abstracting the learning
process by merely loading images on Baxter’s screen.
Instead, further humanizing the robot presenting emotional,
facial responses is of benefit. The robot can express mood
by presenting facial expressions, which extend the
anthropomorphic capabilities of the robot.
By default, Baxter’s screen shows a neutral facial
expression. If needed, the operator can trigger a “happy” or
“sad” expression, to acknowledge that the user carried out
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
299
an action correctly, or to indicate when something was
unsuccessful (Figure 7).
Figure 7. Examples of Baxter’s facial expressions
The Training Environment
After implementing the system, we planned the working
area for the user and robot. Our aim was to create a face-to-
face learning situation. The workspace was a workbench,
containing separated parts of the assembly objects, a
toolbox, and several process-specific areas (Figure 8).
We used a one meter high table, allowing the user to stand.
The standing position is designed to make the user feel
more comfortable while working with the approximately 2
meter high robot on the other side of the table. A user
unfamiliar with humanoid robots might feel uncomfortable
sitting at a height well below the robot. The user’s position
opposite Baxter, allows them to take a step back at any time
taking them out of reach. This positioning, along with the
table as both work space, and physical barrier, is intended
to make the user feel physically safe with the robot.
Figure 8. Spatial configuration of workbench used in our setup
On the workbench, several boxes for the raw parts and tools
can be located. The workbench space is subdivided into
different regions in order to prevent a collision of the robot
and objects on the table, and to simplify the planning of
movements. Masking tape marks specific areas on the
workbench. The center marked area is intended to serve as
the space where assembly is carried out. The second area
can be used, to temporarily store a tool, or partially
assembled object needed later.
To define the robot arm-movements for teaching each of
the assembly steps on the workbench, we developed a
motion-planning tool using Baxter’s “Zero-G” mode. In this
mode, operators manually move Baxter’s limbs to specific
positions capturing coordinates. The captured coordinates
represent key points that can be stored in the Assembly
Plan, for later use in the automatic calculation of the
complete movement trajectory.
EVALUATION
We conducted an exploratory study to answer the question
of whether and how a robot can take on the tasks of an
overseer when teaching industrial assembly tasks. As part
of this work, we analyzed how efficiently a robot can
impart knowledge when operating an instructor. As
mentioned above, our study went through, and was
approved by, a rigorous ethics process at the host
university1. In addition to the required elements, security
measures were added when the robot physically led
participants including, (1) active observation by the
researcher throughout the process allowing intervention as
needed; and, (2) a fail-stop button positioned at the side of
the participant allowing him or her to immediately shut
down the robot. As part of the research ethics committee’s
process, a protocol for introducing, and reviewing these
procedures, and safety measures, was presented in detail
with each participant. One of the areas of interest for our
team was the level of comfort experience by human
operators interacting with the Robot instructor. As such, we
watched closely for moments of even the slightest
discomfort to maintain the integrity of our work, and to
refine our process for broad-based adoption. Participants in
the study were interested in the work because of the human
robot interaction, but we encouraged an awareness of
discomfort to explore any potential challenges for future
implementations of the approach. This focused, and labor-
intensive approach to research is one of the key reasons that
our test group was kept to 15 participants. At the same time,
the intense focus of this approach is one that provides clear
outcomes for the research team to analyze, synthesize, and
further test.
Participants
We recruited 15 participants (10 M / 5 F, mean age 27
years) from a local university for our study. In the
recruitment process, we considered participants of varying
age and gender and did not require prior knowledge of
industrial assembly tasks. All participants who volunteered
were included in the study, and the demographic
composition of participants was a direct result of this open
recruitment process. Out of the 15 participants, two were
familiar with complex assembly tasks, though not with the
gearbox object used in this study.
Task & Context
For this study, we used a small gearbox as the assembly
object (Figure 4). The gearbox contains 16 parts that have
to be put together in specific position and order. We
identified the steps needed for the assembly and specified
an order in which the steps must be performed. Our
approach, requiring a strict order of actions, simulated a
realistic scenario in which workers must strictly follow a
1 Study Id: REB16-0943
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
300
specified method of assembly with no room for improvised
action, or trial-and-error approaches.
The gearbox used for our study is simple. An experienced
worker should be able to assemble it in less than 5 minutes.
While simple, the model provides a valid assembly task,
and includes the opportunity to make several assembly
mistakes during the process.
The experimenter controlled the assembly teaching system
through two computers while sitting three meters away
from the workbench and table. Since the experimenter
assumed the role of the “Wizard” of the system, he or she
had to be aware of what participants were doing at all times.
To simplify the Wizard task, we streamed live video of the
workbench to the second computer used by the
experimenter using Baxter’s head-monitor camera.
Structure
Each session lasted around 45 minutes, starting by
welcoming the participants and giving them an overview of
the project, and the robot. Participants were first trained
(multiple times) in the assembly with the robot as the
instructor. Afterwards, they were asked to carry out the
assembly process on their own to assess the success of the
training. The self-assembly was the last practical part of
each session. Participants were asked to carry out the
assembly on their own without help. In addition, they were
specifically told that self-exploration of the task was
prohibited, and that they must carry out the steps in the
order Baxter instructed.
We implemented post-session questionnaires to obtain
insight into the personal experience of participants. We
used the “Godspeed” Questionnaire [1], which is an
established model for rating human experience with robots.
23 of the 24 questions from the full Godspeed questionnaire
were used, with question 9 “Artificial” removed because it
was confusing to participants (based on our pilot sessions).
We then asked participants to complete a questionnaire we
designed to rate specific parameters of the robot in the
study. We included questions asking if they liked Baxter
and our system, if they would rather train with a human, if
Baxter’s movements were helpful for understanding the
task, and if they wanted to talk to Baxter.
Study Description
We conducted a pilot prior to our evaluation to refine our
study protocol, and assess whether the system, and study
design would fulfil the stated purpose of the study.
Pilot Explorations
During the pilot study, we experimented with two different
assembly plans. The first one was a highly detailed
assembly plan, dividing the assembly instruction into 24
discrete steps (without error handling). Each instruction
from the robot included great detail. We also used pictures
to clarify complex instruction steps, displaying them on
Baxter’s screen, providing hints to the user. The second
assembly plan simplified the first, combining each group of
logically associated steps into a single step. This
streamlined plan required only 12 steps, enabling faster
assembly, if, and when, the user performed the steps
without errors.
In the pilot study, three participants were allowed to train in
the gearbox assembly process up to three times with
Baxter’s help before being asked to assemble the object
without assistance. At the beginning, the detailed plan was
followed by a short questionnaire asking if further training
was needed, and if the comprehensive (detailed) plan
should be used. Depending on the answers, the participant
would train for a second, or even a third time, or he or she
would be directed to self-assemble the gearbox object.
Final Study Design
Following our pilot explorations, we conducted our study
with the 15 participants we recruited. Our final study design
utilized what we learned from the pilot sessions. In this
section, we provide a detailed description of how we
conducted the formal study.
Our pilot created disparities between participants who
selected different levels of exposure to Baxter’s training by
allowing them to decide to opt out of the second or third
training run. To achieve greater consistency, we decided not
to vary the procedure between participants. All participants
were asked to perform the assembly three times with the
robot. This modified structure was used when conducting
the full study.
The feedback from the pilot indicated that the steps of the
first assembly plan were too detailed, and consequently too
time-consuming. Using this feedback for the full study,
Baxter offered the simplified assembly plan as a default,
only reverting to the detailed plan if needed. This procedure
provided adaptive capacity in the training process to
address the learning speeds of individual participants.
Initially, we noticed that some participants focused on
pictures that accompany the assembly instructions, and did
not pay enough attention to Baxter's movement and speech.
Consequently, they made several mistakes since they were
missing relevant information, which could not be derived
from the pictures. Therefore, we decided to limit the use of
pictures during the training, and only provide them as
needed. If the user got stuck on a particular step, the
experimenter could manually trigger the GUI to display a
picture to help the participant complete the step.
We wanted to ensure participants waited until the robot
finished the explanation of a step, before they attempted to
carry it out on their own. This approach supports a more
complete learning experience; and connects to previous
studies [18,23] that explored how light can be an effective
indicator for turn taking. We implemented a feature for
Baxter that turned signal lights on while something was
being explained. When the light turned off, the user would
understand the robot has finished the instruction, and the
user should now carry out the step.
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
301
RESULTS AND DISCUSSION
The goal of our study is to assess the practicality of Baxter,
as a humanoid robot, in teaching technical assembly skills.
As mentioned earlier, our study involved participants who
have trained with the robot using three training sessions.
The results revealed interesting insights into the
effectiveness of our approach to supporting the learning of
assembly tasks without a human overseer. Tellingly, every
participant was able to learn the assembly task from the
robot, and then self-assemble the gearbox object in a timely
manner.
Baxter Training Results
Participants trained with Baxter for three sessions prior to
conducting the self-assembly. As expected, our results
show that participants’ learning improved after each
training session. As detailed in Figure 9, after each
subsequent session, participants needed fewer steps to
complete the assembly, made fewer mistakes, and required
less help from the robot. We argue that it is the presence of
the robot, and not the repetition of the task, that provides
the most effective and efficient learning experience. This
outcome arises from our process, which asks participants to
refrain from self-training or step memorization, in order to
focus on robot-provided instructions.
The combined training and assembly time of each
participant, is a major factor of user performance. As shown
in Figure 10, almost all participants required less time for
each subsequent session with Baxter. This efficiency also
extended to the self-assembly of the gearbox object.
Figure 9. Average participants’ performance with Baxter
During self-assembly, all participants managed to complete
the task without mistakes, and in a reasonable time (average
= 4 minutes). All participants except two followed the
training steps taught by Baxter, indicating that Baxter was
effective in teaching participants the assembly tasks.
Questionnaire Results
The questionnaire responses also support our claim that
Baxter is an effective and efficient instructor of the
assembly tasks. All but two participants also stated they
liked Baxter, and the teaching system. 11 of the participants
said they liked Baxter’s feedback such as moving arms to
guide them. As one expressed, “I think I could remember
how to position the objects thanks to Baxter’s arm
movements”. Interestingly, those who said they liked the
robot suggested Baxter would be helpful training people
with disabilities, or those who lack language skills required
to work in industry.
Figure 10. Participants’ task completion time
Godspeed Results
The results of the Godspeed Questionnaire [1] revealed that
most participants viewed Baxter as having human-like
aspects, and thought that Baxter was conscious of their
presence. This finding is also supported by our
questionnaire responses highlighting that 12 participants
were comfortable with the robot and Baxter’s movement,
with three participants claiming to be less comfortable, but
only at the beginning of the interaction. As one participant
expressed, “When my arm hits the robot's arm, I was at first
a little bit scared. But then I realized that nothing can
happen to me and I started feeling comfortable”.2
Supporting the anthropomorphic power of Baxter is the
uniform declaration of a desire to speak with the robot. One
participant made this clear by stating, “As I got stuck at a
step, I wanted to ask the robot what to do. It would be nice
if I could talk with him [or her]”.
A well-trained, knowledgeable, and empathic human
instructor can adapt to learners’ abilities, and understand
and respond to social cues as part of an optimal learning
experience. A humanoid robot also offers learner
adaptation, can engage in some social cues, but provides
elements not available in human instructors. Consider the
need for instructors to train immigrant workers who speak
different languages. A humanoid robot can easily operate as
2 As discussed in Evaluation, our study followed a strict
safety protocol, adding additional measures beyond those
required by the host university’s research ethics committee.
An observation such as this might give us pause, but it was
just this type of observation we encouraged our participants
to share. Safety is of utmost concern, but so to is the
perception of safety. Beyond the ethical approach our team
is committed to following, we are also aware that learning
cannot occur if the subject does not feel safe in their work
environment.
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
302
a multilingual instructor by offering translated versions of
instruction materials. This option is particularly promising
for teaching discrete procedures and tasks. Additionally,
when working in such a multilingual, multicultural teaching
environment, the disparity in social cues between the
different learners, reduces the need for social cue expertise
carried by the ideal human instructor described. Some
participants expressed feedback supporting these assertions
saying, “[I think that] using, a robot for simple assembly
tasks is more efficient than a human teacher”, and “I guess
a human could detect any mistakes better than a robot. But
involving a robot as overseer has many other advantages”.
Implications for Design
Here, we present the main insights gained from our study to
support inform the important considerations for designing
future humanoid-based teaching systems. First, it is
important to optimize the design of arm movements so
movement is parallels the speed of humans. Some
participants complained that Baxter’s movement was
slower than expected. Second, integrating machine-learning
techniques into the design of future humanoids might
improve the learning experience. This dynamic capacity
would allow the robot instructor to respond to some of our
participants’ expectation that Baxter be able to adapt to
their skill level. To address this desire and realize learning
and outcome efficiencies, the system implementation
should include smart decisions, and adjust the level of
instruction relative to each learner. Third, it may be useful
to use mixed reality in conjunction with humanoid robots to
create a richer learning experience when demonstrating
challenging instructions, rather than using robot face
screens to display simple images.
CONCLUSIONS AND FUTURE WORK
We presented a novel approach for teaching technical
assembly tasks utilizing a humanoid robot as instructor. We
implemented an assembly-teaching prototype utilizing the
humanoid robot Baxter with a flexible approach to teaching
assembly tasks. We also conducted a study assessing the
effectiveness of our approach, which provided feedback
supporting our claim that humanoid robots are effective and
engaging instructors of technical assembly tasks.
We envision various directions for extending our work. An
experiment integrating our approach into an actual
industrial setting would provide valuable data. A
comparative study of our humanoid-based teaching system,
and assembly learning systems based on visual-only
techniques such as augmented reality would support the
refinement of interaction, and identify contextual
appropriateness for the use our approach. Within our
current model, the use of the “Wizard of Oz” component
assumed some of the higher level technical responsibility.
Future development could take on more of the process,
requiring less supervision at each step, eventually replacing
the wizard, rendering our system fully autonomous.
REFERENCES
1. C. Bartneck, D. Kulić, E. Croft, and S. Zoghbi. 2009.
“Measurement instruments for the anthropomorphism,
animacy, likeability, perceived intelligence, and
perceived safety of robots,” International journal of
social robotics, 1(1), 71-81.
2. A. Billard, S. Calinon, R. Dillmann, and S. Schaal.
2008. “Robot programming by demonstration,” In
Springer handbook of robotics (pp. 1371-1394).
Springer Berlin Heidelberg.
3. A. C. Boud, C. Baber, and S. J. Steiner. 2000. “Virtual
Reality: A Tool for Assembly?,” Presence: Teleoper.
Virtual Environ. 9, 5 (October 2000), 486-496.
http://dx.doi.org/10.1162/105474600566970
4. E. Bridgeman and N. Bridgeman. 2008. “A future for
robots in schools: Fact or fantasy?,” Journal of Applied
Computing and Information Technology, 12(1).
5. R. R. Burton, J. S. Brown. 1979. “An investigation of
computer coaching for informal learning activities,”
Intern. Journal of Man-Machine Studies, 11(1), 5–24.
1979. http://doi.org/10.1016/S0020-7373(79)80003-6
6. W. H. Dooley. 1919. “Principles and Methods of
Industrial Education for Use in Teacher Training
Classes,” Houghton Mifflin company.
7. K. Fischer, L. C. Jensen, F. Kirstein, S. Stabinger, Ö.
Erkent, D. Shukla, and J. Piater. 2015. “The Effects of
Social Gaze in Human-Robot Collaborative
Assembly,” In Social Robotics (Vol. 9388, pp. 204–
213). Springer International Publishing.
http://doi.org/10.1007/978-3-319-25554-5_21
8. M. E. Foster and C. Matheson. 2008. “Following
Assembly Plans in Cooperative, Task-Based Human-
Robot Dialogue,” In Proceedings of the 12th Workshop
on the Semantics and Pragmatics of Dialogue (2008),
June 2008. http://www.kcl.ac.uk/content/1/c6/03/89/15
/paperfoster.pdf
9. F. H. Halvorsen, O. J. Elle, V. V. Dalinin, B. E. Mørk,
V. Sørhus, J. S. Røtnes, and E. Fosse. 2006. “Virtual
reality simulator training equals mechanical robotic
training in improving robot-assisted basic suturing
skills,” Surgical Endoscopy and Other Interventional
Techniques, 20(10), 1565–1569.
http://doi.org/10.1007/s00464-004-9270-6
10. J. Han, M. Jo, V. Jones, and J. Jo. 2008. “Comparative
Study on the Educational Use of Home Robots for
Children,” Journal of Information Processing Systems,
4(4), 159–168.
http://doi.org/10.3745/JIPS.2008.4.4.159
11. J. Li, R. Kizilcec, J. Bailenson and W. Ju. 2016.
“Social robots and virtual agents as lecturers for video
instruction”. Computers in Human Behavior, 55,
pp.1222-1230.
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
303
12. A. Langley, G. Lawson, S. Hermawati, M. D'Cruz, J.
Apold, F. Arlt, and K. Mura. 2016. “Establishing the
Usability of a Virtual Training System for Assembly
Operations within the Automotive Industry,” Human
Factors and Ergonomics in Manufacturing & Service
Industries. http://doi.org/10.1002/hfm.20406
13. D. Lees and P. LePage. 1994. “Will robots ever replace
attendants? Exploring the current capabilities and
future potential of robots in education and
rehabilitation,” International Journal of Rehabilitation
Research, 17(4), 285-304.
14. Z. Materna, M. Kapinus, M. Španěl, V. Beran and P.
Smrž. 2016, August. “Simplified industrial robot
programming: Effects of errors on multimodal
interaction in WoZ experiment”. In Robot and Human
Interactive Communication (RO-MAN), 2016 25th
IEEE International Symposium on (pp. 200-205).
IEEE.
15. L. Molin. 2004. “Wizard-of-Oz prototyping for co-
operative interaction design of graphical user
interfaces,” In Proceedings of the third Nordic
conference on Human-computer interaction (NordiCHI
2004). ACM, New York, NY, USA, 425-428.
http://dx.doi.org/10.1145/1028014.1028086
16. H. B. Ndahi, J. M. Ritz. 2002. “Distance Learning in
Industrial Teacher Education Programs,” Journal of
Technology Studies, 28(1/2) (Winter-Fall 2002), p64.
17. Open Source Robotics Foundation. 2017. Robot
Operating System (ROS) open source framework.
Retrieved May 30, 2017 from http://www.ros.org
18. H. R.M. Pelikan and M. Broth. 2016. “Why That Nao?:
How Humans Adapt to a Conventional Humanoid
Robot in Taking Turns-at-Talk,” In Proceedings of the
2016, CHI Conference on Human Factors in
Computing Systems (CHI 2016). ACM, New York,
NY, USA, 4921-4932. DOI:
http://dx.doi.org/10.1145/2858036.2858478
19. K. Schäfer, V. Brauer, and W. Bruns. 1997. “A new
approach to human-computer interaction—
synchronous modelling in real and virtual spaces,” In
Proceedings of the 2nd conference on Designing
interactive systems: processes, practices, methods, and
techniques (pp. 335-344). ACM.
http://doi.org/10.1145/263552.263638
20. SemVox GmbH. 2017. SemVox ODP S3 Speech
Dialogue Platform. Retrieved May 30, 2017, from
https://www.semvox.de/en/technologies/odp-s3-the-
basis-for-interactive-systems/
21. S. Strohkorb, C. Huang, A. Ramachandran, and B.
Scassellati. 2016. “Establishing Sustained, Supportive
Human-Robot Relationships: Building Blocks and
Open Challenges,” At AAAI Spring Symposium on
Enabling Computing Research in Socially Intelligent
Human-Robot Interaction., Palo Alto, USA. March 21-
23, 2016.
22. F. Suárez-Warden, E. G. Mendívil, L. Neira, and O.
Strobel. 2015. “i-Tutoring with AR to support
decisions in assembly via Problem Solving for
aeronautical transportation,” In the 17th International
Conference on Transparent Optical Networks (IEEE
ICTON), pp1–4.
http://doi.org/10.1109/ICTON.2015.7193381
23. A. L. Thomaz and C. Chao. 2011. “Turn-taking based
on information flow for fluent human-robot
interaction,” AI Magazine 32 (4): 53-63.
24. S. Webel, U. Bockholt, T. Engelke, N. Gavish, M.
Olbrich, and C. Preusche. 2013. “An augmented reality
training platform for assembly and maintenance skills,”
Robotics and Autonomous Systems, 61(4), 398–403.
http://doi.org/10.1016/j.robot.2012.09.013
25. C. Zirkle. 2002. “Identification of Distance Education
Barriers for Trade and Industrial Teacher Education,”
Journal of Industrial Teacher Education. 40(1).
Session 11: Collaboration and Human Factors II HAI 2017, October 17–20, 2017, Bielefeld, Germany
304
