Control sharing in human-robot team interaction

Selma Music 1 , Sandra Hirche 2

Technische Universitat Munchen, Munich, Germany

Abstract

Human-robot interaction is a wide area of research which exploits complementary competences of humansand robots. Humans are capable of reasoning and planning, while robots are capable of performing tasksrepetitively and precisely. Rapid developments in robotics and reduction of cost lead to an increased re-search interest in the area of robot teams and human-robot team interaction. One of the key researchquestions is how to combine human and robot team decision making and task execution capabilities, i.e.how control should be shared among them. This paper surveys advances in human-robot team interactionwith special attention devoted to control sharing methodologies. Additionally, aspects affecting the controlsharing design, such as robot team and human behavior modeling, level of autonomy and human-machineinterfaces are identified. Open problems and future research directions towards joint decision making andtask execution in human-robot teams are discussed.

Keywords: Human-robot team interaction, shared control, human behavior modeling, robot team

1. Introduction

Human-robot team interaction describes the in-teraction between a human and multiple robots,which collaborate to achieve a common goal. Its en-visioned benefits are superior performance in highlyunstructured tasks in unknown and/or remote en-vironments, reduced human workload, execution oftasks which are not possible with a single robot,flexibility in task execution, and robustness. Ap-plication domains of human-robot team interactioninclude for example: search and rescue [57], cooper-ative manipulation [63], collaborative manufactur-ing, logistics, and construction.Rapid technological developments in the area of au-tonomous robotics result in large improvements ofrobots’ reliability and adaptability to unknown en-vironments. With these developments the nature ofhuman-robot interaction changes, as robots becomesmart tools to humans, or even their collaborativepartners.Reduction of price, size, and operational complex-ity considerably increases the availability of mod-ern robots, while the advancements in communica-

1selma.music@tum.de2hirche@tum.de

tion technology allow a seamless information ex-change between them. These developments areenablers for multi-robot systems. They provide in-creased flexibility and robustness and are capableto conduct more complex tasks then single-robotsystems [90].Even though the capabilities of modern robots areenhanced, they still need human intervention inthe form of high-level reasoning and planning. Asa consequence, novel forms of human-robot inter-action beyond single-human-single-robot have be-come a current and important topic of research:multiple humans-single robot interaction [73], mul-tiple humans-multiple robots interaction [43], andsingle human-multiple robots interaction [26].

The main scientific challenge of human-robotteam interaction is to fuse the cognitive capabili-ties of the human and the autonomous capabilitiesof the robot team, while maximizing task perfor-mance and intuitiveness of the interaction. Thisleads to the consideration of suitable levels of au-tonomy, control sharing and human cognitive andbehavioral aspects in the interaction design.The aim of this article is to provide a survey onthe existing literature on human-robot team inter-action with the special focus on its control sharingaspects. The overview of the article structure is

Preprint submitted to Annual Reviews in Control July 15, 2017

Section 3. Interaction

Human

Section 2.

Section 3.Interface

Command

Feedback

Sharedcontrol

Section 4.

Robotteam

Section 2.

Figure 1: Article overview in a block structure.

provided with the block scheme in Figure 1. Wereview relevant literature considering important re-search challenges for each component of the sharedcontrol loop for human-robot team interaction.

2. Robot team and human aspects in the in-teraction

In this section we briefly review modeling andcontrol approaches for robot teams and human be-havior modeling.

2.1. Modeling and control of robot teams

This subsection focuses on the modeling and con-trol concepts for robot teams, which are suitableand/or used in human-robot team interaction. Ex-tensive reviews on multi-robot systems in generalare, among many [33], [39], [47] and [79].In this article the term robot team refers to a multi-robot system which cooperates to perform a globaltask. Robot teams can be a set of mobile manipula-tors [108], wheeled robots [27], UAVs [56]. A swarmis considered as a type of robot team which con-tains relatively large number of ”simple” and ho-mogeneous robots. There are also heterogeneouscooperative multi-robot systems, which we here in-clude under the term robot teams as well.Depending on the coupling between the individ-ual robots, robot teams can be uncoupled, looselycoupled or tightly coupled (e.g. through physicalconstraints) systems. Uncoupled and loosely cou-pled robot teams are modeled as a set of differen-tial equations describing the models of individualrobots. Most frequently used models are:

• Kinematic (single integrator) model [111]:

xi = ui i = 1, ..., N, (1)

where xi ∈ Rn is the pose of the i-th robot, ui ∈Rn its control input, and N the number of robots.

• Point mass (double integrator) model [71]:

xi =1

miui, (2)

where mi is the mass of the ith robot.

• Euler-Lagrange model [81]:

Mi(qi)qi + c(qi, qi) + gi(qi) = τ i, (3)

where qi ∈ Rn is the vector of generalized co-ordinates, M i(qi) ∈ Rn×n is the inertia matrix,c(xi, xi) ∈ Rn the vector of Coriolis and cen-trifugal forces, gi(xi) ∈ Rn the vector of gravita-tional forces, and τ i ∈ Rn is the vector of controltorques.

• State-space model:

xi = f i(xi,ui) (4)

where f i ∈ Rn is a smooth vector field represent-ing the dynamics of the robot.

For tightly coupled robot teams it is necessary tomodel the physical interactions between the in-dividual robots, for example see [37]. Togetherwith continuous states, a discrete state, termed asrole [79], can be assigned to each robot in the team.The role can refer to a set of responsabilities or ca-pabilities a robot has within the team [116], andis particularly relevant for heterogeneous teams.Roles can also determine to what extent the indi-vidual robots are capable of making decisions. Ex-amples are leader and follower roles, where a leaderdoes not use information of other robots to make adecision, while a follower considers the informationof other robots to make its decision.

2.1.1. Control of robot teams

Control architectures for robot teams largely de-pend on the way in which robots interact to achieveteam behaviors. In this context, it is possible to dis-tinguish between centralized and distributed controlapproaches. Centralized control architectures com-mand the team from a single point (e.g. throughthe robot leader). Therefore, they have a singlepoint of failure. Distributed control architecturesrun locally on the robots and communicate betweenthemselves. Achieving team behaviors is more chal-lenging in this case, but the reliability is higher. Forhuman-robot team interaction, a combination ofcentralized (through the human involvement) and

2

Subtask layer

Task layer

Action layer

Robot team

Planning layer

...

Planners Planners

Figure 2: Hierarchical control architecture for robot teams.Goal of the robot team is determined and monitored inthe task layer. Based on the goal, a set of global and localbehaviors are activated in the subtask layer. The outputs ofthis layer are control inputs for the low-level controllers ofrobots in the action layer.

distributed (between the robots) control is the mostsuitable.An illustration of a control architecture for robotteams, which can be extended to the human-robotteam interaction, is depicted with Figure 2. It ispossible to distinguish between 6 layers within thearchitecture: task, planning, subtask, action, robotteams and interaction layers. The planning layer isnot treated in this article.The knowledge about the goal (task, mission) isstored within the task layer. Often the task is de-fined as a performance function [79]:

J =

∫ T

0

L(x,α,u)dt+ V (x(T ),α(T )) (5)

where L and V are incremental and terminal costs,respectively. Continuous states of the robots, xi,are stacked in the vector x ∈ RnN , discrete states(roles) are denoted by α, the control inputs areu = γ(x,α) with γ being a smooth vector field,while T is the time horizon in which the task shouldbe accomplished.

The elementary behaviors or subtasks are storedwithin the subtask layer. We consider that globalbehaviors require information exchange betweenthe robots, and local require only the local infor-mation of a robot. Some of the global behav-iors are rendezvous, aggregation, foraging, coopera-tive manipulation, formation, coverage, inter-robotavoidance, etc. An important local behavior is ob-stacle avoidance.Rendezvous describes a behavior in which therobots meet at a common point at a commontime [79]. Foraging refers to a behavior of collectingand delivering an object. Formation refers to themaintenance of robot poses relative to each otheror to a reference [34], [35], [66]. Coverage refers tothe use of the team to cover and visit areas of anenvironment for information acquisiton [23]. Com-binations of these, elementary behaviors, can definedifferent missions. For example, flocking incorpo-rates aggregation and avoidance as a set of neces-sary behaviors [84].Global behaviors are achieved through cooperation.Coordination control approaches from the area ofmulti-agent systems are suitable for accomplishingglobal behaviors by exchanging individual state in-formation through the network of agents (robots)to reach a common agreement/consensus [85]. Forexample, in order to accomplish a rendezvous be-havior, the robots need to perform consensus onthe position. The idea behind the consensus con-trol is that each robot moves towards the weightedaverage of the states of its neighbors. Communica-tion topology of the robot team is frequently mod-eled with graphs [40]. Robustness towards topol-ogy changes and communication uncertainties suchas packet loss and time delay are important con-trol challenges. There are multiple other controlapproaches that are used for cooperation of robotteams, e.g. artifical potential functions [66], Lya-punov analysis [85], sliding mode control [46], be-havioral control [4], virtual structures [67], to namethe few.Nowadays, robot teams need to operate in dynamic,unstructured environments. Therefore, for a suc-cessfull execution of the task, multiple subtasksneed to be performed simultaneously. In order toachieve this, a behavior-based control approach issuitable [7]. It is designed by defining and weight-ing the elementary behaviors. Subtasks are defined

3

as transformations of the system states

xti = f ti(x)

xti = J ti(x)x(6)

where xti are the coordinates of the subtask tiand J ti(x) is the corresponding subtask Jacobian.Therefore, behavioral control defines global and lo-cal behaviors as subtask functions.

Example 2.1. In order to manipulate a commonobject in R2 from an initial to a final configuration,a team of robot manipulators needs to collectivelymove to a desired location, while maintaining a fixedformation. Therefore, we can define two subtaskfunctions

em(x) =1

NΣNi=1xi − xdm,

ef (x) =

(x2 − x1)T − dd12...

(xN − xN−1)− dd(N−1)N

, (7)

where xdm is the desired mean position of the robotteam and dd(i−1)i is the desired distance betweenrobots i− 1 and i.

A behavioral control approach that ensures sub-tasks are conducted according to a predefined pri-ority is termed as Null-space based behavioral con-trol [4]. A common approach is to project lowerpriority subtasks onto the null-space of the higherpriority subtask. For example, in the case of 2 sub-tasks, the control input xd would be:

xd = J†t1xt1 + (I − J†t1J t1)J†t2xt2 , (8)

where (I−J†t1J t1) is the null-space projector. How-ever, the approach is kinematic, which makes itunsuitable for the control of dynamic behaviors(e.g. when the inertia of the team cannot be ne-glected). Additionally, interaction with the envi-ronment (e.g. with objects or humans) cannot behandled appropriately.Allocation of responsibilities to the individualrobots, according to the selected subtasks, and therole they have within the team is an important stepand can be handled in various ways, see for exam-ple [49] and [117].In order to exploit human capabilities within thecontrol architecture for robot teams, it is necessaryto assign a role to the human, understand behaviorsand constraints of the human in the interaction, and

Task layer

Human

supervisor

Figure 3: Possible human roles within the robot team controlarchitecture.

how the control approaches for robot teams needto be extended towards human-robot team interac-tion.

2.2. Human behavior modeling

The human is an element of the control loop inhuman-robot team interaction, see Figure 5. Basedon the team states, delivered through a feedbackinterface, the human performs an action, mappedinto a command for the robot team through thecommand interface.In order to establish the human-robot team interac-tion, the concepts introduced in Subsection 2.1 needto be extended to the human by assigning him/heran appropriate role. We distinguish between su-pervisory and active human role [18], depicted inFigure 3.A supervisory role brings human on the loop andconsiders the interaction on a symbolic and discretelevel. The human supervisor is located on the tasklayer. Therefore, the human is aware of the overallgoal and is capable of modifying it. The respon-sibilities of the supervisor are to select global andlocal behaviors and intervene when necessary.

An active role brings human in the control loopwith the robot team and considers their interactionon a physical and continuous level [64]. Active rolecan be exhibited in the form of the human opera-tor which provides control inputs to the subtask or

4

action layers, or it can be exhibited by includinghuman in the team.Very frequently the human behavior in the inter-action is not modeled. Instead, assumptions areimposed on the human expertise in both roles andon human passivity in the active role, e.g. [104].An appropriate human model can predict underwhich conditions the human exhibits bad perfor-mance or instability, and may be beneficial in thedesign of appropriate control sharing. With theincrease of robot team capabilities, human cogni-tive models are needed. Relevant models and con-straints on the human behavior in the interactionare found within the area of cognitive psychology.These models can provide valuable insight, but needto be transformed into the form suitable for systemsand control analysis.Very often the human supervisor is modeled bya Markov model. In [110], a Markov model is ob-tained from the neurally inspired cognitive model.It predicts the human decision when choosing be-tween two global behaviors of the swarm, Deploy(D) or Rendezvous (R). The probability of transi-tion from one behavior to another is proposed as

psi→sj =csi→sj

Σst∈{D,R}csi→sj, (9)

where si, sj ∈ {D,R} are two possibilities of theteam behavior, and csi→sj is the number of tran-sitions from si to sj obtained during the trainingof the Markov model. The prediction of the nextchosen behavior is obtained with

si+1 = arg maxx∈{D,R}

psi→x. (10)

Models that can capture the dynamics of the de-cision making are termed as accumulator mod-els [91]. They are suitable for modeling the hu-man decision-making behavior in the supervisoryas well as the active role. The accumulator modelsare typically used for two-alternative forced-choicetasks (TAFCTs).The authors of [45] formulate an extended decisionfield theory (EDFT) model to represent multiple se-quential decisions in human-automation interactionwith supervisory role. The preference in two-forcedchoice tasks at sample n is proposed as:

P (n) = (1− s)P (n− 1) + sd+ ε(n) (11)

where s determines the influence of the previouspreference state, P (n − 1), d is the subjective ex-pected payoff and ε is the residual (produced by

fluctuations in attention).In human-robot team interaction this model can beused to model the reliance of the human on therobot team autonomy. The reliance is determinedby trust of the human in the autonomy and by self-confidence of the human in his/her manual control.Therefore, the two alternatives are modeled as thehuman preference for autonomous or manual con-trol.Trust in autonomy is the attitude that an agentwill help achieve an individual’s goals in a situ-ation characterized by uncertainty [65]. Overtrustand undertrust in autonomy can cause overre-liance (misuse) and underutilization (disuse), re-spectively [88]. A review on human trust in auton-omy is provided in [65]. Based on (11), trust andself-confidence are estimated as [45]

T (n) = (1− s)T (n− 1) + sdca(n) + ε(n)

SC(n) = (1− s)SC(n− 1) + sdcm(n) + ε(n),(12)

where T and SC correspond to trust and self-confidence, while dca and dcm are subjective ex-pected payoffs if the task is automated and if it ismanual, respectively. The reliance is computed asthe preference P (n) = T (n)− SC(n). Therefore, itdepends on the dynamical interaction between thetrust and the self-confidence.Another accumulator model, proposed in [107] forthe human-robot team interaction, uses a stochas-tic soft-max choice model that emerges from a drift-diffusion (DD) model. The probability that the thehuman operator will choose option A is defined asa sigmoidal function:

pA(t+ 1) =1

1 + e−µd(t)(13)

where µ represents the slope of the sigmoidal func-tion. The probability (13) can be represented witha drift-diffusion model:

dz = αdt+ σdW, z(0) = 0, (14)

where z represents the accumulated evidence in fa-vor of a candidate choice, α is a drift rate represent-ing the signal intensity of the stimulus, and σW isa Wiener process with standard deviation σ.The authors of [54] use black-box methods to iden-tify human decision-making behavior in the activerole of commanding a swarm. Obtained, lineartime-invariant (LTI) system, reveals that the hu-man decision-making process is not passive in thehigh-frequency range.

5

2.2.1. Constraints on human modeling

Important constraints that affect the human-robot team interaction are human workload and sit-uational awareness [18].The mental workload is the extent to which atask places demands on the human’s cognitive re-sources [100]. The workload increases significantlyif the human operator interacts with the individualrobots within the team [17]. This corresponds tothe interaction on the action layer. Authors of [51]propose that the maximum number of homogeneousand uncoupled robots a single human can manageis determined by the fan out (FO) expression

FO =NT

IT+ 1, (15)

where NT is the neglect time allowed and IT theinteraction time required for each robot.Workload can be reduced by increasing the au-tonomous capabilities of the robot team, and byestablishing the interaction through the subtasklayer.However, with the increase of the robot team au-tonomy, situational awareness (SA) [36] of the hu-man degrades, reducing human apprehension ofthe robot team states. It has been shown that ifthe robot team is involved in the decision-making,the situational awareness is negatively affected [87].Therefore, the higher the support from the robotteam, the greater the risk from complacency, im-paired situational awareness and skill degradation.True danger from these effects can occur when theautomation fails and the human does not react, hasa delayed response or does not have the skill to re-act properly [86]. Situational awareness can be im-proved with a suitable interface design.

Important research questions are how can the un-derstanding of the human behavior and constraintsaid in the control sharing design and to what extentcan the team be included in the decision-makingprocess without inducing negative impact on thehuman behavior. Furthermore, if the robot teamperforms multiple subtasks simultaneously, the hu-man can also exhibit multitasking behavior duringthe interaction. An additional research challengeis to use multitasking decision-making models [91],for the purpose of designing a suitable control ar-chitecture for human-robot team interaction.

3. Interaction aspects for human-robotteams

In this section we consider possible interactionparadigms between a human and a robotic team interms of levels of autonomy, allocation of respon-sibilities and handling multiple subtasks. Further-more, we provide a review of the interfaces used inthis type of interaction.

3.1. Interaction paradigms

The approach we take in reviewing types of inter-action for human-robot teams is motivated by thedegree to which the robot team can perform func-tions autonomously and by the roles the human andthe robot team undertake in the interaction. Theseroles are majorly influenced by the levels of robotautonomy [53]. The concept of levels of autonomy(LOAs) is introduced in the area of human-machineinteraction (HMI):

Definition 3.1. Levels of autonomy are a designaspect that defines which functions should be au-tonomous and which should be managed by the hu-man [101].

The early research proposes fixed number of dis-cretized levels of autonomy between no autonomyand full autonomy. For example, Sheridan proposes10 levels of automation in [101], see Table 1. Theconcept has been extended to the levels of auton-omy for each information-processing system func-tion: information acquisition, information analysis,decision and action selection, and action implemen-tation. For example, high level of autonomy is de-sirable for information acquisition and informationanalysis functions, but not for decision making asit causes human skill degradation, complacency andpoor situational awareness [89]. Therefore, it is nec-essary to allow the human to provide commandsto the robot team. An important question is towhat extent can we include the robot team in thedecision-making process.The concept of levels of autonomy has been con-sidered for human-robot team interaction as well.Each function of the robot team or each robot canhave a level of autonomy. In [21] a concept of au-tonomy spectrum is proposed for human-robot teaminteraction. An example of the autonomy spec-trum is depicted with Figure 4. It is a graphicalrepresentation of operating modes (depicted withnodes), with levels of autonomy and functions to beexecuted along vertical and horizontal graph axes,

6

1 The human executes all actions.2 The computer offers complete set of action

alternatives.3 The computer offers a selection of action

alternatives.4 The computer suggests one alternative.5 The computer executes an action au-

tonomously if the human approves.6 The computer allows the human a re-

stricted time to veto before automatic ex-ecution.

7 The computer executes an action and in-forms the human.

8 The computer executes an action and in-forms the human if asked.

9 The computer executes an action and in-forms the human if it decides to.

10 The computer executes all actions au-tonomously.

Table 1: 10 levels of autonomy by Sheridan [101].

respectively. The authors use 10 levels of auton-omy, proposed in [101]. The approach allows to de-termine several operating modes for each function,and to combine them (depicted with lines). Thismethod emphasizes the importance of having mul-tiple operating modes during the task execution.Another important property that needs to be en-sured for interaction modes is smooth and seamlesstransition, termed as sliding scale autonomy [98].In [70] a sliding autonomy approach is proposed forrobot swarms.

In [80] interaction paradigms are introduced forhuman-robot team interaction through the subtasklayer. The authors propose subtask allocation tothe human operator and the autonomous controllerof the robot team according to the available levelsof autonomy. Three interaction paradigms are pro-posed: direct, complementary and overlapping. Di-rect interaction paradigm is an interaction in whichthe human provides commands for all the subtasks.Complementary interaction paradigm is an interac-tion in which the human provides input to a setof subtasks, while the robot team executes the re-maining subtasks autonomously. Overlapping in-teraction paradigm is an interaction in which theinputs from the human and the robot team auton-omy are blended to perform a common subtask.

If we denote the set of all the subtasks that needto be conducted within a mission as S, Table 2 dif-

Information

acquisition

Information

analysisDecision

selection

Action

implementation

10

9

1

7

5

Figure 4: An example representation of the autonomy spec-trum.

ferentiates between the interaction paradigms andprovides examples. In many applications, especiallythe ones that are time-critical, it is necessary to pri-oritize subtasks. The authors of [87] propose to de-sign an automation matrix which contains weights(representing subtask importance, expected work-load and other factors) that are used to prioritizesubtasks and determine which of them should beautomated. It can be used to allocate responsibil-ity within the interaction and fuse control inputsfrom the human and the robot team.However, the priority between subtasks is, so far,determined in advance and cannot dynamicallychange during the task execution.

3.2. Interfaces for human-robot team interaction

Main challenges in the interface design forhuman-robot interaction are to decide on the suit-able command and feedback information and howto represent them appropriately. According to [18],the interface needs to ensure the human under-stands intentions and behaviors of the robots andthe environment. Furthermore, it needs to ap-propriately allocate human attention to importantevents and ensure the decision authority of the hu-man. Overall, the interface for human-automationinteraction needs to be determined by: purpose (de-gree to which the automation is used w.r.t. the de-signer intent), process (if the autonomy level is suit-able for a given situation) and performance (relia-bility, predictability and capability).

3.2.1. Command interfaces

In the supervisory role, the human typically in-teracts with the robot team via a graphical userinterface (GUI) (e.g. touch screen [14]). An actionof the human supervisor is mapped into high-levelcommands (e.g. setting goals for the robot teamor individual robots, assigning levels of autonomy,

7

Interaction Human Robot teamparadigms responsibilities responsibilities Examples

Direct S ∅ TeleoperationComplementary Sh ⊂ S Sa ⊂ S, Sh ∩ Sa = ∅ Semi-autonomy

Overlapping Sh ⊂ S Sa ⊂ S, Sh ∩ Sa 6= ∅ Mixed-initiative

Autonomy

Table 2: Properties of the interaction paradigms.

interference in the case of events, etc.). In the ac-tive role, the human provides physical commands(e.g. motion and/or force commands), typically viaa haptic device, e.g. a master robot in teleoperationscenarios [64].Human-robot team interaction is asymmetric, asthe robot team typically has more degrees of free-dom than the human. So far, the challenge of in-teracting intuitively with highly redundant systemshas been tackled for specific application examplesonly. The research output indicates that the in-teraction with the robot team through global be-haviors in the subtask layer of Figure 2 is suit-able, as it reduces the dimensionality of commandand feedback information [81]. For example, in or-der to command a formation behavior, instead ofcommanding relative distances between individualrobots, the human operator can command changeof the formation shape using the concept of virtualdeformable volumes [2], [29].From the perspective of control theory, the control-lability property of the system can aid in the inter-face design [80]. Knowing the level of system auton-omy implies which states of the robot team shouldbe controllable by the human. In order to ensurecontrollability of those states, it is necessary to pro-vide sufficient number of command channels. Thisnumber conditions the command interface that canbe used in the interaction.

3.2.2. Feedback interfaces

The feedback signal in human-robot team inter-action is typically visual. In the supervisory role,the human receives feedback via GUI and video.In [14] the authors show that if the roles of thehuman and the robot team are changing during thetask execution, the interface should provide dynam-ical feedback. In [24] authors distinguish between:GUI interfaces for visual representation, warningsystems (visual, auditory and haptic) and sugges-tion systems which propose where the attentionshould be allocated. The performance in manag-ing multiple UAVs individually proved to be the

best with suggestion systems.In the active role, the human typically receives feed-back via a haptic device. Usefulness of the hap-tic feedback in human-robot team interaction hasbeen confirmed in [83]. Analoguously to the super-visory role, the feedback of continuously changingstates should be provided to the human in dynam-ical form. This conclusion has been made throughexperimental validation for the control of multipleUAVs in [30] and [31]. In [1] the authors inves-tigate haptic human-robot team interaction withvariable formation. The haptic signal informs thehuman when the swarm is stretched, compressedor reshaped. Relative behavior of the individualrobots in the team is a useful feedback informationif robots establish multiple contacts with the envi-ronment, e.g. in cooperative manipulation tasks. Itis shown that wearable haptic devices are a suitableinterface in this case [81].In terms of the appropriate feedback, the humanoperator should be informed about the states it con-trols. In that sense, the states which are control-lable by the human should also be observable. In or-der to ensure observability of states, it is necessaryto provide sufficient number of feedback channels.This number conditions the feedback interface thatcan be used in the interaction.Due to complexity of human-robot team interac-tion, it is no longer sufficient to provide only thefeedback about the system states. It is necessary torepresent activity of the automation as well (e.g. thechange of the autonomy level) and sensitivity to fu-ture activities (e.g. warnings) [115]. The activityof automation of multiple UAVs is examined as afunction of interfaces in [106]. The authors showthat the interfaces which allow the human to selectbetween different autonomy levels reduce switchingcosts.The existing literature shows there are many indi-vidual studies on the suitable interface design forspecific examples of human-robot team interaction.However, a systematic control theoretical under-standing which models the interface as a mapping

8

from human action to command signals and fromsensor signals to presented information is still miss-ing.

4. Shared control for human-robot team col-laboration

Human-robot team interaction represents a col-laboration between heterogeneous entities. There-fore, the responsibilities over the task conductionare shared, which is accomplished with shared con-trol approaches. Therefore, we say that the sharedcontrol allows to determine the level of collaborationbetween the human and the robot team. It includesall the control methods between manual and fullyautonomous control.In this section, existing shared control concepts arereviewed. First, we review how the responsibil-ities among the human and the robot team canbe shared. Afterwards, the control theoretical ap-proaches, found in the literature, are summarized.Shared control concepts from a broader area ofhuman-robot interaction are considered if they aredeemed as suitable for human-robot team interac-tion.

4.1. Collaboration in human-robot team interaction

Collaboration refers to the interaction in whichinformation, resources and responsabilities areshared among participants. In [21] the authors dis-tinguish between control by behavior (human in-teracts with each each robot in the team individ-ually) and control by policy (human interacts withthe complete robot team). In the context of Fig-ure 2, control by behavior referes to the control oflocal behaviors by the human, while control by pol-icy referes to the control of global behaviors by thehuman. In the control by policy the local behaviorsare executed autonomosly. Both control approachesare evaluated in [50]. The authors show that controlby policy is more suitable with increasing numberof agents. In the case of control by behavior lim-ited interaction intervals can cause inefficient inter-action and, in the worst case, failures.Interaction of the human with individual robots(control by behavior) limits the number of robotswithin the team. Naturally, it imposes the great-est workload and time-related stress on the humanoperator [75], [114]. Furthermore, the complexityof the interaction is of order N , O(N). In order

to obtain complexity O(1), it is necessary to de-sign autonomous cooperative controller. The re-duction of complexity can be achieved with the in-teraction between the human and a single robot,termed as leader. In this way the human workload isconsiderably reduced [99]. However, human-robotteam interaction is not achieved in this way as thehuman does not need to understand and act in re-sponse to the complete team behavior. Such sys-tems heavily rely on the autonomy of the completeteam, which the human typically cannot manage inthe case of unpredictable situations.If the human controls the complete robot team,its team dynamics is managed (control by pol-icy). This approach enables the human to operateon higher levels of abstraction [25]. However, thehuman is a point of failure of such a system [13].The solution to this problem is an interaction witha subset of robots in the team [54].The results in [11] confirm that humans are capableof perceiving a robot team as a single unit. In [13]human collaborates with a robot team at higherlevels of abstraction, termed as attractors. Theyrepresent states of the overall team dynamics (orits subset). One can also say that attractors areprojections of robots’ states onto the lower dimen-sional state space or that they abstract from theindividual to the group behavior. They correspondto the introduced concept of global behaviors. Weprovide a simple and intuitive example of globalversus local states.

Example 4.1. The pose of the robot team can rep-resent one of the team overall states. If poses of Nrobots are xi ∈ R2, i = 1, ..., N , the team pose canbe their mean value

x =1

NΣNi=1xi. (16)

The dimensionality of the overall behavior is re-duced from 2N to 2 and the control objective forthis team can be limt→∞ x(t)→ xh(t), where xh(t)is the trajectory commanded by the human operator.

Authors of [13] additionally impose the stabil-ity requirement on the global behaviors. Therefore,the overall team dynamics is stable. This is an im-portant requirement for the shared control designsince the human does not need to stabilize the sys-tem. Furthermore, the control design on the levelof global behaviors is easier in terms of dimension-ality [58].

9

4.2. Shared control approaches

In this subsection control approaches for human-robot team interaction are reviewed. Typically,human-robot team interaction is remote and con-siderable research output treats the problem ofshared control design for teleoperation scenarios.We classify the methods into the ones suitable forthe complementary interaction paradigm and theones suitable for overlapping interaction paradigm.

4.2.1. Shared control for the complementary inter-action paradigm

Typically, the human is in charge of control-ling a global subtask of the overall team behav-ior. The robot team is in charge of cooperatingautonomously [103]. Additionaly, the robots withinthe team perform a local subtask of collision avoid-ance, see e.g. [43] and [92]. However, the authorsdo not consider that the desired commands for dif-ferent behaviors may be in conflict. This causesunpredictable behaviors of the robot team. Suchsituations can be resolved by decoupling the dy-namics of the overall robot team into the dynamicsfor the required behaviors. It can be achieved byensuring the autonomous task is uncontrollable tothe human, see e.g. [103] and [80].Since robot teams are inherently redundant, theycan perform multiple subtasks simultaneously. Inorder to avoid conflicts of control inputs, a null-space based behavioral control, introduced in Sub-section 2.1 can be applied to define, decouple andprioritize multiple subtasks. Using double integra-tor model of the robots within the team, the au-thors of [69] assign the responsibility over a set ofglobal behaviors of the team to the human operator.Those are termed as teleoperated tasks and includecommanding the position of the robot team meanpose (16) and its variance

fv(q) =1

NΣNi=1(xi − fm(q))2, (17)

where q ∈ RnN is the stacked vector of generalizedcoordinates of the robot team.The subtasks performed by the robot team are dis-persion, avoidance of obstacles and the other mem-bers of the team. Within the subtask layer, desiredcontrol inputs for the low-level controllers withinthe action layer are computed according to

qd = J+s xs + (InN − J+

s Js)ψs, (18)

where Js is the partial derivative of one of the tele-operated subtasks, while ψs is the sum of partial

derivatives of the autonomous subtasks. In this wayit is ensured that the teleoperated and autonomoussubtasks do not intefere if there are sufficient de-grees of freedom. If they interfere, the teleoper-ated subtasks are of a higher priority over the au-tonomous subtasks. However, the authors do notprioritize autonomous subtasks. Additionally, thepriority is fixed which is not suitable in the caseof unpredictable events, nor is it possible to allo-cate teleoperated tasks to the robot team and viceversa.The possibility that the robot team interacts andcollides with the environment is not treated exten-sively in the literature. However, cooperative ma-nipulation by the robot team in teleoperation sce-narios provides some insights into the appropriatecontrol approaches. In [64] energetic passivity isenforced via passivity-based control. Therefore, thepassivity of the system when interacting with a pas-sive environment in guaranteed. Another approachuses impedance control to ensure passivity in inter-action with environement, see e.g. [3] and [81].The reviewed approaches are suitable for the activehuman role. In the supervisory role, the humantypically behaves as a switch. There are also ap-proaches in which the human performs both roles.For example, in [43] the human selects the mode ofinteraction while the team of UAVs autonomouslycontrols its variable topology. The choice can bemade between the global intervention (steering thecentroid of the formation to the goal) or the ocalintervention (steering a single UAV). In [54] thehuman switches manually between two two con-trollers: the control of the robot team position andthe control of the robot team velocity, and providesthe input commands to the chosen controller.The drawback of the reviewed approaches is thatthe subtask distribution is constant during the taskexecution and the level of the robot team autonomyis fixed. This is problematic as it reduces the flex-ibility of the interaction. Furthermore, if multiplesubtasks need to be executed, their prioritization isof the fixed order. It would be beneficial to be ableto dynamically change the priorities of the subtasksaccording to the stage of the task execution and toallocate the responsibilities of the subtasks online.

Due to developments of robot autonomy, the per-formance of the robot team does not necessarelyimprove with the persistent human command ifthe robot team knows the goal. However, the hu-man command is suitable in the open-ended mis-sions [18]. Therefore, mixed-initiative control ap-

10

proaches that allow the human to be part of thecontrol loop when necessary and to leave the loopwhen desired, can be suitable in the future.

4.2.2. Overlapping interaction paradigm

So far, we reviewed human-robot team interac-tion in which robot team autonomy complementshuman capabilities. However, another aspect ofshared control can be exhibited in the form of assis-tance where the human and the robot team jointlyperform a common task. Shared control approachesthat establish an overlapping interaction paradigmare termed as mixed-initiative. The obtained con-trol commands are a synthesis of the human inputand the autonomy input. The block structure ofthe control loop is depicted with Figure 5.

There is little work done on human-robot teammixed-initiative control. One of the examples isthe work of Chipalkatty et al. in [20]. The au-thors use model predictive control (MPC) to es-tablish a mixed-initiative control of the helicopterrobot team, teleoperated by the human operator.The autonomous controller has a built in planner,i.e. the robot team is capable to reach a target au-tonomously. The human can inject input for theoverall robot team behavior and, in this way, in-terfere with the input from the autonomous con-troller. The coordination between the robots withinthe team is handled autonomously. The stability ofthe global behavior commanded by the human op-erator is not guaranteed. Instead, it is assumed thatthe human is capable of stabilizing the correspond-ing states. Furthermore, the authors disregard thefeedback effect of the MPC approach, i.e. the con-troller predicts the team behavior and the humaninput on which the human reacts in return. There-fore, stability issues can occur when prediction ofthe human behavior is autonomous. In this ap-proach it is not necessary to specify levels of au-tonomy and the MPC approach is suitable only forsufficiently regular human inputs.Potential mixed-initiative control approaches canbe found in the broader area of human-robot in-teraction. Typically, they are obtained by blendinghuman control inputs and autonomy control inputs.In the remainder we review blending approachesthat can be suitable for human-robot team inter-action.

Blending mechanisms. Let us denote the humancontrol input as uh(t) and the autonomy control

input as ua(t). Their blending is typically linear:

u(t) = Khuh(t) +Kaua(t), Kh +Ka = I, (19)

where u(t) is the resulting control command, Kh

and Ka are arbitration matrices, and I is an iden-tity matrix. In general, Ka quantifies the level ofautonomy, and Kh the level of collaboration be-tween the human and the autonomy of the robotteam.There are many ways in which arbitration matri-ces can be selected. The simple approach is to as-sign fixed and constant values to the matrices [38].In [112], the author proposes a probabilistic sharedcontrol and proves that linear blending is its spe-cial case. Furthermore, in this paper shared controlis defined as a joint optimization between agreabil-ity, safety and efficiency of the interaction. Authorsprove that linear blending can generate unsafe shar-ing with safe human and safe autonomous input.The shared control input is determined as

u(t) = fR∗

(h,fR,f)∗ = arg maxh,fR,f

p(h,fR,f |zh1:t, zR1:t, zf1:t)

(20)where h,fR,f are human, robot, and dynamic ob-stacle trajectories, and zh1:t, z

R1:t, z

f1:t their corre-

sponding measurements. The probabilistic sharedcontrol is determined as

p(h,fR,f |z1:t) = ψ(h,fR)p(h|zh1:t)p(fR,f |z1:t)

(21)where ψ(h,fR) = exp(− 1

2γ (h − fR)(h − fR)T ) isthe interaction function between the human andthe robot with the coupling factor γ. Dynami-cal prediction function of the human behavior isp(h|zh1:t). The dynamical prediction of the auton-omy is p(fR,f |z1:t).Arbitration of the human input and the robot teaminput can be achieved using game-theoretical ap-proaches, see e.g. [68]. Recently, the arbitrationbased on the estimation of human trust and self-confidence has been validated. In [96] a mixed-initiative bilateral teleoperation is proposed for thecontrol of a single mobile robot. It uses trust mod-els to scale the manual and the autonomous con-trol inputs with a human-to-robot trust and toscale feedback with a robot-to-human trust. Apassivity-based controller successfully manages thetime-varying scales and communication time de-lays.The arbitration can also be applied on the param-eters of the low-level controller of the action layer.

11

Robot teamAutonomous

Operatoruh

ua

u

y

yh

ya

Allocation

controller

Figure 5: Block scheme of the control loop for the human-robot team interaction in a cooperative manipulation task.

For example, in order to obtain safe and intuitiveassistance, an approach to the allocation of con-trol authority is achieved using a human-inspireddecision-making model (13). The authors of [48]treat the problem of shared control of a mobileassistive robot (MAR) by solving simultaneously3 low level sub-tasks: follow a path, avoid colli-sions and mitigate human fatigue. For each subtaska drift-diffusion decision-making model is used forgain scheduling of the low-level control parametersof admittance or impedance controller:

c(t) = pA(t+ 1)¯c+ (1− pA(t+ 1))c (22)

where¯c and c represent the upper and lower bounds

of the control parameter c. However, the authorsdisregard the problem of subtask conflicts, sincethey do not consider their interference in exper-imental evaluation. A similar approach is usedin [22] for a single task and bilateral teleopera-tion scenario. Experimental results show that thedecision-making models have potential for intuitivemixed-initiative interaction.In the mixed initiave shared control, the humanmodeling is important, since it is necessary to de-termine the most appropriate autonomous controlinput based on the human behavior to accomplish asatisfactory assistance. In [32] the authors proposethe linear arbitration with constant selection matri-ces, and with the autonomous control input that isbased on the prediction of the human intent. Thisapproach adapts to the robot confidence in itself,to the user confidence and the user type. It usesmachine learning for the estimation.

We can conclude that tasks should be decom-posed into multiple subtasks represented by stableglobal and local behaviors. Furthermore, these sub-

Goal

Subtask generation and prioritization

Subtask allocationHumanAutonomy(planner)

Low-level controllers

Robot team

EnvironmentUnpredictable

events

Interactionparadigms

Self-confidence

Trust LOAs

Sliding scaleautonomy

Figure 6: Block structure of the general hierarchicalshared control architecture for human-robot team interac-tion. Based on a desired goal of the interaction and the en-vironment state subtasks are generated and prioritized. Al-location of subtasks to the human and the robot team isdynamical and determined depending on the available lev-els of autonomy, current self-confidence of the human andits trust in automation. Low-level controllers receive desiredcontrol inputs either from human or from the built-in robotteam planners.

tasks should be prioritized according to the currentstate of the environment and allocated dynamicallyto the human and/or the autonomous controller ofthe robot team as depicted in Figure 6. Dynamicaldistribution of subtasks among the human and therobot team is termed as trading control [55], [59].As far as the authors know, this has not been doneso far for human-robot team interaction.

12

5. Conclusion

Control sharing in human-robot team interactionis summarized in this article through the effectsthat each component of the closed-loop, given withFigure 1, has on the overall performance. There-fore, we reviewed human, robot team, interface andcontrol aspects of the interaction. Following conclu-sions can be made based on the reviewed literature:

• With the increase of autonomous capabilitiesof robots, role of humans in the interaction isnot reduced. On the contrary, human gainsmore high-level responsibilities. Therefore, itis important to include human in the controlloop as a decision-making dynamical system.

• Autonomous capabilities of robot teams are de-scribed by the autonomy spectrum, which in-cludes levels of autonomy assigned to each sub-task that the robot team can perform. Combi-nations of levels of autonomy define interactionparadigms between the human and the robotteam. Paradigms represent the design aspectsof the interaction and indicate shared controlrequirements.

• The human should interact with the robotteam on the subtask level by managing itsglobal behaviors. In this way, the high dimen-sional robot team state-space is projected ontothe lower-dimensional space of the global be-havior for which the control design is easier.

• Interface in human-robot team interactionshould provide intuitive mappings to resolvethe inherent asymmetry.

• Mixed-initiative shared control approaches en-able both human and robot team to make deci-sions and can benefit from the human behaviormodeling.

For a better overview, reviewed literature is sortedin Table 3 with respect to the elements of the sharedcontrol loop and the taxonomy for the human-robotteam interaction.

6. Future work

The review of the available literature indicatesthe need to perceive the human as a team member.There are a number of research challenges withinthe area of human-robot team interaction; we high-light some of them:

• Models of the human cognitive process are nec-essary within the control theoretical context.Therefore, decision-making dynamical modelsfrom cognitive psychology might provide a use-ful construct to tackle the problem of hetero-geneity.

• Developing a control architecture which tunesthe assistance of the robot team based on themonitored workload and situational awarenesscan enable human and robots to function as ateam. Therefore, mixed-initiative shared con-trol is a promising control concept for furtherresearch.

• Another important aspect is how to effectivelyand appropriately choose suitable level of au-tonomy. A lot of potential lies in approachesthat optimize level of autonomy with respectto the human confidence in performing certaintask, i.e. by modeling trust in automation andhuman self-confidence.

• A major challenge is to design appropriate one-to-many mappings between the human and therobot team from the control theoretical per-spective in order to be able to formally ana-lyze interaction properties during the task ex-ecution.

• Robot teams as redundant systems can per-form multiple subtasks simultaneously. A ma-jor challenge is to design, prioritize and dis-tribute the subtasks among the human andthe robot team autonomy dynamically. Thislargely depends on the state of the environ-ment and the available levels of autonomy.

• If multiple subtasks are considered, the con-trol loop needs to handle multitasking sit-uations from the human and control per-spective. Therefore, incorporating multitask-ing decision-making models into control loopwould be a challenging research goal.

Overall, sophisticated shared control strategiesthat rely on mixed-initiative interaction, multitask-ing capabilities, dynamical prioritization and dis-tribution of subtasks should bring us closer to apeer-to-peer human-robot team interaction.

7. References

[1] C. K. Alder, S. J. McDonald, M. B. Colton, M. A.Goodrich, Toward haptic-based management of small

13

Topic References

Supervisory role and human-on-the-loop [26], [45], [17], [91], [95], [75]Active role and human-in-the-loop [64], [73], [43], [104] [54], [96], [105], [62], [18],

[69], [71], [92], [103], [60], [61], [42], [22]Human modeling (Subsection 2.2) [17], [51] [36], [65], [24], [26], [44], [45], [93],

[107], [86], [91], [54], [15], [14], [97], [110]

Robot team control (Subsection 2.1)[102], [79], [47], [108], [27],[56], [76], [34], [77],[66], [109], [85]

Robot team surveys (Subsection 2.1)[79], [39], [33], [16], [47], [5]

Interaction paradigms (Subsection 3.1) [101], [86], [89], [95], [6], [28], [8], [21], [113],[80], [106] [115] [41]

Interfaces (Subsection 3.2) [17], [51], [14], [29], [1], [2], [31], [30], [52], [83],[105], [99], [106], [82], [9]

Adjustable control (Section 4) [43], [19], [54] [78]Mixed-initiative control (Section 4) [74], [53], [6], [12], [96], [48], [20], [114], [94],

[14], [59], [22], [68], [32], [112], [72] [10]

Table 3: An overview of surveyed literature for control sharing in human-robot team interaction.

swarms in cordon and patrol, in: Swarm/HumanBlended Intelligence Workshop (SHBI), 2015, IEEE,1–8, 2015.

[2] J. Alonso-Mora, S. H. Lohaus, P. Leemann, R. Sieg-wart, P. Beardsley, Gesture based human - Multi-robotswarm interaction and its application to an interactivedisplay, in: 2015 IEEE International Conference onRobotics and Automation (ICRA), ISSN 1050-4729,5948–5953, doi:10.1109/ICRA.2015.7140033, 2015.

[3] M. Angerer, S. Music, S. Hirche, Port-Hamiltonianmodel based control for human-robot team interaction,in: Proceedings of 2017 IEEE International Confer-ence on Robotics and Automation (ICRA), 2017.

[4] G. Antonelli, F. Arrichiello, S. Chiaverini,The null-space-based behavioral control forautonomous robotic systems, Intelligent Ser-vice Robotics 1 (1) (2008) 27–39, ISSN 1861-2784, doi:10.1007/s11370-007-0002-3, URLhttp://dx.doi.org/10.1007/s11370-007-0002-3.

[5] T. Arai, E. Pagello, L. E. Parker, Editorial: Advancesin multi-robot systems, IEEE Transactions on roboticsand automation 18 (5) (2002) 655–661.

[6] M. Baker, H. A. Yanco, Autonomy mode suggestionsfor improving human-robot interaction., in: SMC (3),2948–2953, 2004.

[7] T. Balch, R. C. Arkin, Behavior-based formationcontrol for multirobot teams, IEEE Transactions onrobotics and automation 14 (6) (1998) 926–939.

[8] J. Beer, A. D. Fisk, W. A. Rogers, Toward a frameworkfor levels of robot autonomy in human-robot interac-tion, Journal of Human-Robot Interaction 3 (2) (2014)74.

[9] H. Boessenkool, D. A. Abbink, C. J. Heemskerk, F. C.van der Helm, J. G. Wildenbeest, A task-specific anal-ysis of the benefit of haptic shared control during tele-manipulation, IEEE Transactions on Haptics 6 (1)(2013) 2–12.

[10] J. Brookshire, S. Singh, R. Simmons, Preliminaryresults in sliding autonomy for assembly by coor-

dinated teams, in: Intelligent Robots and Systems,2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ In-ternational Conference on, vol. 1, IEEE, 706–711,2004.

[11] D. S. Brown, M. A. Goodrich, S.-Y. Jung, S. C. Ker-man, Two Invariants of Human Swarm Interaction,Journal of Human-Robot Interaction 5 (1) (2015) 1–31.

[12] D. S. Brown, S. Y. Jung, M. A. Goodrich, Balanc-ing human and inter-agent influences for shared con-trol of bio-inspired collectives, in: 2014 IEEE In-ternational Conference on Systems, Man, and Cy-bernetics (SMC), ISSN 1062-922X, 4123–4128, doi:10.1109/SMC.2014.6974582, 2014.

[13] D. S. Brown, S. C. Kerman, M. A. Goodrich, Human-swarm Interactions Based on Managing Attractors,in: Proceedings of the 2014 ACM/IEEE Interna-tional Conference on Human-robot Interaction, HRI’14, ACM, New York, NY, USA, ISBN 978-1-4503-2658-2, 90–97, doi:10.1145/2559636.2559661, URLhttp://doi.acm.org/10.1145/2559636.2559661,2014.

[14] D. J. Bruemmer, D. A. Few, R. L. Boring, J. L. Mar-ble, M. C. Walton, C. W. Nielsen, Shared understand-ing for collaborative control, IEEE Transactions onSystems, Man, and Cybernetics-Part A: Systems andHumans 35 (4) (2005) 494–504.

[15] M. Cao, A. Stewart, N. E. Leonard, Integrating hu-man and robot decision-making dynamics with feed-back: Models and convergence analysis, in: Decisionand Control, 2008. CDC 2008. 47th IEEE Conferenceon, IEEE, 1127–1132, 2008.

[16] Y. U. Cao, A. S. Fukunaga, A. Kahng, Coopera-tive mobile robotics: Antecedents and directions, Au-tonomous robots 4 (1) (1997) 7–27.

[17] J. Y. Chen, M. J. Barnes, M. Harper-Sciarini, Super-visory control of multiple robots: Human-performanceissues and user-interface design, IEEE Transactions onSystems, Man, and Cybernetics, Part C (Applications

14

and Reviews) 41 (4) (2011) 435–454.[18] J. Y. C. Chen, M. J. Barnes, Human Agent Teaming

for Multirobot Control: A Review of Human FactorsIssues, IEEE Transactions on Human-Machine Sys-tems 44 (1) (2014) 13–29.

[19] M. Chiou, R. Stolkin, G. Bieksaite, N. Hawes, K. L.Shapiro, T. S. Harrison, Experimental analysis of avariable autonomy framework for controlling a re-motely operating mobile robot, in: Intelligent Robotsand Systems (IROS), 2016 IEEE/RSJ InternationalConference on, IEEE, 3581–3588, 2016.

[20] R. Chipalkatty, G. Droge, M. B. Egerstedt, Less ismore: Mixed-initiative model-predictive control withhuman inputs, IEEE Transactions on Robotics 29 (3)(2013) 695–703.

[21] G. Coppin, F. Legras, Autonomy spectrum and perfor-mance perception issues in swarm supervisory control,Proceedings of the IEEE 100 (3) (2012) 590–603.

[22] J. Corredor, J. Sofrony, A. Peer, Decision-makingmodel for adaptive impedance control of teleoperationsystems, IEEE Transactions on Haptics .

[23] J. Cortes, S. Martinez, T. Karatas, F. Bullo, Coveragecontrol for mobile sensing networks, IEEE Transac-tions on robotics and Automation 20 (2) (2004) 243–255.

[24] J. W. Crandall, M. L. Cummings, M. Della Penna,P. M. de Jong, Computing the effects of opera-tor attention allocation in human control of mul-tiple robots, IEEE Transactions on Systems, Man,and Cybernetics-Part A: Systems and Humans 41 (3)(2011) 385–397.

[25] J. W. Crandall, M. A. Goodrich, D. R. Olsen, C. W.Nielsen, Validating human-robot interaction schemesin multitasking environments, IEEE Transactions onSystems, Man, and Cybernetics - Part A: Systems andHumans 35 (4) (2005) 438–449, ISSN 1083-4427, doi:10.1109/TSMCA.2005.850587.

[26] M. L. Cummings, J. P. How, A. Whitten, O. Toupet,The impact of human–automation collaboration in de-centralized multiple unmanned vehicle control, Pro-ceedings of the IEEE 100 (3) (2012) 660–671.

[27] J. P. Desai, J. P. Ostrowski, V. Kumar, Modeling andcontrol of formations of nonholonomic mobile robots,IEEE transactions on Robotics and Automation 17 (6)(2001) 905–908.

[28] M. Desai, H. A. Yanco, Blending human and robotinputs for sliding scale autonomy, in: ROMAN 2005.IEEE International Workshop on Robot and HumanInteractive Communication, 2005., IEEE, 537–542,2005.

[29] M. Diana, J. P. de la Croix, M. Egerstedt, Deformable-medium affordances for interacting with multi-robotsystems, in: 2013 IEEE/RSJ International Conferenceon Intelligent Robots and Systems, ISSN 2153-0858,5252–5257, doi:10.1109/IROS.2013.6697116, 2013.

[30] B. Donmez, M. L. Cummings, H. D. Graham, Audi-tory decision aiding in supervisory control of multipleunmanned aerial vehicles, Human factors 51 (5) (2009)718–729.

[31] B. Donmez, H. Graham, M. Cummings, Assessing theimpact of haptic peripheral displays for UAV opera-tors, Tech. Rep., DTIC Document, 2008.

[32] A. D. Dragan, S. S. Srinivasa, A policy-blending for-malism for shared control, The International Journalof Robotics Research 32 (7) (2013) 790–805.

[33] G. Dudek, M. Jenkin, E. Milios, A taxonomy of multi-robot systems, Robot teams: From diversity to poly-morphism (2002) 3–22.

[34] W. B. Dunbar, R. M. Murray, Distributed recedinghorizon control for multi-vehicle formation stabiliza-tion, Automatica 42 (4) (2006) 549–558.

[35] M. Egerstedt, X. Hu, Formation constrained multi-agent control, IEEE transactions on robotics and au-tomation 17 (6) (2001) 947–951.

[36] M. R. Endsley, Toward a theory of situation aware-ness in dynamic systems, Human Factors: The Journalof the Human Factors and Ergonomics Society 37 (1)(1995) 32–64.

[37] S. Erhart, S. Hirche, Internal force analysis and loaddistribution for cooperative multi-robot manipulation,IEEE Transactions on Robotics 31 (5) (2015) 1238–1243.

[38] P. Evrard, A. Kheddar, Homotopy-based controller forphysical human-robot interaction, in: RO-MAN 2009-The 18th IEEE International Symposium on Robotand Human Interactive Communication, IEEE, 1–6,2009.

[39] A. Farinelli, L. Iocchi, D. Nardi, Multirobot systems: aclassification focused on coordination, IEEE Transac-tions on Systems, Man, and Cybernetics, Part B (Cy-bernetics) 34 (5) (2004) 2015–2028, ISSN 1083-4419,doi:10.1109/TSMCB.2004.832155.

[40] J. A. Fax, R. M. Murray, Information flow and cooper-ative control of vehicle formations, IEEE transactionson automatic control 49 (9) (2004) 1465–1476.

[41] D. Feth, B. A. Tran, R. Groten, A. Peer, M. Buss,Shared-control paradigms in multi-operator-single-robot teleoperation, in: Human Centered Robot Sys-tems, Springer, 53–62, 2009.

[42] A. Franchi, P. R. Giordano, C. Secchi, H. I. Son,H. H. Bulthoff, A passivity-based decentralized ap-proach for the bilateral teleoperation of a group ofUAVs with switching topology, in: Robotics and Au-tomation (ICRA), 2011 IEEE International Confer-ence on, IEEE, 898–905, 2011.

[43] A. Franchi, C. Secchi, M. Ryll, H. H. Bulthoff, P. R.Giordano, Shared control: Balancing autonomy andhuman assistance with a group of quadrotor UAVs,IEEE Robotics & Automation Magazine 19 (3) (2012)57–68.

[44] F. Gao, A. S. Clare, J. C. Macbeth, M. Cummings,Modeling the impact of operator trust on performancein multiple robot control, AAAI, 2013.

[45] J. Gao, J. D. Lee, Extending the decision field theoryto model operators’ reliance on automation in super-visory control situations, IEEE Transactions on Sys-tems, Man, and Cybernetics-Part A: Systems and Hu-mans 36 (5) (2006) 943–959.

[46] V. Gazi, Swarm aggregations using artificial poten-tials and sliding-mode control, IEEE Transactions onRobotics 21 (6) (2005) 1208–1214, ISSN 1552-3098,doi:10.1109/TRO.2005.853487.

[47] V. Gazi, B. Fidan, Coordination and Control of Multi-agent Dynamic Systems: Models and Approaches,Springer Berlin Heidelberg, Berlin, Heidelberg, ISBN978-3-540-71541-2, 71–102, 2007.

[48] M. Geravand, C. Werner, K. Hauer, A. Peer, An Inte-grated Decision Making Approach for Adaptive SharedControl of Mobility Assistance Robots, InternationalJournal of Social Robotics (2016) 1–18.

15

[49] B. P. Gerkey, M. J. Mataric, A formal analysis andtaxonomy of task allocation in multi-robot systems,The International Journal of Robotics Research 23 (9)(2004) 939–954.

[50] M. A. Goodrich, T. W. McLain, J. D. Anderson,J. Sun, J. W. Crandall, Managing Autonomy inRobot Teams: Observations from Four Experiments,in: Proceedings of the ACM/IEEE InternationalConference on Human-robot Interaction, HRI ’07,ACM, New York, NY, USA, ISBN 978-1-59593-617-2, 25–32, doi:10.1145/1228716.1228721, URLhttp://doi.acm.org/10.1145/1228716.1228721,2007.

[51] M. A. Goodrich, M. Quigley, K. Cosenzo, Task Switch-ing and Multi-Robot Teams, Springer Netherlands,Dordrecht, ISBN 978-1-4020-3389-6, 185–195, 2005.

[52] W. B. Griffin, W. R. Provancher, M. R. Cutkosky,Feedback strategies for telemanipulation with sharedcontrol of object handling forces, Presence: Teleopera-tors and Virtual Environments 14 (6) (2005) 720–731.

[53] B. Hardin, M. A. Goodrich, On Using Mixed-initiative Control: A Perspective for ManagingLarge-scale Robotic Teams, in: Proceedings ofthe 4th ACM/IEEE International Conferenceon Human Robot Interaction, HRI ’09, ACM,New York, NY, USA, ISBN 978-1-60558-404-1, 165–172, doi:10.1145/1514095.1514126, URLhttp://doi.acm.org/10.1145/1514095.1514126,2009.

[54] T. Hatanaka, N. Chopra, M. Fujita, Passivity-basedbilateral human-swarm-interactions for cooperativerobotic networks and human passivity analysis, in:2015 54th IEEE Conference on Decision and Control(CDC), IEEE, 1033–1039, 2015.

[55] S. Hayati, S. Venkataraman, Design and implementa-tion of a robot control system with traded and sharedcontrol capability, in: Robotics and Automation, 1989.Proceedings., 1989 IEEE International Conference on,IEEE, 1310–1315, 1989.

[56] Y. Jin, A. A. Minai, M. M. Polycarpou, Cooperativereal-time search and task allocation in UAV teams, in:Decision and Control, 2003. Proceedings. 42nd IEEEConference on, vol. 1, IEEE, 7–12, 2003.

[57] H. Kitano, S. Tadokoro, I. Noda, H. Matsubara,T. Takahashi, A. Shinjou, S. Shimada, RoboCup Res-cue: search and rescue in large-scale disasters as a do-main for autonomous agents research, in: Systems,Man, and Cybernetics, 1999. IEEE SMC ’99 Con-ference Proceedings. 1999 IEEE International Confer-ence on, vol. 6, ISSN 1062-922X, 739–743 vol.6, doi:10.1109/ICSMC.1999.816643, 1999.

[58] A. Kolling, P. Walker, N. Chakraborty, K. Sycara,M. Lewis, Human Interaction With Robot Swarms: ASurvey, IEEE Transactions on Human-Machine Sys-tems 46 (1) (2016) 9–26.

[59] D. Kortenkamp, R. P. Bonasso, D. Ryan, D. Schreck-enghost, Traded control with autonomous robots asmixed initiative interaction, in: AAAI Symposium onMixed Initiative Interaction, 89–94, 1997.

[60] D. Lee, Semi-autonomous teleoperation of multi-ple wheeled mobile robots over the internet, in:ASME 2008 Dynamic Systems and Control Confer-ence, American Society of Mechanical Engineers, 147–154, 2008.

[61] D. Lee, Passive decomposition and control of non-

holonomic mechanical systems, IEEE Transactions onRobotics 26 (6) (2010) 978–992.

[62] D. Lee, A. Franchi, H. I. Son, C. Ha, H. H. Bulthoff,P. R. Giordano, Semiautonomous haptic teleopera-tion control architecture of multiple unmanned aerialvehicles, IEEE/ASME Transactions on Mechatronics18 (4) (2013) 1334–1345.

[63] D. Lee, M. W. Spong, Bilateral teleoperation of mul-tiple cooperative robots over delayed communicationnetworks: application, in: Proceedings of the 2005IEEE International Conference on Robotics and Au-tomation, IEEE, 360–365, 2005.

[64] D. Lee, M. W. Spong, Bilateral teleoperation of mul-tiple cooperative robots over delayed communicationnetworks: theory, in: Proceedings of the 2005 IEEE In-ternational Conference on Robotics and Automation,IEEE, 360–365, 2005.

[65] J. D. Lee, K. A. See, Trust in automation: Designingfor appropriate reliance, Human Factors: The Journalof the Human Factors and Ergonomics Society 46 (1)(2004) 50–80.

[66] N. E. Leonard, E. Fiorelli, Virtual leaders, artificialpotentials and coordinated control of groups, in: Deci-sion and Control, 2001. Proceedings of the 40th IEEEConference on, vol. 3, IEEE, 2968–2973, 2001.

[67] M. A. Lewis, K.-H. Tan, High precision formationcontrol of mobile robots using virtual structures, Au-tonomous robots 4 (4) (1997) 387–403.

[68] Y. Li, K. P. Tee, W. L. Chan, R. Yan, Y. Chua, D. K.Limbu, Role adaptation of human and robot in collab-orative tasks, in: Robotics and Automation (ICRA),2015 IEEE International Conference on, IEEE, 5602–5607, 2015.

[69] C.-W. Lin, M.-H. Khong, Y.-C. Liu, Experiments onHuman-in-the-Loop Coordination for Multirobot Sys-tem With Task Abstraction, IEEE Transactions onAutomation Science and Engineering 12 (3) (2015)981–989.

[70] L. Lin, M. A. Goodrich, Sliding Autonomy for UAVPath-Planning: Adding New Dimensions to Auton-omy Management, in: Proceedings of the 2015 In-ternational Conference on Autonomous Agents andMultiagent Systems, AAMAS ’15, International Foun-dation for Autonomous Agents and Multiagent Sys-tems, Richland, SC, ISBN 978-1-4503-3413-6, 1615–1624, 2015.

[71] Y.-C. Liu, Task-space coordination control of bilateralhuman–swarm systems, Journal of the Franklin Insti-tute 352 (1) (2015) 311–331.

[72] S. G. Loizou, V. Kumar, Mixed initiative control ofautonomous vehicles, in: Robotics and Automation,2007 IEEE International Conference on, IEEE, 1431–1436, 2007.

[73] P. Malysz, S. Sirouspour, Trilateral teleoperation con-trol of kinematically redundant robotic manipula-tors, The International Journal of Robotics Research30 (13) (2011) 1643–1664.

[74] J. L. Marble, D. J. Bruemmer, D. A. Few, Lessonslearned from usability tests with a collaborative cog-nitive workspace for human-robot teams, in: Systems,Man and Cybernetics, 2003. IEEE International Con-ference on, vol. 1, IEEE, 448–453, 2003.

[75] S. Mau, J. Dolan, Scheduling for humans in mul-tirobot supervisory control, in: 2007 IEEE/RSJInternational Conference on Intelligent Robots

16

and Systems, ISSN 2153-0858, 1637–1643, doi:10.1109/IROS.2007.4399340, 2007.

[76] P. McDowell, J. Chen, B. Bourgeois, UUV teams, con-trol from a biological perspective, in: OCEANS’02MTS/IEEE, vol. 1, IEEE, 331–337, 2002.

[77] N. Michael, M. M. Zavlanos, V. Kumar, G. J. Pap-pas, Distributed multi-robot task assignment and for-mation control, in: Robotics and Automation, 2008.ICRA 2008. IEEE International Conference on, IEEE,128–133, 2008.

[78] C. A. Miller, H. Funk, R. Goldman, J. Meisner, P. Wu,Implications of adaptive vs. adaptable UIs on decisionmaking: Why automated adaptiveness is not alwaysthe right answer, in: Proceedings of the 1st inter-national conference on augmented cognition, 22–27,2005.

[79] R. M. Murray, Recent research in cooperative controlof multivehicle systems, Journal of Dynamic Systems,Measurement, and Control 129 (5) (2007) 571–583.

[80] S. Music, S. Hirche, Classification of human-robotteam interaction paradigms, in: 1st IFAC Conferenceon Cyber-Physical & Human-Systems, 2016, 2016.

[81] S. Musi, G. Salvietti, P. Bude gen. Dohmann,F. Chinello, D. Prattichizzo, S. Hirche, Human-Multi-Robot Teleoperation for Cooperative ManipulationTasks using Wearable Haptic Devices, in: IEEE/RSJInternational Conference on Intelligent Robots andSystems, accepted, 2017.

[82] Y. Nevatia, T. Stoyanov, R. Rathnam, M. Pfin-gsthorn, S. Markov, R. Ambrus, A. Birk, Aug-mented autonomy: Improving human-robot team per-formance in Urban search and rescue, in: 2008IEEE/RSJ International Conference on IntelligentRobots and Systems, ISSN 2153-0858, 2103–2108, doi:10.1109/IROS.2008.4651034, 2008.

[83] S. Nunnally, P. Walker, M. Lewis, N. Chakraborty,K. Sycara, Using haptic feedback in human roboticswarms interaction, in: Proceedings of the HumanFactors and Ergonomics Society Annual Meeting,vol. 57, SAGE Publications Sage CA: Los Angeles, CA,1047–1051, 2013.

[84] R. Olfati-Saber, Flocking for multi-agent dynamic sys-tems: Algorithms and theory, IEEE Transactions onautomatic control 51 (3) (2006) 401–420.

[85] R. Olfati-Saber, R. M. Murray, Consensus problems innetworks of agents with switching topology and time-delays, IEEE Transactions on automatic control 49 (9)(2004) 1520–1533.

[86] L. Onnasch, C. D. Wickens, H. Li, D. Manzey, Humanperformance consequences of stages and levels of au-tomation: An integrated meta-analysis, Human Fac-tors 56 (3) (2014) 476–488.

[87] R. Parasuraman, M. Barnes, K. Cosenzo, S. Mulgund,Adaptive automation for human-robot teaming in fu-ture command and control systems, Tech. Rep., DTICDocument, 2007.

[88] R. Parasuraman, V. Riley, Humans and automation:Use, misuse, disuse, abuse, Human Factors: The Jour-nal of the Human Factors and Ergonomics Society39 (2) (1997) 230–253.

[89] R. Parasuraman, T. B. Sheridan, C. D. Wickens, Amodel for types and levels of human interaction withautomation, IEEE Transactions on Systems, Man, andCybernetics - Part A: Systems and Humans 30 (3)(2000) 286–297, ISSN 1083-4427.

[90] L. E. Parker, Multiple mobile robot systems, in:Springer Handbook of Robotics, Springer, 921–941,2008.

[91] J. R. Peters, V. Srivastava, G. S. Taylor, A. Surana,M. P. Eckstein, F. Bullo, Human supervisory control ofrobotic teams: integrating cognitive modeling with en-gineering design, IEEE Control Systems 35 (6) (2015)57–80.

[92] E. J. Rodrıguez-Seda, J. J. Troy, C. A. Erignac,P. Murray, D. M. Stipanovic, M. W. Spong, Bilateralteleoperation of multiple mobile agents: Coordinatedmotion and collision avoidance, IEEE Transactions onControl Systems Technology 18 (4) (2010) 984–992.

[93] R. M. Roe, J. R. Busemeyer, J. T. Townsend, Mul-tialternative decision field theory: A dynamic connec-tionst model of decision making., Psychological review108 (2) (2001) 370.

[94] E. M. Roth, M. L. Hanson, C. Hopkins, V. Man-cuso, G. L. Zacharias, Human in the loop evaluationof a mixed-initiative system for planning and controlof multiple UAV teams, in: Proceedings of the Hu-man Factors and Ergonomics Society Annual Meeting,vol. 48, SAGE Publications Sage CA: Los Angeles, CA,280–284, 2004.

[95] H. A. Ruff, S. Narayanan, M. H. Draper, Human in-teraction with levels of automation and decision-aid fi-delity in the supervisory control of multiple simulatedunmanned air vehicles, Presence: Teleoperators andvirtual environments 11 (4) (2002) 335–351.

[96] H. Saeidi, F. McLane, B. Sadrfaidpour, E. Sand, S. Fu,J. Rodriguez, J. Wagner, Y. Wang, Trust-based mixed-initiative teleoperation of mobile robots, in: AmericanControl Conference (ACC), 2016, IEEE, 6177–6182,2016.

[97] D. Schuster, S. Ososky, F. Jentsch, E. Phillips,C. Lebiere, W. A. Evans, A research approach toshared mental models and situation assessment in fu-ture robot teams, in: Proceedings of the Human Fac-tors and Ergonomics Society Annual Meeting, vol. 55,SAGE Publications Sage CA: Los Angeles, CA, 456–460, 2011.

[98] B. Sellner, F. W. Heger, L. M. Hiatt, R. Simmons,S. Singh, Coordinated Multiagent Teams and SlidingAutonomy for Large-Scale Assembly, Proceedings ofthe IEEE 94 (7) (2006) 1425–1444, ISSN 0018-9219,doi:10.1109/JPROC.2006.876966.

[99] T. Setter, H. Kawashima, M. Egerstedt, Team-levelproperties for haptic human-swarm interactions, in:2015 American Control Conference (ACC), ISSN 0743-1619, 453–458, doi:10.1109/ACC.2015.7170777, 2015.

[100] T. Sheridan, H. Stassen, Definitions, models andmeasures of human workload, in: Mental Workload,Springer, 219–233, 1979.

[101] T. B. Sheridan, W. L. Verplank, Human and computercontrol of undersea teleoperators, Tech. Rep., DTICDocument, 1978.

[102] D. H. Shim, H. J. Kim, S. Sastry, Decentral-ized nonlinear model predictive control of multi-ple flying robots, in: 42nd IEEE InternationalConference on Decision and Control (IEEE Cat.No.03CH37475), vol. 4, ISSN 0191-2216, 3621–3626vol.4, doi:10.1109/CDC.2003.1271710, 2003.

[103] D. Sieber, S. Music, S. Hirche, Multi-robot manipu-lation controlled by a human with haptic feedback,in: Intelligent Robots and Systems (IROS), 2015

17

IEEE/RSJ International Conference on, IEEE, 2440–2446, 2015.

[104] S. Sirouspour, Modeling and control of cooperativeteleoperation systems, IEEE Transactions on Robotics21 (6) (2005) 1220–1225.

[105] H. I. Son, A. Franchi, L. L. Chuang, J. Kim, H. H.Bulthoff, P. R. Giordano, Human-centered design andevaluation of haptic cueing for teleoperation of mul-tiple mobile robots, IEEE transactions on cybernetics43 (2) (2013) 597–609.

[106] P. Squire, G. Trafton, R. Parasuraman, Human con-trol of multiple unmanned vehicles: effects of interfacetype on execution and task switching times, in: Pro-ceedings of the 1st ACM SIGCHI/SIGART conferenceon Human-robot interaction, ACM, 26–32, 2006.

[107] A. Stewart, M. Cao, A. Nedic, D. Tomlin, N. Leonard,Towards human–robot teams: Model-based analysis ofhuman decision making in two-alternative choice taskswith social feedback, Proceedings of the IEEE 100 (3)(2012) 751–775.

[108] T. G. Sugar, V. Kumar, Control of cooperating mo-bile manipulators, IEEE Transactions on robotics andautomation 18 (1) (2002) 94–103.

[109] D. Swaroop, J. Hedrick, String stability of intercon-nected systems, IEEE transactions on automatic con-trol 41 (3) (1996) 349–357.

[110] K. Sycara, C. Lebiere, Y. Pei, D. Morrison, M. Lewis,Abstraction of analytical models from cognitive mod-els of human control of robotic swarms, in: Interna-tional Conference on Cognitive Modeling, Universityof Pittsburgh, 2015.

[111] H. G. Tanner, A. Jadbabaie, G. J. Pappas, Sta-ble flocking of mobile agents, part I: fixed topol-ogy, in: 42nd IEEE International Conference onDecision and Control (IEEE Cat. No.03CH37475),vol. 2, ISSN 0191-2216, 2010–2015 Vol.2, doi:10.1109/CDC.2003.1272910, 2003.

[112] P. Trautman, Assistive planning in complex, dynamicenvironments: a probabilistic approach, in: Systems,Man, and Cybernetics (SMC), 2015 IEEE Interna-tional Conference on, IEEE, 3072–3078, 2015.

[113] P. Walker, S. Nunnally, M. Lewis, N. Chakraborty,K. Sycara, Levels of automation for human influenceof robot swarms, in: Proceedings of the Human Fac-tors and Ergonomics Society Annual Meeting, vol. 57,SAGE Publications Sage CA: Los Angeles, CA, 429–433, 2013.

[114] J. Wang, M. Lewis, Human control for cooperatingrobot teams, in: 2007 2nd ACM/IEEE InternationalConference on Human-Robot Interaction (HRI), ISSN2167-2121, 9–16, 2007.

[115] D. D. Woods, J. Tittle, M. Feil, A. Roesler, Envision-ing human-robot coordination in future operations,IEEE Transactions on Systems, Man, and Cybernet-ics, Part C (Applications and Reviews) 34 (2) (2004)210–218.

[116] H. A. Yanco, J. L. Drury, A taxonomy for human-robotinteraction, in: Proceedings of the AAAI Fall Sympo-sium on Human-Robot Interaction, 111–119, 2002.

[117] D. Zhang, G. Xie, J. Yu, L. Wang, Adaptive task as-signment for multiple mobile robots via swarm intel-ligence approach, Robotics and Autonomous Systems55 (7) (2007) 572–588.

18