Autonomous Indoor Robot Navigation UsingSketched Maps and Routes

Federico Boniardi Abhinav Valada Wolfram Burgard Gian Diego TipaldiAutonomous Intelligent Systems Group, University of Freiburg, Germany

boniardi, valada, burgard, tipaldi@informatik.uni-freiburg.de

Abstract—Hand drawn sketches are natural means by which ahigh level description of an environment can be provided. Theycan be exploited to impart coarse prior information about thescene to a robot, thereby enabling it to perform autonomousnavigation and exploration when a full metrical description ofthe scene is not available beforehand. In this paper, we presenta navigation system supplemented by a tablet interface thatallows a user to sketch a rough map of an indoor environmentand a desired trajectory for the robot to follow. We proposea novel theoretical framework for sketch interpretation basedupon the manifold formalism in which associations between thesketch and the real world are modeled as local deformation of asuitable metric manifold. We also present empirical results fromexperimental evaluations of our approach in real world scenariosboth from the perspective of the navigation capability and theusability of the interface.

I. INTRODUCTION

The design and implementation of intuitive methods ofcommunication between robots and humans has attractedconsiderable amount of attention from both the robotics andAI communities thus far [19]. In the context of robot nav-igation, significant amount of research has been devoted todevelop natural and human-friendly means for transferringspatial information from users to robots as well as to en-hance the robot’s cognition about the surrounding environment[24, 23, 6, 12]. The ability to navigate in a known environmentis a key requirement for robots to be fully autonomous.Such a capability is usually achieved employing metricallyconsistent maps, which are retrieved up front typically bymeans of human teleoperation or autonomous exploration.Although many popular methods for simultaneous localizationand mapping (SLAM) have proven to be extremely efficient aswell as accurate [5, 4], they all require preliminary operationsthat could be tediously time-consuming or sometimes evenunfeasible. Rescue scenarios, for instance, are common exam-ples where remotely controlling a robot could be impossiblefor an external operator. Furthermore, new service applicationsrequire robots to be employed even by naıve users, such asolder people or children, which would be overburdened bytedious or excessively complex operations.

To overcome these difficulties, researchers have investigatedthe use of hand-drawn maps and sketches to provide a roughdescriptions of the environment. An early attempt to performsimple navigation tasks only relying upon sketched mapswas suggested in [8]. In this research, the authors proposeda POMDP based approach to learn a metrical conversion

MapPath

S

G

Fig. 1. Top: Snapshot of the tablet interface. Figure shows the drawn mapand the path (green). The starting (S) and goal (G) positions are annotated.Bottom: The environment used for the experiments.

between a sketch, encoded as a topological map, and the realworld. More recent approaches have tackled the problem ofproviding a quantitative interpretation of a hand-drawn sketchvia landmarks’ matching, mimicking human-like navigation.Kawamura et al. [7] developed a full navigational system inwhich a robot is instructed to track a trajectory in a sketch.The robot navigates heading towards the waypoints that bestmatch the predicted scenario perceived by the robot’s sensorsand the landscape observable by the waypoints. The currentrobot pose is meanwhile tracked by triangulating the relativepositions of the predicted landmarks.

A wide and deep investigation into sketch-based navigationhas been proposed by Skubic et al. [19, 15, 16, 17, 18, 3].In their works, the authors focused on designing and testingsketch interfaces with the aim of instructing a robot, or even ateam of robots, to perform simple navigation tasks. The userswere required to sketch a map of the scene and a feasiblepath. A fuzzy state controller is then responsible for outputtingsuitable motion commands based on the qualitative state of therobot inferred from local sensor readings. The state is retrievedfrom the spatial relations between landmarks, modeled usinghistogram of forces, and later converted in a linguistic de-scription by means of fuzzy rules. Shah and Campbell [14]have proposed an extension to this approach. The authorsused techniques inspired from landmark-based SLAM to track

RO

SA

ctiv

itySketch Activity

drawing/visualize

sketchpath

current pose

execute, abort

NAVIGATOR

Prep

roce

ss

Controller

Planner

MC Localizer

/sketch

/path

/execute

/abortΩS RS ξ0

ξt

Π′t

ut

Πg

zt

/status

Fig. 2. System architecture showing the software components on the tabletand in the robot. As described in Sec III, ξt is the robot state. Πg andΠ′

t are respectively the global and local path, ut is the control and zt themeasurements. (ΩS ,RS ) is the sketched map.

uncertain landmarks and plan trajectories accordingly. Pathsare therefore encoded as a set of waypoints output by aquadratic optimizer that accounts for the mutual position of therobot and estimated landmarks. Other approaches for matchingthe sketched scene with the real world have been suggested in[11] where Particle Swarm Optimization techniques are usedto fit a hand-drawn sketch to an occupancy grid build usingthe current sensor data.

In our work, we present a theoretical framework for quan-titatively interpreting a hand-drawn sketch of an indoor en-vironment solely relying upon simple assumptions, namelytopological consistency and small deformation. For this, weemploy the Riemaniann manifold formalism and embed thesketch into a metric manifold whose metric tensor is unknown.Consequently, following [1] we estimate the metric togetherwith the current robot pose using Monte Carlo Localizationalgorithm [22]. Once the conversion between current localmetric in the sketch and the real world is known, a Dijkstrabased planner is used to plan collision free paths in closeproximity of the robot. This allows the robot to avoid un-mapped obstacles as well as overcome minor inconsistenciesin the sketch. In addition, we designed and implemented atablet interface that allows a user to sketch a map of theenvironment and a path that the robot should follow forsimple navigation and exploration tasks. A stack of plannersautonomously handle small inconsistencies in the sketch aswell as avoidance of unmapped obstacles, therefore the useris only required to provide a high level description of theenvironment.

The reminder of the paper is organized as follows. Section IIoutlines the design and core components of the tablet interface.In Section III we describe the navigation stack employed toperform the autonomous navigation tasks using hand-drawnmaps. Finally in Section IV, we present the results from ourexperimental evaluation, both in terms of the autonomousnavigation capability of the robot and from the usabilityperspective of the interface.

II. THE SKETCH INTERFACE

The sketch interface was designed to run on a tablet ora mobile phone with a stylus or a touch interface. Theoverall system architecture shown in Fig. 2, was implementedusing the Robot Operating System (ROS) framework andthe interface components were implemented on the Android

INITstart MAP PATH

MOVESTOP

sketch path

redraw redraw

exec

exec

stop

exec

stop

abort

abort

Fig. 3. A finite state machine depicting the tasks that a user can performusing the tablet interface.

operating system. ROSJava, a Java based distribution of ROSwas used in the Android application to publish and subscribeto topics to the ROS core running on the robot. The tablet andthe robot communicate through WiFi.

The tasks that a user can perform using the interface can berepresented as a finite state machine as shown in Fig. 3. Theuser is first presented with a canvas of size 2540 x 1252 pixels,in which he/she can sketch a map of the environment anddraw polygons for obstacles. The sketched map is then sent tothe robot when the user presses the Send Sketch button. Theuser then has the ability to draw the trajectory that the robotshould take in the sketched environment. It is assumed that theuser starts to draw the trajectory from the current position andorientation of the robot. There were no actual constraints setfor the path to be drawn. The user can then send the sketchedtrajectory to the navigation system by pressing the Send Pathbutton. The button is only activated and available to the userafter the map is successfully sent.

The sketched map is encoded in the robot as a grid map,while the path is stored as a set of waypoints obtained bylistening to touch events on the tablet. We interpret the initialposition of the robot as the starting point of the path and set theinitial orientation by estimating the direction of vector fromthe starting point to the next consecutive point beyond a presetthreshold distance. This was done to avoid small squiggles inthe beginning that affect the direction computation. A moredetailed description on how the sketch is interpreted duringthe navigation tasks is given in Sec. III.

During both the map and path sketching, the user has theability to redraw or erase parts of the sketch. This givesthe user a very similar experience as drawing with a penciland paper. The robot can then be instructed to navigate thesketched path by pressing the Execute button. The button isonly activated and available to the user, after the navigationalstacks on the robot have been initialized. This is notified tothe interface using the /status command. He/She can alsoabort the mission at any point of time during the execution. Afeedback message is displayed once the sketch and path aresuccessfully sent and once the task is executing or is aborted.

III. NAVIGATION IN HAND-DRAWN MAPS

In order to interpret the hand-drawn sketch from the met-rical perspective, we assume that a hand-drawn map S :=(ΩS ,RS ) is given as a rasterized image that describe aportion of plane ΩS , with a own reference frame RS . Such

map describes qualitatively a real world indoor environmentW := (ΩW ,RW ) again encoded as a rasterized image. Underthe assumption that the two images are topologically equiva-lent, we can further assume that there exists a diffeomorphismΦ : ΩW ⊂ R2 −→ ΩS ⊂ R2 that transforms pixel by pixelthe free space of the two images. As a consequence, we can de-scribe the robot trajectory (xW

t )t≥0 := ([xWt , y

Wt , θ

Wt ])t≥0 ⊂

SE(2) into the sketched world by applying the diffeomorphismΦ to the planar components of the robot poses, that is,([Φ(xW

t , yWt ), θW

t ])t≥0. From trivial differentiation rules, it isapparent that the following integral relation holds:∫ −→x S

t

−→x S0

d−→x S = TS→W

∫ −→x Wt

−→x W0

[∂Φ(xW , yW )]d−→x W , (1)

where ∂Φ : ΩW ⊂ R2 −→ R2×2 represents the Jacobianoperator of the diffeomorphism Φ and the arrow notation theplanar components of the pose. Thus, the motion of the robotin the real world can be translated into the sketch by meansof a linear scaling operator or, more formally, the sketch canbe taught as a differential (Riemaniann) manifold with metrictensor gx,y := ∂Φ(x, y)T∂Φ(x, y). Indeed, a chart for thesketch can be trivially defined via diffeomorphism by consid-ering as new global coordinate system for the manifold thestreamlines of Φ. As a consequence, owing to the fact that wecan approximate the increment dxW

t by means of the odometryreadings ut, namely dxW

t ≈ xWt+1−xW

t ≈ xWt ⊕ut−xW

t , wecan track the robot pose during the execution of a navigationtask just exploiting the metric gx,y and the operator ∂Φ.

Extending the idea presented in [1], in the rest of thissection we describe how Monte Carlo Localization [22] canbe adapt to track an approximation of the tensor metric onthe sketch together with the current pose of the robot. A finalsection outlines the routine employed to track trajectories inthe sketched map as well as how a local planning strategy canbe used to avoid collisions with unmapped obstacles lying inthe proximity of the robot.

A. Localization and Metric EstimationTo estimate the current pose of the robot while approxi-

mating the tensor metric gx,y we will henceforth assume thatthe local deformations are approximately shearing free. Moreprecisely, we assume that

gx,y ≈[a(x, y)2 0

0 b(x, y)2

], (2)

with a(x, y), b(x, y) > 0. To understand why this assumptionis reasonable, observe that, up to a flip of the diagonal terms,from Eq. 2 and using the Singular Value DecompositionTheorem as well as restricting the case of sketches thatare consistent in terms of orientation, the Jacobian of thediffeomorphism simplifies to

∂Φ =

[cosω(x, y) − sinω(x, y)sinω(x, y) cosω(x, y)

] [a(x, y) 0

0 b(x, y)

]. (3)

Therefore, under the assumption that Eq. 2 holds, Eq. 3 canbe read as the fact that the diffeomorphism applies a localdistortion (stretch or compression) and a further local rotation.

Since we can assume that people are able to perceiveorthogonality and parallelism of walls and it is a commonexperience to observe indoor environments mainly constitutedof parallel and perpendicular walls, we can suppose that asketch preserves a reasonable representation of parallel and or-thogonal features. This hypothesis can be transferred into Eq.3 by setting ω(x, y) ≡ ω ∈ [0, 2π), that is, the local rotation isindeed a global one. Owing to this, the rotational componentof the diffeomorphism can be therefore absorbed into therotational term of the transformation TS→W , provided thata suitable reference frame for the sketch has been choosen,say RS . At a high level, this models the fact that the sketchcould have been drawn with an arbitrary orientation, but still,it is inaccurate in terms of local stretching and compressionsalong the orthogonal coordinate systems of a reference frameRS . As an example, the reader could think of a building withmultiple rooms and a sketch which approximately preserve thedirections of walls but the ratio of the sizes of the rooms arenot accurate (as described in Fig. 4).

According to the above discussion, the Jacobian can befinally written as

∂Φ = R(ω)

[a(x, y) 0

0 b(x, y)

]=: [TrotS→W ]S(x, y). (4)

Since further details about how the reference frame RS iscomputed are not necessary at this point, we postpone theexplanation to the next section.

In order to track both the position of the robot and thetensor metric during a navigation task, following the ideaintroduced in [1], we employ an extended version of theMonte Carlo Localization algorithm [22]. More precisely, wedefine the enhanced robot’s state ξt := (xS

t , at, bt) whereat, bt > 0 are the local scale with respect of the current robotposition, namely a(xW

t , yWt ) and b(xW

t , yWt ). Accordingly, we

apply the standard Bayes’ filter, conditioning on the history ofcommands u1:t−1 and sensors’ measurements z1:t, obtainingthe following recursive update:

p(ξt | u1:t−1, z1:t) ∝ p(zt | ξt) ·∫p(ξt | ξt−1,ut−1) p(ξt−1 | u1:t−2, z1:t−1)dξt−1.

(5)

Standard Monte Carlo Localization approximates the statedistribution using set of weighted samples, called particles.A first propagation step is performed and the particles aredrawn according to a proposal distribution that encodes theevolution of the state with respect of the robot’s commands andthe surrounding environment. In the second step, the particlesare “resampled” with importance sampling according to theirweight, which is the likelihood of the current measurementsif the observations would have been retrieved from the poserepresented by the particle.

To select a proposal distribution, we assume that the scalesat and bt are independent one of the other as well as condi-tionally independent from the robot pose xS

t , consequently

p(ξt | ξt−1,ut) ≈p(xS

t−1 ⊕ 〈[∂Φ]t−1,ut−1〉) p(at|at−1) p(bt|bt−1)(6)

being [∂Φ]t−1 as in Eq. 4 with diagonal terms at−1, bt−1 and〈M, ·〉 : SE(2) → SE(2) the action of a matrix M on theplanar components of SE(2). According to Eq. 6, the followingmodel is a natural choice for describing the evolution of thestate:

xSt := xS

t−1 ⊕ 〈[∂Φ]t−1,ut−1 + εεεt−1〉,at := at−1γt−1,

bt := bt−1ρt−1,

(7)

where εεεt ∼ [N0,σj ]3j=1 is a random vector with independentnormally distributed components (wrapped-normal for theorientation) and γt, ρt are independent multiplicative whitenoise, namely, with distribution Γσ−2

i ,σi(i = 1, 2), the choice

of the parameters ensures unitary mean and tunable variance.As observation model for the likelihood function p(zt | ξt)

we extended the likelihood fields model for range findersdescribed in [22]. Using the metrical conversion provided by[∂Φ]t, we can convert the actual sensor readings in the sketchmetric manifold. More precisely, let zt = (zi,t)

Ni=1 encoded

as endpoints in the robot’s reference frame. We set TS→R

to be the transformation between RS and the robot’s owncoordinate system and we convert the readings endpoint in thesketch manifold by setting z′i,t := TtransS→R + St[T

rotS→R]zi,t.

Here St is the diagonal matrix defined in Eq. 4, computed withrespect of the current scales at, bt. Observe that the operationSt[T

rotS→R]zi,t is actually the projection of the readings on

the tangent space of the sketch manifold. We then define thelikelihood field for the raw sensor measurements as

p(zt | ξt) :=

N∏i=1

No′i,t,σi(z′i,t), (8)

being o′i,t the closest obstacle to z′i,t in the sketch and No′i,t,σi

Gaussian kernels. Although this model is extremely robustin standard metrical consistent maps [13], the presence ofscaling factors can increase the change of “seeing throughwalls” effect, typical of endpoint models. To overcome this, weintroduce another factor to bias the scaling factors. That is, weintroduce a virtual measurement (a′t, b

′t) for the scales obtained

by solving the following least-square approximation: set ri isthe endpoint obtained ray casting along the direction of theendpoint zi,t from the predicted robot’s pose, we compute

minA,B∈R

N∑i=1

[πa(ri −Azi,t)]2 + [πb(ri −Bzi,t)2]

, (9)

where and πa and πb are the projection along the axescentered in the robot position and aligned to the related scalingdirections. Finally, we approximate the actual likelihood of themeasurements as

p(zt | ξt) ≈ p(zt | ξt)Lλ,at(a′t)Lν,bt(b′t), (10)

where Lβ,s is a Laplace distribution with mean s. The reasonfor choosing such distribution is to not suppress excessivelythe value lying far from the virtual scales since the ray castingprocedure is unstable due to the inaccuracies of the sketch.

RS

Fig. 4. Reference frame computed using the method described in Sec. III-B.On the left a SLAM image of an indoor building computed using theCARMEN framework [20]. On the right, the lab map sketched with ourinterface.

B. Estimating the Coordinate System on the Sketch

According to Sec. III-A, we assumed that the sketch con-tains only deformation along the directions of a suitablereference frame RS . Applying the simple heuristic that wallsin indoor environments are mainly orthogonal, we can selecta coordinate system so that one of its axes is parallel to themost frequent direction of the walls and obstacles drawn inthe sketch. To identify such direction we use an approachsimilar to the one suggested in [21]. More precisely, sincea metrical consistent map ΩW can be thought as a chart forthe sketch manifold and since the both ΩS and ΩW can beembedded in R2, we can set without ambiguity the worldreference frame RW to be aligned to the pixel coordinates ofthe rasterized image ΩS . Then we preprocess the image withthe Canny’s algorithm for edge detection [2]. Finally we runthe Progressive Probabilistic Hough Transform [10] to obtaina set of direction θiNi=1 ⊂ [0, 2π) with respect of RW . Inconclusion, to select the rotation angle for TS→W we runk-means on θiNi=1 and set ω to be the mean of the biggestcluster. Such procedure identifies the direction of ω up to arotation of π, however it is easy to see that this does not affectthe behavior of the scaling factors. An example of the outputis reported in Fig. 4.

C. Trajectory Tracking and Local Planning

In order to set up a navigation system that is able to trackand execute a desired path in the sketch, we designed therobot’s controller to have three different layers, namely:• A global planner that stores a path drawn by a user.• A local planner responsible for outputting collision free

trajectories from the current robot position to the targetwaypoint on the global path. In this work, we use aDijkstra planner on the local occupancy grid defined bythe scaled readings (z′i,t)

Ni=1 (see Sec. III-A). A local

planner that computes collision free trajectories is neededas the sketch should provide a high level description ofthe indoor environment without accounting for all thepossible obstacles in the scene.

• A trajectory tracker that matches the current robot po-sition with an approximate position on the desired path,with the aim of coordinating the two planners. It is appar-ent that, due to the presence of obstacles and inaccuracies

xSt

Πg

Π′t

xSk(t)

xSk∗

Fig. 5. Avoidance of unmapped obstacles in the sketch. In blueWL(xS

t , rL). The dashed blue line represents the scaled scan (z′i,t)Ni=1.

in the sketch, only the local path is safe and consequentlyactuated by the robot. Thus the robot’s trajectory canresult in significant displacement from the desired globalpath, therefore a trajectory tracker is required.

In order to match the current position of the robot with awaypoint on the global path, we apply the strategy depictedin Fig. 5. That is, given a global path Πg := xS

k Kk=1 anda current robot pose on the sketch xS

t , we define the localwindow W (xS

t , rL) to be the set of all poses xS so that‖xS

t −xS ‖g < rL, where the norm applies only to the planarcomponents. Here rL > 0 is a parameter that can be expressedfor example as length in pixels. Consequently, we select thesubpath Π′t := WL(xS

t , rL) ∩ Πg and consider the currentposition of the robot on Πg to be the waypoint xS

k(t) that bestapproximates half of the arc length of Π′t. In general Π′t is notconnected if a user has drawn a convoluted path. However, it iseasy to discriminate which connected component of Π′t shouldbe chosen by following the ordering of the waypoints on Πg

and marking those that have already been visited.To coordinate the two planners, we select a lookahead

window WH(xSk(t), rH) depending on a parameter rH > 0

as above and define the waypoint xSk∗∈ Πg∩WH(xS

k(t), rH)

(k(t) < k∗) to be the first waypoint in the path that liesoutside the lookahead window. Finally, we plan a path in thesketch from xS

t to xSk∗

with respect of the scaled readings asdiscussed above.

IV. EXPERIMENTAL EVALUATION

Experiments were carried out in an indoor environmentbuilt using temporary walls at the University of Freiburg.Obstacles were then placed at random locations inside the testenvironment. For carrying out the experiments, we used theFesto Robotino, an ominidirectional mobile platform equippedwith a Hokuyo URG-04LX laser rangefinder. A picture of thetest environment is shown in Fig. 1. The participants were firstbriefed about the task they had to perform and were shownthe environment where the experiment was to be conducted.The participants did not have any technical knowledge on howthe system worked or had seen the environment beforehand.As we envision the target users to be common people withdifferent backgrounds, it was ensured that the participants werenot experienced robot operators.

The task for them was to sketch a map of the environmentand draw a path that they want the robot to follow in the

(g)(h)

(slam)

(c)(b)(a)

(f)(e)(d)

Fig. 6. Example sketches drawn by participants during the experiments.Some participants also sketch the obstacles. Bottom right, a map of the areaobtained using Rao-Blackwallized SLAM.

sketched map. Most of the participants were not very familiarwith drawing on a tablet so they were allowed sketch a fewtrials to get acquainted with the interface. The participantswere not specifically instructed whether they should also drawthe obstacles in the environment, this was intentional done inorder to evaluate different scenarios.

There were a total of thirteen participants and they weresplit into two groups. The first group used the tablet in thelandscape mode and the second group used the tablet in theportrait mode. We decided to conduct experiments using thetablet in different orientations because we noticed that usersfeel the urge to use the entire canvas to sketch the map,even if the proportions of the walls that they drew were verydifferent from the real environment. Interestingly, this leadto different results in either cases. The results are discussedin the following sections. At the end of the experiment, theparticipants were asked to fill out a questionnaire and were alsoasked for suggestions to incorporate more intuitiveness into theinterface. In order to maintain consistency in the evaluations,the environment was not altered in any way between eachexperiment cycle.

A. Usability Tests

There were significant variations in the sketches drawnby the participants. A few interesting examples are shownin Fig. 6. Some participants were concerned about drawingextremely straight lines for the walls but ignored drawing theobstacles (Fig 6(d), Fig 6(f)), whereas others were particularabout drawing every obstacle in the environment (Fig. 6(c),Fig. 6(g)) but did not pay attention to the relative scalesand positions of the walls (Fig. 6(a), Fig. 6(h)). Fig. 6(d)and Fig. 6(e) are some of the accurate sketches sufficientlydepicting the environment. The time that the participants spenton drawing the maps varied from 27 seconds to over 6 minutes,while the average being 2.38 ± 1.42 minutes. Though theredoes not appear to be a pattern such as, the more time youspend on sketching, the higher is the success rate for the robotto complete the task, as each person pays attention to differentparts of sketching and some spend considerable amount of

0% 20% 40% 60% 80% 100%

Sketches are intuitive for describing environments and navigation tasks

The system is sufficient to complete a navigation task

The interface is very easy to use

The tablet is easier for drawning than a pen and paper

The sketch is representative of the map

The screen is big enough for drawing the map

The sketch is easily modifiable

Using more than one color is useful to describe the task

7%

7%

15%

38%

30%

23%

38%

23%

7%

15%

53%

53%

38%

38%

30%

38%

38%

30%

30%

30%

30%

30%

38%

53%

53%

Fig. 7. Plot showing results from the post experiment survey. Stronglydisagree, Disagree, Neutral, Agree, Strongly agree.

time erasing and redrawing the map. We also noticed thatthe attitude of the participants varied from one another, assome wanted to retry the experiment when the robot failed tonavigate in their sketch, whereas some wanted to really pushthe navigation capabilities.

As mentioned in the description of the experiments, theparticipants performed experiments using the tablet in twodifferent orientations. We found that the sketches drawn bythe participants were more proportionally scaled, hence highernavigation success rate, when the tablet was used in theportrait orientation. As the screen real estate is smaller onthe horizontal direction, it prevented them from drawingdisproportionately rectangular sketches.

The questionnaire given to the participants was designed toget an insight on whether they felt at ease using the interfaceto complete the task at hand. We adopted the Likert scale[9] to rate the questions, with 5 (Strongly agree) being mostsatisfied and 1 (Strongly disagree) being the least. The surveyrevealed that using sketches to describe the environment wasvery intuitive for the participants, as they scored an average of4.09. The users reported that having a small number of stepsto perform in the interface to get the task done, the abilityto edit the sketch and having multiple colors to sketch with,were all commendable.

Although only 30% of the participants strongly agree thatthe sketch is entirely representative of the environment andis easier to sketch than on paper, their comments revealedthat this was because free-hand sketching on a tablet requiressome practice and most participants had not sketched ona tablet before. This could be improved by providing theoption of using predefined geometries for drawing. Almostno participant strongly agreed that the system is sufficient tocomplete the task, though 53% agreed. The users commentedthat this was because there was not enough feedback fromthe robot after the execute command is sent. Timely positionupdates and warnings or alerts can help provide more feedbackto the user.

B. Navigational Autonomy

The experiments described above were also used to evaluateour navigation system. Overall, thirteen experiments wereperformed with a successful rate of 69.23% (9 successfulruns of 13 sketches). The reliability of the entire system is

dramatically affected by the quality of the sketch.The parameters in the navigation stack were initially cal-

ibrated and were kept constant during the experiments. Wetuned the parameters for odometry and sensor models usingMonte Carlo Localization on metrically consistent map. Thevariances for the scales’ model were chosen trading off thecapability of adapting to the deformation of the sketch and therisk of increasing false detection. Similarly, the radius of thelocal lookahead window rH affects the way the robot tracksthe desired trajectory. If the radius is big, the robot is forcedto track the locally optimal trajectory output by the Dijkstraplanner. This results in considerable displacement from thedrawn path if it is significantly suboptimal. However, if theparameter is chosen to be too small, the planner is not able toreact quickly enough to unmapped obstacles and the safety ofthe navigation is severely affected.

All the failures occurred as a consequence of localizationerrors, in particular, it appears that the system is not robustto handle quick changes in the scales (the effect is visiblein the right border of Fig. 6(g)). Moreover, we observed thatparticipants drew sketches with different levels of details, butwe observed that a navigation task was successful independentof the amount of clutter drawn in the sketch, for instanceFig. 6(b), Fig. 6(d),Fig. 6(e) were successful, while Fig. 6(c)was not.

V. CONCLUSIONS

In this paper we addressed the problem of equipping anon-expert user with an interactive tool using which spatialinformation about the environment can be communicated tothe robot. To accomplish this, we designed and implementeda tablet interface that allows a user to sketch a map of anindoor environment and specify a desired trajectory that therobot should follow. We presented a theoretical framework thatenables the robot to localize itself with respect to the hand-drawn map, which is achieved by tracking the pose of therobot together with a local metric of the sketch. We further usethis metrical description to convert the sensor’s readings intothe sketched environment and use these virtual measurementsto perform avoidance of unmapped obstacles as well as toovercome small inconsistencies in the drawing.

We performed a usability study of our interface to determinehow practical it is to sketch a map of the environment thatsufficiently describes the real-world, in order to successfullycarry out a navigation task. We found that each user has avery different style and focus while sketching a map andthe system has to be robust to all the variations of thesketch. Nevertheless, we have shown that even in a clutteredenvironment, a minimal representation of the scene as a sketchis adequate for successfully navigating it.

ACKNOWLEDGEMENTS

This work has been supported by the European Commis-sion under contract numbers FP7–610532–SQUIRREL, FP7–610603–EUROPA2, Horizon 2020–645403–RobDREAM.

REFERENCES

[1] B. Behzadian, P. Agarwal, W. Burgard, and G. D. Tipaldi.Monte Carlo localization in hand-drawn maps. 2015.arXiv:1504.00522v1.

[2] J. Canny. A computational approach to edge detection.IEEE Transactions on Pattern Analysis and MachineIntelligence, (6):679–698, 1986.

[3] G. Chronis and M. Skubic. Sketch-based navigation formobile robots. In Proc. of the 12th IEEE InternationalConference on Fuzzy Systems, 2003. FUZZ’03, volume 1,pages 284–289. IEEE, 2003.

[4] G. Grisetti, G.D. Tipaldi, C. Stachniss, W. Burgard,and D. Nardi. Fast and accurate SLAM with rao-blackwellized particle filters. Robotics and AutonomousSystems, 55(1):30–38, 2007. URL http://ais.informatik.uni-freiburg.de/∼tipaldi/papers/grisettiRAS07.pdf.

[5] G. Grisetti, R. Kummerle, C. Stachniss, and W. Burgard.A tutorial on graph-based SLAM. IEEE IntelligentTransportation Systems Magazine, 2(4):31–43, 2010. doi:10.1109/MITS.2010.939925. URL http://ais.informatik.uni-freiburg.de/publications/papers/grisetti10titsmag.pdf.

[6] S. Hemachandra, M. R. Walter, S. Tellex, and S. Teller.Learning spatial-semantic representations from naturallanguage descriptions and scene classifications. In IEEEInternational Conference on Robotics and Automation(ICRA), pages 2623–2630. IEEE, 2014.

[7] K. Kawamura, A.B. Koku, D. M. Wilkes, R. A. Peters,and A. Sekmen. Toward egocentric navigation. Interna-tional Journal of Robotics and Automation, 17(4):135–145, 2002.

[8] S. Koenig and R. G. Simmons. Passive distance learningfor robot navigation. In ICML, pages 266–274, 1996.

[9] Rensis Likert. A technique for the measurement ofattitudes. Archives of psychology, 1932.

[10] J. Matas, C. Galambos, and J. Kittler. Robust detectionof lines using the Progressive Probabilistic Hough Trans-form. Computer Vision and Image Understanding, 78(1):119–137, 2000.

[11] G. Parekh, M. Skubic, O. Sjahputera, and J. M. Keller.Scene matching between a map and a hand drawn sketchusing spatial relations. In Proc. of the IEEE InternationalConference on Robotics and Automation (ICRA), pages4007–4012. IEEE, 2007.

[12] A. Pronobis and P Jensfelt. Large-scale semantic map-ping and reasoning with heterogeneous modalities. InIEEE International Conference on Robotics and Automa-tion (ICRA), pages 3515–3522. IEEE, 2012.

[13] J. Roewekaemper, C. Sprunk, G.D. Tipaldi, C. Stach-

niss, P. Pfaff, and W. Burgard. On the position ac-curacy of mobile robot localization based on particlefilters combined with scan matching. In Proc. of theIEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS), pages 3158–3164, Villamoura, Pro-tugal, 2012. URL http://ais.informatik.uni-freiburg.de/publications/papers/roewekaemper12iros.pdf.

[14] D. C. Shah and M. E. Campbell. A robust qualita-tive planner for mobile robot navigation using human-provided maps. In Proc. of the IEEE InternationalConference on Robotics and Automation (ICRA), pages2580–2585. IEEE, 2011.

[15] M. Skubic, C. Bailey, and G. Chronis. A Sketch interfacefor mobile robots. In Proc. of the IEEE InternationalConference on Systems, Man and Cybernetics (SMC),volume 1, pages 919–924. IEEE, 2003.

[16] M. Skubic, S. Blisard, C. Bailey, J. A Adams, andP. Matsakis. Qualitative analysis of sketched route maps:Translating a sketch into linguistic descriptions. IEEETransactions on Systems, Man, and Cybernetics, Part B:Cybernetics, 34(2):1275–1282, 2004.

[17] M. Skubic, D. Anderson, S. Blisard, D. Perzanowski,W. Adams, J. G. Trafton, and A. C. Schultz. Using asketch pad interface for interacting with a robot team. InProc. of AAAI, volume 5, pages 1739–1740, 2005.

[18] M. Skubic, D. Anderson, S. Blisard, D. Perzanowski, andA. Schultz. Using a hand-drawn sketch to control a teamof robots. Autonomous Robots, 22:399–410, 2007.

[19] Marjorie Skubic, Pascal Matsakis, Benjamin Forrester,and George Chronis. Extracting navigation states from ahand-drawn map. In Proc. of the IEEE InternationalConference on Robotics and Automation (ICRA), vol-ume 1, pages 259–264. IEEE, 2001.

[20] C. Stachniss. URL http://www2.informatik.uni-freiburg.de/∼stachnis/datasets/datasets/fr079/fr079-complete.gfs.png.

[21] S Thrun and D Dolgov. Detection of principal directionsin unknown environments for autonomous navigation.Robotics: Science and Systems, pages 73–79, 2009.

[22] S. Thrun, W. Burgard, and D. Fox. ProbabilisticRobotics. MIT Press, 2005.

[23] M. R. Walter, S. Hemachandra, B. Homberg, S. Tellex,and S. Teller. Learning semantic maps from naturallanguage descriptions. Robotics: Science and Systems,2013.

[24] H. Zender, O. Martiınez Mozos, P. Jensfelt, G-J.M. Krui-jff, and W. Burgard. Conceptual spatial representationsfor indoor mobile robots. Robotics and AutonomousSystems, 56(6):493–502, 2008.