Demonstrating the effect of computational offloading using the RobotOperating System and Lego Mindstorms EV3

Justin McManus

School of Computing ScienceSir Alwyn Williams BuildingUniversity of GlasgowG12 8QQ

Level 4 Project — 27/03/15

Abstract

The main focus of this project is to design and build a robot which may be used to demonstrate the effects ofcomputational offloading. The robot will be built using Lego Mindstorm pieces and will be controlled by aRaspberry Pi. The robots primary function is to perform face tracking, a computationally intensive task, as wellas respond to the users movements by attempting to follow them in order to keep its camera centered on them. Ifthe robots Central Processing Unit becomes too encumbered while performing the face tracking, then the robotsscheduler should dynamically attempt to offload parts of the system, thereby reducing the load on the processorand keeping the system responsive. A Graphical User Interface is used in order to demonstrate the offloading andalso to allow manual interaction with the system. The manual interaction takes the form of injecting false CPUlevels so that the user can see the effect of different CPU loads on the systems responsiveness and accuracy.

Education Use Consent

I hereby give my permission for this project to be shown to other University of Glasgow students and to bedistributed in an electronic format. Please note that you are under no obligation to sign this declaration, butdoing so would help future students.

Name: Signature:

i

List of Tables

2.1 Summary of related academic studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Table of must have requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 Table of should have requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Table of could have requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.5 Table of would have requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.6 Table of non-functional requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.1 FaceBox Message Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2 FeatureCoordinates Message Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3 SchedulerCommand Message Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.4 FaceBox Message Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.5 Image Message Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.6 AddFeatures Message Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.7 PruneFeatures Message Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.1 Zeigler-Nichols table for determining PID values . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.2 Offloading thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.1 Table of must have acceptance tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.2 Table of should have acceptance tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

A.1 Table of offloading times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

A.2 Table of CPU utilisation for subscribed nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

A.3 Table of CPU utilisation for unsubscribed nodes . . . . . . . . . . . . . . . . . . . . . . . . . . 44

A.4 Table of average CPU utilisation for entire system . . . . . . . . . . . . . . . . . . . . . . . . . 45

ii

A.5 Table of node downtimes for the system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

iii

List of Figures

2.1 An overview of simple ROS system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.1 A picture of the default tracker configuration and the Logitech C270. . . . . . . . . . . . . . . . 13

3.2 The new robot arm design inside Lego Digital Designer. . . . . . . . . . . . . . . . . . . . . . 14

3.3 Before and after of the physical robot design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4 A simple system overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.5 System UML Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.6 Initial System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.7 A hand drawn mock up of the systems GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.8 A hand drawn mock up of the systems GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.1 Activity diagram for the face tracking system . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.2 Relationship between ROS CVBridge and OpenCV . . . . . . . . . . . . . . . . . . . . . . . . 22

4.3 Basic PID Controller for the system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.4 Final GUI Design in RQt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.1 Boxplot of system offloading times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5.2 Area graph showing average CPU utilisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5.3 Box plot of system down times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.4 Sample ROS Bag output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.5 Results from the Likert Scale questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

iv

Chapter 1

Introduction

This chapter provides an introduction to the project as well as the motivation behind the project and how it relatesto the AnyScale group. The chapter also presents a general outline that the dissertation will follow.

1.1 Aim

The aim of this project is to design, build, and test a small robot using Lego Mindstorms EV3 that is capable ofperforming face tracking using an on board high definition webcam. The face tracking algorithm will be split intomultiple sections, hereby known as nodes, and these nodes will be used to demonstrate computational offloadingtriggered based primarily on the robots Central Processing Units (CPU) current load. As the load on the robotsCPU increases, its scheduler will dynamically offload nodes in order to keep the system responsive, providingthat there is an Internet connection to offload on. The robot should respond to the presence of a user by attemptingto centre itself with respect to the user by rotating or elevating its motor positions. In order to demonstrate thecomputational offloading in an intuitive and interactive way, a Graphical User Interface (GUI) will be developedthat allows the user to not only view the offloading occurring but to also interact with the system. This interactionwill take the form of artificial CPU usage levels being sent to the robots scheduler and hence forcing offloadingto occur at a time convenient to the user. Finally the system should be robust enough to cope with the possibilitythat nodes may fail, messages may be lost, or the remote server could become unresponsive.

1.2 Motivation

1.2.1 Computational offloading

Computational offloading is generally used to migrate an application or task to a more resourceful computer ornetwork of computers. Deciding what tasks may be offloaded and when to offload them can only be determinedonce the aim of the offloading has been decided. With each passing year the number of portable devices soldincreases dramatically [14] as well as the number of robots [24]. As these devices continue to offer increasingamounts of processing power and communication bandwidth, there is a larger burden placed on their batterycapacity in order to ensure they can keep up with the energy requirements of said processors, whilst still providinga good user experience. Therefore one potential aim for the offloading of tasks is to conserve energy in order toextend the battery life by reducing the CPU load but also keeping the bandwidth to the minimum [5].

1

In contrast to this there has also been a rise in cheap, low power systems on a chip such as the RaspberryPi and Arduino [11]. These computers have a plethora of potential uses ranging from being used as personalcomputers to automation systems, but due to their relatively low specifications[12][2], some applications willnot be able to meet the real time requirements of the system they are controlling as the CPU load increases.Therefore in order to alleviate the strain on the CPU, tasks can be offloaded based on their expected CPU usage,hence allowing the real time system requirements to be met providing the offloading overhead is sufficientlysmall [23].

Since offloading can occur at any time depending on the current state of the system, it can be difficult to showthe offloading occurring, which is not ideal for the educational demonstration of the process. Users are unlikelyto want to view a string of command line prompts informing them that a node has been offloaded. Therefore itwould be desirable to have a GUI capable of relaying information about the current state of the system, such aswhich nodes are active and which have been offloaded, as well as a method of manually offloading parts of thesystem.

1.2.2 The AnyScale Project

The AnyScale Project is a group consisting of members from the University of Glasgow, the University ofEdinburgh, and the University of Manchester. The group aims to work on run time adaption, resource awarenessand app migration. Their aim is to support software developers in deploying software transparently at multiplecomputing scales, from small scale devices such as the Raspberry Pi to entire cloud clusters. [1] This projectis related to AnyScale due to the their research using the Raspberry Pi cloud. Demonstrating the benefits ofcomputational offloading using the Raspberry Pi cloud is desirable because it would allow the user to scale theirapplications computation when necessary, depending on the systems current requirements. If the system requiresa large amount of computation, it migrates the more demanding parts to the cloud, thereby allowing a potentiallylarge application to be run on any device.

1.3 Outline

The remainder of this document will be divided into the following chapters:

Chapter 2 - AnalysisThis chapter will focus on the background literature of the project and the requirements gathering.

Chapter 3 - DesignThis chapter will focus primarily on the physical design of the robot and its software.

Chapter 4 - ImplementationThis chapter will focus on the how the project was implemented using ROS, EV3Dev and Python.

Chapter 5 - EvaluationThis chapter will focus primarily on the systems’ ”Chaos Monkey” and the user evaluation of the system.

Chapter 6 - ConclusionsThis chapter will focus on improvements that could be made to the system following the evaluation as well

as allowing me to reflect on my work.

2

Chapter 2

Analysis

This chapter focuses on the requirements gathering and specification of the system as well as the backgroundliterature for the project which includes: a summary of computational offloading and previous academic workrelated to the project.

2.1 Background

2.1.1 Computational Offloading

Between 1996 and 2006 the majority of research papers regarding computational offloading were focused ondetermining if it was feasible. From 2001 to 2005 the vast majority of research papers were then focusing on thealgorithms that would make the actual decision on when the application should offload, with only a few papersfocusing on the actual infrastructure required to implement it. Then as Internet speeds continued to increaseand the prevalence of small, low power devices increased alongside them, the focus was turned primarily to theinfrastructures required to implement offloading. [18]

Basic Computational Offloading

A simple offloading system consists of two sections: a client running on the less resourceful device such asa mobile phone or a robot, and a more resourceful server which may be another computer nearby or in thecloud[4][8]. In these systems it is proposed that a computation-offloading system’s client performs three primaryfunctions. Firstly, it monitors the performance of the network that the portable device is on. This is likely to be ameasure of signal strength or bandwidth. Secondly, it monitors the current CPU utilisation or execution time ofthe portable device. Finally, it uses the information it has gathered regarding the current state of the system anddecides whether or not to offload a partition of the system, in our case a node[7].

The server on the other hand has only one role to fulfil. The server simply provides an environment that allowsthe offloaded parts of the system to be executed, whilst relaying any relevant data back to parts of the system thatare currently not offloaded, i.e. executing locally. Most simple offloading systems assume that a stable networkconnection is always available however[4][8], therefore it is important to ensure that the offloading system isrobust and able to deal with the possibility of nodes becoming unresponsive or a connection failure between theclient and the server.

3

There are a number of reasons for wishing to perform computational offloading in a system, however thesecan be broadly grouped into two categories, (1) to improve the systems performance and (2) to save energy in asystem[18]. Offloading is a useful solution to the problem of meeting soft real time requirements when a systemhas limited resources. Assuming that the system consists of two main parts, we can determine the requirementsfor offloading to improve system performance. Let the execution speed of the client be sclient and let w be theamount of computation required. Therefore the time to execute that part of the program on the client is given by:

w

sclient(2.1)

If we are to offload to the server then sending the data d at bandwidth B takes:

d

B(2.2)

seconds. If we also take sserver to be the execution speed of the server, then we can determine that the offloadingwill increase performance[18] provided that:

w

sclient>

w

sserver+

d

B(2.3)

Partitioning the System

The decision on when and what to offload can either be static or dynamic. In a static offloading system, thesections of the program that are to be offloaded a generally decided during development. For example, immediateperipheral controllers would be designated as not offloadable whilst resource intensive parts of the system would.An advantage of this static partitioning is that it has low overhead since the decision on what to offload has alreadybeen made and no algorithm is required to compute this. The parameters for how to offload may be predicted ina static system and a number of methods exist to perform this, such as probabilistic prediction[26], history basedprediction[3] and fuzzy control[28].

Dynamic offloading on the other hand is able to adapt to changing run-time conditions such as fluctuatingCPU loads, network speed and battery life. A dynamic offloading system takes into account these various fac-tors and makes a decision on whether or not to offload, as well as what to offload given those circumstances.The downside of this is an increase in the offloading overhead as the system must continuously monitor theseconditions and then make an informed choice with regards to the offloading. Although it is possible to partitionthe program during execution, it is often the case that the system is partitioned during development like in astatic system. The reason for this is that dynamically partitioning the system can incur a much larger offloadingoverhead, this is due to the system needing to analyse the program to find a suitable subsection to offload. Insteada pre-execution partitioning algorithm or technique is used[4][5][23].

2.1.2 Related Academic Work

In 2002 Messer, Iragreen, Bernadat and Dejan proposed a system that could be used to offload portions of aprogram to a service. They show the benefits of using nearby, more resourceful distributed machines to offloadtasks to in order to reduce CPU utilisation and memory consumption in the local machine. The benefits of thissystem is that applications can be written more generically and for a more diverse range of devices because thelimitation of resource constraints would be lessened. [28] This relates heavily to this project and is one of theprimary goals, however for this project we aim to provide a dynamic offloading method whereas they propose astatic method for the entire application.

In 2008 Wolski presented a frame work that could determine when it would be more beneficial to moveparts of a computation to a more capable machine based on the bandwidth of both machines. They go on to

4

provide a number of statistical methods for predicting the effectiveness of the offloading for a given bandwidth,concluding that a Bayesian approach is the best performing method of doing so. [27] In our system we wouldinstead simply use the availability of WiFi to determine when offloading should or should not occur. Also in2008 Yang demonstrated the efficiency of computational offloading when used on mobile phones in order tohave them execute more demanding applications than their resources would allow. [19]

Perhaps most interestingly Nimmagadda et al presents a method of performing real time object recognitionand tracking for a small surveillance robot. In order to help the system meet its soft real time requirements, theypropose migrating the computation to nearby servers to help reduce the computation time on the less resourcefulrobots. This ties in perfectly with what we hope to achieve with this project, with the main difference being thatwe want the robot to track a users face and respond to it. Furthermore we do not need to estimate the computationtime before offloading because we are not intending on offloading the entire task.[23]

This information is summarised in table 2.1:

Year Paper Decision Method Description2002 Static [28] Using a distributed system to offload portions of a program to a service2008 Dynamic [27] Statistically predict when offloading will cause the system to outperform

local execution2008 Dynamic [19] Proposed a system that allows some tasks within a mobile application to

be offloaded in order to improve performance2010 Dynamic [23] Demonstrated that a robot can partition its computation between local

and remote machines in order to improve its performance

Table 2.1: Summary of related academic studies

2.1.3 Robot Operating System

The Robot Operating System (ROS) is a ’meta’ operating system specifically designed for robots. This meansthat it acts as a type of middleware operating between the systems actual operating system (such as Linux,Windows etc...) and the end user application.

Prior to ROS the majority of robots were created by designing not just the hardware, but also the embeddedsoftware and drivers individually. Although there would obviously be re-use of programs due to their strong linkto the underlying hardware, it would have been common for teams to reinvent the wheel and re-create a systemthat had already been made beforehand, but specifically for their hardware. Removing the need to do this is theprimary benefit of such an operating system. Another key benefit of ROS is that it is language-independent, withthree main libraries currently being used: Python, C++ and Lisp.

The underlying concept behind ROS is that the system should maintain separation of concern by runninga number of executables in parallel, with these executables sending and receiving data through a well definedcommunication method. These executables, known as Nodes in ROS, run independently from one another. Inorder for the nodes to find one another and communicate a single master node is defined by the operating systemwhich acts as a declaration and registration node.

In order to communicate back and forth, the nodes can publish data to a ”topic” or subscribe to receive datafrom a topic. Topics act as a method of providing a many to many asynchronous communication path, wheremultiple nodes may subscribe to receive the information published on a given topic. ROS also provides a methodof synchronous one to one communication known as a service, which acts as a remote function call to anothernode.

5

ROS is composed of a collection of tools, libraries, and conventions designed with the intention of beingas distributed and modular as possible in order to streamline the task of creating robots across a wide range ofplatforms. ROS encourages collaboration between the various groups that use the software and provides a richcatalogue of existing packages that can be incorporated into ones own project [10]

Figure 2.1: An overview of simple ROS system

2.2 Requirements

2.2.1 Functional Requirements

The MoSCoW method was used in order to categorise the systems functional requirements. This method dividespotential requirements into four separate categories based on the priority level of completing those requirements.The four categories in descending priority are known as:

Must Have requirements are essential for the system to be considered complete.

Should Have requirements are important but not critical in order to complete the system.

Could Have requirements are not required to be met but the system would benefit from them.

Would Have requirements will not be met during the initial release of the system but could beadded to future versions to further refine the system.

At the beginning of the project the end user provided a rough list of requirements that they felt were neededin order for the system to be considered complete. As the system evolved, these requirements were continu-ously refined and added to. Below are the functional requirements of the system in their respective MoSCoWcategories.

2.2.2 Non-Functional Requirements

6

FR Requirement Description1 Build a robot The project requires a robot to be designed and built using Lego Mind-

storms. The robot must be sturdy enough to carry on board all the nec-essary parts of the system including a webcam.

2 Motor Control The robot must be capable of moving left and right, it must also becapable of changing the height of the webcam so that the Y-axis may bemanipulated

3 Face detection The robot should perform face detection on the given input video streamusing Haar cascade classifiers to increase performance of the face detec-tion algorithm

4 Feature extraction When given a region that is known to contain a face, the system mustdetermine the optimal features on the face that could be tracked.

5 Face tracking When supplied with a set of feature points from the feature extractionalgorithm, the system must track these points as the user moves theirface around the image, the new position of the face must be updated asit moves

6 Robot Response The robot must react to the presence of a users face within its view byattempting to keep the person in question centred about the webcamscentre point. It should accomplish this in a smooth manner and in areasonable period of time

7 Computational Partitioning The parts of the system that are anticipated to be computationally inten-sive must be split into individual ”nodes” that are capable of being runin isolation from the other nodes

8 Computational Offloading The separated nodes must be capable of being executed on either theless resourceful client or the more resourceful server. Thereby allowingthe computation to be offloaded based on the current state of the system

9 System monitoring The system must monitor its current state in terms of current CPU usage,WiFi availability and battery life

10 Automatic scheduling Based on the systems current state, the system must be capable of decid-ing whether or not a node should be offloaded automatically, i.e. withoutintervention from the user, in order to maximise the execution speed ofthe system

11 Internode Communication The system must be able to pass information between the various nodesin order to allow offloaded nodes on different machines to communicatewith one another as well as with local client nodes

12 Video Stream The system must interface with the usb cam ROS package in order toprovide a video stream of what the robot can see

Table 2.2: Table of must have requirements

7

FR Requirement Description13 Manual offloading The system should allow the user to specify an artificial CPU usage level

which would thereby allow simulated offloading to occur by injectingthese values into the systems scheduler

14 GUI Image view The system should overlay graphical information on top of the videostream, it should reflect what is currently occurring in the robots facetracking

15 GUI Node view The system should provide a means of viewing which nodes are cur-rently in use and which have been offloaded already

16 GUI CPU usage Should allow the user to see the current CPU usage of each node in thesystem and reflect their idleness when offloaded

17 Local and remote launches Allows all the nodes for the respective localisation to be launched withone command

Table 2.3: Table of should have requirements

FR Requirement Description18 GUI for launching nodes Could allow the user to carry out the local and remote launching from

the offloading GUI19 Dynamic GUI Node View Highlights could be drawn on nodes being offloaded20 Recorded video input Allow a pre-recorded video to be used instead of a live stream21 Node View manipulation The Node View GUI could allow the user to click on a given node and

have it offload or unload instead of relying on the percentage slider forforcing offloading

Table 2.4: Table of could have requirements

FR Requirement Description22 Animations for offloading Allows the user to view the offloading occurring in an interactive and

intuitive way23 Multiple remote computers Allows the system to offload the computation to various remote PCs if

they are connected24 GUI for input stream Would allow the user to choose whether they want to use a video stream

or a user defined pre-recorded video on start up

Table 2.5: Table of would have requirements

8

NFR Requirement Description1 Integration with ROS The system should use the Robot Operating System in order to facillitate

message passing between nodes2 Lightweight The system should run on a Raspberry Pi Model B and above or a similar

lightweight system in order to remain portable3 Lego Ev3 Lego Mindstorms should be used to construct the robot and its motor

system4 Responsiveness The robot should respond to the users movements in a reasonable time

in order to keep them within view5 Real time The user should be able to see their movements on screen in real time

with minimum lag5 Robust The system should be robust so that a node or connection failure does

not cause a system failure6 Robot Size The entire system must fit on the Lego mindstorms chassis whilst re-

maining sturdy

Table 2.6: Table of non-functional requirements

9

Chapter 3

Design

This chapter focuses primarily on the design of the robot using Lego Mindstorms EV3 and the development ofthe systems software structure. It also describes the division of the initial face tracking algorithm into multipleROS nodes. Furthermore it details the various message and service types required by the ROS middleware toeffectively communicate with other nodes distributed in the system. Finally it details the various technologiesused to design and implement the system.

3.1 Implementation Technologies

This section will describe the technologies that will be used to implement the system

3.1.1 Lego Mindstorms EV3

Lego Mindstorms are a series of kits which contain software and hardware that allows users to create cus-tomisable, programmable robots from a set of Lego parts, motors and sensors[20]. The EV3 has the followingspecifications [21]:

1. Processor: 300 MHz TI Sitara AM1808 (ARM926EJ-S core)

2. RAM: 64 MB

3. Memory: 16 MB Flash, Optional SD card slot

4. Display: 178 x 128 pixel Monochrome LCD

5. Wifi: Optional with a dongle

The Lego Mindstorms kit is particularly useful for this project as it allows us to rapidly prototype and developsmall robot designs and has a well established community of developers. Of particular note however is EV3Devand the Lego digital designer.

10

EV3Dev

The software supplied with the EV3 Smart Brick is a graphical programming language developed by Legothemselves. In order to give ourselves more freedom, as well as maintaining a homogeneous tool chain acrossthe system, I instead have opted to utilise EV3Dev. EV3Dev is a full Debian Linux distribution available for theEV3 Smart Brick. [9] Its main benefits for this project are that it will allow us to program in the language of ourchoice and it also comes with built in libraries for interfacing with EV3 peripherals, including the various motors.Furthermore, there is currently work being done to provide a distribution of ROS that will run on EV3Dev.

Lego Digital Designer

The Lego Digital Designer is a tool developed by Lego that allows us to prototype Lego Mindstorms robotswithout needing to build them first. It provides a graphical user interface that allows us to manipulate existingdesigns, determine if motors and gears will function as expected and more.

3.1.2 Raspberry Pi

The Raspberry Pi is a small, low cost computer that caters to users of all ages with the intention of enabling themto learn how to program and understand how computers work. Since it is a complete computer it is capable ofdoing everything a normal desktop can, from browsing the Internet to playing games, but on top of this it is alsocapable of interacting with the outside world using its output pins so that it can be used for a variety of projectsranging from temperature sensors to infra-red cameras[13]. The specifications for the Raspberry Pi 2 can be seenbelow[12]:

1. Processor: 900MHz quad-core ARM Cortex-A7 CPU

2. RAM: 1GB RAM

3. Memory: Optional SD card slot

4. Display: HDMI

5. Wifi: Optional with a dongle

It is of particular interest to this project as it is compatible with the ROS operating system and also accuratelyrepresents the types of small, low cost devices we discussed in chapter one. In addition it provides us with aeasier to use and more powerful platform to test the system on compared to the EV3 Smart Brick.

3.1.3 OpenCV2

OpenCV2 is the newest implementation of the Open Source Computer Vision Library. OpenCV provides a richcatalogue of computer vision and machine learning algorithms and functions that are free to be modified andused. It is useful for this project because not only is it widely employed, has a vibrant community and detaileddocumentation, it also has optimised implementations of the computer vision methods we required in order toimplement our algorithm successfully. [17]

11

3.1.4 PSUtil

Python System and Process Utilities (PSUtil) is a library that enables us to gather information regarding processesrunning on the system as well as the current system state such as CPU utilisation, memory use, and networkavailability. It is perfect for this project as it will allow us to monitor the systems CPU usages in order to decidewhen offloading should occur. [25]

3.1.5 Qt

Qt is a cross platform framework for design application user interfaces. It provides a graphical IDE for designingand building the GUI and is able to be written in C++, Javascript, CSS and Python. [6] Of particular note is that itattempts to separate the GUI layout from the actual functionality of the code using .ui files. These files describethe layout and provide and interface for the user to create procedure callbacks, thus maintaining an MVC methodof abstracting in the system. Furthermore, ROS provides a wrapper for Qt which allows it to interface with thesystems nodes.

12

3.2 Robot Design

3.2.1 Physical Design

The initial requirements for the robots physical design from table 2.2 were that it should:

1. Be sturdy and capable of support various added weights, such as the EV3 module, Raspberry Pi and BatteryPack.

2. Utilise at least a HD 720p webcam, in particular the Logitech HD Webcam C270.

3. Be capable of moving in at least four degrees of free in motion in order to allow the webcam to centeritself on the user.

The physical design of the robot was based on one of the default Lego Mindstorms configurations known asthe ’Tracker’. This configuration of the robot had a wide sturdy base and could be moved using the caterpillartracks located on either side of the base frame. Unfortunately this version of the robot was unsuitable for anumber of the requirements as it could only rotate left or right and was unable to hold a webcam.

Figure 3.1: A picture of the default tracker configuration and the Logitech C270.

Due to the original designs wide and sturdy base, it was the perfect starting point to develop a robust robotcapable of carrying any extra weight that was added to it. Therefore using this initial design, a new prototypewas designed using the Lego Digital Designer software. Much of the original base was left intact but this newiteration featured a webcam mount and arm that allowed for a webcam with a grip to be secured tightly to theframe. The new arm allowed for the required 4 degrees of motion by using one Medium EV3 Motor to controlthe pitch of the robots webcam arm, whilst 2 Large EV3 Motors controlled its horizontal rotation.

13

Figure 3.2: The new robot arm design inside Lego Digital Designer.

Figure 3.3: Before and after of the physical robot design

14

3.3 System Layout

The system will be comprised of a local machine, and remote server and a GUI which allows the user to interactwith the local machine and get feedback from the system. The local and remote systems will be identical to oneanother, containing the same algorithm. The primary difference will be the presence of a scheduler in the localsystem. The scheduler will act as a mediator that determines when offloading occurs within the system. A simpleoverview of these interactions can be seen below.

Figure 3.4: A simple system overview

Many offloading systems are implemented in such a way that the entirety of the program is offloaded to helpreduce the systems computation load, however the purpose of this system is to allow parts of the system to bedynamically offloaded if and when required. In order to accomplish this the original face tracking algorithm firstneeded to be divided into smaller more manageable chunks that could be offloaded individually. These chunksalso required a way to interact and communicate with one another so that data could be passed between themirregardless of where they were physically located. These chunks, which are hereby referred to as nodes, couldbe easily implemented in the Robot Operating System.

3.3.1 ROS Nodes

Although ideally the entire system should be divisible and offloadable, it follows that some of the nodes must notbe offloadable in order to ensure adequate response times for the end user when using the program. In particularany nodes which interact with system peripherals such as the node responsible for reading image data fromthe usb webcam and the node responsible for controlling the systems motors. The original algorithm could besegregated into three primary sections: Image Pre-processing, Face Detector and the Lucas Kanade tracker. Inthe original algorithm, the Lucas Kanade tracker performed the post processing and drawing functions, howeverfor this system, in order to maintain separation of concerns, an extra node was needed that would aggregatean system information and display it to the user. Furthermore a node would be needed to send instructions tothe motors in order to keep the user within the robots view. Finally, the system would need a node that couldmonitor the systems current state, such as CPU usage and use this to determine when to offload, thereby sendingan offloading command to the correct node. We can therefore define our system using the following nodes:

Image Pre-processingThe image pre-processing node is responsible for taking the raw image input data supplied by the webcam drivernode and performing operations on it that make it suitable for use in the other nodes. It does this by convertingthe input image to a grey scale and equalising it to reduce the noise present in the image.

Face DetectorThe face detector node takes the grey scale output of the image pre-processing node and using a selection oftrained Harr cascade classifiers, determines whether or not a face is present within the image. If a face is detected,its location and size is passed onto the next node.

15

Lucas Kanade TrackerThe Lucas Kanade Tracker takes a given face location and using Good Features to Track, extracts features fromthe face so that they may be used to track the faces movement across future frames in the image stream. This pre-vents the need to perform the costly face detection algorithm every frame, thereby increasing the responsivenessof the system.

Motor ControllerThe motor controller is responsible for taking the users current facial position with respect to the robot and usinga Proportional Integral Differential (PID) Controller, centre the users face about the webcams centre point.

Post ProcessingThe post processing node is responsible for superimposing the face detectors and Lucas Kanade trackers infor-mation on top of the input webcam image and then outputting the finalised image to the GUIs Image View.

SchedulerThe scheduler is responsible for monitoring the current system CPU usage, its WiFi connection and also itsbattery life in order to use this information to decide when to offload nodes. It is also responsible for determiningwhether or not nodes are available or alive, as well as determining whether or not the remote server is actuallycapable of taking further computation.

Figure 3.5: System UML Diagram

3.3.2 ROS Messages

The next step is to determine how these nodes will communicate. In order to facilitate communication betweenthe various system nodes there first needs to be a defined message protocol. Each node in the system will requiredifferent information and does not need to see the data intended for other nodes. ROS messages allow data of agiven type to be published on a ’topic’ and any node that subscribes to this topic will receive data of that type.The message types used by this system are shown below:

Face BoxThis message type is used by the Face Detector node to pass information to the Lucas Kanade tracker regardingthe current position of the users face. It is comprised of four parameters: the top left x and y coordinates of theusers face, the faces height and the faces width. This information can then be used to determine a rectangularregion that can be used to extract facial features. All of the values passed in this message are unsigned 32 bitintegers.

Feature CoordinatesThis message type is used to pass information to the post processing node so that features may be superimposedonto the output image. As well as this, this message type is used to send current feature data to the Add Featuresand Prune Features servers, so that new features may be added and unsuitable ones removed. This message typeconsists of an array of 32 bit float coordinate tuples in the form (x,y).

16

FaceBox.msguint32 xuint32 yuint32 heightuint32 width

Table 3.1: FaceBox Message Definition

FeatureCoordinates.msgfloat32[2] x

Table 3.2: FeatureCoordinates Message Definition

Scheduler CommandThis message type is used to notify a given node whether it should un-subscribe itself (i.e. offload to anothermachine) or re-subscribe itself (i.e. begin its own processing again). It is listened for by any node that declaresitself as offloadable. This message is composed of a string representing a node name and a Boolean value ofun-subscribe or re-subscribe (i.e. True and False)

SchedulerCommand.msgstring node namebool offload

Table 3.3: SchedulerCommand Message Definition

Offload CommandThe offload command message type is used by the Graphical User Interface in order to inject false CPU usagelevels into the Scheduler node. It contains a Boolean type, which represents the type of offloading it is currentlyset to (i.e. Automatic or Manual) as well as a percentage which is used as the artificial CPU usage level wheninjected.

OffloadCommand.msgbool typeuint32 percentage

Table 3.4: FaceBox Message Definition

ImageThis message type is used to transfer Images between the various ROS nodes in the system. It is comprised ofthe images width and height, its encoding type, whether the data is big endian, and finally an array containing theimages data. For the systems raw input image and final output image, this data takes the form [(r,g,b)...] whilst

17

the remaining nodes use the grey scale form [(rgb)...].

Image.msguint32 heightuint32 widthstring encodinguint8 is bigendianuint32 stepuint8[] data

Table 3.5: Image Message Definition

3.3.3 ROS Services

For parts of the system that may need significant CPU time but are not required to operate at all times we are ableto use ROS services. ROS services act like remote procedure calls. These allow the system to be further dividedby separation of concern and also provide the possibility of requesting their services both locally and remotely,further reducing the CPU load of the system if required. Services are defined in a similar way to messages andare comprised of a request and a response type.

Add FeaturesAdd features is used to generate new features for use in the Lucas Kanade tracker node within a given Face Boxregion. Its request is comprised of an array of FeatureCoordinates, a FaceBox region and an input image for thealgorithm to scan. When features have been added, it responds with a new FeatureCoordinates array.

AddFeatures.srvFeatureCoordinates[] prev featuresFaceBox track boxImage ros imageFeatureCoordinates[] features

Table 3.6: AddFeatures Message Definition

Prune FeaturesThis service type takes an input set of features as well as a minimum number of features required for the systemto maintain the tracking. It responds with the new, pruned set of features and a Boolean score. If the score isFalse then there are not enough features to maintain tracking the face.

From this we general description of how the nodes function and interact with one another, we can develop a basicsystem architecture layout.

The nodes highlighted in green are those that are capable of being offloaded by the system. Nodes high-lighted with red are able not able to be offloaded. This is primarily due to them interfacing directly with local

18

PruneFeatures.srvFeatureCoordinates[] prev featuresuint32 abs min featuresFeatureCoordinates[] featuresbool score

Table 3.7: PruneFeatures Message Definition

Figure 3.6: Initial System Architecture

peripherals such as the webcam. Finally nodes which are white are actually services. These service nodes pro-vide synchronous communication to the face tracker and need to be available for communication at all times toprevent large delays in the system.

3.4 Graphical user Interface

3.4.1 Designing the GUI

One of the functional requirements for the system was to have a GUI that could relate offloading informationback to the user and also display the robots image stream. It was initially decided that the GUI should beseparated by levels of abstraction when interacting with the systems offloading. This comprised the highlyabstracted offloading controls which would be used to simply decide how much offloading was required and alsoto enable or disable automatic offloading, a medium level abstraction that allowed the user to see the systemsnode overview as the offloading occurred and finally a low level abstraction that showed the current system state.Using this information, a mock up of the potential system was created which included the robots image streamand a system terminal to provide feedback from error messages and allow the user to launch nodes.

Using this basic layout as a base to refine and improve upon, the GUI was further developed into a wire framemodel using ForeUI[22]. In order to make the GUI more user friendly, the terminal was removed and insteadsimple launch buttons were added for convenience. The node visualisation was further refined and given morespace so that it occupied the primary partition of the GUI, whilst the offloading controls were made more selfexplanatory and compact.

19

Figure 3.7: A hand drawn mock up of the systems GUI

Figure 3.8: A hand drawn mock up of the systems GUI

Page1

Offloading GUI

Webcam Stream

0% 25% 50% 100%

75%

Automatic Manual

Offloading Type:

Offloading Percentage:

N1 N2 N3 N4

Status IndicatorsLaunch Local Client

Launch Remote Server

Shutdown Nodes

20

Chapter 4

Implementation

This chapter focuses primarily on the implementation of the systems offloading functionality, its scheduler, andalso its motor control system. In addition to this it details how the motor controllers PID was tuned and how thesteps taken to ensure the system remains robust enough to handle node failure. Finally it details how a ”ChaosMonkey” was created in order to help test the systems resilience.

4.1 Breaking up the algorithm

The face detection and tracking was designed based on an existing algorithm from a ROS node known as PiVision. This algorithm ran on a single node and was implemented using python and an older version of OpenCV[15]. One of the key parts of the implementation was separating this original algorithm into the various nodes thatwere defined in the design chapter and updating the code to utilise OpenCV2. This section summarises the keypoints in each of the different nodes that were required in order to provide the original algorithms functionality.In order to gain a better picture on how this would be accomplished, I developed an activity diagram for the mainnodes required for that part of the processing which can be seen in 4.1.

From this diagram we can see how the nodes will interact with one another, as well as understanding thespinning loops that are needed to continue the process. Furthermore we can see that when a node receives thecommand to offload, its execution stops until its node is restarted and begins to listen on its topic again. Theidea behind the system is that all of the nodes are generic and can be run either locally or remotely, therebyallowing the offloading to occur without having to actually offload any of the program. Instead the locationof the computation is offloaded, allowing us to reduce the overhead cost associated with the offloading processwhilst still reducing the local CPU usage.

4.1.1 Pre-processing

When the system first begins running, the pre-processing node listens on the input image topic for any Imagemessages. When it receives an image it is a three channel colour image. In order to carry out the face trackingand detection however, the system requires a one channel grey scale image. The pre-processing node accom-plishes this by using OpenCVs cvtColor() and equalizeHist() functions, but OpenCV requires its owninput image format, therefore the helper functions convert img to cv() and convert cv to img()are needed to convert from a ROS Image type to an OpenCV cv::Mat type. These helper functions utiliseCVBridge, a ROS library that allows us to interface with OpenCV. The encoding paramter can be used to

21

Figure 4.1: Activity diagram for the face tracking system

specify the exact desired encoding type and it defaults to passthrough, which specifies to CV Bridge that itshould maintain the encoding of the original input image.

Figure 4.2: Relationship between ROS CVBridge and OpenCV

1 d e f c o n v e r t i m g t o c v ( s e l f , r o s image , e n c o d i n g =” p a s s t h r o u g h ” ) :2 t r y :3 cv image = s e l f . b r i d g e . imgmsg to cv2 ( ro s image , d e s i r e d e n c o d i n g = e n c o d i n g )4 r e t u r n cv image5 e x c e p t CvBr idgeEr ro r , e :6 p r i n t e7

8 d e f c o n v e r t c v t o i m g ( s e l f , cv image , e n c o d i n g =” p a s s t h r o u g h ” ) :9 t r y :

10 r o s i m a g e = s e l f . b r i d g e . cv2 to imgmsg ( cv image , e n c o d i n g = e n c o d i n g )11 r e t u r n r o s i m a g e

22

12 e x c e p t CvBr idgeEr ro r , e :13 p r i n t e

Listing 4.1: Conversion helper functions

The process first attempts to convert the image to the more suitable cv::Mat type, it then initialises the inter-mediary images if they do not already exist. Next cvtColor() converts the inputted three channel OpenCVimage into a one channel grey scale image by specifying the cv2.COLOR BGR2GRAY parameter. This newgrey scale image is then passed to the equalizeHist() function which helps to reduce lighting effects inthe image and results a better contrasted image for use in the face detection and tracking. Finally, the Open CVimage is transformed back into a ROS image so that it may be sent over ROS to other nodes.

1 cv image = s e l f . c o n v e r t i m g t o c v ( r o s i m a g e )2 #Code o m i t t e d f o r b r e v i t y3

4 # Conve r t t o g rey s c a l e and e q u a l i z e t h e h i s t o g r a m t o r e d u c e l i g h t i n g e f f e c t s5 s e l f . g r ey = cv2 . c v t C o l o r ( cv image , cv2 . COLOR BGR2GRAY)6 s e l f . g r ey = cv2 . e q u a l i z e H i s t ( s e l f . grey , s e l f . g r ey )7

8 p r e p r o c e s s e d i m a g e = s e l f . c o n v e r t c v t o i m g ( s e l f . g r ey )

Listing 4.2: Pre-processing grey scale conversion and equalisation

4.1.2 Face Detection

The face detection algorithm works by using Haar cascade classifiers, therefore before the node begins running,it must set find and set up these classifiers. The location of the classifiers is read from the ROS parameter server.ROS Param allows system wide variables to be set and read from at run time to help with configuration of thesystem. Hence the node first reads the location of the classifiers from a launch file before proceeding to set up theclassifiers by creating a classifier object from each type. The system can potentially use three classifiers,one profile classifier and two frontal classifiers.

1 s e l f . c a s c a d e f r o n t a l a l t = cv2 . C a s c a d e C l a s s i f i e r ( s e l f . c a s c a d e f r o n t a l a l t )2 s e l f . c a s c a d e f r o n t a l a l t 2 = cv2 . C a s c a d e C l a s s i f i e r ( s e l f . c a s c a d e f r o n t a l a l t 2 )3 s e l f . c a s c a d e p r o f i l e = cv2 . C a s c a d e C l a s s i f i e r ( s e l f . c a s c a d e p r o f i l e )

Listing 4.3: Setting up the Haar Cascade Classifiers

Before the face detection algorithm is run, the image is first resized in order to help increase execution time andreduce lag. The constant self.IMAGE SCALE is used to determine how much the original image should bereduced by for use in the Haar detector. When the convenience function self.haar detector is executed,it returns an array of potential face matches that it has found, if no faces are found then the system consecutivelyattempts to locate a face using each of the Haar classifiers until it finds at least one face.

1 s e l f . s m a l l i m a g e = cv2 . r e s i z e ( cv image , ( i n t ( im wid th / s e l f . IMAGE SCALE) , i n t ( i m h e i g h t /s e l f . IMAGE SCALE) ) , i n t e r p o l a t i o n =cv2 . INTER LINEAR )

2

3 # F i r s t check one o f t h e f r o n t a l t e m p l a t e s4 i f s e l f . c a s c a d e f r o n t a l a l t :5 f a c e s = s e l f . h a a r d e t e c t o r ( s e l f . c a s c a d e f r o n t a l a l t )6

7 # I f f a c e i s n o t found , t r y t h e p r o f i l e t e m p l a t e8 i f l e n ( f a c e s ) == 0 :9 i f s e l f . c a s c a d e p r o f i l e :

10 f a c e s = s e l f . h a a r d e t e c t o r ( s e l f . c a s c a d e p r o f i l e )11 i f l e n ( f a c e s ) == 0 :12 # I f t h a t f a i l s , check a d i f f e r e n t f r o n t a l p r o f i l e13 i f s e l f . c a s c a d e f r o n t a l a l t 2 :

23

14 f a c e s = s e l f . h a a r d e t e c t o r ( s e l f . c a s c a d e f r o n t a l a l t 2 )15

16 # I f we have found f a c e s , g e n e r a t e a FaceBox message from them17 f a c e b o x = s e l f . g e n e r a t e f a c e b o x ( f a c e s )

Listing 4.4: Resizing and checking the cascades

The function self.haar detector() is simply a convenience function that actually executes Open CVsdetectMultiScale() function on the Haar cascade object.

1 d e f h a a r d e t e c t o r ( s e l f , c l a s s i f i e r ) :2 r e t u r n c l a s s i f i e r . d e t e c t M u l t i S c a l e ( s e l f . s ma l l im age , s e l f . HAAR SCALE, s e l f . MIN NEIGHBORS

, s e l f . HAAR FLAGS, s e l f . MIN SIZE )

Listing 4.5: Haar cascade convenience function

4.1.3 Face tracking

The face tracking algorithm first checks to see whether or not there are features that are able to be tracked already.If features exist, then the cv2.calcOpticalFlowPyrLK() function is executed which tracks these featurepoints across future image frames. If however there are no features to track, but there is a track box, thenthe node will send a request to add features to the features list using the image region defined in track box.track box is a ROS message container of the type FaceBox that is used to determine where the currentlytracked face has been detected on the image. Once the new list of features is generated, those which are consid-ered to be good valid points (i.e. high status) are kept and placed into the self.features array.

1 i f s e l f . t r a c k b o x and s e l f . f e a t u r e s != [ ] :2 s e l f . f e a t u r e s , s t a t u s , t r a c k e r r o r = cv2 . c a l c O p t i c a l F l o w P y r L K ( s e l f . p r e v g r e y , s e l f . grey ,

np . a s a r r a y ( s e l f . f e a t u r e s , d t y p e =” f l o a t 3 2 ” ) , None , ∗∗ s e l f . l k p a r a m s )3 s e l f . f e a t u r e s = [ p f o r ( s t , p ) i n z i p ( s t a t u s , s e l f . f e a t u r e s ) i f s t ] # Keep on ly h igh

s t a t u s p o i n t s4 e l i f s e l f . t r a c k b o x :5 s e l f . f e a t u r e s = s e l f . a d d f e a t u r e s ( ro s image , s e l f . t r a c k b o x , s e l f . f e a t u r e s )

Listing 4.6: Tracking features in the image

Next the node checks to see if it has enough feature points to perform cv2.fitEllipse(). This method triesgenerate an ellipse of feature points from those found in self.features so that it may be fitted to the usersface. In order to do this however the function requires a minimum number of points self.MIN ELLIPSE FEATURESwhich is equal to 6. As the function requires a CV Matrix, Numpy is used to convert between the regular arrayand this format.

1 i f l e n ( s e l f . f e a t u r e s ) > s e l f . MIN ELLIPSE FEATURES :2 s e l f . f e a t u r e m a t r i x = np . f l o a t 3 2 ( [ p f o r p i n s e l f . f e a t u r e s ] ) . r e s h a p e (−1 , 1 , 2 )3 f e a t u r e b o x = cv2 . f i t E l l i p s e ( s e l f . f e a t u r e m a t r i x )4 e l s e :5 f e a t u r e b o x = None

Listing 4.7: Attempting to fit the current fits inside an ellipse

Next, the node checks to see if the current number of features is sufficient to continue the algorithm. If thisnumber falls below the threshold self.min features then the area that we are looking for the face in isincreased. This helps us account for the user getting closer to the webcam or feature points just being lost. Thecurrent feature boxs values are extracted into the track box and then the values of the faces dimensions areincreased. Once the new dimensions have been calculated, a new request is sent to have features added oncemore but inside the newly defined area. Finally, self.prune features() is executed in order to removefeatures that are now too far from the main cluster.

24

1 i f ( l e n ( s e l f . f e a t u r e s ) < s e l f . m i n f e a t u r e s ) and ( s e l f . t r a c k b o x i s n o t None ) and (f e a t u r e b o x i s n o t None ) :

2 s e l f . e x p a n d r o i = s e l f . e x p a n d r o i i n i t ∗ s e l f . e x p a n d r o i3 ( ( s e l f . t r a c k b o x . x , s e l f . t r a c k b o x . y ) , ( s e l f . t r a c k b o x . width , s e l f . t r a c k b o x . h e i g h t ) , a )

= f e a t u r e b o x4 s e l f . t r a c k b o x . wid th ∗= s e l f . e x p a n d r o i5 s e l f . t r a c k b o x . h e i g h t ∗= s e l f . e x p a n d r o i6 s e l f . f e a t u r e s = s e l f . a d d f e a t u r e s ( ro s image , s e l f . t r a c k b o x , s e l f . f e a t u r e s )7 e l s e :8 s e l f . e x p a n d r o i = s e l f . e x p a n d r o i i n i t9

10 s e l f . f e a t u r e s , s c o r e = s e l f . p r u n e f e a t u r e s ( s e l f . f e a t u r e s , s e l f . a b s m i n f e a t u r e s )

Listing 4.8: Expanding the area that we are tracking within

4.2 Launching Nodes

In ROS nodes can be dynamically renamed and can also have their variables changed at run time by using ROSLaunch. This feature allows us to create a duplicate of the system on each machine and have the node names andvariables be dynamically changed based on which machine the system is on. One of the primary benefits of thisis scalability as it allows us to launch an arbitrary number of systems, thereby allowing us to scale the system inthe future. This is achieved by using special .launch files written in XML that detail which nodes are to belaunched and what their names and variables are.

4.3 Offloading Nodes

ROS provides topics for sending and receiving messages between nodes. The process of sending a messageis known as publishing and this makes the message available to any and all nodes listening on the same topic.The process of listening for these messages is known as subscribing and one or more nodes can subscribe to atopic. In ROS, when a node receives a message on a topic, it processes the data from the message in a callbackprocedure. However when a node is no longer receiving data because it is not listening on any topics, it isdeclared ”out of scope.” This means that the node essentially shuts itself down, ceasing any further computationsince it cannot execute its callbacks without data to process. It is this functionality that allows us to performthe offloading because by removing any subscribers allows us to shut down that node and instead we can have aduplicate node on the remote system begin listening on those topics instead, thereby changing the location of thecallback execution.

4.4 Offloadable Node

Every node in the system that wishes to be offloadable must be a subclass of Offloadable FR Node. Im-plementing this class forces any offloadable node to implement their own versions of the abstract methodsunsubscribe node() and resubscribe node(). Furthermore, it causes them to all listen for SchedulerCommandsfrom the Scheduler node whilst they are running. This is accomplished by using a callback method scheduler listener()that listens on the scheduler commands topic when the node is first initialised. When a node receives amessage from this topic, it first compares the messages node name field to its own name. If the messagewas indeed intended for itself, then it proceeds to determine whether it was a command to unsubscribe thenode, or to resubscribe the node. If the command was to resubscribe, the the nodes own implementation of

25

self.resubscribe node() is called. In the code in listing ?? scheduler command is a message ob-ject containing data regarding the offloading that is to occur.

1 d e f s c h e d u l e r l i s t e n e r ( s e l f , schedule r command ) :2 i f schedule r command . node name == s e l f . node name :3 i f schedule r command . o f f l o a d == True :4 s e l f . o f f l o a d n o d e = True5 e l i f schedule r command . o f f l o a d == F a l s e :6 s e l f . o f f l o a d n o d e = F a l s e7 i f s e l f . i s o f f l o a d e d == True :8 s e l f . r e s u b s c r i b e n o d e ( ) # Must be implemented by each o f f l o a d a b l e node9 s e l f . i s o f f l o a d e d = F a l s e

Listing 4.9: Listening for scheduler commands

Each individual node needs to place the check for offload() method within its main callback method.This ensures that the offloading command is executed at the correct time, thereby allowing any important calcu-lations the node was making to be completed or any messages to be sent before executing the offloading method.

1 d e f c h e c k f o r o f f l o a d ( s e l f ) :2 i f s e l f . o f f l o a d n o d e == True and s e l f . i s o f f l o a d e d == F a l s e :3 s e l f . u n s u b s c r i b e n o d e ( ) # Must be implemented by each o f f l o a d a b l e node4 s e l f . i s o f f l o a d e d = True

Listing 4.10: Checking for an unsubscribe command

Furthermore, the offloading class is used to store the default topic names that are used. These names are thenoverridden using ROS Param at run time so that we can dynamically name nodes located on different physicalmachines.

4.4.1 Un-subscribing and Re-subscribing

In order to perform the offloading, each node is required to implement the abstract methods mentioned in theprevious section. Generally each of these methods contain a object function call to unregister(), whichforces the subscribing object, i.e. the object listening for incoming information, to delete itself. This has theeffect of the node no longer receiving messages from that specific subscriber. One of the useful things aboutROS is that when a node goes out of scope, which in other words means it no longer subscribes to anything, thenthat node shuts itself down and stops using as much CPU time. The only part of the node that remains activewhen this occurs is the scheduler listener which waits for the command to restart the node. To furtherprotect against nodes unsubscribing themselves during important operations or checking for unsubscribing whilsta node is already offloading itself, the function must first acquire the offloading lock. An example of theunsubscribing code from Image Pre Processing is shown below.

1 d e f u n s u b s c r i b e n o d e ( s e l f ) :2 t r y :3 wi th s e l f . o f f l o a d i n g l o c k :4 i f s e l f . i s o f f l o a d e d == F a l s e :5 s e l f . image sub . u n r e g i s t e r ( )6 s e l f . p r e p r o c e s s e d i m a g e p u b . u n r e g i s t e r ( )7 e x c e p t O f f l o a d i n g E r r o r , e :8 p r i n t ” Could n o t o f f l o a d node ” + s e l f . node name + ”\n ” + ”−−−−−\n ” + e

Listing 4.11: Unsubscribing a node.

If the offloading is unsuccessful, it raises the OffloadingError exception. When resubscribing a node, asimilar approach is taken. Resubscribing a node does not require the check for offload() function to becalled because the scheduler listener() itself calls it. The reason for this is that if a node is able to be

26

resubscribed, then there is currently no computation taking place on the node, therefore the process cannot beinterrupted. To resubscribe a node we simply create a ROS publisher or subscriber object and bind it tothe nodes corresponding variable for storing these objects. An example of this usingImage Pre Processingis shown below.

1 d e f r e s u b s c r i b e n o d e ( s e l f ) :2 wi th s e l f . o f f l o a d i n g l o c k :3 s e l f . image sub = r o s p y . S u b s c r i b e r ( s e l f . i n p u t r g b i m a g e , Image , s e l f . p r e p r o c e s s i n g ,

q u e u e s i z e = s e l f . q u e u e s i z e )4 s e l f . p r e p r o c e s s e d i m a g e p u b = r o s p y . P u b l i s h e r ( s e l f . p r e p r o c e s s e d o u t p u t i m a g e , Image ,

q u e u e s i z e = s e l f . q u e u e s i z e )

Listing 4.12: Resubscribing a node.

4.5 Motor Control

In the initial implementation of the system, the motor was controlled using a simple threshold. If the users facecrossed this threshold, then the robot would attempt to reposition itself slightly in order to return the face towardsthe centre of the camera. This method meant that set increments were needed and once the robot had begunchanging its facing, it could not be interrupted until it had reached its new position. A more elegant solution tothe problem of keeping the users face centred is a Proportional Integral Derivative(PID) Controller.

4.5.1 PID Controller

A PID controller is a feedback loop system, meaning that its output is fed back in to be used as part of the newinput to the system. The PID works by calculating the error between the systems current position and the actualdesired position of the system, i.e. centred in the middle of the camera. This error is calculated by taking theweighted sum of the new potential value, the new integral value and the new derivative value that were calculatedusing the previous input. In a PID, the P value represents the present error, I value represents the sum of all pasterrors, and the D value is used to predict future error. The weighted sum of the P,I, and D values are used to

Figure 4.3: Basic PID Controller for the system.

generate a value that can be used to control the motors. In order to do this however, they need to be weighted.

27

This is done by using error constants which can be tuned to produce the desired system response for a given error.In the code below we can see that the input to the system is a set of face coordinates. These coordinatesrepresent the centre of the cluster of features generated from the detected face.

1 d e f c a l c u l a t e x a x i s p i d ( s e l f ) :2 # Code o m i t t e d f o r b r e v i t y3

4 t u r n i n g p o w e r = ( s e l f . kp∗ e r r o r ) + ( s e l f . k i ∗ s e l f . i n t e g r a l ) + ( s e l f . kd∗ s e l f . d e r i v a t i v e )5 l e f t m o t o r p o w e r = s e l f . t p + t u r n i n g p o w e r # These depend on t h e i n i t a l6 r i g h t m o t o r p o w e r = s e l f . t p − t u r n i n g p o w e r # o r i e n t a t i o n o f t h e motor s7 s e l f . r i g h t t r a c k m o t o r . p u l s e s p e r s e c o n d s p = r i g h t m o t o r p o w e r8 s e l f . l e f t t r a c k m o t o r . p u l s e s p e r s e c o n d s p = l e f t m o t o r p o w e r9

10 s e l f . p r e v i o u s d t = d t11 s e l f . p r e v i o u s e r r = e r r o r

Listing 4.13: Calculating the motor power using a PID

Tuning the PID

As we can see from the code, the error constants kp, ki and kd are used to manipulate the P, I and D valuesrespectively. There are a number of methods that may be used to tune a PIDs error constants but for this systemI used the ZieglerNichols Method. The Niegler-Nichols method is a heuristic method of tuning a PID controllerthat is performed by first setting the I and D values to zero. The remaining P is then increased gradually fromzero until it reaches its peak gain ku, which is the point at when the control loop oscillates with a constantamplitude. ku and the oscillation period Tu are then used to set the P, I, and D values using table 4.1

Table 4.1: Zeigler-Nichols table for determining PID values

Control Type Kp ki kd

Classic PID 0.60*ku 2kp/tu kp*tu/8

Unfortunately during the tuning method it became apparent that when the robot rotated, it would often findfalse positives in its face tracking algorithm. This resulted in it continuously attempting to turn towards a newface, making it very difficult to determine the gain and oscillation in the system and preventing me from com-pleting the tuning correctly.

4.6 Graphical User Interface

4.6.1 RQt

RQt is a Qt[6] based framework that provides GUI tools for ROS that run as nodes. It allows a GUI to be designedusing the Qt Integrated Development Environment and then implemented using RQt as if it were part of the ROSinfrastructure. This allows the GUI to display information from topics as well as send data to nodes. In additionto this, it provides a system for adding GUI components to the screen in a modular fashion, so that they maybe rearranged to suit the users needs and screen space. This has the added benefit of allowing us to divide oursystem into the previous levels of abstraction and having each layer occupy its own module.

The offloading controls were implemented by exploiting the fact that RQt runs as a node within ROS. Thisallows us to publish and subscribe to topics like a normal node and hence have callbacks that relay informa-

28

tion to the overall system. In the case of the offloading controls, I implemented a publisher that would createoffload commands when the user selected Manual offloading and then proceeded to manipulate the slider.This forces the scheduler to offload a node. Since the scheduler works by monitoring the CPU usage, the offload-ing controls send a falsified CPU utilisation level to the scheduler in the message packet, along with a variabletelling the scheduler that this is indeed a fake CPU level.

1 i f s e l f . i s A u t o m a t i c :2 s e l f . o f f l o a d i n g p e r c e n t a g e s l i d e r . s e t E n a b l e d ( True )3 s e l f . i s A u t o m a t i c = F a l s e4 s e l f . p u b l i s h o f f l o a d c o m m a n d ( s e l f .MANUAL COMMAND, s e l f . o f f l o a d i n g p e r c e n t a g e s l i d e r .

v a l u e ( ) )

Listing 4.14: Sending a manual offload command

One key issue that arose during the design of the GUI was that nodes operating on the local and remotemachines were not easily distinguishable at a glace when viewing the node visualisation. Therefore it wasdecided that a colour coding system should be used to help the user differentiate between the nodes that arerunning locally and those that have been offloaded. The colour scheme is defined as follows:

Blue is used to denote nodes that are not offloadable

Red is used to denote nodes that are running on the local machine

Green is used to denote nodes that are running on a remote machine

Grey is used to denote failed/unresponsive nodes

This functionality was implemented by extending ROS’ built in ROS graph code. The changes made in the codeshown below listens for incoming node names that are awaiting a redraw and dynamically changes their colourbased on their system prefixes. In this case PI PREFIX and PC PREFIX. The function add node to graphgenerates a dot description of the node using the supplied input parameters. Dot is a simple description languagethat is both human and machine readable that can be used to generate graphs. This dot code description of thecompleted system overview is then passed back to RQt where it is converted into a format that may be displayedto the user.

1 i f node [ 0 : 4 ] == PI PREFIX :2 d o t c o d e f a c t o r y . a d d n o d e t o g r a p h ( do tg r aph ,3 nodename=node ,4 shape = ’ e l l i p s e ’ ,5 u r l =node ,6 c o l o r =” r e d ” )

Listing 4.15: Dynamically changing node colours

The remainder of the GUI was developed using ROS’ existing GUI modules and adding them to my RQtdesign. In order to help the user visual the face tracking algorithm an Image View was added that allows theuser to see the output of the Post Processing node. The face tracking is displayed to the user as a series of greendots located in proximity to their face. These dots represent the features that the algorithm is currently tracking onthe users face. In addition to this a Process Monitor was added. This monitor lists useful information suchas CPU utilisation and memory use for each node connected to the current ROS master. This is of particular usefor our offloading system as it allows us to view information regarding nodes even when they are not currentlyresiding on the machine that has the GUI running. The final GUI design can be seen in figure 4.4. Another keyfeature of the GUI is being able to filter information by topic to make it easier for the user to view. This can bedone in both the Process Monitor and the Node View to help de-clutter the screen when there are a lot of nodespresent in the system.

29

Figure 4.4: Final GUI Design in RQt

4.7 Scheduler

The scheduler node serves two primary functions in our system:

1. To monitor the system state, in particular the CPU usage

2. To decide when a node should offload

The first function is achieved using PSUtil. This library allows the scheduler to interact with the operatingsystem and retrieve CPU utilisation levels. In our system the scheduler can be set to one of two monitoring modes:automatic or manual. In automatic mode the scheduler periodically sample the total system CPU utilisation inorder to help determine when is the most opportune moment to offload a particular node. In this mode thescheduler will also take into account the fact that there may be more than one core in the CPU it is monitoring,this can be seen in listing 4.16. In manual mode however. before checking the CPU usage, it first checks whetheror not a manual CPU level has been received from the GUI. If it has, then this value is chosen in place of theactual system CPU level, thereby allowing users to specify when the system will offload using the GUI.

1 i f s e l f . c p u c o u n t > 1 :2 c p u u s a g e = ps . c p u p e r c e n t ( p e r c pu =True )3 f o r cpu i n c p u u s a g e :4 cpu usage sum = cpu usage sum + cpu5 c p u u s a g e = cpu usage sum / s e l f . c p u c o u n t6 e l s e :7 c p u u s a g e = ps . c p u p e r c e n t ( )

Listing 4.16: Monitoring the system CPU level

4.7.1 Determining when to offload

The scheduler uses utilisation thresholds to determine when to subscribe or unsubscribe a node in the system.These thresholds are currently hard coded into the system, but can be changed dynamically at run time by using

30

ROS Launch. The scheduler uses different thresholds for unsubscribing and resubscribing because otherwise thesystem hysteresis occurs, resulting in an oscillating loop of loading and offloading nodes, hence resulting in nouseful work being completed by the system. These thresholds are described in table 4.2

Table 4.2: Offloading thresholds

Node Unsubscribe (%) Subscribe (%)Pre-Processing 80 40Face Detection 60 10Face Tracking 90 30

Once a node reaches one of these percentages, an offloading command is generated and published to thatnode so that it may begin subscribing to the relevant topics defined in its subsribe method. In addition to thisanother offloading command is published, this time to the node that is currently still subscribed. This offloadcommand tells the subscribed node to unsubscribe itself from all relevant topics in order to prevent both nodespublishing to the same topic. This code can be seen in listing 4.17 where destination is a Boolean value thatdetermines whether it is being made local or remote and node name is the actual name of the node within thesystem. This is used to help the node identify that the message is intended for itself. In the code in listing 4.17,scheduler command is a message object that is to be sent to the corresponding node.

1 schedu le r command = SchedulerCommand ( )2 schedu le r command . node name = node name3 schedu le r command . o f f l o a d = d e s t i n a t i o n4 s e l f . s c h e d u l e r p u b . p u b l i s h ( schedu le r command )5 s e l f . r a t e . s l e e p ( )

Listing 4.17: Sending an offloading message

4.8 Chaos Monkey

In order to test how robust the system is I developed a ”Chaos Monkey.” The Chaos Monkey works by effectivelyattempting to disrupt the entire system as much as possible at random intervals. While the Chaos Monkey is doingthis I would inspect how the system reacts. My implementation achieves this in two main ways:

1. Randomly choosing a node from the system and killing its process

2. Randomly choosing an active topic in the system and flooding it with garbage messages.

The main loop of the Chaos Monkey node generates a random number between 1 and 10 and based on thischooses an action to complete, this can be seen in listing 4.18

1 i f rand num >= 6 :2 s e l f . k i l l n o d e s ( t a r g e t n o d e )3 e l i f rand num == 5 or rand num == 6 :4 s e l f . n o d e s t o k i l l . append ( t a r g e t n o d e )5 e l i f rand num >= 1 and rand num <=3:6 s e l f . i n f e c t t o p i c ( t a r g e t t o p i c , t a r g e t t o p i c t y p e )7 e l s e :8 s e l f . r a n d o m i s e c h a o s ( )

Listing 4.18: Chaos Monkey choosing an action

31

The method kill nodes takes an input array of victim nodes and executes rosnode.kill() on each ofthem. This completely kills the nodes process and does not allow it to shut down correctly, hence it is possiblefor the system to remain idle until the node resumes execution.

In order to implement the infect topic) method I specified two main target node types, however morecould be added if required. These were scheduler command and face box. When a topic is selected thatmatches one of these types, the Chaos Monkey generates a ROS message of that type with garbage informationinside it. An example of this can be seen in 4.19 where face box is a message object.

1 f a c e b o x . x = random . r a n d r a n g e ( 0 , 6 4 0 )2 f a c e b o x . y = random . r a n d r a n g e ( 0 , 4 8 0 )3 f a c e b o x . wid th = f a c e b o x . x + random . r a n d r a n g e ( 0 , 6 4 0 )4 f a c e b o x . h e i g h t = f a c e b o x . y + random . r a n d r a n g e ( 0 , 4 8 0 )5 s e l f . o u t p u t f a c e b o x p u b . p u b l i s h ( f a c e b o x )

Listing 4.19: Chaos Monkey setting up an infected message

It then begins to spam the topic with as many of these as it can produce in order to test the systems response.

32

Chapter 5

Evaluation

The focus of this chapter is to review the system and determine whether or not it has met the requirements set outby the user in the functional requirements analysis in chapter two. Furthermore this chapter attempts to evaluatehow effective the system is at achieving the project goal set out in chapter one.

5.1 Testing

In ROS there are two primary types of testing we can perform: unit testing and node testing. Unit testing isessentially the same as normal unit testing in any other language, allowing us to inspect the actual code of thenodes we are interested in. This form of testing is known as white box testing. Node testing however allowsus to run tests as if the entire system is currently running, but does not allow close code inspection. This isknown as black box testing and tests the interactions between the edges of the nodes we are interested in. ROSalso provides a method of playing back a previously executed version of the system in the form of a ”ROSBag.” This method was used to determine the down times between offloading and to evaluate the overall systemsperformance. ROS Bag essentially stores all of the messages and communication exchanged between nodes andallows us to Furthermore I designed a ”Chaos Monkey” that would interact with the system and probe it forresilience to ensure the system was robust and capable of recovering from node failures.

5.1.1 Acceptance Testing

The acceptance tests allow us to determine to what extent we have completed the requirements set out the by theuser during the Analysis chapter. From table 5.1 and table 5.2 we can see that our system meets the acceptancecriteria for all of the must have and all of the should have functional requirements.

33

FR Acceptance test Implemented1 Have a built physical robot that can carry the re-

quired add onsPage 13, Physical Design

2 Be able to move the robot left and right Page 27, Motor Control3 Have the robot detect the users face Page 23, Face Detection4 Find features in a given area of the image Page 24, Face Tracking5 Have the robot track the users face using the webcam Page 24, Face Tracking6 The robot should attempt to keep the user within

viewPage 27, PID Controller

7 Have the algorithm run in separate nodes Page 21, Breaking Up the Algorithm8 Offload the nodes between local and remote ma-

chinesPage 25, Offloading Nodes

9 Allow monitoring of the systems current CPU usage Page 30, Scheduler10 Allow the system to automatically determine when

to offloadPage 30, Determining when to offload

11 Show that the system sends messages between bothlocal and remote machines

Page 28, GUI

12 Display a video stream to the user Page 28, GUI

Table 5.1: Table of must have acceptance tests

FR Acceptance test Implemented13 Allow the user to manually specify offloading Page 29, GUI14 Overlay graphical data onto the image showing the

face trackingPage 29, GUI

15 Let the user see the nodes offloading graphically Page 29, GUI16 Show the CPU usage of the nodes Page 29, GUI17 Allow dynamic local and remote launching of nodes Page 25, Launching Nodes

Table 5.2: Table of should have acceptance tests

5.1.2 System Testing

In order to perform the system testing I used ROS Bag, which is a graphical tool that allows you to monitorand examine messages that are passed between specified nodes. Using this tool I was able to determine the timebetween each offload once the offloading command was received and hence the average time required to performthe offloading operation. From the analysis section we know that

w

sclient>

w

sserver+

d

B(5.1)

Where w is the amount of computation to be done, d is the data to be sent, B is the bandwidth of the system andsclient and sserver is the CPU speed of the client and server respectively. From this we can see that the systemsperformance will only increase providing that the offloading can occur faster than the time it would take toperform the task on its on locally. Assuming that the bandwidth is constant, this means that the time to offloaddepends heavily on the size of the data to be offloaded. Since our system is duplicated both locally and remotely,we do not need to offload the actual task. Instead we are sending a message packet that has a constant size andthen awaiting the system to resume normal execution. From this we can assume that the system will performbetter providing the speed of the server sserver is sufficiently faster than the local system.

This hypothesis is reflected in the offloading times observed in the system in table A.1. The box and whisker

34

plot in figure 5.1 shows how closely packed the timing for the system is. The minimum value is 1.922 seconds,its max is 2.482 seconds and it has a standard deviation of 0.12.

Figure 5.1: Boxplot of system offloading times

I also calculated the average CPU utilisation for each node when it was subscribed and when it was unsub-scribed which can be seen in A.2 and A.3. This shows that when a node is offloaded, its CPU utilisation becomescompletely negligible. In fact when the system is set to its full level of offloading, the total system CPU utilisa-tion can drop by between 46% and 82%, this can be observed in table A.4 and is summarised in the area graphin figure 5.2, which shows the cpu utilisation of three different system configurations over a 60 second window.As we can see, when the entire system is running on the low powered machine, it can easily consume the entire

Figure 5.2: Area graph showing average CPU utilisation

35

CPU causing large amounts of delay in the system, whereas when the system has been offloaded the machinemaintains a size-able amount of spare CPU time.

5.2 Chaos Monkey

The chaos monkey that was implemented server two primary functions:

1. Test the resilience of the system by randomly killing sections of the system

2. Test how the system handles errors by flooding randomised topics with garbage data

Since the system has been designed to attempt to handle node failure by restarting dead nodes, we can alsouse the chaos monkey to determine the average downtime between when a node fails, and when it begins sendinguseful data again. To achieve this I once again used ROS Bag in order to time the downtimes. Furthermore, ROSBag could be used to determine the effect of garbage data in the system, in particular garbage data that is send tothe scheduler node. The results of this investigation can be seen in table A.5 and are summerised in the boxplotgraph 5.3. From the box plot we can see that there is a large variance of down times, with a minimum of 1.811seconds, a maximum of 14.574 seconds and an average down time of roughly 8 seconds. This is primarily causedby the varied sizes of the nodes. Nodes with few publishers and subscribers require less time to begin executingagain. This is particularly apparent in the Image Output node and the Usb Cam node. It is also dependant onthe number of failed nodes as the Chaos Monkey can decide to kill multiple nodes simultaneously, as the systemattempts to initialise them sequentially.

Figure 5.3: Box plot of system down times

In figure 5.4 below we can see the effect the Chaos Monkey has on the running system. An interestingexample of a point of failure in the system can be observed at 2 minutes 43 seconds and 3 minutes 8 secondsrespectively. We can see that whilst the Pre-Processing node is down, the Face Tracking node can no longercontinue its execution. Although this would not result in a stalled system since the input from the webcam wouldinstead be displayed, this would result in no further face tracking occurring until the node became active oncemore.

A final interesting note to make regarding figure 5.4 is the scheduler commands topic at 3 minutes 15 seconds.We can see that it becomes inundated with a considerable number of offloading commands. These commandsare just garbage commands generated by the Chaos Monkey. From the graph we can see that none of thenodes are offloaded repeatedly and in fact continue to function like normal until pre processed image receivesa genuine scheduler command from the Scheduler. This shows that the system is able to deal with errors in

36

messages and furthermore, due to the nature of the ROS operating system, the nodes will ignore any messagesthat are not destined for them on the offloading topic. If however an image node receives a genuinely corruptedImage message, it will attempt to display it during that frame because it cannot determine if the image has beencorrupted or not.

Figure 5.4: Sample ROS Bag output

5.3 User Demo

In order to test the systems usability a user demo was carried out. The purpose of this user demo was to haveusers follow a set of initial instructions on how to interact with the system and then set them a series of furthertasks to determine how easy the system is to use as well as how easily the system conveys its information backto the user. After the users had completed the user demo, they were then asked to fill out a brief questionnairethat incorporated a Likert scale to help quantify the systems suitability. As well as this the users were asked tospecify any changes they felt would be beneficial to include in the system.

5.3.1 Results

The questions asked on the Likert scale were as follows:

1. The Node View was easy to understand.

2. The Process Monitor aided you in understanding what the system was doing.

3. The Offloading Controls were intuitive to use.

4. It was easy to visualise the offloading occurring using the Node View.

5. It was easy to see when the algorithm was tracking you.

6. It was hard to determine when the offloading would occur.

7. The GUI was well laid out and simple to navigate.

8. Using the GUI helped you to understand the effects of offloading.

During the demo there was some issues with the face tracker accurating plotting the feature points onto theusers faces. This is heavily reflected in the score in question 5, which shows that the users had a hard timedetermining when the algorithm was effectively tracking them. Furthermore it was generally agreed that theexact moment when the offloading had occurred was difficult to gauge. This is primarily due to a limitation inROS’ graphing ability, which prevents its graph from refreshing itself when a view has changed. Despite theseslight problems it was generally agreed that the system was easy and intuitive to use and that Node View andProcess view aided the users in understand the offloading process in the system.

37

Figure 5.5: Results from the Likert Scale questions

When the users were asked what changes they felt should be made to the system, the majority of themspecified that the overall systems CPU utilisation should be displayed in addition to the individual node CPUutilisation. Furthermore one user mentioned that instead of colour coding the nodes, that it would be considerablyeasier to keep track of the changes by separating them into columns and having the nodes move around thesecolumns. The other main point that they made was that the system would be more responsive if the Node Viewautomatically refreshed itself, instead of requiring user intervention to do so.

38

Chapter 6

Conclusions

The focus of this chapter is to reflect on the final outcome of the project and analysis how well the system meetsthe requirements set out in the analysis chapter. As well as this I reflect personally on my experience throughoutthe project and suggest potential improvements that could be made to the system in the future.

6.1 Conclusion

6.1.1 Suitability

The primary aim of this project was to design and build a robot that was capable of dynamically offloading aface tracking application in order to demonstrate the benefits of the offloading process. This was to evaluate itspotential use in developing AnyScale applications that can be run at any level of scalability. Furthermore theproject aimed to implement a GUI that would convey the offloading process to the user in an intuitive mannerand allow them to interact with the system for demonstration purposes.

From the results in the previous chapter we can see that the there is an evident benefit to implementingdynamic offloading in a system. Of particular note is that a task that has implemented the offloading systemcould potentially be run on any level of hardware, from the Raspberry Pi to a cloud network since the amountof offloading occurring would scale alongside the systems current CPU utilisation requirements. From the initialtables of a functional requirements I have implemented all of table 2.2, all of table 2.3 and all of table 2.4. Thisleaves only the items in table 2.5. Due to this I feel that the system meets the suitability needs of the client asexpected.

6.1.2 Usability

The results from the user demonstration provided good feedback on the systems usability. Overall the usersfelt that the system was intuitive to use and easy to navigate. The general consensus was however that theresponsiveness of the system would be much better if the Node View was capable of refreshing itself withoutmanual intervention from the user. Some users stated that the names of some of the labels should be changed inthe next iteration of the GUI design.

39

6.2 Problems encountered

1. An initial stumbling block while developing the system was being unable to get ROS running on theEV3Dev machine, this resulted in me having to create a packet sending system that would allow the EV3to communicate with the rest of the nodes. Although it didn’t prevent me from interfacing with the motorslibrary, it did cause a delay in the development of the motor controller.

2. On top of this there was numerous issues with the ROS environment in Raspbian that caused large delayswhen developing the system. In particular attempting to get to original algorithm running on the RaspberryPi proved to be impossible and instead I simply redesigned the entire structure so that it would work withnewer versions of ROS and catkin.

3. My final stumbling point was the Node View in the GUI. This was built on top of an existing ROS frame-work for displaying graphs, however the existing framework did not support automatic refreshing of theview and instead forced users to refresh the view manually. Unfortunately I reached this point too late inthe project to be able to spend an adequate amount of time to resolve the issue correctly.

6.3 Future Work

6.3.1 Scalability

As the system currently stands it is set up to deal with one local client and one remote server. ROS launchhowever provides the means to allow many more nodes to be renamed dynamically on launch and hence thesystem could potentially scale to a much larger level. The primary adjustment that would need to be madewould be changing the scheduler so that it utilises the ROS Param server when retrieving the names of machinesconnected to the system. This would allow it to dynamically take a list of node names within the system. Oncethis had been implemented the scheduler would need to be adapted to choose a destination for the offloadingfrom the list of machines connected to the system. Implementing this feature would allow the system to interactwith the Raspberry Pi cloud and would more effectively demonstrate the kinds of offloading that would occur ina real situation.

6.3.2 GPGPU

The local machine was primarily limited in its computation due to the size of its processor. General-Purposecomputation on Graphics Processing Units (GPGPU) allows you to treat the system Graphics Processing Unitas a high power, high throughput CPU. [16] By using GPGPU we could extend the machines ability to handlethe computationally intensive tasks, especially since these tasks are graphics based and require high throughput.This would allow us perform a much more accurate face tracking method on the same local machine.

6.3.3 Robustness

The current system, although robust in its ability to deal with dead nodes by respawning them, does have adrawback. It is currently unable to determine on its own which nodes are still alive in the system, instead relyingon ROS to take care of this functionality for it. Modifying the scheduler so that it could ping nodes currentsubscribed to the system to determine if they are functioning, would allow us to restart them if they no longerrespond. This method, coupled with the nodes being able to respawn themselves once killed, would result in asystem that could not go down due to node failure.

40

6.4 Personal Reflection

As an Electronics and Software Engineering (ESE) student I feel that this project has provided me with anextremely valuable insight to the world of robot development. Throughout the duration of the project I havebecome more and more proficient in the use of the Robot Operating System, to the point where I would beconfident in tackling a larger project in the future using the same tools. Due to studying ESE we are not exposedto a lot of GUI programming and so it was invaluable spending time designing and implementing a functioningGUI, particularly once the feedback showed that it was largely intuitive to use. I feel that the biggest thing I willtake away from this project is the importance of setting small goals that can be achieved in the short term.

6.5 Acknowledgements

I would like to thank Callum Cameron and Jonathan Aitken for lending their experience throughout the project. Iwould also like to sincerely thank David White for his insights, help and most of all patience towards me duringthe project.

41

Appendices

42

Appendix A

Results Tables

Table A.1: Table of offloading timesOffloading Times

Prev Message (s) Next Message (s) Difference (s)4.028 5.95 1.9224.071 6.011 1.946.923 9.18 2.2579.876 12.043 2.16714.947 17.139 2.19215.991 18.335 2.34420.969 23.161 2.19224.062 26.232 2.1729.795 32.08 2.28530.052 32.258 2.20632.947 35.227 2.2835.832 38.109 2.27737.993 40.251 2.25838.038 40.276 2.23838.946 41.249 2.30343.896 45.949 2.05344.944 47.249 2.30548.044 50.302 2.25856.077 58.381 2.30459.01 61.312 2.30270.091 72.375 2.28473.89 76.243 2.353

100.102 102.265 2.163108.044 110.166 2.122108.078 110.422 2.344113.83 116.108 2.278113.914 116.023 2.109142.404 144.472 2.068184.174 186.38 2.206206.093 208.575 2.482

43

Table A.2: Table of CPU utilisation for subscribed nodesCPU Utilisation

Pre-Processing (s) Face Detector (s) Face Tracker (s)10.2 31.1 13.99.9 36 13.99.3 32.9 11.19.2 32.2 17.99.1 37.5 19.69 31.3 18.6

10.4 31.2 11.29.7 32 119.2 30.1 11.410.2 38.4 11.58.1 33.5 17.711.2 33.6 13.89.3 33 23.18.2 32.5 10.511.2 33.2 11.69.3 30.5 16.78.2 21.6 11.29.6 29.2 11.211.3 30.7 14.49.7 36.2 13.1

Table A.3: Table of CPU utilisation for unsubscribed nodesCPU Utilisation

Pre-Processing (s) Face Detector (s) Face Tracker (s)0 0 00 0 00 1 00 1 01 1 00 0 10 0 20 0 11 2 01 1 00 0 00 1 02 0 0

1.1 0 01.2 0 00 0 10 1 10 1.1 01 0 10 0 1

44

Table A.4: Table of average CPU utilisation for entire systemAverage CPU usage for entire system

No Offloading (%) Offloading with GUI (%) Offloading without GUI (%)100 40.1 47.5100 37.7 34100 47.1 46100 40.6 34.6100 44.8 34100 39.1 29.1100 44.3 26.9100 45.7 24.1100 49 26.9100 38 24.3100 46.9 26100 38.5 29.7100 40.9 27.9100 36.6 24100 38.4 20.5100 53.5 22.8100 40.6 35.5100 45 27.8100 37 17.8100 36.6 33

Table A.5: Table of node downtimes for the systemNode down times in the system

Node Down (s) Node Up (s) Difference (s)72.845 82.298 9.45375.211 81.576 6.36545.951 48.389 2.438107.753 122.327 14.574163.252 171.753 8.501188.256 196.222 7.966175.782 177.593 1.811163.515 171.583 8.068200.476 206.258 5.782

45

Appendix B

User Demonstration

46

Offloadable Face Tracking Demonstration

Justin McManus

March 27, 2015

1 Introduction

The main focus of this project was to design and build a robot which may be used todemonstrate the effects of computational offloading. The robot was be built using LegoMindstorm pieces and will be controlled by a Raspberry Pi. For this demonstration however,the system will be executed entirely on the host PC. The robots primary function is toperform face tracking, a computationally intensive task. If the robots Central ProcessingUnit becomes too encumbered while performing the face tracking, then the robots schedulershould dynamically attempt to offload parts of the system, thereby reducing the load onthe processor and keeping the system responsive. A Graphical User Interface is used todemonstrate the offloading and provides a means of manual interaction with the system.The manual interaction takes the form of injecting false CPU levels so that the user can seethe effect of different CPU loads on overall system.

2 GUI

For this demonstration you will attempt to cause manual offloading to occur in the systemand then view how the systems node graph changes. When the system starts, you will bepresented with the GUI. This GUI has several key sections that provide useful feedback,these are:

Image View: This provides visual feedback from the robots point of view. It also overlayswhere the robot thinks the users face is using green dots to represent important featurepoints.

Control Panel: This is used to manually interact with the system. When the radiobutton is set to manual, you are able to use the slider to change the injected CPUlevel.

Process Monitor: The process monitor provides feedback on each node that is runninginside the system, including its CPU usage and memory usage.

Node View: This view allows us to visualise which nodes are currently offloaded andwhich are running locally. The node view uses the following colour scheme.

1

Appendix C

Installation Instructions

1. To begin the install you firs need ROS. Follow the instructions availablehere to get ROS http://wiki.ros.org/indigo/Installation/Ubuntu

2. Next clone the packages from github using git clonehttps://github.com/aracanid/offloadable face recognition, git clone https://github.com/aracanid/rqt offloading controls,git clonehttps://github.com/aracanid/rqt offloading graph.

3. Alternatively build it by placing the un-ziped contents intoyour catkin workspace like with the instructions from step one.

4. next run catkin make on the workspace

5. You should now be able to run the system by typingroslaunch offloadable face recognition raspberry pi client nodes.launch, roslaunch offloadable face recognitionremote pc server nodes.launchand rqt all in separate terminals.

48

Bibliography

[1] AnyScale. The Project, year = 2015, url = http://anyscale.org/, urldate = 2015-03-04.

[2] Arduino. Arduino Uno Specifications, year = 2015, url =http://www.arduino.cc/en/Main/arduinoBoardUno, urldate = 2015-03-03.

[3] Petros Maniatis Mayur Naik Ashwin Patti Byung-Gon Chun, Sunghwan Ihm. NWSLite: a light-weightprediction utility for mobile devices. In MobiSys ’04 Proceedings of the 2nd international conference onMobile systems, applications, and services, pages 2–11, 2004.

[4] Petros Maniatis Mayur Naik Ashwin Patti Byung-Gon Chun, Sunghwan Ihm. CloneCloud: elastic execu-tion between mobile device and cloud. In EuroSys11, page 181194, 2011.

[5] Zhiyuan Li Changjiu Xian, Yung-Hsiang Lu. Adaptive computation offloading for energy conservation onbattery-powered systems. In Parallel and Distributed System ,2007 International Conference on, volume 2,pages 1–8. IEEE, 2007.

[6] The Qt Company. Homepage, year = 2015, url = http://www.qt.io/, urldate = 2015-03-18.

[7] Pranesh Pandurangan Mostafa Ammar Mayur Naik Ellen Zegura Cong Shi, Karim Habak. COSMOS:computation offloading as a service for mobile devices. In MobiHoc ’14 Proceedings of the 15th ACMinternational symposium on Mobile ad hoc networking and computing, pages 287–296, 2014.

[8] Dae-ki Cho Alec Wolman Stefan Saroiu Ranveer Chandra Paramvir Bahl title = MAUI: Making Smart-phones Last Longer with Code Offload booktitle = MobiSys ’10 Proceedings of the 8th international con-ference on Mobile systems year = 2010 pages = 49-62 Eduardo Cuervo, Aruna Balasubramanian.

[9] EV3Dev. EV3Dev, year = 2015, url = http://www.ev3dev.org/, urldate = 2015-03-26.

[10] Open Source Robotics Foundation. What is ROS?, year = 2015, url = http://www.ros.org/about-ros/, urldate= 2015-03-03.

[11] Raspberry Pi Foundation. Raspberry Pi at Buckingham Palace, 3 Million Sold, year = 2014, url =http://www.raspberrypi.org/raspberry-pi-at-buckingham-palace-3-million-sold/, urldate = 2015-03-03.

[12] Raspberry Pi Foundation. Raspberry Pi Models and Revisions, year = 2015, url =http://www.raspberrypi.org/documentation/hardware/raspberrypi/models/README.md, urldate = 2015-03-03.

[13] Raspberry Pi Foundation. What is a Raspberry Pi?, year = 2015, url =http://www.raspberrypi.org/help/what-is-a-raspberry-pi/, urldate = 2015-03-03.

[14] Inc Gartner. Forecast: PCs, Ultramobiles and Mobile Phones, Worldwide 2010-2017, 4q13 update, 2014.

[15] Patrick Goebel. ROS Pi Vision, year = 2015, url = http://wiki.ros.org/pivision, urldate = 2015− 03− 26.

[16] GPGPU.org. GPGPU, year = 2015, url = http://gpgpu.org/, urldate = 2015-03-27.

49

[17] Itseez. About OpenCV, year = 2015, url = http://opencv.org/about.html, urldate = 2015-03-27.

[18] Yung-Hsiang Lu Bharat Bhargava Karthik Kumar, Jibang Liu. A Survey of Computation Offloading forMobile Systems. Mobile Networks and Applications, 18(1):129–140, Febuary 2013.

[19] Hsiao-Hwa Chen Kun Yang, Ou S. resource-constrained mobile devices running heavier mobile Internetapplications. Communications Magazine, IEEE, 18(1):56–63, January 2008.

[20] Lego. About Lego Mindstorms EV3, year = 2015, url = http://www.lego.com/en-gb/mindstorms/about-ev3,urldate = 2015-03-03.

[21] Lego. How is the EV3 different from the NXT?, year = 2015, url = http://service.lego.com/en-us/helptopics/products/themes/mindstorms/mindstorms-ev3/ev3-and-nxt-differences, urldate = 2015-03-03.

[22] EaSynth Solution Inc. Ltd. Homepage, year = 2015, url = http://www.foreui.com/, urldate = 2015-03-18.

[23] Yung-Hsiang Lu Lee Nimmagadda Y, Kumar K. Real-time moving object recognition and tracking usingcomputation offloading. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Confer-ence on, pages 2449–2455. IEEE, 2010.

[24] International Federation of Robotics. Service Robot Statistics, year = 2014, url = http://www.ifr.org/service-robots/statistics/, urldate = 2015-03-03.

[25] Giam Paolo. PSUtil Github, year = 2015, url = https://github.com/giampaolo/psutil, urldate = 2015-03-26.

[26] Massoud Pedram Peng Rong. Extending the lifetime of a network of battery-powered mobile devices byremote processing: a markovian decision-based approach. In DAC ’03 Proceedings of the 40th annualDesign Automation Conference, pages 906–911, 2003.

[27] Krintz C Nurmi D Wolski R, Gurun S. Using bandwidth data to make computation offloading decisions. InParallel and Distributed Processing ,2008 IEEE International Symposium on, pages 1–8. IEEE, 2008.

[28] K. Messer A. Greenberg I. Milojicic D. Xiaohui Gu, Nahrstedt. Adaptive offloading inference for deliveringapplications in pervasive computing environments. In Pervasive Computing and Communications, 2003.(PerCom 2003)., pages 107 – 114, 2003.

50