Vision based localization of remotely operable agricultural vehicles

Vision based localization of remotely operable

agricultural vehicles

Jayaweera Mudiyanselage Lasitha Chandana Piyathilaka

09/8068

Degree of Master of Philosophy

Department of Electronic and Telecommunication Engineering

University of Moratuwa

2011 June

Declaration

”I declare that this is my own work and this thesis does not incor-

porate without acknowledgement any material previously submitted

for a Degree or Diploma in any other University or institute of higher

learning and to the best of my knowledge and belief it does not con-

tain any material previously published or written by another person

except where the acknowledgement is made in the text”.

Also, I hereby grant to University of Moratuwa the non-exclusive right

to reproduce and distribute my thesis/dissertation, in whole or in part

in print, electronic or other medium. I retain the right to use this

content in whole or part in future works (such as articles or books).

......................

Signature

Date:

The above candidate has carried out research for the Masters/MPhil/PhD

thesis/ dissertation under my supervision.

.......................

Signature of the supervisor

Date:

Abstract

This thesis addresses the problem of outdoor autonomous robot lo-

calization for agricultural operations. A 12hp power tiller, commonly

known as the walking two wheel tractor is used as the test plat-

form after after introducing electrical actuators for its remote and au-

tonomous operation. The mathematical model of the power tiller has

been developed and simulated in MatLab SImulink. A vision based

outdoor localization system is developed using off-the-shelf electronic

components, and its accuracy has been verified in small agricultural

fields. Visual odometry using a downward faced camera is tested with

better resolution for relative localization, and popular visual odom-

etry algorithms were tested for speed and accuracy in agricultural

fields. A stereo vision based range measurement system has also been

developed and field tested as an absolute localization system that can

bound the incremental error caused by the visual odometry system.

The extended Kalman filter with measurement gaiting has been im-

plemented using both visual odometry and stereo range measurement

data. Experimental results verified that the proposed system as an

effective low cost technique for outdoor localization of field robots in

small agricultural fields.

To my parents Mr and Mrs Piyathilaka.

Acknowledgements

First and foremost, I wish to express my sincere gratitude to my

research supervisor, Dr. S.R. Munasighe Senior Lecturer at Depart-

ment of Electronic and Telecommunication Engineering, University

of Moratuwa for the valuable advice and guidance throughout my

research.

I would also like to thank Dr. A. A. Pasqual for the support shown as

the PG coordinator and memebers of the progress evaluation commit-

tee for giving invaluable cosntructive ideas.Also I would like to thank

Dr. E. C. Kulasekere, head of Department of Electronic and Telecom-

munication Engineering and all academic and none academic staff of

the Department of Electronic and Telecommunication Engineering for

facilitating and supporting this research work.

This research wouldn’t be possible if not the support given by Dayan

Rajapakshe and Bhanu Watawana during the implementation and

testing stages.

I also gratefully acknowledge the financial assistance provided by the

National Science Foundation of Sri Lanka under the grant No:RG-

2007-E-05.

On a more personal note, I am extremely indebted to my beloved

parents and my dearest wife Uthsuka Ekanayake for being a constant

source of inspiration, strength and encouragement throughout this

endeavor .

Contents

List of Figures i

List of Tables iii

1 Introduction 1

1.1 Problem Statement and Motivation . . . . . . . . . . . . . . . . . 1

1.2 Contribution of the Research . . . . . . . . . . . . . . . . . . . . . 3

1.3 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Literature Survey 5

2.1 Autonomous vehicles in Agriculture . . . . . . . . . . . . . . . . . 5

2.1.1 Vision Based Localization . . . . . . . . . . . . . . . . . . 6

2.1.2 Global Positioning System . . . . . . . . . . . . . . . . . . 7

2.1.3 Inertial Sensors . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2.1 Relative Localization . . . . . . . . . . . . . . . . . . . . . 9

2.2.2 Absolute Localization . . . . . . . . . . . . . . . . . . . . . 9

2.2.3 Sensor Fusion . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2.3.1 The Extended Kalman Filter . . . . . . . . . . . 10

3 The Power Tiller 11

3.1 Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Modification for Autonomous Operation . . . . . . . . . . . . . . 11

3.3 Mathematical Modeling . . . . . . . . . . . . . . . . . . . . . . . 15

3.3.1 Kinematic Model . . . . . . . . . . . . . . . . . . . . . . . 15

3.3.2 Dynamic Model . . . . . . . . . . . . . . . . . . . . . . . 17

3.3.2.1 Traction Forces . . . . . . . . . . . . . . . . . . . 17

3.3.2.2 Rolling resistance forces . . . . . . . . . . . . . . 18

3.3.2.3 Lateral Forces . . . . . . . . . . . . . . . . . . . . 18

6

CONTENTS

3.3.2.4 Dynamic Equations . . . . . . . . . . . . . . . . 19

3.3.3 Drive Model . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3.4 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.3.5 Model Validation . . . . . . . . . . . . . . . . . . . . . . . 23

3.3.5.1 Analogue Matching . . . . . . . . . . . . . . . . . 23

3.3.5.2 Maneuvers . . . . . . . . . . . . . . . . . . . . . 23

3.3.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4 Visual Odometry Based Relative Localization 28

4.1 Visual odometry using FFT based Image Registration. . . . . . . 29

4.1.1 Heading Angle Calculation . . . . . . . . . . . . . . . . . . 32

4.2 Visual Odometry Using Feature Tracking . . . . . . . . . . . . . . 32

4.2.1 Feature extraction and correlation . . . . . . . . . . . . . 34

4.2.2 Determination of Vehicle Orientation . . . . . . . . . . . . 34

4.3 Single Camera Visual Odometry . . . . . . . . . . . . . . . . . . . 35

4.3.1 Hardware Setup . . . . . . . . . . . . . . . . . . . . . . . . 35

4.3.2 Odometric model . . . . . . . . . . . . . . . . . . . . . . . 35

4.3.3 Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.3.3.1 Experiment 1: Effect of frame rate . . . . . . . . 36

4.3.3.2 Experiment 2: Effect of frame size . . . . . . . . 37

4.3.3.3 Experiment 3: Error Propagation . . . . . . . . . 38

4.3.3.4 Experiment 4: Loop-Closing . . . . . . . . . . . . 40

4.3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.4 Multi Camera Visual Odometry . . . . . . . . . . . . . . . . . . . 41

4.4.1 Hardware setup . . . . . . . . . . . . . . . . . . . . . . . . 41

4.4.2 Estimating Robot Motion . . . . . . . . . . . . . . . . . . 42

4.4.3 Mapping Camera Coordinates to Ground Plane . . . . . . 42

4.4.4 Estimating Translation Movement . . . . . . . . . . . . . . 43

4.4.5 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.4.6 Testing and Results . . . . . . . . . . . . . . . . . . . . . . 45

4.4.6.1 Variation of Uncertainty of Measurements . . . . 45

4.4.6.2 Loop Closing Test 1 . . . . . . . . . . . . . . . . 46



4.4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7

CONTENTS

5 Stereo Vision for Absolute Localization 51

5.1 Stereo Camera Geometry . . . . . . . . . . . . . . . . . . . . . . . 53

5.2 Hardware setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.3 Object Tracking and Stereo Image Correspondence Process . . . . 56

5.4 Camera Coordinate Transformation . . . . . . . . . . . . . . . . . 57

5.5 Stereo Camera Calibration . . . . . . . . . . . . . . . . . . . . . 58

5.6 Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

6 Sensor Fusion with EKF for Localization 62

6.0.1 Process Model . . . . . . . . . . . . . . . . . . . . . . . . . 62

6.0.2 Measurement Model . . . . . . . . . . . . . . . . . . . . . 63

6.0.3 Process error . . . . . . . . . . . . . . . . . . . . . . . . . 63

6.0.4 Measurement Gating . . . . . . . . . . . . . . . . . . . . . 64

6.0.5 Time Propagation . . . . . . . . . . . . . . . . . . . . . . . 64

6.0.6 Measurement Update . . . . . . . . . . . . . . . . . . . . . 64

7 Localization Implementation and Validation 66

7.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 66

7.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

8 Conclusions 73

References 75

List of Figures

3.1 Main parts of the Power Tiller . . . . . . . . . . . . . . . . . . . . 12

3.2 Disk brake system . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3 Remotely operable power tiller . . . . . . . . . . . . . . . . . . . . 14

3.4 Hand-held remote controller and X-Bee radio transceiver . . . . . 14

3.5 Power Tiller dimensions . . . . . . . . . . . . . . . . . . . . . . . 16

3.6 Power tiller’s position . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.7 Forces acting on power tiller . . . . . . . . . . . . . . . . . . . . . 18

3.8 Power Train . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.9 Diesel Engine Model . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.10 Structure of the simulation model . . . . . . . . . . . . . . . . . . 22

3.11 Ground tetsing of power tiller . . . . . . . . . . . . . . . . . . . . 24

3.12 Clutch signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.13 Position of the power tiller in world coordinate frame . . . . . . . 25

3.14 Change of heading angle . . . . . . . . . . . . . . . . . . . . . . . 26

3.15 X and Y position errors between actual and simulation results . . 27

4.1 Phase Correlation using FFT . . . . . . . . . . . . . . . . . . . . 29

4.2 Two images captured from consecutive frames . . . . . . . . . . . 30

4.3 The phase correlation of two frame . . . . . . . . . . . . . . . . . 31

4.4 Calculating heading angle . . . . . . . . . . . . . . . . . . . . . . 32

4.5 Optical Flow Field . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.6 ICR detection and calculating heading angle . . . . . . . . . . . . 34

4.7 Rover platform with downward faced camera . . . . . . . . . . . . 36

4.8 Effect of frame rate for visual odometry . . . . . . . . . . . . . . . 37

4.9 Effect of frame Size for visual odometry . . . . . . . . . . . . . . . 38

4.10 Effect of frame size vs visual odometry error . . . . . . . . . . . . 39

4.11 Cumulative error propagation . . . . . . . . . . . . . . . . . . . . 39

4.12 Result of loop closing test . . . . . . . . . . . . . . . . . . . . . . 40

i

LIST OF FIGURES

4.13 Optical flow field estimated by the two cameras mounted on the

rover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.14 Skid steering platform used for ground testing . . . . . . . . . . . 43

4.15 Projection of image plane to ground plane . . . . . . . . . . . . . 44

4.16 Variation of the uncertainty at high speed . . . . . . . . . . . . . 45

4.17 Variation of the uncertainty at low speed . . . . . . . . . . . . . . 46

4.18 Results of Loop Closing Test 1 . . . . . . . . . . . . . . . . . . . 47



5.1 Pin-hole camera geometry . . . . . . . . . . . . . . . . . . . . . . 53

5.2 Stereo camera geometry . . . . . . . . . . . . . . . . . . . . . . . 54

5.3 Rotatable stereo camera system . . . . . . . . . . . . . . . . . . . 56

5.4 Coordinate Transformation . . . . . . . . . . . . . . . . . . . . . 57

5.5 Calibration Chart . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.6 Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

7.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 67

7.2 Field Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

7.3 Extended Kalman Filter Localization . . . . . . . . . . . . . . . . 69

7.4 Visual Odometry Localization . . . . . . . . . . . . . . . . . . . . 69

7.5 Absolute localization using stereo range measurements . . . . . . 70

7.6 Effect of Measurement Gating . . . . . . . . . . . . . . . . . . . . 70

7.7 Error Distribution Along the Path Traveled . . . . . . . . . . . . 71

7.8 Cumulative Frequency Distribution of the Cartesian Error . . . . 72

ii

List of Tables

3.1 Change of total inertia of the wheels with clutch engage disengage

combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.1 Performances of the loop closing test results (Multi-camera Visual

Odometry) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7.1 Performances of the Kalman filter based localization system . . . 68

iii

1

Introduction

1.1 Problem Statement and Motivation

Farming in tropical regions such as Sri Lanka is extremely tedious and exhaust-

ing. These farmers still bear with it to make their living. The situation becomes

more worse because many of those farmers still practice traditional farming meth-

ods that require more labor and time. And, the farming community gradually

declines as young generation is looking for other easier employments. This sit-

uation shows signs of a significant future national problem, and it is required

to take efforts to device a technical solution to this problem. At present, some

agricultural machineries such as tractors are used by individual farmers for land

preparation, still it is very tedious to operate these machines manually. In this

view, it is desirable to device remote control or autonomous operations of these

machines. This way, those machines could operate long hours, even at night so

that productivity can be significantly improved while making the work less labo-

rious. Worldwide, there is a large demand for introducing autonomous systems

for agriculture and numerous research has been done to innovate and to pro-

mote rational and safe agriculture. Precision agriculture is a new research area

where automatic guidance of agricultural vehicle are tested and developed. The

motivations for automatic guidance of agricultural vehicles are as follows:

• First, automatic guidance reduces the arduousness of the driving task: for

instance, achieving perfectly parallel runs when driving manually is very

tiring over hours.

• Allows the driver to fully devote his time to the monitoring and the tuning

of the tool. This clearly can improve the quality of the agronomic work

1

1.1 Problem Statement and Motivation

carried out.

• Ensures an optimal work precision all day long and on the whole field.

Double applied and skipped areas between successive passages can then be

minimized, so that the exact placement of field inputs (seeds, fertilizers,

pesticides, etc.) can be achieved. Their cost can then be reduced.

• Allows to operate wider tools at higher speeds. Therefore productivity can

be increased.

Many types of agricultural vehicles are commercially available. Most common

is the four wheel tractor which is used for various applications such as tilling,

ploughing and seeds/fertilizer placement. But in south and south east Asia power

tiller or walking tractor is used heavily for farming because of its low operating

cost. Also it can be utilized for field operations in small agricultural fields easily.

However, operating these machineries is extensively laborious. In this view, we

decided to design and develop a remotely operable farm vehicle with some au-

tonomous capabilities using latest technologies. This innovation will enable the

farmer to operate the machine remotely from a convenient place and perform the

work in the field quickly and effectively. The economic and social impact of such

an innovation is very significant. Because remote and autonomous operations

are latest and fascinating experience for farmers even young people find farming

as an attractive employment. Even elderly, disabled, and even women will be

able to engage in farming if such an innovation is provided. In this research we

focused of modifying a two wheel tractor for remote and autonomous operation.

A vision based localization system has been developed for the power tiller for its

autonomous operation. Automatic guidance of vehicles requires a very accurate

on-board localization system. Kinematic GPS sensors provide real-time position

information with high accuracy and used in many commercial autonomous agri-

cultural vehicles (1). But these available systems are so expensive that local

farmers wont be able to afford to pay for them. The low cost implementation

is a very critical requirement in developing autonomous agricultural vehicles for

the third world country farming communities. More time and effort has therefore

been devoted in the development of vision based localization system to make it

as low as possible.

2

1.2 Contribution of the Research

1.2 Contribution of the Research

The objectives of this research are to modify a two wheel walking tractor for

remote operation, and to design an accurate low cost localization system for its

autonomous operation. For the latter objective, visual odometry based relative

localization system and stereo vision based absolute localization systems have

been developed and tested. These two localization systems were fused together

using the Extended Kalman Flter.

Following developments have been achieved in this research

• An autonomous agricultural vehicle test-bed has been built by introducing

necessary modification to a conventional two wheel walking tractor(Power

Tiller).(Chapter 3)

• Amathematical model has been developed for the power tiller and simulated

in Matlab Simulink environment.(Chapter 3).

• Performances of visual odometry based relative localization system has been

analyzed using the two techniques: Phase correlation in frequency domain

and Optical flow field estimation using feature tracking (Chapter 4)

• A technique that uses stereo vision for range measurement with object

tracking has been developed as an outdoor absolute localization system.

(Chapter 5)

• Extended Kalman Filter has been employed to fuse data from absolute and

relative localization systems. (Chapter 6)

• An outdoor localization system has been built with off-the-shelf low cost

component, and tested in outdoor environment. (Chapter 7)

1.3 Structure of the Thesis

The thesis is organized as follows. Chapter 2 presents related work and localiza-

tion systems used in autonomous agricultural vehicles. Chapter 3 discusses the

two wheel tractor and modification that were done for autonomous operation.

It also presents mathematical modeling of the two wheel tractor and simulation

model with its simulation results. In Chapter 4 visual odometry is proposed as

3

1.3 Structure of the Thesis

a high resolution relative localization system for autonomous operation in agri-

cultural. Image registration in frequency domain using Fast Fourier Transform

(FFT) and optical field flow estimation using feature tracking were used to track

the pose of the moving robot by using image sequence captured from a downward

faced camera. These two methods were tested for their performances in an out-

door field. Chapter 5 describes absolute localization system that use stereo vision

for range measurement. It consists of stereo depth measurement and a tracking

algorithm that track a color coded tag attached to the moving tractor. Prelim-

inary testing were done to verify the algorithm and results are also presented.

Chapter 6 presents use of Extended Kalman Filter(EKF) as a sensor fusion sys-

tem that combined absolute range measurement from stereo vision system and

relative measurement from visual odometry. Chapter 7 describes implementation

of the complete vision based outdoor localization system with its result. Chapter

8 presents the conclusions and future design recommendations.

4

2

Literature Survey

2.1 Autonomous vehicles in Agriculture

Introducing field robotics to agriculture is an emerging area of interest for many

researchers with the increasing demands on agricultural sector that force farm-

ers to consider robotic in agriculture. In recent years GPS guided tractors have

become commercially available. But these systems still require the farmer to

supervise them through out the field operation. Unlike other field robots that

are used in military and mining where safety is the major concern agricultural

vehicles will become feasible only if they save money with comparison to the tra-

ditional farming methods. The agricultural robots face many difficulties than a

laboratory mobile robots. Agriculture robots operate in large areas where envi-

ronment conditions may change quickly( Bright sunlight, rain,fog). These make

automation in agriculture is difficult to realize. But still numerous research are

done on agricultural robots and there are few reasons that encourage robotics

and agriculture. The operating areas generally do not change and artificial land-

marks can be easily setup around the boundaries of the field. In agriculture crops

color and features are previously known. Also Plants are cultivated in systematic

manner in rows with equal distances in standard patterns. Commercial localiza-

tion systems like GPS are available in outdoor environment where few centimeter

accuracy can be obtained. Therefore even though there are more disadvantages

than advantages still lot of research are done on autonomous agriculture robot

around the world. The most challenging task faced by autonomous agricultural

vehicles are localization and navigation. The accuracy of the localization system

define the accuracy of the field operation and the degree of operator intervention.

5


Sensors that use commonly in autonomous agriculture vehicle include machine

vision, global positioning systems (GPS),dead-reckoning sensors, and inertial sen-

sors .

2.1.1 Vision Based Localization

Machine vision sensors measure the relative position and heading using the image

sensor mounted on the vehicle. One of the most commonly used machine vision

methods is for detecting a guidance directrix on row crops, soil tillage, and the

edges along harvested crops. Marchant and Brivot (11) used the Hough transform

for row tracking in real time (10 Hz) and noted that their technique was tolerant

to outliers (i.e., weeds) only when their number was reasonably small compared

to the number of true data points. Marchant (12) reported an overall RMS error

of 20 mm in the lateral position at a travel speed of 0.7 m/s using this technique

to guide an agricultural vehicle in a transplanted cauliflower field. Han et al.(10)

developed a row segmentation algorithm based on k-means clustering to segment

crop rows. This information was then used to steer a tractor. The guided tractor

was able to perform field cultivation in both straight and curved rows. Okamoto

et al.(13) developed an automatic guidance system for a weeding cultivator. A

color CCD camera acquired the crop row images, which were then processed by

computer and used to determine the offset between the machine and the target

crop row. For more complete crop or field information, some researchers used a

stereo vision system to provide a three-dimensional (3D) field image by combining

two monocular field images taken simultaneously from a binocular camera. Such

3D images are reconstructed based on the different disparity monocular images

to decrease the ambient light influence. Kise et al.(14) developed a stereo vision

based agricultural machinery crop row tracking navigation system. The RMS

error of lateral deviation was 3-5 cm following both straight and curved rows

at speeds up to 3.0 m/s. The method required some weed-free areas to provide

sufficient information for detecting the navigation points. Astrand and Baerveldt

(15) developed a machine vision guidance system that achieved good performance

in detecting plants in near-infrared images acquired under non-uniform natural

illumination by performing gray scale opening on the raw near-infrared image

and subtracting it from the original prior to segmentation. Their method, based

upon the Hough transform, used multiple rectangular regions (one for each row

viewed) with the rectangle width adjusted for crop size. The information from

6


multiple rows was fused together to obtain a common estimate of the row posi-

tion. The accuracy of position estimation was less than 1.2 cm with a standard

error depending on plant size. Field tests showed that the system had sufficient

accuracy and speed to control the cultivator and mobile robot in a closed-loop

fashion with a standard deviation of position of 2.7 and 2.3 cm, respectively,

with incomplete row structures due to missing plants combined with high weed

pressure (up to 200 weeds/m2).

2.1.2 Global Positioning System

GPS-based navigation systems are the only navigation technologies that have

become commercially available for farm vehicles. Many tractor manufacturing

companies now offer the Real-Time Kinematic (RTK) GPS based auto steering

system as an option on their tractors. The position information from the RTK

GPS can be used for both guidance and other applications such as seed mapping,

traffic control, and tillage control. GPS guidance systems provide an absolute

guidance system in contrast to the relative guidance provided by machine vision,

which requires that the crop be planted using a GPS-guided planting system

or the crop rows mapped using some type of geo-referenced mapping technique.

GPS guidance systems also require that a GPS base station be located within

approximately 10 km of the RTK GPS guided tractor or agricultural robot. How-

ever, since GPS systems do not depend upon the visual appearance of the crop,

they are not adversely affected by weed density, shadows, missing plants or other

conditions that degrade the performance of machine vision guidance systems. An-

other advantage of GPS guidance systems is that they can be easily programmed

to follow curved rows (17). There appear to be three limitations to using GPS

for vehicle guidance. The first is that GPS guidance systems cannot be used in

microwave-shielded areas. Also, GPS cannot promise consistent positioning ac-

curacy in the range of centimeters for a variety of field conditions (e.g., presence

of buildings, trees or steeply rolling terrain, and interruption in satellite or differ-

ential correction signals). The second limitation is the inherent time delay (data

latency) required for signal processing to determine locations that might present

control system challenges at higher field speeds. The third is the high cost for

agricultural application (although there is a consistent trend of cost reduction

with widespread use). Stoll and Kutzbach (18) studied the use of the RTK GPS

as the only positioning sensor for the automatic steering system of self-propelled

7


forage harvesters. They found that the standard deviation of steering was less

than 100 mm under all conditions. Standard deviation of lateral offset (error)

along straight-line paths ranged from 25 to 69 mm depending upon the travel

speed. Kise et al. (19),(20) studied the use of an RTK GPS guidance system

for control of a tractor as an autonomous vehicle traveling along a curved path.

Test results for following a sinusoidal path with a 2.5-m amplitude and 30-m

wavelength at 6.5 km/h showed a 6-cm RMS error with a 13-cm maximum error.

To compensate for GPS positioning error associated with machinery attitude, re-

searchers at Hokkaido University integrated an inertial measurement unit (IMU)

with an RTK GPS to provide more accurate navigation information. This inte-

grated navigation system could guide agricultural machinery performing all field

operations, including planting, cultivating and spraying, at a travel speed of up

to 3 m/s, with a tracking error of less than 5 cm on both straight and curved

paths. Nagasaka et al.(7) used an RTK GPS for positioning, and fiber optic gy-

roscope (FOG) sensors to maintain vehicle inclination, for an automated six-row

rice transplanter . Root-mean-square deviation from the desired straight path

after correcting for the yaw angle offset was approximately 55 mm at a speed of

0.7 m/s. The maximum deviation from the desired path was less than 12 cm. by

using

2.1.3 Inertial Sensors

Inertial sensors take measurements of the internal state of the vehicle. A ma-

jor advantage of inertial sensors is that they are packaged and sealed from the

environment, which makes them potentially robust under harsh environmental

conditions. The most common types of inertial sensors are accelerometers and

gyroscopes. Accelerometers measure acceleration relative to an inertial refer-

ence frame. This includes gravitational and rotational acceleration as well as

linear acceleration. Gyroscopes measure the rate of rotation independent of the

coordinate frame. They can also provide 3D position information and have the

potential to detect wheel slippage. Unfortunately, these types of sensors are prone

to positional drift (5) Inertial sensors are mostly used in combination with GPS

or machine vision. Zhang and Reid (6) presented an on-field navigation system

with a vision sensor, FOG and RTK GPS. The results indicated that the multi-

ple sensor based agricultural navigation system was capable of guiding a tractor

8

2.2 Localization

between crop rows and showed that the inertial sensor was a good assistant func-

tion. Noguchi et al. (8) developed an agricultural navigation system consisting

of an RTK GPS and an inertial measurement unit. Experiments conducted in a

soybean field for tilling, planting, cultivating and spraying demonstrated that the

accuracy of the vehicle surpassed that of skilled farmer operation. The lateral

error of the guided vehicle was less than 5 cm.

2.2 Localization

Localization is the process that use sensory measurements to determines the

robot pose . The accuracy of the localization is determine by sensor accuracy,

dynamics of the systems , localization algorithms and even trajectory of the path

to be followed. For any autonomous system the design of an adequate localization

system in terms of accuracy, cost, and other design criteria is a challenging task.

In general Cartesian position and the heading is sufficient to accurately localize a

robot. Localization systems are generally divided into two basic categories namely

relative localization and absolute localization according to the sensor type used.

2.2.1 Relative Localization

Relative localization often called Dead Reckoning provides localization solution by

integrating the available incremental measurements, usually obtained from wheel

encoders, gyroscopes, and accelerometers. The relative sensing is self-contained

and allows high accuracy and high bandwidth for short-term localization. How-

ever, inherent dead reckoning signal integration results in an unbounded error

accumulation, which precludes the use of relative sensing as a sole mean of local-

ization for autonomous robots .

2.2.2 Absolute Localization

Absolute localization (AL), provides the localization solution based on the ex-

teroceptive sensing of current environmental features or landmarks and without

using the previous sensor information. The measurements are taken in the ab-

solute coordinate frame directly. In general, the localization problem is solved

if every position in the environment could be marked in a distinct way. On the

other hand the robots position in an absolute frame can be deduced from sensing

a finite number of fixed environmental landmarks. Landmarks can be natural like

9

2.2 Localization

trees or artificial like laser reflectors or satellites. Global Position System (GPS)

is a good example for absolute localization system.

2.2.3 Sensor Fusion

Both relative and absolute localization approaches can be used with an advantage

if properly combined. High fidelity and bandwidth localization can be obtained

by merging measurements from the high-bandwidth relative and low-bandwidth

absolute sensors. The absolute localization observes, directly or indirectly, at

least part of the robots pose and therefore is used as a feedback that bounds the

relative localization error. Combining the sensor measurements from the relative

and absolute localization systems results in redundancy of at least some of the

information. The process of extraction of the desired estimates from redundant

measurements is called sensor fusion. The fusion process can be done in the

time or frequency domains . Such a very effective sensor fusion technique is the

Kalman Filter.

2.2.3.1 The Extended Kalman Filter

The extended Kalman filter (EKF) is a nonlinear generalization of a linear Kalman

filter (31). The EKF has been used extensively for applications such as missile

tracking and guidance, marine and aeronautical navigation, as well as for the

localization of the indoor and outdoor mobile robots (29). The extended Kalman

filter maintains the same compact error distribution representation as the lin-

ear Kalman filter and thus is computationally efficient. The first order linear KF

framework requires that the Jacobians of the non-linear system and measurement

equations be evaluated at the state estimate. It is assumed that the filter will

maintain the estimate close to the true state so that the linearization will remain

valid. The Jacobians substitute the system and measurement matrices in the KF

algorithm. Since the assumption above is not guaranteed to hold and due to bias

introduced by the nonlinearity of the system, EKF estimates can be suboptimal

or even diverging i.e. giving an unbounded estimation error. The filter error co-

variance as well as the covariance of the innovation sequence has to be constantly

monitored to prevent unrecoverable divergence (29).

10

3

The Power Tiller

Power tiller, also known as walking tractor is used heavily for agriculture in many

part of the world. It is mainly used for tilling, ploughing and ground prepara-

tion in small agricultural fields. This versatile machine has gained popularity

among farmers due to its rugged and reliable operation. In this research a power

tiller was mechanically modified to cater both autonomous and remote opera-

tion. Mathematical model was also developed in order to facilitate subsequent

trajectory planning and control algorithm development.

3.1 Operation

The tractor used for field testing has a single piston diesel engine. The steering of

the power tiller is controlled by two clutches that enable differential drive. When

the tractor is steered left, left clutch is disengaged in order to release the wheel

from the engine while keeping the right clutch in the engaged position. Same

procedure is done for right maneuvers by engaging left clutch and disengaging

right clutch. There is a Main clutch to disengage two wheels from the engine and

the transmission system. Caster wheel which is fixed as the third wheel of the

tractor stabilize the vehicle in rough terrains. Basic parts of the power tiller is

shown in the following figure 3.1.

3.2 Modification for Autonomous Operation

It was required to modify some parts of the tractor in order to facilitate au-

tonomous operations. The tractor is driven in field by manually activating

11


Figure 3.1: Main parts of the Power Tiller -

clutches and gears. Therfore electric linear actuators are introduced to actuate

clutches in autonomous mode. linear actuator’s drivers receive remote commands

from the remote controller and control clutches accordingly. Initial field testing

proved that two wheels require additional braking system. This was an unfore-

seen requirement of the project. Researchers were initially under the impression

that the tractor wheel would stop when the clutch is disengaged . The tractor was

in fact particularly design in such a way that even if one clutch is disengaged that

wheel passively keeps moving with almost the same speed as of the other wheel

(clutch engaged) which caused problems in remote operation. Therefore it was

required to introduce disk brake system in order to stop the wheel completely.

The disc brake system that was attached to the tractor is shown in figure 3.2.

An additional linear actuator was fixed for the brake system. This brake system

enabled accurate steering of the tractor. Figure 3.4 shows the remote controller

unit. One of these units is located on the tractor while the other one (hand

held) is used by the farmer to issue motion commands to the tractor. Though

two linear actuators have been used to operate braking system and steering left

and right clutches there was no provision to stop the tractor when it is moving

forward. Therefore a powerful linear actuator is attached to actuate main clutch

of the tractor. This main clutch disengages tractor engine from the gear box.

Therefore by disengaging the main clutch tractor can be stopped. Attached lin-

12

3/figures/tractorparts.eps


Figure 3.2: Disk brake system - These brakes completely stop wheels when the

clutch is disengaged

ear actuator is operated using the remote controller in order to stop / run the

tractor. The control boards and actuators are powered by two 12 V car batteries.

24 V is required to fast operation of right and left clutch actuators. Tractors

inbuilt alternator is used to charge two batteries. This is done using a separate

battery charging circuit. Alternator was only generating 9V-10V at full speed of

the engine. Therefore DC voltage booster is used to step up voltage to 12V to

charge the batteries. Figure 3.3 shows the remotely operable tractor. Fig-

ure 3.4 shows the remote controller unit. One of these units is located on the

tractor while the other one (hand held) is used by the farmer to issue motion

commands to the tractor. This remote controller transmits control signals to the

control board mounted on the tractor. X- Bee Pro radio transmitter was used

to establish the wireless radio link with the tractor. It is capable of transmit-

ting control signal nearly 100m. Therefore operator can control the tractor from

a comfortable location. Remote controller circuit was enclosed in small plastic

casing. The remote controller was designed to operate single handed. Two push

buttons control the steering clutch actuators and toggle switch controls the main

clutch actuator. When the main clutch switch is at on position tractor starts to

move forward direction. When one of the clutch control push button is pressed

13

3/figures/diskbreaks.eps


Figure 3.3: Remotely operable power tiller - Electrical actuators receive remote

steering commands inoreder to actuate clutches

Figure 3.4: Hand-held remote controller and X-Bee radio transceiver - This is

used by farmer to send motion command to the tractor

14

3/figures/modifiedtractor.eps

3/figures/remote.eps

3.3 Mathematical Modeling

relevant clutch of the tractor is disengaged allowing the tractor to turn.


Although manual control of the tractor in agricultural fields are relatively easy,

autonomous control of the power tiller in such extreme environments is far more

challenging. Therefore it is necessary to build a correct mathematical model for

simulation and to facilitate control algorithm and trajectory planning. Such a

mathematical model was developed and simulated using Mathlab Simulink. First

power tiller model was decomposed in to three subsystems kinematic, dynamic

and drive model. These subsystems are then used to model the power tiller in

the Matlab Simulink environment. For a mathematical model to be used there

has to be trust in it and knowledge that it has been validated. Therefore ground

testing was used to verify the mathematical model.

3.3.1 Kinematic Model

To consider the kinematic model of a power tiller, it is assumed that the power

tiller is placed on a plane in the earth fixed reference frame E (X,Y,Z). This kine-

matic model assumes that no tire slippage occurs, so the inputs to the system are

right and left wheel angular velocities ωr and ωl respectively. In order to deter-

mine the robot motion, it is so important to define the position and orientation

the robot as the location and orientation of the center of gravity, P(x0, y0,θ0)

relative to the fixed world frame. If the Lf is the distance from the rear axle to

the centre of gravity, Tf is the distance between the two driving wheels, Rt is the

radius of a wheel, u, v and r are the forward, the lateral and the yaw velocities,

full kinematic model for the robot can be obtained as follows.

u = (ωr + ωl)Rt

2(3.1)

r = (ωr − ωl)Rt

Tf

(3.2)

v = (ωr − ωl)LfRt

Tf

(3.3)

If vx and vy are the velocity components of the vehicle relative to the world

frame, x0, y0 and θ0 are the position and heading of the robot in the world frame

15


Figure 3.5: Power Tiller dimensions -

Figure 3.6: Power tiller’s position - Orientation and Geometry on X-Yplane

coordinates system assignment

16

3/figures/tractoraxis.eps

3/figures/xycord.eps


as shown in 3.6 then the heading, velocities and position of the power tiller in

the world coordinate system can be obtained.

[vxvy

]=

[cosθ −sinθsinθ cosθ

] [uv

](3.4)

θ0 =

∫rdt (3.5)

x0 =

∫vxdx (3.6)

y0 =

∫vydy (3.7)

3.3.2 Dynamic Model

In order to obtain the dynamic model force exerted on the power tiller should be

identified and analyzed. The forces acting on the power tiller are exerted by left

and right wheels. Rigid body dynamic equations of motion of any 3 degree of

freedom system can be stated as (3.8) – (3.10), with the origin of the body fixed

frame coinciding with the center of gravity (22).

Fx = m(u− vr) (3.8)

Fy = m(v + ur) (3.9)

N = Jr (3.10)

Where m is the mass of the robot, u and v are velocities in forward and lateral

directions, r is the rotational velocity about the z axis, J is the moment of inertia

and Fx and Fy are external forces acting on wheels in forward and lateral direc-

tions and N is moment around centre of gravity. Active and reactive ground

interaction forces acting on wheels of the power tiller are shown in the 3.7 where

F1 ,F2 are tractive forces of each wheel at the ground, FL1, FL2 are lateral forces

and FS1,FS2 are rolling resistance .

3.3.2.1 Traction Forces

Traction forces acting on wheels are the control inputs generated by the driving

torques from the engine and are controlled by the right and left clutches. To

steer the power tiller to right direction right clutch should be disengaged while

engaging the left clutch. Same way to steer the power tiller to left direction,

17


Figure 3.7: Forces acting on power tiller -

left clutch should be disengaged while engaging the right clutch. Relationship

between the engine torque and traction force is considered more detail in the

drive model.

3.3.2.2 Rolling resistance forces

Rolling resistance occurs when a round object such as a tire rolls on surface

and caused by deformation of the object surface or both. Since power tillers are

used for the ground preparation in agricultural field usual operating surface is

often not even. Therefore considerable rolling resistance force opposes the power

tiller forward motion. There are several models that are used to model rolling

resistance and this paper takes more simplicity look at this problem (21). The

force of rolling resistance can be calculated by (3.11),

Fr=CrrNf (3.11)

Where F is the resistant force, Crr is the rolling resistance coefficient or coefficient

of rolling friction (CRF), and Nf is the normal force acting on the wheel.

3.3.2.3 Lateral Forces

Under cornering conditions in which tires must develop lateral forces, the tire will

also experience a lateral slip as it rolls. The angle between its direction of heading

and direction of travel is slip angle α. At low slip angle relationship between the

18

3/figures/Forces.eps


lateral force and slip angle is linear and hence, lateral force can be described by

the proportional constant Cα, known as the cornering stiffness. For a given tire

it is largely depended on load and the inflation pressure(21). Lateral force acting

on a wheel can be calculated by (3.12).

Fl = Cαα (3.12)

3.3.2.4 Dynamic Equations

After considering all the above forces substituting them to (3.13) – (3.15), com-

plete dynamic equations for the power tiller can be obtained.

(F1 + F2)− (FS1 + FS2) = m(u− vr) (3.13)

− (FL1 + FL2) = m(v + ur) (3.14)

N = Jr (3.15)

3.3.3 Drive Model

In a power tiller, driving torque to the wheels are supplied by a diesel engine.

Therefore it is required to model the mechanical system, by which engine power

is transmitted to the ground. Key elements of power tillers power train are shown

in the figure 3.8. Output torque from the engine is transmitted to the gear box

via the main clutch. Torque available at the main clutch can be written as (3.16)

where Tc is the torque at the clutch, Te is the engine torque at a given speed , Ie

is the engine rotational inertia and αe is engine rotational acceleration.

Tc = Te − Ieαe (3.16)

Output torque from the transmission is amplified by gear ratio of the transmission

and is decreased by the inertial losses in the gears and the shaft. Therefore torque

output to the drive shaft can be approximated by (3.17) where Td is the torque

available at the drive shaft, Nt is the numerical ratio of the transmission and It

is the rotational inertia of the transmission as seen from the engine side.

Td = (T c − Itαe)Nt (3.17)

Torque available at the drive shaft is then transmitted to left and right wheel of

the power tiller via two control clutches. Finally drive shaft torque accelerates

19


Figure 3.8: Power Train - This figure shows how engine power is transmitted to

wheels

the wheels and provides traction force at the ground. This can be expressed by

(3.18), where Fx is the total traction force acting on wheels, r is the effective

radius of the wheels, ω is the angular acceleration of the wheels and Iw is total

rotational inertia of the wheels and axle shaft.

Td = (F xr + Iwω) (3.18)

Total rotational inertia of the wheels Iw is not a constant but changes with the

Left Clutch Right Clutch Total Inertia

1 0 ILW

0 1 IRw

1 1 ILW + IRw

0 0 0

Table 3.1: Change of total inertia of the wheels with clutch engage disengage

combinations . Here 1= Engaged Clutch 0=Disengaged clutch .

control clutch input. Therefore right, left clutches engage and disengage combi-

nations change the effective total rotational inertia of the wheel. When analyzing

the wheel dynamics, these different cases should be considered separately. Table

1 shows the change of total rotational inertia of the wheel, with right, left clutches

20

3/figures/powertrain.eps


engage and disengage combinations, where ILW is the left wheel rotational iner-

tia and IRw is the right wheel rotational inertia. Rotational acceleration of the

engine and the transmission are related to the wheel acceleration by (3.19).

αe = Ntω (3.19)

Equations (3.16)-(3.19) can be combined to solve for the tractate force available

at the ground and shown in (3.20).

Fx =TeNt

r−

(Ie + It)Nt

2 Iw ω

r(3.20)

In a power tiller driving torque to the wheels are supplied by a diesel engine.

Relationship between the speed and the output torque of a diesel engine is highly

none linear. Therefore it cannot be modeled by simple mathematical equations.

The engine is modeled by a torque and speed look up table. It provides the

Figure 3.9: Diesel Engine Model - This figure shows how engine power is trans-

mitted to wheels

maximum torque available for a given engine speed Ω. The throttle input signal

T specifies the actual engine torque delivered as a portion of the maximum torque

achievable in a steady state at a fixed engine speed. It modulates the actual torque

delivered τ from the engine. The model accepts the throttle signal through as

a input . This signal specifies the engine torque as a fraction of the maximum

torque possible in a steady state at a fixed engine speed and should lie between

21

3/figures/diesel.eps


0 and 1. The throttle signal directly controls the output torque that the engine

generates and indirectly controls the speed at which the engine runs. The actual

engine drive shaft speed, Ω is fed back to the engine input.

3.3.4 Simulation

Complete mathematical model has been derived for each subsystems of the power

tiller; drive, dynamic and kinematics . Complete simulation model was developed

using Matlab§Simulink environment. Inputs to the simulation model are Right

clutch signal, Left clutch signal and throttle signal. Following parameters were

taken for the simulation. The structure of the simulation model is shown in Fig.8.

Following parameters were taken for the simulation. Total mass of the power tiller

m = 350 Kg, J = 100 Kg /m2, Iw = 50 Kg /m2, Rt = 0.3 m , Lf = 0.3 m ,

Tf = 0.6 m. Rolling resistance coefficient Crr =0.05. Left clutch and right

Figure 3.10: Structure of the simulation model -

clutch are the main input for the simulation model. Drive model outputs traction

force of the each wheel whereas Dynamic model outputs longitudinal, lateral and

rotational acceleration relative to the body fixed reference frame. Kinematics

22

3/figures/simulation.eps


model output velocity components of the vehicle relative to the world frame, and

integrated to get the position of the power tiller in the world coordinate frame.

3.3.5 Model Validation

Validation is required to prove that the model is a good representation of the

physical power tiller, without accurate validation the model is of little use and

the results produced will have no value (23). Two validation methods can be

used for the effective validation of simulation data and actual data.

3.3.5.1 Analogue Matching

Analogue matching or visual inspection is an established method of model val-

idation, where the output from the model is compared to experimental data

graphically by superimposing the plots. As the model is altered the output is

compared to that of the results from the physical power tiller. The data that

‘best fits’ the experimental data indicates the model is a valid representation of

the physical power tiller.

3.3.5.2 Maneuvers

To record the actual data for the model validation, set of simple maneuvers were

carried out on the power tiller using the remote controller. The maneuvers chosen

were:

1. Straight line for 20s in third gear with both left and right clutches engage.

2. Disengage the right clutch while keeping the left clutch engaged for 5 s

allowing power tiller to change its direction.

3. Engage the right clutch while keeping the left clutch engaged and guide the

power tiller forward for 25 s.

An IMU was fixed on the center of gravity of the power tiller and heading data

was logged, Path of the power tiller was marked on the ground and later measured

and used as the experimental data for model validation. A snapshot of the actual

path traveled by the power tiller is shown in the figure 3.11.

23


Figure 3.11: Ground tetsing of power tiller - Figure shows actual path travelled

by the power tiller

3.3.6 Results

For the subsequent controller development knowledge of the heading angle and

the position are required. Analogue matching of heading and position of the

power tiller for above discussed maneuvers are used to validate the model. It can

be seen from the results that the model is a good representation of the physical

power tiller. However some discrepancies exist, this is because disturbance forces

acting on the power tiller cannot be modeled with sufficient accuracy. The figures

3.12-3.15 show the experimental data, the simulation data and the error between

them.

24

3/figures/testing.eps


Figure 3.12: Clutch signals - These signals simulate engage and disengage com-

binations of clutches

Figure 3.13: Position of the power tiller in world coordinate frame - Graph shows

actual,simulated and error of the position

25

3/figures/clutches.eps

3/figures/position.eps


Figure 3.14: Change of heading angle - Graph shows actual,simulated and error

of the heading

26

3/figures/heading.eps


Figure 3.15: X and Y position errors between actual and simulation results - (a)

X Position error , (b) Y Position error

27

3/figures/xyposition.eps

4

Visual Odometry Based Relative

Localization

One of the most challenging problems of field robots is self-localization. Relative

localization systems involve incremental update of position while in motion and

used when high accuracy and resolution is needed in short-run. Wheel odometry

and use of Inertial Measurement Unit (IMU) are the mostly used relative local-

ization systems.(6)-(7). Wheel odometry relies on the integrity and accuracy of

a kinematic model which is very difficult to assume in uneven out-door terrains.

Also its accuracy degrades rapidly when wheels tend to slip. On the other hand

inertial measurement can not be used reliably when the dynamics of the field

robot is slow. Therefore it can not be used for self-localization for autonomous

vehicle in agriculture as these vehicles move in the field at slow speeds. Also for-

ward dynamics can be easily distorted by the dynamic effects caused by uneven

terrains. As an alternative visual odometry can be used which does not rely on a

kinematic model of the vehicle.(24). This involve use of one or more cameras to

acquire images from the surrounding which are used to estimate relative change

in pose of the robot. The two most popular visual odometry estimation use op-

tical flow field based on feature tracking (26) and planar translations between

images using phase correlation in frequency domain (24). In this experiment it

was required to identify the best visual odometry method that can be used for

self localization of the two wheel tractor. Therefore basic navigation experiments

including loop-closing and error propagation were done to analyze the error in

view of frame size and frame rate.

28

4.1 Visual odometry using FFT based Image Registration.

4.1 Visual odometry using FFT based Image

Registration.

Visual odometry relies on estimating the pose of a moving object by tracking rel-

ative movement against fixed features of the surrounding using a video camera.

Therefore any visual odometry system should utilize image registration technique

that can estimate relative translative movement between consecutive frames cap-

tured from a video sequence. One such a widely used image registration technique

is phase correlation in frequency domain using Fast Fourier Transform (FFT).This

involves use of fast frequency-domain approach to estimate the relative transla-

tive offset between two similar images. The following figure 4.1 demonstrates

the usage of phase correlation to determine relative translative movement be-

tween two images corrupted by independent Gaussian noise. The image was

translated by (30,33) pixels. When the phase correlation is done in frequency

domain a peak is created approximately at (30,33) which equals the translative

offsett of the two images. The same principle can be used in visual odometry

Figure 4.1: Phase Correlation using FFT - The image was translated by (30,33)

pixels. When the phase correlation is done in frequency domain a peak is created

approximately at (30,33)

by acquiring images of the ground using a downward faced camera. When the

vehicle is in motion two subsequent frame from the video will contain two similar

images with some translation that corresponds to the vehicle forward or lateral

movement. This translative movement then can be calculated by using phase

correlation frequency in frequency domain. This motion updates overtime and

provides relative position and heading estimates enabling the vehicle to track its

position. This kind of techniques is far more suitable for outdoor terrains where

29

4/figures/Phase.eps


ground surface texture is not unique. Accuracy and speed of image registration

method determines the overall success rate of the proposed localization method.

Frequency-domain methods find the transformation parameters for registration

of the images while working in the transform domain. Such methods work for

simple transformations, such as translation, rotation, and scaling. Unlike many

spatial-domain algorithms, the frequency domain phase correlation method is re-

silient to noise, occlusions, and other defects. Additionally, the phase correlation

uses the Fast Fourier Transform (FFT) to compute the cross-correlation between

the two images, generally resulting in large performance gains. Advantage of FFT

based approach is that it is fully autonomous and requires no user interactions.

This makes FFT based image registration very fast and robust (24). Most of

the commercially available cameras capture image at a rate of 20˜30 frames per

second (FPS) leaving a less than 50 ms time window between subsequent frames.

Therefore to effectively utilize the camera hardware, it is required to process the

image and obtain the results within 50 ms. Therefore FFT based image registra-

tion should be fast enough to register to frames within 50ms. The Computation

behind the phase correlation is shown below. Given two input images, ga and gb

then the discrete 2D Fourier transform of the both images (Ga and Gb) can be

obtained as follows.

Figure 4.2: Two images captured from consecutive frames -

Ga = Fga (4.1)

30

4/figures/ground.eps


Figure 4.3: The phase correlation of two frame - Impulse corresponds to the

location of maximum translation. The indices of the peak are then manipulated to

represent offset from (0,0)

Gb = Fgb (4.2)

Then cross power spectrum of images can be obtained by taking the complex

conjugate of the second result ( G∗

b), multiplying the Fourier transforms together

element-wise, and normalizing this product element wise.

R =Ga G∗

b

|Ga G∗

b |(4.3)

Inverse Fourier transform then can be applied to obtain the normalized cross

correlation of (4.9).

r = F−1R (4.4)

Location of the peak is then equal to the translation in X an Y direction. (∆X,∆Y).

(∆X,∆Y) = max(x,y)r (4.5)

By applying 4.1 -4.5 to images shown in figure 4.2, phase correlation of two images

can be obtained. At the location of maximum translation a impulse is created

as shown in figure 4.3. The correlation value represents the matching percentage

in which value 1.0 correspond to exact match. The indices of the peak then

correspond to translation in X an Y directions.

31

4/figures/graph.eps

4.2 Visual Odometry Using Feature Tracking

4.1.1 Heading Angle Calculation

Longitudinal movement of the rover is more significant than that of lateral move-

ment of the rover unless wheel slips. To accurately position the rover we need to

calculate the heading angle corresponding to each pose. But image registration

technique provides only two measurements, translation movement in X and Y di-

rections. But by positioning the camera with an offset to the wheel base as shown

in figure 4.4, X direction image translation can be easily converted to change of

heading angle. The localization system receives information about the rover’s

Figure 4.4: Calculating heading angle - By positioning camera with an offset to

the wheel base , X direction translative distance can be easily converted to change

of heading angle

driving intention (or state) so that it can filter out improbable rotation angles as

they are computed. This allows the system to achieve higher levels of accuracy

by not considering changes that are obviously wrong (e.g., vibration and pitching

of rover due to unevenness of the ground). Also it is required to minimize the

lateral slip, when the rover turns to minimize computation errors.


The other popular visual odometry estimation use optical flow field based on

feature tracking . In this approach comparatively distinguishable features are

32

4/figures/heading.eps


identified from the environment and tracked across video frames inorder to esti-

mate the motion of the robot. In feature tracking algorithms computation burden

is mainly determine by number of feature points to track. Therefore optical flow

field estimation using feature tracking is more suitable for real time position

update. Also since each feature is tracked individually statistical data filtering

techniques can be applied easily. Over the years many optical flow estimation

techniques have been developed with their own strengths and weaknesses. Out

of these Lucas Kande method (28) found to be more robust and accurate over

other. Also much improved version of this algorithm is available with OpenCv

image processing library. As in the frequency domain method a camera was fixed

facing downward to the ground and video sequence was recorded when the robot

moves. Then Lucas Kande optical flow estimation technique was used with sta-

tistical data filtering to incrementally update the pose of the robot. Optical flow

generated by Lucas Kande method is shown in figure 4.5.

Figure 4.5: Optical Flow Field - Figure shows optical flow field generated on a

video frame captured from ground

33

4/figures/vocam.eps


4.2.1 Feature extraction and correlation

Shi/Tomasi algorithm (32) for feature extraction is used to detect features in

each video frame. Intuitively, a good feature needs at least ,texture (ambiguity

in tracking) and Corner (or aperture problem). Since out-door ground texture

is not unique Shi/Tomasi algorithm can be used effectively to detect good fea-

tures. When good features are identified LucasKanade optical measurement can

be used to estimate motion vector. The Lucas-kanade algorithm is easily the

most popular method of tracking movement. The LK tracker uses three as-

sumptions, brightness constancy between the same pixels from one frame to the

next, small movements between frames (requiring image pyramids to track larger

movements), and spatial coherence, thats points near each other are on the same

surface. Then the basic concept of the tracker is to estimate the velocity of a

moving pixel by the ratio of the derivative of the intensity over time divided by

the derivative of the intensity over space.

4.2.2 Determination of Vehicle Orientation

Figure 4.6: ICR detection and calculating heading angle - When the robot rotates

ICR point cloud is created. (a) ICR detection, (b) ICR point cloud.

Orientation of the robot is estimated using optical flow patterns. It is noted

that optical flow field creates an Instantaneous Center of Rotation (ICR) when

robot is rotating. If the ICR is estimated then the robot heading changes can be

34

4/figures/icr.eps

4.3 Single Camera Visual Odometry

calculated. It is determined by finding the intersection point of each flow vector’s

bisecting line for consequent frames as shown in the figure 4.6. But all these lines

do not intersect at a single point. Instead they create point cloud that consists

of series of ICR points as shown in 4.6.(b). Therefore median of x positions and

median of y positions of ICR point clouds are taken as the probable estimation of

the ICR. Once the ICR is calculated heading change of the robot is calculated by

dividing the hypotenuse of each tracked point optical flow vector by the distance

to ICR.


4.3.1 Hardware Setup

To test the performances of the two visual odometry techniques discussed above

series of experiment were done in out-door environment using a commercially

available toy rover which has a maximum drive speed of 30 cm/sec. A digital

camera has been fixed facing downwards at back of the rover at a height of 50 cm

from the ground. With this height camera field of view was 23 x 17 cm surface

patch, and with the cameras image resolution, the horizontal and vertical dis-

tances per pixel can be computed. Thus, we can compute both distance traveled

from the pixel offsets computed on a frame-by-frame basis using image registra-

tion techniques and relative position on the XY plane. For this setup, which

records images at 640 x 480 resolutions, calibration factors are 0.074 cm/pixel

along the X direction and 0.072 cm/pixel along the Y direction. Image process-

ing software also contributes considerably to the speed of image registration. It

was found that satisfactory speed can be obtained by using Intels OpenCv Im-

age processing library with C++ compiler. Libraries that facilitate both feature

tracking and phase correlation in frequency domains are readily available with

OpenCv.

4.3.2 Odometric model

A simplistic odemetric model is employed to update rover’s position in each frame.

In order to determine the robot motion, it is important to define the position and

orientation the robot as the location and orientation of the center of gravity,

P(x0, y0, θ0), relative to the fixed world frame. For the rover the position can

35


Figure 4.7: Rover platform with downward faced camera - Images captured from

the camera were used to calculate pose of the rover

be estimated starting from known location and summing the incremental travel

distance in X direction (∆x), Y direction (∆y).If the calculated change of heading

(∆θ)then the updated position P′

can be obtained as follows.

P′

= P +

∆s cos(θ+∆θ)∆s sin(θ+∆θ)

∆θ

(4.6)

4.3.3 Test Results

A series of experiments were carried out to find out the performance of the pro-

posed localization system. Video was captured and processed off-line to calcu-

late heading angle and longitudinal travel distance for each subsequent frame.

OpenCv Image processing library with C++ compiler was used to compute said

data. Only a limited amount of ground truth information was recorded for the

experiments, such as distances, final rover heading, and knowledge of the general

path the robot traveled during the experiment.

4.3.3.1 Experiment 1: Effect of frame rate

For this experiment initially video sequence was recorded at a rate of 30 FPS.

For both frame registration techniques to work there should be common features

or common areas in a subsequent frame pair. If the robot moved too quickly,

36

4/figures/hardware.eps


depending on the height of the camera there can be a possibility that two sub-

sequent frames do not satisfy the above requirement. Then image registration

algorithm will fail to calculate the translative distance the robot traveled. In

this experiment different speeds are emulated by varying the frame rate of the

recorded video. 30 FPS was selected as the base speed. 15 FPS emulates twice

the speed of the base speed. To study the effect of the frame rate/speed the rover

was driven 10m forward straight line and video sequence was recorded. Then

this video sequence was processed off-line in order to analyze the performance

of the two image registration techniques namely, image registration using feature

tracking and phase correlation using FFT. The results are shown in the figure 4.8.

It can be seen from the results at high frame rate (slow speeds) both FFT based

Figure 4.8: Effect of frame rate for visual odometry - Frame rate emulates change

of the speed

image registration and feature tracking algorithms give better results. Even at

15 fps feature tracking algorithm gives acceptable accuracy. Overall performance

of feature tracking algorithm is better than the FFT based image registration. It

also depicts feature tracking algorithm is more resilient to change of speed than

the FFT based image registration.

4.3.3.2 Experiment 2: Effect of frame size

In experiment 3 effect of the frame size of the captured video is analyzed by

driving robot in a straight line for a distance of 10 m. The results are shown in

4.9 and 4.10. If the image registration or feature tracking algorithm can complete

37

4/figures/framerate.eps


odometric calculation fast, then more frames can be acquired and processed in a

given time which will eventually increase the accuracy. The maximum frame

Figure 4.9: Effect of frame Size for visual odometry - When the frame size of the

image get reduced processing time per frame also get decreased

rate available with commercially available camera is 30fps. Therefore to utilize the

bandwidth of the camera image processing algorithms should be able to complete

its operation with in 33.3 ms which only can be achieved for frame sizes less than

160X120 pixels as shown in the figure 4.9. But according to the figure 4.10 for

both algorithms at 160X120 pixels frame size the error is unacceptable. Therefore

optimum window size would be 320x240 pixels in terms of both image processing

time and accuracy. At the frame size of 320X240 feature tracking algorithm gives

better results than FFT based image registration both by accuracy and processing

time.

4.3.3.3 Experiment 3: Error Propagation

To analyze the error propagation of the proposed odometric model robot was man-

ually driven 10m straight line and calculates the error for each 50 cm distance.

Such a experiment was done to analyze the error accumulation over distance. Er-

ror propagation will determine the maximum distance the robot can be localized

successfully. Obtained results are shown in the figure 4.11 One of the primary

disadvantages of any odometry model is that small motion estimation errors will

compound over time casing large error over long distance. Same is evident from

38

4/figures/resolution.eps


Figure 4.10: Effect of frame size vs visual odometry error - At 160X120 resolution

frame size error has increased significantly

Figure 4.11: Cumulative error propagation - It can be seen Visual Odometry

error propagate both positive and negative directions

39

4/figures/reserror.eps

4/figures/errorpropogation.eps


the fig.4.11. But unlike wheel odometry in which error propagation is one sided in

visual odometry error propagate both the directions. Therefore visual odometry

would become more accurate than wheel odometry even at long run though error

accumulation is still a primary disadvantage.

Figure 4.12: Result of loop closing test - Loop was not closed mainly due to

errors in heading angle.

4.3.3.4 Experiment 4: Loop-Closing

The success and validation of most localization systems is determined by a loop-

closing experiment. To test this 10m X 10m square path was defined and rover

was carefully maneuvered along that and video sequence was recorded using the

downward faced camera. Calculated results of the loop-closing experiment are

shown in the figure 4.12. For both the algorithms the loop was not successfully

closed, mainly due to errors in heading angle. Therefore robust method of calcu-

lating heading angle is required in order to get good visual odometry results.

4.3.4 Summary

Test results prove that feature tracking algorithm is better than FFT based visual

odometry both in term of accuracy and speed. It is clear that real time processing

which utilizes maximum available frame rate of the camera, can be achieved

using feature tracking algorith in 320X240 pixels resolution. As in any odometric

40

4/figures/loop.eps

4.4 Multi Camera Visual Odometry

model visual odometry error also accumulate over time. Therefore it is required

to combine visual odometry data with absolute localization in order to get better

results.


Figure 4.13: Optical flow field estimated by the two cameras mounted on the

rover - (a) Video frame from downward faced camera , (b) Video from the forward

faced camera

In this second approach we build a a visual odometry system that can be

used to localize an outdoor skid-steering field robot using on board multi- cam-

era system. Intention was to increase the accuracy of the visual odometry results

by fusing the visual odometry data from two cameras. Most of the outdoor vi-

sual odometry based systems are based on using forward looking camera (25)

or downward looking camera (26). Forward looking camera provides more infor-

mation than downward faced camera but it is less sensitive to small movements

and can be easily obscured by environmental changes such as lighting effects. On

the other hand the accuracy of visual odometry system that use downward faced

camera (26) deteriorates when the robot is moving fast causing difficulties to

match points between two consequent frames. Therefore in the proposed system

we used two consumer grade cameras, one faced downward with right angle to

the ground and other facing forward with a small tilt to the ground plane. Data

41

4/figures/twocam.eps


gathered from optical flow fields was fused using Kalman Filtering for better ac-

curacy. Also real-time Instantaneous Center of Rotation (ICR) detection using

optical flow field data is used to calculate the heading angle

4.4.1 Hardware setup

Figure 4.14: Skid steering platform used for ground testing - Two cameras were

mounted one looking downwards and one looking forward

To test the proposed system a commercially available toy rover was used. Max-

imum drive speed is 30 cm/sec. A digital camera was fixed facing downwards

at back of the rover at a height of 50 cm from the ground. With this height

camera field of view was 23 x 17 cm surface patch, and with the camera’s im-

age resolution, the horizontal and vertical distances per pixel can be computed.

Projection of image plane to ground plane the pixel offsets computed on a frame-

by-frame basis using image registration techniques and relative position on the

XY plane. For this setup, which records images at 320 x 240 resolutions, calibra-

tion factors are 0.074 cm/pixel along the X direction and 0.072 cm/pixel along

the Y direction. Though it is more appropriate to transmit the captured video

wireless to the computer some video noise was experienced when tried through

a wireless transmitter. Therefore it was later decided to record the video from

the downward faced camera and process off-line to test the performances of the

algorithm.

Image processing software also contributes considerably to the speed of image

registration. It was found that satisfactory speed can be obtained by using Intel’s

OpenCv Image processing library with C++ compiler. This software was able to

42

4/figures/skid.eps


register a subsequent frames (320 x 240 pixels) captured from the camera with

in 50 ms when run on a 1.8 Ghz Intel Core Duo processor.

4.4.2 Estimating Robot Motion

4.4.3 Mapping Camera Coordinates to Ground Plane

For each tracked feature it is required to map the camera coordinate (u,v) to

ground plane(x,y). In a previous study (27) simple method has proposed to match

camera coordinate to ground plane. The tilt of the forward looking camera is

calculated using the height (H) of the camera from ground plane and the distance

(D) from the robot to the intersection point of ground and principle ray.

tanα =H

D(4.7)

Pinhole camera model is used to calculate the angle β which is the angle between

the principle ray and the tracked feature (u,v) in the camera coordinate system.

If the vertical dimension (V), of the camera and the vertical field of view (VFOV )

is known β can be calculated as follows.

tan = (2v − V ) tan

(V FOV

2

)/V (4.8)

When the β is known, y distance to a feature point can be calculated using the

equation (4.9).

y =H

tan(α+ β)(4.9)

4.4.4 Estimating Translation Movement

Estimating of the translation distance of the robot is done by back projecting each

point in the camera coordinates system to the ground plane. This will correct

the distortion created by camera. For an ideal image sequence, the resulting flow

field would consist of identical vectors; in reality, the flow field is perturbed by

unmodeled effects such as tracking error, variations in the height of the ground

surface. To take a robust estimate for the displacement, optical flow data from two

cameras are fused together for a better result using basic Kalman filter method.

The basic Kalman filter method allows measurement from two cameras to be

43


Figure 4.15: Projection of image plane to ground plane -

incorporated optimally into a single estimate of state. Since two cameras acquire

images of the ground simultaneously we can assume the robot does not move

between two measurements. Therefore static Kalman Filter estimation can be

applied for two camera readings. If the downward camera optical flow field reading

has mean value of q1 and variance of σ21 and forward tilt camera has mean value

of q2 and variance of σ22 then the best estimate for the robot (q) pose can be

obtained using Kalman Filter by following equation.

q= q1 +σ21

σ21 + σ2

2

(q2 − q1) (4.10)

In (4.10) as each measurement is weighted by inverse of variance even a poor

measurement will only increase the precision of an estimate.

4.4.5 Algorithm

The algorithm used in the proposed system is an 8 stage process. When video

frames are captured from two cameras these steps are executed in sequence to

calculate the incremental position of the rover. The algorithm was coded using

C++ and OpenCV and was capable of processing data at a rate of 15 fps. The

main steps of the algorithm are shown below.

1. Capture the images from 2 cameras and correct the lens distortion.

44

4/figures/camco.eps


2. Identify significant ground features and obtain the optical flow fields by

tracking them across consecutive frames.

3. Project image coordinate of flow vector to the ground plane coordinate. By

using this transformation, feature displacement in X and Y direction can

be calculated.

4. Calculate statistical data (mean, variance and median) of each camera’s

optical flow field and fuse two camera data using Kalman Sensor fusion.

5. Determine the ICR and calculate the heading changes

6. Add incremental measurements to calculate the robot current position rel-

ative the starting position.

4.4.6 Testing and Results

To test the performance of the visual odometry algorithm skid- steering vehicle

was driven in an outdoor terrain. Captured video was then processed off-line

to calculate heading angle and longitudinal travel distance for each subsequent

frame. Only a limited amount of ground truth information was recorded for the

experiments, such as distances, final rover heading, and knowledge of the general

path the robot traveled during the experiment.

4.4.6.1 Variation of Uncertainty of Measurements

Measurements from two cameras are fused using basic Kalman filter method. The

performances of the Kalman filter can be best analyzed by comparing the vari-

ances of the measurements for two cameras frame by frame. As shown in (4.10)

optimum measurement is obtained by weighting the each camera measurement

by the inverse of the measurement variance. Therefore measurement with least

variance becomes dominant over the measurement with high variance. Variance

models the uncertainty of the measurements. To analyze the variation of uncer-

tainty robot was run over 10m both at low and high speeds. Measured uncertainty

for these different speeds is shown in figures 4.16 and 4.17. It is quantitatively

modeled as variance for each frame. It can be seen from the results that the

uncertainty of the downward faced camera increase when the robot is run at high

speed. At low speed downward camera gives good results as its variance is low

45


Figure 4.16: Variation of the uncertainty at high speed -

Figure 4.17: Variation of the uncertainty at low speed -

46

4/figures/highspeed.eps

4/figures/lowspeed.eps


with comparison to that of the forward faced camera. At high speed increase

uncertainty of the downward faced camera is compensated by the measurement

of the forward faced camera. This proves multi camera system gives better results

than single camera system both at low and high speed runs of the robot.

4.4.6.2 Loop Closing Test 1

Figure 4.18: Results of Loop Closing Test 1 - View of the terrain as seen from

the forward faced camera also shown in the bottom)

Since it is required to find the performance of the algorithm skid steering vehicle

was tested on several outdoor terrains. In order to validate the results robot

was run on a closed loop. Since in a closed loop starting and ending points are

same ,pose difference between these two points gives good indication about the

error in localization. In Wheel odometry calculations differential drive system

was assumed to ease the calculation of heading angle changes. In the loop test 1,

skid steering vehicle was run in a closed loop over 30-35 m. Video was recorded

30fps. The closed loop was consisted of five turns. Robot was started from (0,)

position and final pose was (2.19,-1.19) m. The results of the loop closing test 1

are shown in figure 4.18. It can be seen from the results wheel based odometry

generate a large error mainly due to heading angle errors.

47

4/figures/loop1.eps





Loop closing test 2 was done over approximately long distance. The skid steering

vehicle was run on a much complex route. It was consisted of an inner loop and

both clockwise anticlockwise turns. The experimental results are shown in figure

4.19. Robot was started from (0,0) position and final position was (-0.879,-0.80)

m.


Loop closing test3 was done on cut grass approximately over 30 m. The accuracy

of feature tracking based visual odometry is depended on the ability of feature

tracking algorithm to differentiate features from the background. On cut grass

terrain features are more or less same from the background. The results of the

loop closing test 3 are shown in figure 4.20. Robot was started from (0,0) position

and final pose was (-6.08, 9.80) m. . In cut grass visual odometry algorithm has

failed to uniquely identify ground features. Therefore visual odometry error has

increased. Also from the results it can be seen that uncertainty of the downward

faced camera has increased though robot was run on slow speed. It is a indication

that downward faced camera has failed correctly calculate incremental distance

48

4/figures/loop2.eps




traveled. .

4.4.7 Summary

Following table shows percentage errors of loop closing test for visual odometry

system. In test 3 where the robot has been run on cut grass error is significant

over others. This is because across video frames visual odometry algorithm failed

to uniquely identify features. But visual odometry system has shown acceptable

accuracy when the algorithm can distinctively identify features from the back-

ground. Experimental results show that proposed visual odometry model gives

Total Distance (m) xF inal(m) yF inal(m) Cartesian Error (m) Precentage Error

Test1 32.6 1.49 2.19 2.64 8.13

Test2 49.1 -1.76 -0.8 1.93 5.93

Test3 35.5 9.8 -6.08 11.53 35.39

Table 4.1: Performances of the loop closing test results (Multi-camera Visual

Odometry).

better result with comparison to wheel odometry for skid steering vehicle. From

the results it is clear that at slow speeds, data captured from downward faced

49

4/figures/loop3.eps


camera become predominant over the forward faced camera. This is because at

low speeds downward faced camera provides good features to track for optical flow

estimation. At high speeds downward camera fails to track features as its field

of view is limited. Therefore optical flow data read from the downward faced

camera become erroneous at high speed. Forward faced camera has relatively

large angle of view with comparison to the downward faced camera. Therefore,

even at high speed feature points move slowly across the forward camera view

causing data read from forward face camera to predominant. All three tests were

done at slow speed. Therefore uncertainty of the measurements of the forward

faced camera is significant. It was required to detect ICR in order to calculate

the heading angle of the Robot. Statistical data filtering was done to correctly

identify the ICR. For skid steered vehicle where lateral skidding is significant to

correctly localize the vehicle it is required to find the accurate position of the

ICR. Also it was observed that proposed ICR detection method can be use to

correctly calculate lateral and forward slip. This is very important when the field

robot is moving in a loose surface like sand. But correct heading angle can be

obtained using a digital compass if better accuracy is needed. Also it was noted

that if the views of the cameras are obscured by the own shadow of the robot

the accuracy of the system degrade quickly. This is because there is no relative

movement between the robot and its own shadow. In such inevitable situations it

is recommended to use artificial lighting even at day light. As in any odometric

model, in the proposed visual odometry system the error accumulates over time.

Therefore for long run it is required to combine visual odometry with absolute

localization systems such as GPS.

50

5

Stereo Vision for Absolute

Localization

In the previous chapter visual odometry is introduced as an alternative to wheel

based odometry for out-door localization of autonomous vehicles in short-run.

Accurate localization is necessary for navigation and for controlling the robot in

its environment, in which the degree of accuracy required being governed by the

application.

For many mobile robot environments, the pose can be simplified by estimating

only the Cartesian position and orientation. Therefore a traditional mobile robot

is equipped with wheel encoders, which measure the robots incremental distance

traveled, as well as gyroscopes or accelerometers, which measure the incremental

change in the robots heading or video cameras that track path traveled by the

robot. These are called proprioceptive measurements, since they estimate the

robots relative motion from onboard sensors rather than measuring from some

absolute point of reference. The process of integrating these proprioceptive mea-

surements over time to maintain an estimate of the robots pose is known as dead

reckoning. Dead reckoning localization is easily achieved and can provide high

accuracy for short paths. However Dead reckoning alone will fail for most real-

world applications, since it inherently provides no means of correcting error and

therefore suffers from unbounded error accumulation. This is clearly depicted

from the visual odometry results discussed in the previous chapter.

To improve visual odometry results , there should be some means to sense

the robots pose absolutely, with reference to a fixed point or points. Absolute

sensor measurements are called exteroceptive, and absolute localization requires

the robot to sense features in its environment and estimate its pose in the world

51

coordinate frame directly based on these (usually fixed) reference positions. In

this paradigm, localization at any point in time does not require knowledge of

the prior positions of the robot, and error in localization is determined only by

the noise and bandwidth of the sensors. GPS is a common means of absolute

localization. Many out-door autonomous robots including autonomous vehicles

in agriculture rely mainly on GPS. Many low cost GPS units are available but even

when many satellites are visible their accuracy is in the range of 5m. Therefore

though they are suitable for commercial applications like map navigation it can

not be used for full autonomous navigation. There has been previous attempts

that use differential GPS combined with Inertial Navigation Systems for accurate

positioning with an error of few centimeters and hundredths of degrees (6). But

GPS may be unavailable or become less effective when only few satellites are

available in urban or remote areas.

Beacons based localization that includes RF or optical beacons are used as

alternative to GPS in relatively small operating areas. But these systems need

to fix beacon emitters or reflectors in several locations which require excessive re-

arrangement when the operating environment change.(29). Range measurement

using time of flight can reduce the number of beacons require but need expensive

hardware which discourage use of it in low cost applications. Because of these

constrains , we look for some means to sense the robots pose absolutely, with

reference to a fixed point by using stereo vision. Stereo vision is used over the

years to extract depth measurement from two image pairs obtained by two cam-

eras placed with some distance. As the range measurement from stereo camera

system is intended to bound the incremental error caused by the visual odome-

try, ±1m accuracy is believed to be sufficient for stereo range measurement unit.

Other advantage is same hardware used by the visual odometry system can be

used by the stereo range measurement system without much added cost which

reduce overall cost of the system. Stereo measurement system was capable of mea-

suring range distance to objects less than 15 m with acceptable accuracy. For

this experiment two low cost web cameras were used as stereo pairs and Intel’s

OpenCv image processing library was used to track and calculate the distance to

the object.

Fundamental problem of range measurement using stereo pair is the object

should lie on the common area of the two cameras in-order to measure the dis-

parity. This reduces the area that the object (robot) can lie. Therefore as the

object move camera’s fixed coordinate system also should rotate with the object.

52

5.1 Stereo Camera Geometry

This was achieved by fixing the stereo camera pair on a platform that is rotated

by a servo motor.


To derive the geometry of an ideal stereo camera, geometry of a single pin-hole

camera shown in figure 5.1 is considered. Suppose point P with coordinates (x,y )

is the image of point P in the scene with coordinates (X, Y, Z). If O is the camera’s

lens center, h is the camera’s piercing point with coordinates (x0, y0)(this is the

point where the principal ray intersects the x-y plane), f is the focal length of the

camera, XYZ is the world coordinate system with origin O and the X-Y plane is

parallel to the image plane. Then from the perspective rule we have centers of

two cameras.X

Z=

x−x0

f(5.1)

Y

Z=

y − y0f

(5.2)

Now, suppose two such cameras are separated horizontally by 2B distance

Figure 5.1: Pin-hole camera geometry - Figure shows coordinate transformation

from camera’s reference frame to world fixed coordinate system

with their optical axes parallel and normal to the baseline as shown in figure

53

5/figures/pinhole.eps


5.2. Suppose Ol, and Orare the left and right camera lens centers, respectively,

and the world coordinate system is midway between the two cameras with the

Z-axis parallel to the optical axes of the cameras. Therefore, the left and right

camera lens centers are at (-B,0,0) and (B,0,0). Also Suppose hl, and hr, are the

left and right cameras piercing points with coordinates (x0l, y0l) and (x0r, y0r). P

[X. Y. Z) is a point in the scene whose projections of the left and right image

planes are pl and pr , with coordinates (xl, yl) and (xr, yr) respectively. If both

cameras have the same focal length f then using relations (5.1) and (5.2) relation

between a point in the scene and its images in the left and right cameras can be

correspondingly written as

Figure 5.2: Stereo camera geometry - Figure shows coordinate transformation

from camera’s reference frame to world fixed coordinate system

X+ B

Z=xl−x0l

f(5.3)

Y

Z=

yl − y0lf

(5.4)

X −B

Z=

xr − xor

f(5.5)

Y

Z=

yr − y0rf

(5.6)

yl − y0l=yr − y0r (5.7)

54

5/figures/stereo1.eps

5.2 Hardware setup

yl=yr (5.8)

Relation (5.8) shows that if the piercing points or the two cameras appear on the

same scan line in the images. Corresponding points will lie on the same scan line

in the images. From equations (5.3) and (5.5) we find

f

z(x+B) + x0l=xl (5.9)

f

z(x+B) + x0r=xr (5.10)

The horizontal displacement between corresponding points in the images, known

as the disparity, may be computed by subtracting equation (5.10) from equation

(5.9). From equation (5.11) the relation between the depth of a point in a scene

and the disparity between its images may be written as

xl−x0l=2Bf

z+(x0l−x0r) (5.11)

z =2Bf

(xl−xr)−(x0l−x0r )(5.12)

Relation (5.12) shows that depth is inversely proportional to Image disparity

and for a given depth, the larger the baseline length the larger the disparity.

Therefore, baseline length may be increased to improve the accuracy of measured

depth.

5.2 Hardware setup

Stereo camera system used in the experiment is consisted of two low cost web

cameras having resolution of 640X480 pixels. They are fixed on a platform with

baseline distance of 40 cm. Stereo range measurement will work only if the object

lie on the common area of two cameras. Therefore when the target object (Robot)

moves on the field , it should be tracked and stereo camera system should adjust

accordingly. This is achieved by fixing two cameras on a platform that can be

rotated by a servo motor. When the object is detected by cameras, it position

is estimated. If the object starts to move servo motor starts to rotate according

to the position information received from the image processing software. Image

processing is done on a Intel Core Duo computer using OpenCv image processing

libraries. Servo controller is connected to the computer via a RS 232 serial link

which sends position information to the servo. Figure 5.3 shows the rotatable

stereo camera system mounted on a tripod .

55

5.3 Object Tracking and Stereo Image Correspondence Process

Figure 5.3: Rotatable stereo camera system - Web cameras are used as stereo

pairs and servo motor rotate the platform while keeping the moving object in the

common field of view of the cameras

5.3 Object Tracking and Stereo Image Corre-

spondence Process

The major step in the stereo depth perception is the computation of correspon-

dence between points in the two stereo image pair. This step is known as the

correspondence process which involves matching feature points in two stereo im-

ages. In this experiment our scope was to measure the range distance to a moving

object in order to localize it in the world coordinate frame. To reduce the com-

plexity of the correspondence process colored spherical tag is attached to the

robot. By tracking the position of the color tag position of the moving robot was

tracked. Main steps of the correspondence matching and tracking algorithm is

shown below. It was assumed that there aren’t any objects with the same color

as color tag in the environment.

• Capture two images using the two web cameras.

• Use a color filter to extract image points having the same color as the color

tag.

• Smooth the image in small window to filter noise.

56

5/figures/setup.eps

5.4 Camera Coordinate Transformation

• Find the point Pmax with maximum intensity of the smoothed image on the

right image.

• Extract a 20X20 pixel image template around the point Pmax.

• Match the template with left image. If a matching window is found its

center position coordinates are extracted.

• Verify whether matched windows center is located on the same scan-line as

the template image’s original position .(If the cameras are perfectly aligned

, then correspondence points should lie on the same scan line). If the two

point are not in the same scan-line repeat the correspondence process from

step 1.

• If the above verification is succeeded calculate the disparity of the corre-

sponding pints.

.

5.4 Camera Coordinate Transformation

Figure 5.4: Coordinate Transformation - Figure shows coordinate transformation

of point P from camera’s reference frame to world fixed coordinate system

57

5/figures/coordinates.eps

5.5 Stereo Camera Calibration

Since the moving object’s position is measured with reference to the camera’s

coordinate system it is required to transform the measured position to world fixed

coordinate system. Suppose that the rotating stereo camera system is fixed on

the origin of the world coordinate system (XYZ) and stereo camera’s orthogonal

reference frames are Xc and Zc respectively. If the stereo camera is rotated by

a angle of θ with reference to the world coordinate system in order to track the

moving object P and measured position of the object P by the stereo camera

system is (x′

c , z′c) then using the equation 5.13 it can be transformed to the

world’s fixed coordinate system.

[xy

]=

[cosθ −sinθsinθ cosθ

] [x′

c

z′c

](5.13)

5.5 Stereo Camera Calibration

Stereo images obtained from the cameras need to be calibrated in order to get

accurate 3D results. Calibration process includes relating the ideal model of

the camera to the actual physical device and of determining the position and

orientation of the camera with respect to a world reference system. Depending

on the model used, there are different parameters to be determined.

The pinhole camera model is broadly used and the parameters to be calibrated

are internal (or intrinsic) parameters and external (or extrinsic) parameters. In-

ternal parameters include internal geometric and optical characteristics of the

imaging device and external parameters include position and orientation of the

cameras in a world reference system.

To obtain accurate real world measurements stereo camera system should be

calibrated in order to compensate unmodeled instric and extrinsic parameters.

Though there are many methods (33) that can be used for calibration, we fol-

lowed a very simple mechanism to avoid coding and computation burden. We

obtained several range measurements and recorded the corresponding disparity .

The calibration chart is shown in the figure 5.5. The coefficient of determination

R2 of the calculated trend line for the measurement was 0.98 which indicates that

regression line accurately approximate the real data. The calibration process ex-

plained above is capable of delivering the required accuracy ± in 15 m range.

58

5.6 Test Results

Figure 5.5: Calibration Chart - The coefficient of determination R2 of the calcu-

lated trend-line for the measurement was 0.98 which indicates that regression line

accurately approximate the real data

5.6 Test Results

In-order to test the feasibility of the proposed stereo range measurement system

simple outdoor test was done. Colored marker ( Red colored ball) was placed on a

field robot and carefully maneuvered in a 7mX10m square path. Object tracking

and disparity calculations were done using openCv in a C++ environment. On

a 1.8 Ghz Core Duo computer we were able to obtain range measurements at a

rate of 5Hz. The test result are shown in the figure 5.6.

Test results closely follow the actual path the robot traveled. Results indicate

a drop in accuracy in range measurements with the increase of the distance. The

stereo vision localization system was able to localize the robot with in +-1m in

most of the time in an outdoor environment. But sudden deviations caused in

the depth measurements make it difficult to use this absolute localization system

for autonomous operation without filtering out unacceptable of measurements.

But the major advantage is error in each measurement is independent from the

prior measurements. Therefore this absolute localization system can be used to

bound the incremental error caused in relative localization systems.

59

5/figures/calchart.eps

5.6 Test Results

Figure 5.6: Test Results - The test results shows that stereo range measurement

system can accurately localize the robot within ±1m .

60

5/figures/result.eps

5.7 Summary

5.7 Summary

This chapter propose a stereo vision range measurement system as a low level

absolute localization system. Rotatable stereo vision platform was build using off-

the- shelves consumer grade equipments. Test results in the outdoor environment

showed that the proposed system can be used efficiently to compensate and bound

the accumulated errors caused in relative localization systems.

61

6

Sensor Fusion with EKF for

Localization

Kalman filter has been frequently applied to the problem of robot localization.

It works recursively, and so does not require a history of the robots previous

states to be kept. Therefor it can be run in real-time in a normal computer.

Originally introduced in 1960, the Kalman filter assumes a multivariate Gaussian

distribution (31).In the Kalman Filter the robots pose estimation is maintained

as a Gaussian distribution and sensor data from dead reckoning and stereo vision

range measurement is fused to obtain a new position distribution.

In this experiment we fused absolute range measurements from the stereo

measurement system with the high resolution dead reckoning visual odometry

data using Extended Kalman Filter( EKF). EKF has to be used due t the none

linearity of the range measurements. This chapter describes formulation of EKF

for the proposed vision based localization system.

6.0.1 Process Model

If the tractor pose (position and attitude) at time k is represented by the state

vector qk = [xk, yk, k]T , then the dynamics of the two wheeled tractor used in this

experiment are well-modeled by the following set of non-linear equations:

qk+1 =

∣∣∣∣∣∣

xK +∆DKcos(θK)yK +∆DKsin(θK)θK +∆θK

∣∣∣∣∣∣= f(qK,uK,) + vK (6.1)

where vk is a noise vector. Here,Dk is the distance traveled by the point at

the center of the robot’s front axle, obtained by the visual odometry using the

62

downward faced camera. The incremental orientation change ∆DK is obtained

also by the angle measurement technique described in the chapter 4. These dead

reckoning measurements constitute the control input vector uk = [∆DK , θK ]T .

The system matric A(k) is represented by the Jacobian:

A(k + 1) =∂f

∂q|qK =

∣∣∣∣∣∣

1 0 −∆DKsin(θK))0 1 ∆DKcos(θK)0 0 1

∣∣∣∣∣∣(6.2)

The input gain matrix B(k) is constructed simply

B(k + 1) =∂f

∂u|qK =

∣∣∣∣∣∣

cos(θK) 0sin(θK) 00 1

∣∣∣∣∣∣(6.3)

6.0.2 Measurement Model

Absolute measurement information obtained from the stereo measurement system

is modeled in the measurement model. The stereo measurement system located at

the origin measures the angle between the moving robot and the ground (x axis)

and the distance between the robot and the stereo system. These measurement

at time k+1 is given by the following equation.

hk+1 =

θ

r

=

atan( yk+1

xk+1) + v1

√x2k+1 + y2k+1 + v2

(6.4)

where v1 and v2 are the random variables which represent the process noise. Then

the m by n jacobian matrix of partial derivatives H can be obtained as follows

H(k+1) =∂h

∂q|qK+1 (6.5)

H(k+1) =∂h

∂q|qK+1 =

−yk+1

x2k+1+y2

k+1

xk+1

x2k+1+y2

k+1

xk+1√x2k+1+y2

k+1

yk+1√x2k+1+y2

k+1

(6.6)

6.0.3 Process error

Using the method of Chenavier and Crowley [24] process error can be considered

as uncorrelated and and model them as:

Q (k) =

∣∣∣∣∣∣

Q11(k) 0 00 Q22(k) 00 0 Q33(k)

∣∣∣∣∣∣(6.7)

63

∣∣∣∣∣∣

Q11(k) = Kss|∆DK cos (θK)Q22(k) = Kss|∆DKsm(θK)|

Q33(k) = Ksθ|∆DK |+Kθθ|∆θK

∣∣∣∣∣∣(6.8)

Coefficients of the equation (6.8) are experimentally estimated.

Sk = HkP−

k HTk +Rk (6.9)

6.0.4 Measurement Gating

Measurement validation gate is required in order to filter out occasional erroneous

measurement that can degrade robot’s state estimations. If the uncertainty of the

current measurement is great it can be caused by a erroneous measurement and

if the confident of the current measurement is great, gate should assign a higher

criteria for the incoming measurement. The first step of the gating process is

calculating the normalized innovation squared which is proposed in (29).

Ev(k) = v(k)S(k)−1v(k)T (6.10)

Here v(k) is the innovation (i.e., the difference between the measurement and

its expected value according to the model) and S is the matrix Sk = HkP−

k HTk +

Rk.The matrix has a shi-square distribution, so the gate’s bounding values can be

read from the chi-square table, and a measurement will be discarded if its Ev(k)

value is outside the bounding values.

6.0.5 Time Propagation

When a new control signal uK is received a above developed process update robot

pose according the equation 6.1 and the following process covariance matrix holds

the uncertainty and propagate through each step.

P−

k = A(k)P+k−1A(k)

T +B(k)ΓB(k)T +Q(k) (6.11)

6.0.6 Measurement Update

Final step of the kalman filtering is the measurement update. When a new mea-

surement is received following step are done sequence which update the robot’s

position in world coordinate system. Using the current position of the robot

(xk, yk) compute Hk. Look up the variance Rk and the mean yk associated with

the current measurement from its stored PDF. Using the measurement model,

64

compute the expected measurement rk. Then Let v(k) = yk − hk be the innova-

tion. Compute S(k) = HkP−kHT

k +Rk Compute the Kalman gain Kk

Kk = P−

k HTk S

−1k (6.12)

Compute the normalized innovation squared and test the measurement against

the chi-square gating as described above. If the measurement passes the gating

test, update the state by letting ,

q+ = q− +Kkv(k) (6.13)

and update the covariance matrix by letting

P+k = P−

k −KkSkKTk (6.14)

Above mention steps are repeated in order to estimated Kalman Filtered position

of the tractor.

65

7

Localization Implementation and

Validation

This chapter describes the experimental results for the work conducted for this

thesis. The localization experiments were performed by manually driving a two

wheel tractor in an out door field on a path that is similar to maneuvers used for

tilling in agricultural field preparation. The trajectory of the tractor is tracked by

the stereo vision based absolute localization system and visual odometry based

relative localization system. The two systems were later fused using the Extended

Kalman Filter. To analyze the performance the localization error should be cal-

culated with respect to the ground truth (GT). One of the main difficulties in

the validation of the localization outdoors is the absence of a ground truth for

the vehicle motion. By using real time kinematic GPS (RKTGPS) as the ab-

solute sensor ground truth information can be extracted with absolute error of

2cm. Since such system is not available during this experiment simple but yet

reliable method was used to extract the ground truth information. A metal rod

was attached to the two wheel tractor touching the ground that marks the path

traveled by the tractor on the ground. This path is later measured and recorded

to use as the ground truth information.

7.1 Experimental Setup

The experimental setup is shown in figure. A web camera with 640X480 pixels

resolution is fixed facing downward at front of the tractor from about 75 cm

from the wheel base. This camera is connected to a laptop pc that runs visual

odometry algorithm in C++ and OpenCv. The same camera is used to record

66

7.1 Experimental Setup

the ground video,along the path traveled by the tractor in order to verify the

ground truth information . Visual odometry algorithm discussed in chapter 4

runs on the computer in real-time while loging the data into a text file along with

the system time in millisecond resolution.

Figure 7.1: Experimental Setup - A web camera is fixed at front of the tractor

as the visual odometry sensor. Red ball is used as the object which is tracked by

the stereo range finder.

Stereo vision based absolute localization system described in the chapter 5 is

also fixed at a corner of the test field and its position is taken as the origin of the

world fixed coordinate system. A red colored marker (red ball) was mounted on

the tractor as the object that can be tracked by the stereo range measurement

system. This colored marker is clearly visible to the stereo range measurement

unit throughout the test field. When the tractor moves in the field stereo range

measurement system tracks the red colored object across the field and calculate

the range and the absolute angle to the tractor. Two web cameras of the stereo

measurement system were connected to the PC and stereo video sequence was

processed by C++ and openCV . The system is capable of updating the position

of the moving object at a rate of 5Hz on CoreDuo 1.8Ghz computer. Range

measurement also logged to a text file along with the system time in milliseconds.

System time of the two computers were synchronized prior to the experiments.

The data from the visual odometry and stereo range measurement system are

later synchronized together using the system clock data and fused using Extended

Kalman Filter (EKF).

67

7/figures/test1.eps

7.2 Results

Figure 7.2: Field Testing - Stereo range finder in the front and the tractor with

colored marker in the back

7.2 Results

Results of the experiments by maneuvering the tractor manually in a 10mX10m

is shown in the following figure. The accuracy and precision (error standard de-

viation) of the measurements are obtained by comparison with the ground truth.

The biases and standard deviations are given in Table 6.1 and the corresponding

error distributions are shown in Figure 7.3.

Cartesian Error

Average Error 53.84 cm

Maximum Error 152.40 cm

Standard Deviation 35.59 cm

Table 7.1: Performances of the Kalman filter based localization system.

The analysis of the experimental data of EKF based localization showed that

the proposed system is capable of localizing a moving two wheel tractor in an out-

door environment with acceptable accuracy. Standard deviation of the Cartesian

error was 35.6 cm. Figures 7.4 and 7.5 show that both visual odometry and

stereo vision based localization systems fail to correctly localize the tractor when

68

7/figures/test2.eps

7.2 Results

−1200 −1000 −800 −600 −400 −200 00

200

400

600

800

1000

1200

X Position (cm)

Y P

ositi

on (

cm)

Kalman Filter Localization

EKF Localization

Ground Truth

Figure 7.3: Extended Kalman Filter Localization - Figure shows ground truth of

the trajectory the tractor traveled and the path estimated by the EKF localization

system

−1200 −1000 −800 −600 −400 −200 0−200

0

200

400

600

800

1000

1200

X Position (cm)

Y P

osi

tio

n (

cm)

Visual Odometry Localization

Visual OdometryGround Truth

Figure 7.4: Visual Odometry Localization - Figure compares the ground truth

and the path estimated by visual odometry

69

7/figures/kalman_filter.eps

7/figures/vo.eps

7.2 Results

−1600 −1400 −1200 −1000 −800 −600 −400 −200 0 2000

200

400

600

800

1000

1200

1400

1600

1800

2000

X Distance (cm)

Y D

ista

nce

(cm

)

Stereo Range Measurements (Absolute Localization)

Stereo Range MeasurementsGround Truth

Figure 7.5: Absolute localization using stereo range measurements - Stereo range

finder tracked the tractor across the field and the absolute position is updated at

a rate of 5Hz.

−1400 −1200 −1000 −800 −600 −400 −200 0 200 400−200

0

200

400

600

800

1000

1200

X Position (cm)

Y P

ositi

on (

cm)

Kalman Filter Localization Without Measurement Gating

EKF Localization Without Measurement Gating

Ground Truth

Figure 7.6: Effect of Measurement Gating - Without the measurement validation

gate EKF failed to localize the tractor

70

7/figures/stereo.eps

7/figures/gating.eps

7.2 Results

−1200 −1000 −800 −600 −400 −200 00

200

400

600

800

1000

1200

X Distance (cm)

Y D

ista

nce

(cm

)

Error Distribution Along The Path

Figure 7.7: Error Distribution Along the Path Traveled - Errors at curve path

is significant with comparison to the straight line motions.

run alone. In Visual odometry,error gets accumulated and the resolution of the

stereo vision based range measurement system detoriates with the increase of the

distance as shown in figures 7.4 and 7.5. Therefore these localization systems can

not cater autonomous navigation. EKF data fusion showed acceptable results

that can be used for autonomous operation of the tractor. EKf has bound the

cumulative error caused by visual odometry using the low resolution absolute

stereo vision range measurements. A measurement validation gate is implemented

to augment the Kalman filter’s weighting of the range measurements; this is

meant to disallow occasional erroneous measurements. Figure 7.6 shows such gate

validation process is needed in order to avoid degradation of the pose estimations

through EKF. Cumulative Frequency Distribution of the Cartesian Error showed

in the figure 7.8. 70 precent of the time Cartesian error is less than 70cm which

is acceptable for a low cost outdoor localization system. Overall results have

proved the proposed localization system can be used for autonomous operation

of agricultural vehicles.

71

7/figures/error_distribution.eps

7.2 Results

Figure 7.8: Cumulative Frequency Distribution of the Cartesian Error - 70 of

the time Cartesian error is less than 70cm

72

7/figures/cdf.eps

8

Conclusions

This thesis outlines the problem of outdoor autonomous robot localization in

agricultural fields. A commonly used two wheel tractor is modified and devel-

oped as an autonomous vehicle test bed. Visual odometry using downward facing

cameras are proposed as the absolute localization system. Two visual odometry

algorithms visual odometry with feature tracking and visual odometry with FFT

are field tested. Test results proved that feature tracking algorithm is more suit-

able for real time operation both intern of accuracy and speed. Stereo vision

based absolute localization system also proposed to compensate error accumu-

lated by the visual odometry. It is capable of tracking colored objects in 15m

range and estimate range measurements. The Kalman filtering methodology is

used to fuse data from visual odometry and stereo vision localization in conjunc-

tion with the chi-square gaiting technique for a robust localization using vision

based systems. The test results showed that the position error is acceptable

for autonomous operation, but requires additional investigation and vision algo-

rithm development. Field test were done using a two wheel tractor in an out

door field on a path that is similar to maneuvers used for tilling in agricultural

field preparation. Test results proved EKF as a reliable method that can improve

the performance of the proposed vision based localization system. The advantage

of the developed system is it can be instrumented using low cost off the shelves

electronics components.

Using vision based sensors in outdoor environment is challenging. The cam-

eras used in this experiment are low cost web cameras. But to cater with changes

in lighting conditions caused by the sunlight good quality cameras are required

with better exposure control. Also object tracking algorithm and visual odometry

need to be improved in order to get better results in out door environment. Field

73

testing of this experiments were done in a flat surface. It is required to analyze

and improve the visual odometry accuracy when the robot is moving in an uneven

terrain. Gyro sensors can be used to compensate effects caused by pitch roll and

yaw and three dimensional model for the tractor needed to be developed.

74

References

[1] B. Thuilot, C. Cariou, P. Martinet, M. Berdu-

cat,”Automatic guidance of a farm tractor relying on a

single CP-DGPS”, Autonomous Robots, Kluwer Academic

Publishers, vol. 13, n. 1, pp. 53-71, July, 200. 2

[2] J.A Marchant and R.Brivot.”Real-time tracking of plant

rows using a Hough transform”, Real-Time Imaging, Vol-

ume 1, pp.189-194,1995.

[3] S.Han,Q.Ni.B Zhang and J.F Reid.”A guidance direc-

trix approach to vision-based vehicle guidance sys-

tems”,Comput Electron Agric, Volume 3, pp.179-194,

2004.

[4] L.Wei, C.Cappelle, Y.Ruichek and F. Zann. ”GPS and

Stereovision-Based Visual Odometry: Application to Ur-

ban Scene Mapping and Intelligent Vehicle Localiza-

tion”,International Journal of Vehicular Technology, Vol-

ume 2011,2011.

[5] R.C Harrell,P.D Adsit ,T.A Pool and R.Hoffman . The

FloridaRobotic Grove Lab. ASAE Transactions, 1990,Vol

33,pp 391-399. 8

[6] Q. Zhang ,J.F Reid,N. Noguchi. ” Agricultural vehicle

navigation using multiple guidance sensors”’ In: Pro-

ceedings of International Conference on Field and Service

Robotics, Pittsburgh, PA, USA, 1999, pp 293298. 8, 28,

52

[7] Y.Nagasaka ,N.Umeda,Y.Kanetai ,K.Taniwaki and

Y.Sasaki .”Automated rice transplanter using global

positioning an gyroscopes”,Comput Electron Agric,2004 ,

Vol 43 (3),pp 223-234. 8, 28

[8] N. Noguchi and H. Terao. ”Path planning of an agricul-

tural mobile robot by neural network and genetic algo-

rithm”, Comput Electron Agric,1997;Vol 18(2-3),pp. 187-

204. 9

[9] H.T Sogaard ,H.J Olsen. ”Determination of crop rows by

image analysis without segmentation”, Comput Electron

Agric, 2003,Vol 2(38), pp.141158.

[10] S.Han,Q.Zhang , J.F Reid .”A guidance directrix ap-

proach to vision-based vehicle guidance systems”, Com-

putElectron Agric, 2004,Vol 3(43), pp.179-197. 6

[11] J.A Marchant and R Brivot. ” Real-time tracking of

plant rows using a Hough transform”. Real-Time Imag-

ing, 1995,Vol 1(5),pp.36371. 6

[12] J.A Marchant.”Tracking of row structure in three crops

using image analysis”, Comput Electron Agric, 1997,Vol

15(2),pp.161-179. 6

[13] H. Okamoto ,K. Hamada ,T Kataoka ,M. Terawaki and

S. Hata .”Automatic guidance system with crop row sen-

sor”, In:Proceedings of Conference on Automation Technol-

ogy for Off-Road Equipment (Chicago, Illinois, USA),2002,

pp. 307-316. 6

[14] Kise M, Zhang Q, Rovira M F.”A stereovision-based

crop row detection method for tractor-automated guid-

ance”,Biosystems Engineering, 2005,Vol 90(4),pp. 357-

367. 6

[15] B. Astrand ,A J Baerveldt . ”A vision based row-

following system for agricultural machinery”, Mechatron-

ics, 2005,Vol 15(2), pp.251-269. 6

[16] K. Imou ,T. Okamoto and Y. Kaizu and M. Ishida,

Sawamura A and Y.Sumida.”Ultrasonic Doppler sensor

for measuring vehicle speed in forward and reverse mo-

tions including low speed motions”, Agricultural Engi-

neering International: CIGR Journal of Scientific Research

and Development, Vol. III, 2001.

[17] D.C Slaughter ,D.K Giles and D.Downery.”Autonomous

robotic weed control systems”, A review. Comput. Elec-

tron Agric,2008,Vol 61(1),pp.63-78. 7

[18] A.Stoll and H.D Kutzbach.” Guidance of a Forage

Harvester with GPS”. Precision Agriculture, 2000,Vol

2,pp.281-291. 7

[19] M.Kise,N. Noguchi, K. Ishii and H. Terao.”Development

of agricultural autonomous tractor with an RTK-GPS

and a FOG”, Proceedings of the Fourth IFAC Symposium

on Intelligent Autonomous Vehicles, Sapporo, Japan,2001,

pp. 99-104. 8

[20] M.Kise,N.Noguchi,K. Ishii K and H.Terao. ”The devel-

opment of the autonomous tractor with steering con-

troller applied by optimal control”, Proceedings of Con-

ference on Automation Technology for Off-Road Equipment,

Chicago, USA,2002, pp. 367-373. 8

[21] T.D Gillespie.Fundamental of vehicle dynamics. Society

of Autonomous Engineers, 2004,Ch1-4. 18, 19

[22] T.I Fossen.Guidance Control of Ocean Vehicles.Willy and

Sons Ltd, 1994. 17

[23] D.J.M Smith.Continuos System Simulation.Chapman

and Hall, Cambridge,1995. 23

[24] E. De Castro and C. Morandi, Registration of Trans-

lated and Rotated images Using Finite Fourier Trans-

forms, IEEE Transactions on Pattern analysis and ma-

chine intelligence,vol.PAMI-9, 1987, pp.740741 28, 30

[25] N. Vatani, J. Roberts, M.V Srinivasan. ”Practical visual

odometry for car-like vehicles,” IEEE International Con-

ference on Robotics and Automation, 2009, pp.3551-3557.

41

[26] J. Campbell, R. Sukthankar, I. Nourbakhsh, A. Pahwa.

”A Robust Visual Odometry and Precipice Detection

System Using Consumer-grade Monocular Vision,” Pro-

ceedings of the 2005 IEEE International Conference on

Robotics and Automation, 2005, pp. 3421- 3427. 28, 41

[27] L.Piyathilaka and R.Munasinghe.”Multi-camera visual

odometry for skid steered field robot,” In Proceedings 5th

International Conference on Information and Automation

for Sustainability (ICIAFs), pp.189-194, 17-19 Dec. 2010.

42

75

REFERENCES

[28] J. Y. Bouguet. Pyramidal Implementation of the Lucas

Kanade Feature Tracker, OpenCV Documentation, Intel

Corporation, Microprocessor Research Labs. 33

[29] J. Djugash, S. Singh, P. Corke. ”Further Results with

Localization and Mapping using Range from Radio,”In

Proceedings, Fifth International Conference on Field and

Service Robotics, Pt. Douglas, Australia,July 2005. 10,

52, 64

[30] G. Kantor and S. Singh,”Preliminary results in range-

only localization and mapping”, In Proc. of the IEEE Con-

ference on Robotics and Automation, May 2002.

[31] R. E. Kalman.”A new approach to linear filtering and

prediction problems” Transactions of the ASME, pages 35-

45, 1960. 10, 62

[32] J.Shi and C.Tomasi.”Good Features to Track”,IEEE

Conference on Computer Vision and Pattern Recogni-

tion,pp 593600, 1994. 34

[33] J. Weng, P. Cohen, M. Herniou, ”Camera Calibration

with Distortion Models and Accuracy Evaluation,” IEEE

Transactions on Pattern Analysis and Machine Intelli-

gence, pp. 965-980, October, 1992 58

76

Date post:	23-Jan-2023
Category:	Documents
Upload:	uts
View:	0 times
Download:	0 times

Vision based localization of remotely operable agricultural vehicles

Documents