Abstract This paper describes the requirements for user interfaces for teleoperation of mining vehicles and systems. The current status of teleoperation in mining is outlined. Current commercial interfaces are relatively unsophisticated, evolving from line-of-sight remote control systems with the addition of video displays. The user interfaces for two experimental teleoperated systems, the Numbat emergency response vehicle and remote highwall mining systems are described. In both cases the purpose of the user interface is to provide efficient data presentation for the operator. The major client requirements for mining systems are robustness, reliability and ease of use by site operational and support personnel. 1. Introduction CSIRO Exploration and Mining has been active in developing and applying modern automation technology to coal mining equipment and systems. This paper, in keeping with the vehicle teleoperation interfaces theme of the workshop, will outline the current status of teleoperation systems for mining and will go on to describe prototype operator interfaces we have developed. The first project concerns development of the Numbat, a remotely controlled vehicle that provides real-time visual surveillance and atmospheric analysis of underground coal mines in situations where it is too hazardous for manual exploration. The second project area involves the teleoperation of mining equipment extracting coal using the highwall mining method. Here a continuous mining machine is driven under remote control into the coal seam exposed by previous open-cut mining operations. Because the tunnels formed have no roof support, human entry is prohibited, so the mining method is completely reliant on teleoperation. While these interfaces are undoubtedly not the most sophisticated ones that will be described in this workshop, they have been found to be effective in the mining context where survivability, reliability and efficiency are of importance in a harsh production environment.

2. Current Status of Teleoperation for Mining in Australia

Line-of-sight remote control is now commonplace in underground mining. It has proved to be highly effective in removing operators from the dust and other hazards immediately surrounding production equipment. However, there are two disadvantages with this form of control. Often, restrictions to direct vision mean that the operator cannot be removed from the equipment far enough, and the second is the converse that there is the temptation to get too close to an active machine without the protection afforded by actually being on the machine in a secure driver’s station. Unfortunately there have been fatalities using remote controlled equipment in Australia and the enquiries and litigation currently in progress may introduce restrictions to remote control operations. The solution to both these problems is teleoperation or automation where the operator can be removed preferably to the surface and appropriate safety systems are implemented to keep underground personnel and machinery absolutely separated. Thus there is incentive to develop teleoperated and automated systems. Currently there are no production teleoperation systems in underground coal mining. The metal mining industry has led the way in teleoperation because the common ore and muck transport method using load-haul-dump (LHD) vehicles offers benefits if teleoperated. Apart from the safety issues already mentioned, one person can operate multiple LHD’s, giving significant economic advantages. The operator interfaces currently used commercially are relatively unsophisticated, having two basic components: the vision system and the control interface. The control interface is often an adaptation of the earlier generation remote controller and the vision system consists of usually two screens giving simultaneous rear and front views. Figure 1 shows a typical teleoperation control station (courtesy RCT).

Teleoperation User Interfaces For Mining Robotics

D. W. Hainsworth

CSIRO Exploration & Mining Brisbane Qld 4069 Australia d.hainsworth@dem.csiro.au

Figure 1: Mining teleoperation user interface This level of sophistication has allowed teleoperation to be a viable and profitable technique. Feedback from commercial users of teleoperation systems shows that the successful introduction of teleoperation into mining requires attention to the following points. • Need committed workforce & management • Fail-safe design • Camera position, lenses and lights are important • Ergonomics of controls is important

• Training • Maintenance • Operator input to design While the operator interface appears in the list, the majority of the issues are not interface-related. In the writer’s opinion the ongoing management of the other issues is the key to the continued advance of teleoperation in the mining industry. However researchers are active in interface design, particularly when other forms of feedback to the operator besides vision are important, and it is necessary to find efficient ways to display this data. The following projects illustrate some of this work. 3. The Numbat Mine Safety Vehicle In order to gain an understanding of the requirements of the Numbat user interface it is useful to summarize the role of the Numbat remote vehicle. Its primary purpose is to provide rescue teams with immediate information on underground coal mine conditions after an emergency incident. Sending the Numbat vehicle under remote control into hazardous areas instead/ahead of mines rescue teams not only minimizes risk to personnel, but also provides an invaluable source of timely information regarding the mine’s environmental state for the people planning the rescue/recovery operation.

Figure 2: Architectural overview of the functionality and interconnections of the Numbat surface station

SurfaceComputer

(Linux)

SurfaceComms

Optical Fibre Link

GUI

Control

Audio

Video Out

RearComputer(LynxOS)

VehicleComms

Tractio

Cable

Gas

FrontComputer(LynxOS)

Audio

Camera

Lights

Surface ControlStation

NumbatVehicle

Video In

The Numbat integrates a diverse range of communications, actuation, mobility, power, control and software technologies. The Numbat system consists of a mobile surface control station and the remotely controlled vehicle. Recent upgrades and improvements to the Numbat’s hardware, software, and mechanical design have been made to enhance the capabilities of the vehicle. A special emphasis has been placed on the increased system modularity, re-configurability, and ease of maintenance. Figure 2 shows the basic configuration and interconnectivity of the Numbat vehicle and surface station core functionality. The vehicle computers act as slaves with respect to the surface computer. All computers are ruggedised to withstand vibration and harsh treatment. An optical fibre link provides bi-directional communications between the surface station and the Numbat. Three x86-based computers running real-time UNIX are used for real-time control and processing; one in the surface control station and two mounted in the Numbat vehicle. While vision is the major navigation and surveillance tool, the operator interface is also required to display environmental sensing data and vehicle operating parameters in an efficient way.

3.1 Operator Interfaces The Numbat is remotely controlled in real-time from a portable control station located near the mine entry. The air-conditioned control station houses all of the computers, communication racks, video display units, and monitoring equipment required in teleoperating the Numbat vehicle. Figure 3 shows an inside view of the control station, which readily accommodates several people and maintenance equipment.

Figure 3: Numbat Surface Control Station The Numbat can be considered generally as a ‘mainstream’ remote vehicle teleoperation application. The user interface design was driven by the usual

requirement to build an interface to allow the vehicle to be operated adequately and the sensor information to be displayed appropriately, all within a budget. Further constraints arose during the extended development cycle of the system where newer parts of the system needed to remain compatible with parts not being upgraded. The major inputs to the interface design were: • The operator is the major component in the sensor

analysis-vehicle control loop because the vehicle is required to operate in environments which can be rearranged substantially after an explosion due to roof falls, dislodgement of pit props (roof supports) and generation of other debris from damaged mine infrastructure. Negotiation of these obstacles requires human observation and processing power.

• The vehicle moves slowly and has limited control

inputs so the interface is not required to help the operator with real-time control issues such as replication or simulation of the vehicle dynamics at the operators seat to provide sensory input (like a flight simulator). Nor is it necessary to combine the control of several degrees of freedom (as in a manipulator).

• Although the control task is relatively

straightforward, there is a considerable workload associated with operating the vehicle in low light and in unfamiliar surroundings. Moreover considerable data on environmental and vehicle operating conditions are generated, so an interface is required to provide control inputs and data display for either two or one operators.

Given these inputs, the operator interface took the form shown in figure 4. Each of the main features of the interface; the video displays, the control console, and the GUI are now considered. 3.2 Video Displays Video is the main navigation and surveillance tool. The Numbat has four cameras, fixed cameras facing forward and rearward and two cameras, one with zoom facility, in a turret which pans 360 degrees and tilts –20 to +90 degrees. Each video channel is displayed on a dedicated screen so the operator does not have to switch different cameras to a screen and soon becomes accustomed to the pattern of views on the screens. In this way a quick scan of the screens gives the operator a complete update of the environment around the vehicle. This can be achieved more quickly (and cheaply) than by using a virtual reality helmet servoed to a moveable camera. Moreover, in an emergency situation, mine personnel will be in the control station to advise the vehicle operator(s) using their local knowledge. The multi-screen video display is more useful for this purpose where observers view the

same displays as the operator and communication is easier. Each video channel is separately logged to its own VCR, giving a continuous record of all camera outputs. In addition a fifth, high-resolution screen is situated at a comfortable viewing angle in front of the operator. This screen can be switched between front and rear fixed cameras. Experience has shown that these fixed cameras are the main driving tools, and given that several hours of driving are required in a typical mission (>5km range at 2km/hr), operator comfort is greatly enhanced by being able to use this additional screen. In the underground environment, normally the Numbat produces all the available illumination. A tight power budget was necessary for lighting because the vehicle is battery powered, and this has resulted in low illumination levels which add to the workload of vehicle navigation. It was not possible to use low-light cameras with image intensification because of the widely variable lighting conditions that can be encountered. For example when conducting exercises along with underground personnel, caplamp dazzle can overload sensitive cameras. 3.3 Control Console With the computing power and graphical tools currently available, it is feasible to implement the operator’s controls via the GUI screen and arrange access through mouse or touch screen. This has several advantages in that the hardware requirement is simpler and the controls

are reconfigurable for developmental purposes. However, unless the operator has a head-up display, there is the ergonomic disadvantage in that the operator must look at the screen in order to identify the position of the desired control and the degree of control input being applied. In a high workload situation this becomes tiring and leads to operator errors. A head-up display would require a helmet for the operator, and as outlined above, this option was rejected. It was decided to implement the critical operational controls in physical form in a separate console. This can be seen in figure 3 on the bench between the screens. Controls operated in this manner are: • Vehicle traction joystick • Camera turret pan and tilt controls • Camera zoom and focus controls • Light controls • Manual fibre optic cable reeling controls • Light switches • Parking brake With practice the operator can operate these controls without having to take his/her eyes off the video screens. A two-way audio link is provided as part of the control console. Primarily this link enables communication with victims or with rescue personnel, although it plays an important role in monitoring the vehicle’s operation. In production teleoperation systems there is debate as to the

Figure 4: Numbat Operator Interface GUI default page

worth of audio as a feedback mechanism, because the noise level around production equipment is usually extremely loud. The normal human ability to extract an audio signal of interest from a noisy background is lost when the signal has been recorded by microphone(s) and is then heard at a distance. Under these conditions audio is more of a distraction than an asset. However in the case of the Numbat, the ambient noise level is relatively low and with practice the operating team can detect subtle changes to the normal operating sounds of the vehicle which could signal a fault condition. 3.4 Graphical User Interface The computer-driven user interface has evolved from a text-only DOS based system to its current configuration, a high-resolution GUI built using the Tool Command Language (Tcl) and its associated ToolKit (Tk), referred to as Tcl/Tk. Tcl/Tk is ideal for the Numbat as it is simple to use, is platform independent, supports TCP/IP sockets, and is easily embedded into the control station program. In the Numbat case, the real-time controls are implemented physically, as are the controls of commercial teleoperation systems. The function of the computer GUI is to structure and present a large amount of data of various kinds to the operator(s) in a manageable form. There are a number of approaches to the way in which the data can be organized. It is impractical to present all environmental and system information on the screen at once. A common approach to data display management is to arrange the data into pages referring to particular aspects of system operation. For example in the Numbat case these could be based on: • Environmental - External atmospheric composition,

temperatures, status of gas analysis system • Status Indicators – Servo systems controlling cable

payout, traction, brakes etc • Vehicle Operating Parameters – Pitch, roll,

velocity… • Communications cable handling system – Length of

payout… The default page could be as simple as a message which displays a summary of the information from the various pages or a “system OK” message. Fault or exception conditions could automatically cause the page corresponding to the part of the system at fault to appear giving more information. Additionally a menu of pages could be part of the default display allowing the operator to choose a particular page for more information. A disadvantage of this approach is that the operator is reliant on the software which analyses the raw data to give an accurate picture of the condition of the system. If a scenario, unanticipated when the software was written is encountered in a particular mission, appropriate or necessary information might not be displayed.

It has been found during the development of the system that the most efficient GUI default page content is, as expected a summary of the data. The default page is shown in figure 4. The screen is divided into sections corresponding to the dot points above. This gives the operator an immediate display of important system variables and status indicators. If status alarms appear, a set of pulldown menus enables more detailed displays and control functions to be accessed. The hardware, software, and communications architecture of the control system also allows the data and executables in any of the three unix-based platforms in the system, including in the Numbat vehicle to be reconfigured on the fly from the control station (or anywhere else on the planet). This means that if absolutely necessary, any vehicle control characteristic, a servo loop damping constant or gas analyzer calibration value, for example, can be modified in the process of a mission. 4. Highwall Mining Guidance The second area of teleoperation activity concerns highwall coal mining. In this mining method which is depicted in figure 5, a continuous mining machine is driven under remote control into the seam exposed by previous open cut operations. A continuous haulage system carries the coal from the miner to an open-air installation for stockpiling and transport. A series of parallel, unsupported drives is formed by this process. It is vital that the coal pillars remaining between adjacent drives are capable of supporting the overburden structure. The ideal situation of straight, parallel openings at the tightest separation consistent with geotechnical design can only be achieved if the miner’s position and heading can be determined accurately and controlled remotely.

Figure 5: Highwall Mining Operation

4.1 Operator Interface In this case teleoperation interfaces have been set up for highwall mining machines based on both continuous miners (which can be regarded as “heavy duty” vehicles) and steerable auger cutter heads. The latter are more snake-like robots than vehicles but the teleoperation principles are the same. In this application of teleoperation, vision plays at most a secondary role in machine guidance. In the continuous miner systems, vision is used only to observe the cutter drum to determine if possible from the appearance of the dust and chips whether coal or rock is being cut, and to verify the condition of the walls or ’ribs’ of the drive. In the auger-based systems, there is no vision. Consequently for machine guidance operator feedback is derived almost entirely from sensors. Two components of navigation are required: Lateral Guidance: An inertial navigation system (UNS) is used to establish and maintain a desired track for each drive. The INS is used in attitude-heading reference (AHRS) mode, which when coupled with odometry enables an accurate course to be maintained. Horizon control: It is desirable to guide the cutting machine in the vertical plane so it remains within the coal seam. This is achieved using a natural gamma radiation-based (NGR) coal thickness sensor. The sensor measures the attenuation of gamma radiation due to the surrounding strata by the remnant coal in the roof or floor after excavation.

Other requirements for the user interface include: • Gas Concentration measurement: Mining must cease

and the equipment be withdrawn if an explosive gas mixture is likely to form. Sparks from cutting picks could ignite this.

• Display of hole parameters: Depth, tonnage produced • Logging of mining data: Required for mapping the

as-mined area for planning and regulatory purposes. This teleoperation application clearly fits into the category of an intermediate step to full automation, particularly because vision does not play a part in navigation of the systems. All sensors produce outputs in a form that can be conveniently input to the control system without the requirement for extensive pre-processing which is required for image-based data. However, because this is a relatively new mining method, industry was more comfortable with the intermediate stage of human control via teleoperation. 4.2 GUI The machine controls were implemented in physical form with joysticks and switches, and the data display was implemented in the form of a computer-generated GUI. The GUI software was developed using Linux, Metro X Server, Motif and OpenGL and is implemented on a Pentium II platform. This proved to be a cost effective solution

Figure 6: Guided Auger Operator interface GUI default page

The default page is shown in Figure 6. The main function of the display is to show the operator where the machine is located relative to its desired position. This is achieved through the plan and elevation views that take up the lower portion of the display. Further orientation feedback is provided by the individual displays of machine azimuth and pitch in the upper centre of the display. At the top right of the display is the gas concentration graphic which plots current gas concentration on an explosivity chart. This gives the operator a clear indication of the current gas status and trend. If gas concentrations exceed predetermined levels, the system trips automatically. 5. Conclusions The teleoperation user interfaces described in this paper are not highly sophisticated. The teleoperated equipment is not demanding in its real-time control requirements. Under these conditions the interface’s major function is to provide the operator with sufficient quality feedback to enable the equipment to be controlled appropriately in the overall mining system context. The major challenge in introducing automation technology to the mining industry is to ensure that the automated equipment operates in the long term with the resources available to the mining operator. With the tendency for mining (and other) companies to downsize in-house expertise in skilled areas, it will be as important to educate the workforce and management to commit to training and maintenance as it will be to develop new automation technologies and systems. 6. Bibliography D. C. Reid, D. W. Hainsworth, and R. J. McPhee, “Lateral Guidance of Highwall Mining Machinery Using Inertial Navigation”, 4th Intnl. Symp. on Mine Mechanisation and Automation, July 6-9, pp B6:1 –11 Brisbane, Australia (1997). J. C. Ralston and D. W. Hainsworth. “The Numbat: A remotely controlled mine emergency response vehicle”, Proceedings of the International Conference on Field and Service Robotics, pp. 48-55,Canberra, Australia, December 1997. M. Hebert, C. Thorpe and A. Stentz, Intelligent unmanned ground vehicles (Kluwer academic, 1997). J. Ousterhout, Tcl and the Tk Toolkit, Addison-Wesley, 1994 J. Cunningham et al, Mining and Mineral Processing at Mt Isa Mines Ltd, 4th Intnl. Symp. on Mine

Mechanization and Automation, July 6-9, ppB2:13-21, Brisbane , Australia 1997