PowerPoint Presentation

KEEPCALMANDTELE-OP

Design Issues for Real-Time Remote Robotic Science Operations Support Tools: Observations from the FieldHyunjung Kim*, Young-Woo Park, Electa Baker, Julie Adams, and Terry Fong

Thank you for your introduction. My name is Hyunjung Kim. Im here today to talk about Design Issues for real-time remote robotic science operations support tools1

I Introduction

II Analog Mission Overview

III Methods

IV Results and Findings

V Discussion and Conclusion

Id like to give you a brief outline of my presentation.The presentation is divided into five main sections: introduction, analog mission overview, data collection and analysis methods, results and findings, and discussion and conclusion.2

I Introduction

II Analog Mission Overview

III Methods

IV Results and Findings

V Discussion and Conclusion

3

Real-Time Remote Robotic Science OperationsRely on diverse, emerging information & require fast, effective decision-makingOperations software that supports efficient monitoring of science data and planning is essential!!

Science teamsRover operatorsPlanetary RoverDefine the science plan & analyze real-time data collected as a result of executing that planPerform navigation of robots & ensure system functionalityExplore remote locations

Lets start with what real-time remote robotic science operations is.

As the picture describes, real-time remote robotic science operations require highly effective coordination and collaboration btw: Science teams, who define the science plan and analyze real-time data collected as a result of executing that plan,Robot operators, who perform navigation of robots and ensure system functionality, andPlanetary rovers, which explore remote locations

Since real-time remote robotic science operations rely on diverse, emerging information and require fast, effective decision-making,operations software that supports efficient monitoring of science data and planning is essential.4

Real-Time Remote Robotic Science Operations Support Tools

Rarely tested in practice: the tasks and activities that need to be supported are not yet well understood

Hierarchical, and involve professionals from multiple disciplines, resulting in a diversity of information needsDeal with incomplete information and unpredictable problems

12Real-Time Science OperationsScience Teams

The information to be delivered and its representation should be carefully designed!!Design Challenges

However, in order to design operations software to support real-time remote robotic science operations, several challenges must be addressed.First, as the concept of real-time science operations is relatively new and has been rarely tested in practice, the tasks and activities that need to be supported are not yet well understood.Second, science teams, the main users, are hierarchical, and involve professionals from multiple disciplines, resulting in a diversity of information needs.In addition, they deal with incomplete information and unpredictable problems. Therefore, the information to be delivered and its representation should be carefully designed.5

Summary

To better understand real-time remote robotic science operations

1

To provide practical guidance for improving the design of operations support tools

2

In-field observations of a team of 14 scientists remotely operatinga planetary rover during a five-day prospecting mission

Three characteristics of real-time science operations in a short-duration robotic prospecting mission

1

Identified challenges, opportunities, and guidelines associated with improving the design of support tools

2

Goals

Approach

Contributions

This study has two goals: First, to better understand real-time remote robotic science operations, and second, to provide practical guidance for improving the design of operations support tools.To achieve these goals, we conducted in-field observations of a team of 14 scientists remotely operating a planetary rover during a five-day prospecting mission.Our contribution is twofold: First, three characteristics of real-time science operations in a short duration robotic prospecting mission, and second, the identified challenges, opportunities and guidelines associated with improving the design of support tools. 6

I Introduction

II Analog Mission Overview

III Methods

IV Results and Findings

V Discussion and Conclusion

7

The Mojave Volatiles Prospector (MVP) Analog Mission

Let me show you a 3-minute video about the Mojave volatiles prospector analog mission during which we conducted in-field observations. 8

The Mojave Volatiles Prospector (MVP) Analog Mission October 2024, 2014, with the rover operating remotely for 5 hrs/day

To perform remote prospecting, using rover-mounted instruments (e.g., GroundCam, Hazcams, NIRVSS, NSS), to investigate the water content in the Mojave Desert as an analog for subsurface volatiles on the Moon

Science TeamRover OperatorsRover and payload instrumentsSOC (Science OperationsCenter) @ NASA ARCMROC (Mojave Rover Operations Center)

The MVP analog mission took place on October 20 24 , 2014, with the rover operating remotely for five hours a day.The teams science goal for the mission was to perform remote prospecting, using rover-mounted instruments, to investigate the water content in the Mojave desert as an analog for subsurface volatiles on the Moon.The science team was located in the science operations center at the NASA Ames, approximately 400 miles from the Mojave rover operations center, where the rover operators were located.

9

Science Operations

Nominally covers the entire test durationSets of regions of interest (ROIs)Created prior to the start of the field testUpdated every few days as the mission progresses

The MVP science operations consist of three main processes, each having a characteristic time span and update time:

Covers one day of operationsTraverse plans for prospecting, predesignated stops, area of interest mappings (AIMs), predicted power usage an margin, part of the next plan to support get-ahead activitiesCreated 1 or 2 days before use, updated 1 day prior to execution

Conducting activities in the tactical plan, with decisions based on science data returned and assessed during the activities

1 Strategic Planning

2 Tactical Planning3 Real-tine Execution

The MVP science operations consist of three main processes, each having a characteristic time span and update time.

First, strategic planning nominally covers the entire test duration. It contains sets of regions of interest areas identified to potentially have characteristics relevant to the science goals and objectives. The strategic plan is created prior to the start of the field test, and the plan is updated every few days as the mission progresses.

Second, tactical planning covers one day of operations. Tactical plans include traverse plans for prospecting, predesignated stops, and search locations, allocations for discretionary activities such as area of interest mapping, predicted power usage and margin, and part of the next plan to support get-ahead activities. The plans are created one or two days before use, and updated one day prior to execution.

Finally, real-time execution refers to conducting activities in the tactical plan, with decisions based on science data returned and assessed, during the activities.

Observation in this study was specifically focused on tactical re-planning and real-time execution processes during remote rover operations.10

Science LeadSci Ops Mngr (SOM)NIRVSS ScienceNSS ScienceTimelinerTraverse PlannerSciComCamera ScienceStenographer

Science TeamSci Lead(Responsible for science part of tactical plan)Sci Ops Mgr(Responsible for strategic plan and mission goals)SciCom(single interface to Real Time Sci)StenographerSci Analysis Team(analysis and recommendations)NIRVSS Sci(monitor sci & hk)NSS Sci(monitor sci & hk)Camera Sci(monitor sci & hk)

Traverse planner(create/update traverse plan)Time-liner(create/update timeline plan)

@ Science Backroom

RoverTeam@ MROC

The science team consisted of scientists and mission specialists with different roles, responsibilities, knowledge, and expertise.The team includes the Science Operations Manager, the science lead, science communications, the traverse planner, the timeliner, the three science payload leads, and the stenographer. The science analysis team who focuses on more in-depth data analysis was located to a separate room called the science backroom.11

Science Operations Support Tools

NSS ScienceCamera ScienceNIRVSS ScienceScience LeadScience Ops ManagerTimelinerTraverse PlannerScienceCommunications

Google Earth mapxGDSInstrumentsxGDS PlansxGDS Images3D Display (Verve)Timeline (Playbook)Steno-grapher

This is the layout of the science operations center. The layout is designed to enable face-to-face communications between certain console roles.For example, the science operations manger and the science lead, who are responsible for strategic and tactical decision-making, are adjacent to each other and are located behind the science payload leads to allow observation of their individual displays. The Traverse Planer and Timeliner, who are in charge of spatial and temporal planning, are adjacent to each other as well.

The science operations center had six wall-mounted shared displays. 12

Science Operations Support Tools

ImagesPlansRaster MapsInstruments

In terms of operations support software, the NASA exploration ground data system, xGDS, provided four displays: Plans, Images, Instruments, and Raster maps.In addition, the 3D Display showed a three-dimensional representation of the rover and the terrain. The timeline display showed the NASA Playbook software, which tracks the timeline. 13

I Introduction

II Analog Mission Overview

III Methods

IV Results and Findings

V Discussion and Conclusion

14

Data Collection

To understand how the science team members experience real-time remote science operationsTo collect their feedback and suggestions on the use of science operation support tools

4 Sessions of semi-structured interviews, 30-50 minutes/session

To observe and investigate how the science team used operation support tools for science activities and operational tasksTo identify inadequacies of the tools and opportunities for improvement

General Observation, Focused Observation, Voice loop conversations, Videos

1 Interview

2 Observation

Participants14 scientists and mission specialists (8 females and 6 males, ranging in age from 18 to 65, with experience of involvement in a series of NASA robotic field tests)

To understand real-time remote robotic science operations, and to assess how well the tools support science operations,we gathered data by means of semi-structured interviews, direct observation, and audio and video recordings of science operations.

The participants consist of 14 scientists and mission specialists. Many of them had experience of involvement in a series of NASA mission operations, such as the MER mission, the LCROSS mission, the LADEE mission, and the Mars Science Laboratory prelaunch.

First, we conducted four sessions of semi-structured interviews To understand how the science team members experience real-time remote science operationsTo collect their feedback and suggestions on the use of science operation support tools

Second, during the five-day period of the mission, for five hours each day, we conducted observations within the science operations center:To observe and investigate how the science team used operation support tools for science activities and operational tasksTo identify inadequacies of the tools and opportunities for improvement

15

Example Interview Transcript

This is an example interview transcript16

Example Observation Data

General Observation(Team-level activities)Focused Observation(Ethnographic observation on each console position)Voice loop Annotation(Direction of execution & re-planning, internal science team conversations, conversations btw the science team & robot operators)

And these are example general observation logs, focused observation notes, and voice loop annotation.General observation focused on how the science team used the tools for team-level activities.On the other hand, focused Observation focused on the individual use of the tools.In addition, voice loop conversations were recorded and annotated as well.17

Data AnalysisObservation data

Generalobs logsFocusedobs notesVoice loopannotations

Interview dataInterviewtranscriptions

The tools involvedThe type of activity or task supportedThe type of related awarenessHow well the concerned tools supported users goals or tasksIf the statement or incident involve users design suggestionsScience downlink assessmentBalancing science desiresOptimizing science activities vs. controlling operational complexity(The common elements of science operations identified by Cheng et al. 2008)

To analyze the collected data, we used the framework of common elements of science operations identified by chengs paper presented in 2008:science downlink assessment, balancing science desires, and optimizing science activities versus controlling operational complexity.

In addition, we identified statements and incidents related to the use of support tools. During the analysis we considered: the tools involved, the type of activity or task supported, the type of related awareness, how well the concerned tools supported users goals or tasks, and if the statement or incidents involve users design suggestions.18

I Introduction

II Analog Mission Overview

III Methods

IV Results and Findings

V Discussion and Conclusion

Lets move on to the results and findings.19

Real-Time Sci Ops in a Short-Duration Robotic Prospecting Mission

Discovery-based approachScience as the teams highest priority

For this [mission], being able to really work with the rover operators to maximize what you can do in a very small amount of time [is unique]. Science Lead

We have the instrument leads that arent monitoring the instruments which is typically what you see in mission operation, they are actually observing the science out of the instruments. - Timeliner

1 Closeness of the scientists to the operation decision-making

The ability to react to the realities

a several hour turnaround and science planning, and then the reaction-all stop, do this-handed over to the rover driver to accomplish a certain goal of science, Science Lead

It takes time just get to know your instrument and the real environment. You wont really know how the system is going to behave. NSS Science

2 Plan revision during execution

The traverse provides the data that the scientists need

This is a very unique simulation in which the traverse is actually part of the science. So the scientists are very interested in the traverse Usually, in other robotic missions, you are interested in the destination, and you do science at the destination. Timeliner

3 A Traverse-based, not a destination-based approach

From the interviews and observations, we identified three characteristics of real-time science operations in a short-duration robotic prospecting mission.

First, as this mission takes a discovery-based approach, the closeness of the scientist to the operation decision-making is unique compared to other missions.

Second, most participants agreed that plan revision during execution in reaction to real-time data is a critical feature of real-time science operations in a robotic prospecting mission.They emphasized that the ability to react to the realities is invaluable.

Third, a science-driven robotic prospecting mission takes a traverse-based approach, not a destination-based approach.The traverse is critical because it must provide the data that the scientists need.20

Identified Challenges for Real-Time Remote Robotic Sci Ops Support Tools

Support real-time analyses that will actually promote immediate plan decisions

When you are sitting in a console position, youre to monitor inconsistencies, you can point things out, but when I analyze it, Im going to crunch a bunch of numbers on it, and if Im doing that, then Im not doing my console position. NSS Science

Enable scientists to quickly identify trends and correlations within and across different data sources by allowing the rapid and precise manipulation of data and its settings

Help scientists easily return to a specific time or area of interest and extract the data needed more efficiently

1 Facilitate Science for Operations, Not Science Itself

Maximizing the utilization of robots means maximizing rover traverse distance

They [the scientists] intend to rather have more than you could do than less. That was intentional to have no margins. Well, we can stop and start and give up whenever we want to do. Thats the direction I went. Traverse Planner

Consider and facilitate collaborative tactical planning activities

Help the science team identify the availability of resources easily

2 Optimize Rover Traverse to Maximize Science Returns

Our major findings include two design challenges for improving monitoring, analysis, and planning tools for real-time remote robotic science operations:

First, is to facilitate science for operations, not science itself.It is important to support real-time analyses that will actually promote immediate plan decisions. This means that the tools should enable scientists to quickly identify trends and correlations within and across different data sources by allowing the rapid and precise manipulation of data and its settings.In addition, the tools should help scientists easily return to a specific time or area of interest and extract the data needed more efficiently.

Second, is to optimize rover traverse to maximize the science returns.In this kind of cases, for scientists, maximizing the utilization of the robots means maximizing rover traverse distance. Traverse planning and re-planning is about optimization of rover traverse in reaction to the reality.In practice, the major bottleneck in traverse re-planning was in collaborative activities, for example, collection of information and operational recommendations, delivery of instructions to the Traverse Planner, and collaborative reviewing.Therefore the tools should consider and facilitate collaborative tactical planning activities.In addition, the tools should help the science team identify the availability of resources easily, such as the time remaining.21

Design Opportunities to Improve Spatial Awareness

2 Geospatial information-based science activitiesComparing multiple traverse paths on a single map, or different sets of data collected at a certain location should be better supported

We are having a discussion about where we have been and where we wanted to go. The questions I had to go through with the team yesterday were what knowledge have you garnered? How do you expand or deepen that breath of the knowledge, and then how do you all test that knowledge and based on those answers, where do you want to go? Science Operations Manager

Camera Sci wants to see TextureCam from the exact same spot. I think thats hard - General obs note (Oct 21 10:21)

No means of coordinating visual attention or support for deictic communication

Unclear which specific area on the map is being referred to and whether others are following the conversations or not

1 Communication of spatial info within and btw teams

[The scientists] Said We want this. Do this area, and try to avoid what looks like an obstacle. I am not a scientist so I dont know exactly what they want me to cover Traverse Planner

One of the main objectives of remote robotic operations support tools is to provide users with a sense of awareness.In this study, we particularly focused on spatial awareness and temporal awareness, as they are directly related to the key question: How can we make a traverse that meets our science goals but doesnt make the durations so long that we cannot actually finish it?

To better support spatial awareness,

First, communication of spatial information within and between teams should be improved.Although the scientists often reference the maps and telemetry plots displayed on shared screens, there is no means of coordinating visual attention or support for deictic communication.It is sometimes unclear which specific area on the map is being referred to, and whether others are following the conversations or not.As shown in the pictures, the science lead and science operations manager use laser pointers to point to certain areas or draw paths on the map. This is not precise, and the trace disappears instantly.Especially, the Traverse Planner commented that having a clear consensus of what scientists want to do was difficult.

Second, geospatial information-based science activities, for example, comparing multiple traverse paths on a single map or different sets of data collected at a certain location, should be better supported.Although the current tool allows stacking of multiple layers of plans, the plans all appear in the same color and become cluttered by the labels of all of the stations.

22

Design Opportunities to Improve Temporal Awareness

Time stamps based on absolute time, time since task performance, time until the next required tasks

If you are going to change a certain path or certain segments [of the traversal plan], you need to bring it in and do your whole planning process in 5 minutes. They [the scientists] spend so much time discussing if they will do or not, then they run out of the time to actually do it, approve it, send it Timeliner

Science Lead is talking about collecting data and the time available to develop a plan of what to do next. How do we integrate a timer so that we can see? - Global observation note (Oct 21 15:16:42)

1 Offer scientists the temporal information needed at a glance

2 Time-based navigation and searchSupport to easily and quickly navigate and search past instrument plots, images, and raster map data

I am quantifying real time in a tactical sense, being able to augment the plan significantly or make a reaction to what you are seeing within an hour or two. Science Lead

* Real time in actual operations

I dont think they [the scientists] are looking at it [the timeline] that much. When I call their attention to it, they do. - Timeliner

* Significantly less interested in temporal info than spatial info

Regarding temporal awareness of the team, it is noteworthy that real-time in actual operations had a longer time frame than we initially expected. In addition, there was less time pressure than we predicted, as well.It was also interesting to observe that the scientists were significantly less interested in temporal information than spatial information.

To better support temporal awareness,

First, shared information displays should be redesigned to offer scientists the temporal information needed at a glance. During the observation, it was observed that the science team did not know if they were ahead of or behind the schedule unless the Timeliner announced it.In addition, there was no support for measurement of the execution time of ad hoc science activities, such as NIRVSS reference spectra collection.

Second, time-based navigation and search should be improved to support scientists to easily and quickly navigate and search past instrument plots, images, and raster map data.23

I Introduction

II Analog Mission Overview

III Methods

IV Results and Findings

V Discussion and Conclusion

24

Suggestions and Guidelines

1Support efficient time-based and geolocation-basednavigation & searchIndicate operational events on the time axes of the strip charts

For viewing location-related data, allow users to input locations easily, by, e.g., clicking directly on the position of interest on the rover traverse

Permit direct tagging in the plots

Add shortcuts to facilitate the extraction of necessary data

2 Support communication of spatial informationAn implementation of screen sharing that allows users to indicate, highlight, draw, and write on top of the underlying visual information would be useful

Enable scientists to leave graphical instructions and comments directly on the map and allow the Traverse Planner to import this info into the planning tool as a layer for reference use

Here are some suggestions and guidelines for the design of real-time remote robotic science operations support tools.

First, the tools should support efficient time-based and geolocation-based navigation and search.One of our suggestions for more efficient data navigation is to indicate operational events, for example, starting and finishing times for the plan or reference data collection, on the time axes of the strip charts.In addition, for viewing location-related data, allowing users to input locations easily, by, for example, clicking directly on the position of interest on the rover traverse would help.Other suggestions include permitting direct tagging in the plots and adding shortcuts to facilitate the extraction of necessary data.

Second, the tools should better support communication of spatial information.An implementation of screen sharing that allows users to indicate, highlight, draw, and write on top of the underlying visual information, such as maps, plots, and images, would be useful, as has been proved in many other collaborative support tools.For example, to enable scientists to give precise instructions and guides to the Traverse Planner, scientists should be enable to leave graphical instructions and comments directly on the map, and allow the Traverse Planner to import this information into the planning tool as a layer for reference use.

25

Suggestions and Guidelines

A clock displays the absolute time, which is important for coordinating work within and between teams

A critical reference for time-based navigation and search

Must be consistent across all different tools used

3Integrate a clock, a timer, and a stopclock as temporal representations

until

fromA timer that counts down from a specified amount of time is necessary for deadline-driven plan revision A stopclock that shows the time since the task performance is necessary to present temporal context of science operation

e.g., NIRVSS reference spectra collection, taking panorama images

Finally, the tools, particularly shared information displays, should integrate a clock, a timer, and a stopclock as temporal representations.First, a clock displays the absolute time, which is important for coordinating work within and between teams.Time stamps based on absolute time are also a critical reference for time-based navigation and search of archived data.So, they must be consistent across all different tools used.

Second, a timer that counts down from a specified amount of time is necessary for deadline-driven plan revision, to notify science teams the time available to develop a plan of what to do next.

Lastly, a stopclock that shows the time since the task performance, in this case, the time since NIRVSS reference spectra collection or taking panorama images, is necessary to present the temporal context of the science operation.

Importantly, the information for updating temporal awareness needs to be accessible and absorbable at a glance.

26

Conclusion

Lets summarize briefly what weve looked at.

Through and in situ observation study, we investigated how science teams monitor, analyze, and plan during real-time remote robot operations and evaluated how the current tools support teams goals, tasks, and activities. On the basis of our results, we identified the characteristics of real-time science operations in a short duration robotic prospecting mission. In addition, we discussed the identified challenges, opportunities, and guidelines associated with improving the design of operations support tools.

Although our study was limited to observation of a single analog mission, we expect that our findings will inform future research on robotic operations support tools andexpand the scope of inquiry to several different use cases, including a future lunar volatile prospecting mission.

Thank you for your attention.27