American Institute of Aeronautics and Astronautics

1

Exploration Operations Support and Training Require Real-Time Virtual 3D

Abe E. Megahed* and Thomas M. Crabb† PLANET LLC, Madison, WI, 53717-1961

Mark C. Lee‡ Orbital Technologies Corporation, Madison, WI, 53717-1961

and

Marty A. Gustafson§ PLANET LLC, Madison, WI, 53717-1961

Real-time 3D simulation for operational support and training will be a requirement to improve operational effectiveness in the Exploration era as the demands for training become more complex and the length of time between training and mission operations grows. We propose a similar cadre of 3D aids to apply and bridge the gap among training, real-time operations, and remote control automation. This new type of automated and intuitive assistance will be necessary for displaying integrated vehicle health monitoring status and to minimize the crew time needed for training for operations prior to and during missions. Other capabilities include transfer of operational tasks to ground crew for basic monitoring and automated maintenance through remote operation of a virtual representation of the system which sends appropriate data packets to the actual system. Each system could have its own embedded training and operational support, reducing reliance on the crew’s memory and the proficiency of training that occurred months or years previously. This approach applies to vehicle and transportation systems, remote operations of unmanned equipment, and robotics. Significant efficiency and cost savings is also possible using interactive, real-time 3D visualization tools that are reused from design data and expanded into mobile training. Specific examples highlight the benefits and needs in the exploration era.

I. Introduction Long-duration missions will present the most demanding situations ever encountered in manned spaceflight from

a safety and performance perspective as well as efficiency and cost affordability perspective. Success will depend on the efficiency and effectiveness of continuous system knowledge, flexibility of the ground and orbital crews, and the critical tools at their disposal. A new platform is therefore needed that can provide a number of visualizations for ground and orbital crews, including an interface to the integrated vehicle health management (IVHM) system. Intelligent tutoring and highly interactive 3D simulations in an on-demand situational training and operations support system can also provide remote operations aids for robotic or other unmanned equipment. This platform will provide a faster means for astronauts to assess necessary actions in routine and emergency situations, quickly select from software-aided, simulation-based, training materials or job aids, and link to information including logistics or vehicle health status to more effectively conduct tasks in a just-in-time scenario. Simulation-based user interfaces greatly improve abilities to perform rapid diagnosis and repair by visually mapping the location of problems and immediately linking to simulated repair procedures for off-nominal rehearsal training. Intelligent simulations can also be used in a non-scripted mode for testing of new procedures in emergency situations. Pre-flight, they also

* Virtual 3D Software Manager, 1212 Fourier Drive, Madison WI 53717 † President/CEO, 1212 Fourier Drive, Madison WI 53717, and Associate Fellow, AIAA ‡ Program Manager, 1212 Fourier Drive, Madison WI 53717, and Member, AIAA § Commercial Applications Manager, 1212 Fourier Drive, Madison WI 53717 and Member, AIAA

Space 200619 - 21 September 2006, San Jose, California

AIAA 2006-7282

Copyright © 2006 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved.

American Institute of Aeronautics and Astronautics

2

serve as more effective ground trainers and provide an interface to simulated equipment for optimization of human interactions.

This type of solution is driven by interactive and collaborative three-dimensional (3D) simulations that have emerged as one of the most versatile and valuable new technologies to support crew operations, training and service. Improvements in computer graphics modeling capabilities mean that 3D models and simulations are no longer restricted to computer training or design. Their ability to display difficult information in a more humanly accessible manner by providing a powerful means of effectively communicating spatial information expands their role in the future of space exploration to include 3D virtual training, mission support tools, and remote operations. If combined with embedded multi-person collaboration and small file sizes for low bandwidth transfers, 3D simulation solutions can create a complete package for more effective communications both during missions and in a Lunar-Earth network by bringing together experts and users distributed throughout the world for training and mission support.

II. Integration of Training, Mission Operations, and Remote Control Today’s crew training (orbital and ground) occurs over many years prior to the mission on a wide range of tasks.

The training requires enormous resources in crew time and travel. We estimated that with a single, complex ISS payload, about $500,000 could have been saved in training and trainer costs alone.

As shown in Figure 1, 3D visualization technologies allow a common platform among training, mission control operations support, real-time crew support, and remote automation control, especially when integrated to system health management and control. By using the same models, scripted and programmed for different purposes provides commonality throughout training and operations interfaces. The same models can be used for interactive training with embedded intelligent tutorial knowledge, be used in mission operations driven by real state data, support situation awareness and guide personnel through complicated procedures, and be used for virtual interfaces in remote operations. The fact that these models and the behavioral programming behind them are portable and usable in nearly any digital environment, they may be used in real-time – significant pre-training can be limited to extreme safety related reflex reaction procedures and actions.

The lightweight procedures and intelligent simulations can also be embedded directly into the system, be fed state data from the system health management system and provide visualization-based instruction for status monitoring, procedure identification, and procedure guidance at the point of service, or remotely in mission control. A 3D visualization interface can be used for personnel to interact with a system as though they were interacting with the actual system, with the information collected from the virtual interface and sent to the remote systems and the real state data from the system sent back to the virtual interface to update the simulation model. This type of system may be used for training or operations. Thus, the intelligent, data-interactive 3D visualizations close the gap between training, real-time operations, mission and system control, and remote automation control.

American Institute of Aeronautics and Astronautics

3

Data-driven, intelligent

simulation and visualization

Operations Support

Remote automation controlTraining

- Embedded crew operations

- Mission operations

Data-driven, intelligent

simulation and visualization

Operations Support

Remote automation control

Remote automation controlTrainingTraining

- Embedded crew operations

- Mission operations

Figure 1. Portable simulation and visualization provides a common platform for and bridges gaps between

training and operations.

III. Benefits to Exploration Though future mission equipment has become more complex, 3D simulation is an excellent medium for

communicating status, procedure or other visually-oriented information. Table 1 summarizes its benefits.

Table 1. Benefits of 3D simulation for status and procedure display.

Human Factors Benefits Technological Benefits • Visually realistic simulations provide a faster

understanding of spatial relationships • 3D Animation helps understanding of dynamic

procedures • Visual status displays and procedure materials

are naturally more amenable to understanding by an international crew

• Increased personnel proficiency and confidence due to the availability of effective just-in-time information for emergency conditions

• Electronic real-time distribution is possible due to the language interface and small graphic file size relative to other visual approaches

• Geometric descriptions of objects are shared among many different procedures resulting in file size benefits as the amount of instructional material increases

• 3D simulations are easy to internationalize due to their modular nature

• Reduced training time results in costs savings in personnel and training staff expenses

PLANET LLC and its implementation partner Orbital Technologies Corporation (ORBITEC) have spent over

four years developing solutions to demonstrate the efficacy of these tools for NASA missions and equipment. One of the first systems delivered to NASA was used to develop ground-based training and operational support computer-based tools for several payloads and International Space Station systems. One of these systems now used for astronaut training was built for the Microgravity Experiment Research Locker Incubator (MERLIN) and Lotec Active Door Recharger (LADR). It contained parts familiarization, procedure simulations, and interactive models of the MERLIN payloads with sufficient information to perform nominal, malfunction, and contingency operations (Figure ). When this 3D virtual training system was evaluated by NASA trainers, astronauts and Space Station

American Institute of Aeronautics and Astronautics

4

engineers in August of 2004, Astronaut Peggy Whitson remarked, “I wish I could have had this on all the systems I trained with.”

(a) (b)

(c) (d)

Figure 2. The MERLIN interactive 3D training model (a), training interface (b), manipulatable connectors (c) and display screen (d).

As this example shows, the flexibility of a 3D simulation platform provides several benefits to a number of applications for exploration.

Improved safety and reliability of off-nominal troubleshooting and repair: Long-duration manned missions to the moon and beyond require astronauts to be flexible and prepared for emergency or repair conditions. These situations will call on astronauts to perform procedures not previously trained for, or trained for years past. 3D simulations of relevant systems and components on-board organized in a procedure management system will allow rehearsal of all nominal/off-nominal procedures prior to execution. Low file sizes and collaborative capability will also allow ground-based experts to manipulate simulations on-orbit for troubleshooting support or to provide real-time training.

In-space assembly and maintenance: Manned moon missions will require extensive in-space assembly for the infrastructure to support human life. PLANET LLC is currently looking into ways to integrate viewing capacity of 3D procedures, models, and assembly information into the display helmet on the next generation spacesuit, as conceptualized in Figure. Through a voice-activated system, astronauts could visually train in real-time for in-space assembly. This capability, when combined with on-board procedure management, will increase the integrity of the missions while reducing training and hardware costs dramatically.

American Institute of Aeronautics and Astronautics

5

Figure 3. Conceptual view of 3D simulation in EVA and mission support.

Affordable ground-based mission preparation and training: Immediate cost and time savings is realized by

replacing hands-on hardware training with 3D virtual training derived from existing design information. These training simulations are not meant to totally replace all hardware and high-fidelity mockup trainers, but can reduce the amount of mockups required, and provide astronauts with portable trainers that are common to real-time and refresher training aids available on-orbit. This will save millions of dollars in high fidelity mockups and ground-specific training equipment, and save an estimated 50% of astronauts’ time for training support.

Embedded manned space vehicle integrated health maintenance: Combining a simulation-based user interface with a vehicle health maintenance system greatly improves the crews’ ability to perform rapid diagnosis and repair. Simulation systems can immediately visually map the location of a problem and link to a simulated repair procedure.

A. Future Benefits It is important to note that close to real-time collaborative operations support may also be achievable using this

technology. “Tele-support” is one of the most fascinating and promising aspects of 3D simulation with the low file sizes attainable by systems such as the Hypercosm platform from PLANET and ORBITEC. This collaboration capability will make it possible for the flight crew to work out procedures with the assistance from the ground anywhere within the protected ISS, Orion, or future lunar network. It would include a network of experts on the ground including vehicle engineers and NASA personnel. This will result in more effective communications with the ground and geographically distributed experts, while increasing the ability to recover from off-nominal situations, which may have significant performance and safety implications.

Figure shows a mock-up of a future collaborative system that could be used on-orbit. The current bandwidth available to space however is quite limited. For data uplinks, an S-band connection is used with a bandwidth of 57 Kbps, which is only slightly better than a dial-up 56 Kbps modem connection. For data downlinks, a Ku-band connection with a better capacity of 43.2 Mbps is used. (For collaborative use, we are limited by the slowest connection since data communications must be two-way.) Fortunately, in a Hypercosm collaborative 3D simulation the only data that must be sent over the network is information describing each user’s interactions with the simulation. Each computer simulation can then generate the exact same graphical display while keeping well within the bandwidth limitations.

American Institute of Aeronautics and Astronautics

6

Collaboration Center

In-flight Console

Collaboration Center

Mission Control Console

Figure 4. Conceptual mock-up of a collaboration system used for spaceflight.

IV. Current Technology Developments As previously mentioned, PLANET LLC and Orbital Technologies Corporation are developing a software

solution called Hypercosm for these types of aerospace applications. Hypercosm software represents the next generation of online delivery or embedded interactive 3D content. The patented Hypercosm format for web-based 3D simulation is based on the procedural modeling method. Unlike other methods with strictly graphical formats, Hypercosm simulations run from instructions that reference data or equations, making it possible to handle physics required for complex scenario visualization. These instruction files and not graphics are transmitted in a Hypercosm simulation resulting in uncharacteristically small file sizes.

B. Specific Technology Benefits of the Hypercosm System The Hypercosm technology has a number of key benefits which are critical to enabling the development and

delivery of complex rich content including low file sizes, flexibility, easier content creation and high performance. 1. Low File Size

• Uses procedural compression for very small file sizes • Files range from 50% smaller for unstructured CAD files to between 10 and 100 times smaller for files

that have been optimized to take advantage of the procedural compression • Allows practical and effective web distribution

Hypercosm tools enable compact transmission of three dimensional computer graphics and simulations over low bandwidth communications. The low bandwidth is a result Hypercosm’s patented procedural description of objects and behaviors (U.S. Patent #6,426,748). Using this approach, rather than explicitly encoding and sending detailed descriptions of complex objects in terms of low level polygon or mesh data, we encode the object as a series of instructions on how to create the object from a small amount of data and then construct the object when it is needed. The conventional approach would be to encode each model as a huge collection of tiny 3D triangles (polygons) that approximate the shape of structures. Our approach alternatively encodes these structures as a series of points that define the paths that outlines the structure and code on how to generate these structures from this data. This approach involves an order of magnitude or less data. This enables the content to be effectively distributed electronically and placed in the hands of those that need the information the most.

American Institute of Aeronautics and Astronautics

7

2. Flexibility • Graphics and simulation capabilities are expandable through a high performance, compiled scripting

language with specific extensions for 3D graphics (OMAR) • User interface capabilities are expandable through integration with JavaScript and Flash • System capabilities are expandable through TCP/IP networking capabilities and integration with Java

The Hypercosm system was designed to be as flexible and expandable as possible. To do this, the system was based upon a full modern object oriented high performance scripting language. This scripting language allows a virtually unlimited range of new features, behaviors, and graphical objects to be added to the system. No fixed file format, such as VRML or X3D has the flexibility to be significantly expanded, which is why these file formats are always followed by a large number of proposed extensions. By contrast, the Hypercosm system allows new features to be developed and added on using a high performance compiled scripting language.

In addition, because of its web oriented nature, the Hypercosm system has been designed to be modular. This allows an author to incorporate Hypercosm applets into larger systems and to build complex user interfaces using a range of standard tools to drive 3D simulations. Hypercosm even has real-time networking extensions built in to drive Hypercosm simulations using sources of real-time data or build collaborative applications where multiple participants can interact.

3. Easier Content Creation

• A high level scripting language allows simulations to be written in an order of magnitude fewer lines of code and dramatically less time

• Less code to write means less code to maintain • Faster development cycles save time and money

The Hypercosm system uses a high-level, interpreted programming language that was built specifically for 3D graphics, called OMAR. This language provides the means to generate objects and interfaces with any set of behaviors. The fact that behaviors are controlled through a high-level interpreted language rather than a low-level language means that most interactive content may be developed quickly and economically. In the past, complex simulations had to be created using low level programming in C or C++. This resulted in development times of many weeks or months for even basic simulations which made most simulations economically infeasible to develop. Using the Hypercosm scripting oriented approach, a simulation that might ordinarily require many thousands of lines of code and many months of time for a highly skilled (and paid) programmer to develop can be created by a relatively inexperienced programmer in a few hundred lines of code in perhaps a few days or weeks. By lowering the development costs using this scripting-based approach, Hypercosm brings 3D simulation within reach of a wide range of new applications.

C. Applications Hypercosm 3D simulation software has been used for several aerospace and military applications for 3D

interfaces, maintenance, training and just-in-time support. In one new, innovative approach, circuit board schematics or electrical diagrams are transformed into digital graphic elements. Each element can then have information and intelligence may be embedded in the graphical schematic, which is linked to physical 3D model of the component or system. This turns a simple circuit diagram into an interactive schematic with embedding troubleshooting information. The concept is outlined in Figure 5 below. The linked 3D model acts as a visual roadmap, highlighting the exact location of the object in question (a) and voltage or current path in question to follow (b). Individual components can also be highlighted (c) and linked to embedded user manuals or other relevant information (d). A user can now quickly and easily locate the exact pin, component, or trace to follow during an emergency situation, rather than be required to search through cryptic diagrams that do not resemble actual hardware. All of these benefits combine to create a faster and more effective operations aid.

American Institute of Aeronautics and Astronautics

8

(a) (b)

(c) (d)

Figure 5. 3D Circuit board simulation (a), interactive schematic with highlighted 5 V trace (b), interactive board simulation with highlighted component switch (c), and linked switch user’s manual (d).

Hypercosm’s most recent implementation of this technology has been for a major defense contractor supplying an electronic integrated shipboard monitoring, troubleshooting, and repair system for the U.S. Navy. In this project, Hypercosm 3D simulations have been combined with Macromedia Flash and an existing web-based monitoring system to display an interactive signal flow diagram of the major system flows (direction, speed, communications, etc.) as seen in Figure 6. As a change to a particular flow occurs, the system links to 3D interactive repair procedures to quickly provide crew members with repair location and requirements.

Figure 6. Real-time flow diagram (left) and repair procedure (right).

Another NASA application to demonstrate Hypercosm’s ability to interface with external equipment was completed for robotic maintenance operations of spaceflight equipment of Figure 7. The Special Purpose Dexterous Manipulator (SPDM) trainer demonstrated how Hypercosm simulations could be combined with different navigation and control options to train astronauts on the operation of a robotic arm. Using simple voice commands,

American Institute of Aeronautics and Astronautics

9

users could navigate through step-by-step procedure training, while utilizing a joystick to manipulate the model of the robot arm to test the procedure’s requirements.

(a) (b)

(c)

Figure 7. SPDM control station procedure (a), robotic arm (b), and stereo goggle display (c).

Hypercosm has also been selected as a tool for use in the V-22 Phase III Pilot and Enlisted Air Crew Training program. An example of how Hypercosm simulations are improving the courseware effectiveness can be shown in the Flight Control System (FCS) Controls of Figure 8.

Figure 8. Interactive thumbwheel switch (left) and joystick (right)

This hand controller demonstrates an item to bring in haptic-like behaviors to PC-simulation, along with the

importance of gauging motions. The small yellow thumbwheel highlighted in the bottom left quadrant of Figure 8 (left) must be rotated by a pilot to control the V-22 nacelles. In original courseware outline, a pilot “clicks” on the thumbwheel to cause a percent change of nacelle angle. However, in actual operation, the thumbwheel is controlled by a non-linear force spring that snaps back into its starting position if force is not maintained in the position desired. Using integrated physics equations, the correct rotation and force required to adjust the nacelles can be programmed, followed by the correct angle of rotation output. The joystick motion (Figure 8 right) is also tied to a file that plays an increasing pitched realistic sound as the flight system increases speed. These additional levels of detail gives the pilot or maintainer using the courseware a much greater understanding of the actual physical

American Institute of Aeronautics and Astronautics

10

operation and resulting behavior of the control system. Though they may appear simple additions, they simulate non-intuitive physical forces and results that are not easily grasped using only mouse-click simulation or 2D graphics.

Hypercosm has been used to complete several projects to demonstrate how 3D virtual systems can replace hardware trainers for NASA payloads and robotics, and to evaluate their use as trainer refreshers prior to or during flight. As shown earlier, the Microgravity Experiment Research Locker Incubator (MERLIN) and Lotec Active Door Recharger (LADR) virtual training system delivered a complete free-play simulation of equipment along with procedural training scenarios. Another NASA maintenance and operations training system was completed for the Biomass Production System payload. This training system was developed to maximize the use of crew training time. It features linked voice annotations that make it easy to integrate multiple languages for international users. The file size of the completed system is also compatible for distance learning, with a complete set of animated interactive 3D simulations (containing 56 animated steps) for just 2 MB (Figure 9).

Figure 9. NASA training prototype of BPS payload (left) and actual payload (right).

V. Conclusion NASA applications are extreme in their need for complex operational support tools that must be delivered

accurately and just-in-time. These tools deliver even higher value if based on 3D simulations with multiple uses including training, design visualization and simulated interfaces to equipment for robotic control. When used early in the mission operations timeline, they minimize the initial training time and hardware expenses, and provide a means of “just-in-time” training for off-nominal operations. Immediate applications of these technologies can be used to create payload operations or Space Station assembly task trainers instead of the current text-based paper and CBT instructions to reduce currently planned 1-hour refresher training sessions each day with 10-minute simulation training sessions. Saving astronaut-training time on the ground and on orbit can easily pay back in efficiency and effectiveness that is easily valued in the millions of dollars. In addition, the simulation trainers can be updated and maintained on a much more routine and frequent timeframe, as changes and even new simulations can be generated in hours to days and immediately transmitted where needed.

Combining a simulation-based user interface with a vehicle health maintenance system is natural progression for a technology that greatly improves the crews’ ability to perform rapid diagnosis and display comprehension. This system will also improve safety and reliability of off-nominal troubleshooting and repairs. During long-duration manned missions to the moon and beyond, astronauts are required to be flexible and prepared for emergencies and repairs. By integrating the IVHM system status display with 3D simulations of systems and components organized into a procedure management system, crew will be able to rehearse or predict behaviors of nominal/off-nominal procedures prior to execution. Low file sizes and collaborative capability also allow ground-based experts to manipulate simulations on orbit for troubleshooting support or to provide real-time training as needed.

Another exciting application is simulations for in-space assembly and maintenance. Manned moon missions will require extensive in-space assembly for the infrastructure to support human life. 3D procedures, models, and assembly information could also be integrated into the display helmet on the next generation spacesuit. Through a voice-activated system, astronauts could visually train in real-time for in-space assembly. This capability, when combined with on-board procedure management, will provide an encyclopedia of information at an astronaut’s immediate disposal while reducing training and hardware costs dramatically.

American Institute of Aeronautics and Astronautics

11

PLANET and ORBITEC continue to work together with NASA and their suppliers to create and deliver effective 3D training tools, and is looking to expand into mission operations support. We are also looking to expand into medical simulation tools for on-board emergency support and remote interfaces for ground-based control of equipment.