Robustness Testing&Research Infusion

IV&V Facility

Robert BallFairmont State University

2

IV&V Facility

Robustness Testingwith Fault Injection

Robustness - the degree to which a system operates correctly in the presence of exceptional inputs or stressful environmental conditions

The goal of robustness testing is to develop test cases and test environments where a system's robustness can be assessed.

3

IV&V Facility

Process Overview Fault Injection

Preparation Test Execution Log Analysis

Documentation Describes scope of test

campaign Lists directives/routines

tested Describes test results

4

IV&V Facility

RTOS API Test Results

RTOS Directives found in Project

FSW

Directives found in API Test Results and

Project FSW

Flight Software Traces

Source Navigator

Source Code

5

IV&V Facility

Test Execution

6

IV&V Facility

DocumentationSample Fault Table

Declaration int taskSpawn ( char * name, int priority, int options, int stackSize, FUNCPTR entryPt, int arg1, int arg2, int arg3, int arg4, int arg5, int arg6, int arg7, int arg8, int arg9, int arg10 )

Fault Location C:\TornProj\Test2\usrAppInit.cline: 186

Test Case 1 set to a value outside of range 0 to 255

Results Error ID #: 0x3d001Error Description: S_objLib_OBJ_ID_ERRORAnalysis: Since the range of priority levels are limited to be between 0 and 255, anything out of this range is expected to return this error message.

7

IV&V Facility

Results VxWorks

HIGHLY AUTOMATED using scripts and support programs Testing VxWorks 5.5 Testing POSIX and Classic APIs GLAST Flight Software Traces – Complete nearly 300 functions used VxWorks Libraries

Task Management, Task Hook, Task Information, Task Show, Task Variables Support, Timer, Timex, Clock Tick Support, Watchdog Timer

Over 70 routines Progress

S:\NASA Shared\Research and Development Fisher \FacilityResearchProgram\Robustness Testing\VXWORKS \GLASTProgress.xls

Documents GLAST/VxWorks Robustness Testing

8

IV&V Facility

Automation Tools Test Case Generator Test Script Generator Various Scripts (Batch Files) Xception Robustness testing lends itself to automation,

saving time and money S:\NASA Shared\Research and Development -

Fisher\FacilityResearchProgram\Robustness Testing\VXWORKS

9

IV&V Facility

Applicability Any project that utilizes a Real-Time

Operating System with an Application Programmable Interface (API)VxWorksRTEMS

Both developers and analysts may find the tool useful

10

IV&V Facility

Research Infusion Project This project’s goal is to transition mature

software engineering research-derived technologies into practice by NASA and its contractors.

Our task was to evaluate the tools, create a lessons learned document, and support future users of the tools.

11

IV&V Facility

Tool Evaluation Criteria Is it a burden to process? Are features organized intuitively? Was good documentation included? Is it useful? What errors were encountered?