20
1 Test Process Management (TPM) Product Overview Software Tool Suite for Verification & Validation of Electronic Equipment
1
Test Process
Management (TPM)
Product Overview
Software Tool Suite for Verification
& Validation of Electronic Equipment
22
Table of Contents
Introduction 3
Test Life Cycle Overview 4
TPM Concept 5
TPM Modules 6
TPM Creation 7
TPM Editor 8 TPM Test Definition Elements 9
TPM SIgnal Browser 12
Test Execution 13
TPM Executor 14
Test Documentation 15
TPM Reporter 16
Test Scenario Example 16
Summary 18
3
Introduction
TPM Overview
TechSAT’s Test Process Management (TPM) application is a comprehensive software tool suite for managing the entire life-cycle involved in the testing of electronic equipment. TPM supports you in all phases of the test process, from importing test requirements and system ICDs, creating test cases and test plans, executing tests, to generating test reports and performing impact analyses.As an Automatic Test Equipment (ATE) for testing, validating, and certifying electronic equipment, TPM can perform both interactive and fully automated tests. Therefore, TPM is ideally suited for production, integration, and maintenance testing.
Introduction
TPM Benefits
TPM offers the following benefits:
Reduced costs through the use of a single tool suite for the entire test life-cycle >
Reduced test implementation effort through the use of intuitive form-based test definitions >
Complete traceability from system requirement to test result >
Increased productivity through a high degree of automation during all stages of the test >
processImproved test quality through consistent use of the same test process >
TPMTest Process Management
44
The figure below shows the test life-cycle using DOORS™ together with TechSAT’s TPM and Avionics Development System ADS2.
DOORS is used for managing system require-ments, test requirements, test results, and all interrelations between them, thereby providing traceability.
Test Life-Cycle Overview
By importing test requirements from DOORS into TPM, it is possible to define test cases, based on which test plans can be established. The test plans are then compiled and built into test executables. These can be executed on ADS2, which serves as the test execution engine.
During test execution, the test results are fed back to TPM to generate a test result report. Subsequently, the test results can be exported to DOORS to create result and traceability matrices.
Functional/System
Requirements
Functional/System
Requirements
Test Cases& Test Plan
Test Cases& Test Plan
TestRequirements
TestRequirements
TestResultsTest
Results
TestResult Report
TestResult Report
ResultMatrix
ResultMatrix
TraceabilityMatrix
TraceabilityMatrix
TestExecution
TestExecution
DOORS
TechSAT ADS2
TechSAT TPM
TestExecutable
TestExecutable
5
TPM Concept
TPM Life-Cycle Overview and Concept
The test category covered by TPM is »black box testing« of any equipment that is connect-ed to its environment through I/O connections. The idea is to stimulate the inputs of the unit under test (UUT) and verify the outputs against the expected values.
TPM test programs do not access the I/O lines directly, but use the concept of signals as an interface. Signals are named data objects of different data types carrying a value and a timestamp. TPM manipulates and verifies only such signals. The transfer of signals to/from physical I/O and thus to/from the UUT is performed by the underlying test execution engine based on ADS2. The following diagram shows the data flow between TPM and a unit under test:
created with TPM Editor,then compiled to executable test program
TPM Compiler
ADS2 Execution Engine
TPMTest Program
unit under test
Signals
I/O Connections
I/O Hardware
66
TPM Editor � – used to create and modify test cases based on the built-in test defi-nition elements. These can be linked to test requirements and access signals as defined in the system ICD. Test cases can be combined to test plans.
TPM Test Definition Elements � - used to create test cases. TPM provides differ-ent types of test definition elements, e.g. for signal stimulation and verification, flow control, and event handling.
TPM Compiler � – generates C code from the defined test plan which is then built to a test executable that can be executed on the test rig. The compiler is an integral part of the TPM Editor.
TPM Modules
The TPM software suite is comprised of the following modules:
TPM Signal Browser � – used to select signals and insert them in test definition elements. The browser allows importing system ICDs and provides powerful search and filter options.
TPM Executor � – manages the test execu-tion and offers monitoring, debugging, and tracing capabilities.
TPM Reporter � – creates customizable test result reports based on the test case defi-nitions and the test results. Additionally, the TPM Reporter can generate test descrip-tion reports before the tests are executed.
The following diagram shows the TPM modules and the data flow between them.
TPMEditor
Requirements
ICD
TPMCompiler
TestPlan
TestExecutable
TPMExecutor
TestResult
TPMReporter
Test ResultReport
TPMReporter
TestDescription
Report
7
TPM Modules and Test Creation
Test Creation
The TPM Editor can import test require-ments from requirement engineering tools like DOORS through XML or CSV files.
By importing a requirement document (e.g. a DOORS module), a test definition document is created, which has the same structure as the imported requirement document. For each requirement an empty test case is created,which is linked to the corresponding require-ment. The test definition document can be navigated through a tree browser, which reflects the structure of the document.
According to the associated requirement each test case has to perform certain operations on the system under test to determine if the requirement is fulfilled. These operations are carried out on process variables or signals, which are defined in the system Interface Control Document (ICD). With the TPM Sig-nal Browser, the ICD can be imported in XML format and stored in a signal database, which can be accessed by the test cases. The sig-nals can therefore be referenced directly in the test cases or inserted from the signal browser via drag and drop.
When requirements are imported, TPM creates merely the test case structure. It is therefore up to the test engineer to implement each test case based on the test concepts. The test cases are implemented using the test defini-tion forms in the TPM Editor.
Once the test cases are implemented, it is possible to package subsets into a test plan. After invoking the editor‘s integrated compiler, it first converts the selected test plan into C-code. Subsequently a C-compiler compiles and builds the test plan into a test executable which will execute on the test engine, perform the specified system operations, and deliver the test results. Currently, the TPM Compiler can generate test executables for Windows, Linux, and VxWorks platforms.
An additional output of the TPM Compiler is the test job, which is a complete descrip-tion of all test plan components, including the name of the author, revision number, etc. This information is used to create test description reports as well as test result reports.
Requirements(e.g. DOORS)
System ICD
Test Plan TestExecutable
Test Job
Editor Compiler
Test Creation
Test Concept
SignalBrowser
Test Case
The flowchart above shows the test creation process with TPM.
88
TPM Editor The TPM Editor GUI and its main components are depicted below.
The test project view (see above) displays the components of a test project: the »test case« list and the »test plan« list. All test cases are organized in the test case list, and all test plans are organized in the test plan list.
By clicking the expand/collapse icons, you can expand or collapse the content of the test definition element, thereby changing the level of detail displayed in the view.
When you select a particular definition element in the test project view, its properties are dis-played in the test definition element property view for editing.
The online help view provides detailed informa-tion on how the definition element works and how it can be configured.
The console output view lists diagnostic infor-mation of the TPM Editor and compiler.
1 Test Project View
2 Test Definition Element Property View
3 Online Help View
4 Console Output View
12
3
4
9
1 Test Project View
2 Test Definition Element Property View
3 Online Help View
4 Console Output View
TPM Editor and Test Definition Elements
TPM Test Definition Elements
Test definition elements are used to implement test cases. They are represented in corresponding forms provided by the TPM Editor. The main test definition elements are depicted below.
1010
Test Case
The Test Case is a container for the test pro-cedure and the corresponding test require-ment. It is usually created automatically when you import test requirements into the TPM Editor. However, you can create additional test cases if desired. Test cases can be referenced in test plans to create specific test scenarios.
Test Requirement A Test Requirement element is created for every imported test requirement. Test Require-ments contain a general description of the requirement as well as a description of the required system conditions and the expected results.
Test Procedure The Test Procedure is a container for the executable part of the test case. It typically contains an initialization component called a Prologue, one or more Test Steps, and an Epi-logue.
Test StepThe Test Step implements a sequence of sys-tem operations executed in real time. Typically, it first stimulates the system, then waits for an event to occur, and finally verifies that the sys-tem reacts as expected, i.e. that it fulfills the system requirement associated with the Test Step.
Prologue
The Prologue is a non-realtime element used to initialize and prepare the system for the subsequent Test Steps. The Prologue is typi-cally used to preset signals and load resourc-es, such as simulations and panels.
Epilogue The Epilogue is a non-realtime element which concludes the Test Procedure. It is typically used to reset the system to its initial state and to unload previously loaded resources.
Time Slot The Test Step contains one or more Time Slot definitions. Time Slots are defined by absolute or relative time, by events, or by conditions. Within a Time Slot, multiple signal stimulation or verification operations are performed con-currently.
Time Slots may overlap, thus allowing a very natural way of defining test conditions. Examples:
2 sec after PowerOn until PowerOff: >
verify voltage > 10Von WOW == False until WOW == True >
verify doors all closed
Time Slots are event controlled, i.e. the execu-tion of the contained statements depends on the occurrence of the event that triggers the time slot. Typical events are, for instance, the elapse of a time delay or a signal reaching a certain value. Per default, a Time Slot is trig-gered when the previous Time Slot is finished.
To trigger the execution of Time Slots, you can either use system events or define your own events.
11
TPM Test Definition Elements
VerifyNoChange – verifies that the signal >
value remains unchanged during the time slot period.
Trigger � Statements
Trigger statements are used to generate events on certain conditions. When an trig-ger expression evaluates to true, its asso-ciated event is raised possibly starting or terminating a time slot.
BranchCondition Statement �
This statement is used to define a list of trigger statements. The BranchCondition Statement can raise one event only.
Dialog Statements �
These statements are used to create message boxes and input dialogs.
UserCode Statement �
The available TPM statements are suf-ficient to implement most test cases. However, for special complex behavior, the TPM statements may not be sufficient to define a test. In this case TPM provides the framework to integrate customized C-code through the UserCode Statement. Using this statement, TPM will automatically generate templates for the include file, the source file, and the make file. This makes it simple to add the necessary code and integrate it into the Test Case.
LibrariesOften test cases have certain repeating ele-ments. For this purpose, TPM provides the Library Definition Element, which makes it possible to create libraries of Test Procedures, Test Steps, Prologues, and Epilogues. These can be referenced (called) in different test cases.
StatementsStatements are actions such as signal stimula-tion, signal verification, and dialog pop-ups. All Statements grouped in the same Time Slot are executed in parallel. By default, a Time Slot finishes when the last active statement in the Time Slot has finished. However, if you specify a timeout, the Time Slot is forced to finish upon the elapse of the timeout period, regard-less whether there are still running statements.
TPM supports the following types of state-ments:
Stimulation Statements �
Stimulation statements are used to set signals to constant values or to values computed by a function generator.Examples:- set A to 5- hold B at 7- ramp A from 0 to 100The function generator outputs are con-tinously assigned to the signal as long as the statement is running.
Verify Statements �
Verify Statements are used to compare actual signal values with their expected val-ues for deciding whether a requirement is fulfilled or not. TPM offers several types of Verify Statements, including:
VerifyLimits – verifies whether the signal >
value is between the given minimum and maximum value.VerifyTolerance – verifies whether the >
signal value is within the tolerance of the given nominal value.VerifyExact – verifies whether the signal >
value exactly matches the nominal value.VerifyExpression – verifies whether the >
given expression is true.
1212
TPM Signal Browser
The Signal Browser is a support utility for the Test Editor. It is used to select and insert signals into test definition elements. For this purpose, the TPM Signal Browser can import the ICD of the system under test. The Signal Browser GUI is shown below.
In addition to its ICD import functionality, the Signal Browser can be used to search for required signals in the database. Found signals can be selected with the mouse and inserted into the test cases via drag and drop.
You can search the database for particular signals by applying custom match patterns on signal names or on information which is specific to the signal type. For example, you can search for AFDX signals with a certain VL identifier and for MIL-STD-1553 signals with a certain RT address.The Signal Browser has four built-in filter types:
1) CVT2) AFDX 3) A429 4) MIL-STD-1553
The CVT filter is a search filter based on signal names only. All other filters provide search cri-teria that are specific to a particular I/O type.
13
TPM Signal Browser and Test Execution
Test Execution
The flowchart above illustrates the Test Execu-tion phase, which follows immediately after the Test Creation. The Test Execution is man-aged by the TPM Executor, which uses, as described earlier, the test executable and the test job as inputs. To perform the execution, the Executor additionally needs hardware and software resources, such as the test rig with the test execution engine (ADS2), simulations, etc.
The Executor does not only allow loading and executing different test executables, but also provides means to control the execution using step execution with user-defined step time intervals.
The TPM Executor generates a test log file, which contains the value history of all signals referenced in the executable. The test log file is very useful for tracing and post-analyzing signal values as they were recorded during the test run. The test log file can be customized to contain only information on a selected number of signals. Besides in ASCII format, the test log file is also stored in the SQLite database format (sld) for better browse, filter, and search capabilities.
Finally, the Executor generates a test result file in XML format. The test result file is based on the test job file and the output of the test exe-cutable. The following information is contained in the test result file:
Exectuted test components (test compo- �nents that were not executed are discard-ed) and their associated results (passed/failed)
Information on potential user interference �during execution, for example, aborting, pausing to step through, etc.
Absolute timestamp information regarding �the start and end of individual test compo-nents as well as the entire test
Miscellaneous information such as the �name of the test person, the TPM Executor revision, etc.
From this pool of information, the TPM Test Reporter builds customizable reports.
Requirements(e.g. DOORS)
Signal Browser
Test Plan
HW & SW Resources
TestExecutable
Test Job
Test Result
Test Log
Editor Compiler Executor
Test Creation Test Execution
Test Concept
Test Case
1414
TPM ExecutorThe TPM Executor GUI is divided into six areas as shown below.
1 Test Job View
2 Test Component Properties View
3 Execution Console View
4 Execution Control View
5 Execution Engine (ADS2) Log View
6 Signal View
The Test Job View on the left-hand side dis-plays the tree structure of the loaded test job. By clicking a test component in the tree struc-ture, you can display the corresponding prop-erties in the Test Component Properties View.
You start the test execution by clicking the „START“ button in the Execution Control View, and you terminate it by clicking the „STOP“ button. When you stop the execution, the Executor will finish the current cycle and then execute the Epilogue component before termi-nating the test. Furthermore, you may pause the automatic execution and continue the test manually step by step in time intervals speci-fied either in seconds or number of cycle times (or time frames) of the underlying execution engine. You can resume the normal execution at any time.
During execution the execution details are automatically displayed in the Execution Con-sole View. This view does not only list the test component currently executed, but also the absolute timestamp of its execution and the associated events. For better visualization, the Execution Console View highlights passed verify statements in green and failed verify statements in red.During test execution the tree structure in the Test Job View is automatically expanded, thereby indicating the execution progress. Analog to the Execution Console View, exe-cuted test procedures are highlighted either in green or red.Finally, the Execution Engine Log View pro-vides information on the status of the ADS2 execution engine. You can set the level of information detail by selecting the message categories to be logged: errors, warnings, and information.
1
2
3
4
5
6
15
Test Execution and Test Documentation
Test Documentation
The last step of the test process involves the test documentation, i.e. the generation of a test report in PDF format, and the transfer of the test results back to the requirements tool, if such a tool is used.
As depicted in the flowchart above, test reports can be generated at two points in the test process.First, you can create a test description report based on the test job file generated by the TPM Compiler. Usually, you create the test description report before you start the test. This way you can print an overview of the test you intend to execute.
Second, the TPM Reporter allows you to generate a test result report based on the test result file. The test description report and the test result report are identical in struc-ture. However, only the latter contains actual results.
The test result report serves to evaluate the test run. Only if each individual verify state-ment evaluates to „passed“, the test as a whole will do so, too. If an error occurs, the report indicates its location and, if possible, also its cause.
Besides generating the test result report, you can also use the test result file to transfer the results back to a requirements tool for con-ducting statistical analyses. With a configu-rable script included in TPM, you can extract result information from the test result file and store it to a separate file in comma-separated values (CSV) format. Then you can import the CSV file to a requirements tools such as DOORS.
Requirements(e.g. DOORS)
System ICD
Test Plan
Test Description
Report
HW & SW Resources
TestExecutable
Test Job
Test Result
Test Log
Editor Compiler Executor
Reporter
Test Creation Test Execution
Test Documentation
Test Result Report
TRGReporter
Test Concept
Signal
Browser
Test Case
1616
TPM ReporterThe TPM Reporter GUI is divided into five sections as depicted in the screenshot below.
The Icon Bar Section contains two icons to choose between the two modes of report gen-eration: test description report („Description mode“) or test result report („Result mode“). Additionally, it contains an icon to start the report generation.
The Report Selection Section lets you select particular test jobs from the available test proj-ects. In „Description mode“ a test description report can be generated. In „Result mode“, however, you can select a particular test run of a test job and generate the corresponding test result report. When you select a test run, the Result Overview Section provides a result overview without the need to generate the report beforehand.
In the Report Configuration Section you can tailor the report content and layout accord-ing to your wishes. On the one hand, you can select from numerous check boxes to decide what type of information you want to include in the report. On the other hand, you can define XML style sheets (XSL) to customize every aspect of the report page layout, including logos, headers, and footers.
TPM comes with various default style sheets suitable for different purposes, such as »debug level« reporting, »management level« reporting, etc.
During report generation, the generation prog-ress is displayed in the Log and Control Sec-tion of the GUI. In this section, you can also stop the ongoing report generation.
1 Icon Bar Section
2 Report Selection Section
3 Result Overview Section
4 Report Configuration Section
5 Log and Control Section
1
2
3
4
5
17
TPM Reporter and Test Scenario
Test Scenario Example
Power Switch VerificationTest Requirements �
(1) Verify that the power_ok condition >
becomes true within one second after turning power on.(2) Verify that the voltage is 10 V +/- >
0.1 V and the current is > the 2.5 A after power_ok for at least 10 seconds.(3) Verify that 0.4 sec after switching >
power off, the voltage is 0 V +/- 0.1 V and the current is < 0.1 A.
The following is an example of how to formulate a test requirement with TPM.
Create Test Case Now we define the test case corresponding to the above test requirements:
We divide the whole requirement into two test steps, one is power_on and the other is power_off.
TestStep power_on:Start: �
trigger PowerOnFail on timeout==1.0sec >
set power = 1 >
Trigger PowerReady on power_ok==1 >
Ver: OnEvent PowerReady, for 10.0sec �
Verify, ALWAYS, Voltage == 10.0V, >
Tolerance=0.1VVerify, ALWAYS, Current > 2.5A >
OnEvent PowerOnFail: �
Popup Message “FAILED” >
TestStep power_off:Start: Delay = 1.0, �
Set power = 0 >
Ver: After 0.4sec for 10.0sec: �
Verify, ALWAYS, Voltage == 0.0, Toler- >
ance = 0.1Verify, ALWAYS, Current < 2.5 >
1818
Within the Test Step power_on, there are three Time Slots: Start, Ver, and Fail. Within the Time Slot Start, the signal power is set to 1 (which means power is turned on). Furthermore, an event called PowerReady will be triggered when the condition power_ok == 1 becomes true.
The two Statements Set and Trigger should be finished within one second; else, the event PowerOnFail will be triggered. Within the Time Slot Ver voltage and current values are verified against the expected values for 10 seconds.
The Time Slot Fail is only used to display an error message in case of a timeout of Time Slot Start, that means that the power_ok sig-nal is not raised for 1.0 second after setting the power to 1.
The power_off test step is similar.
The following screenshot shows the result of the test definition of this example in the TPM Editor.
After defining the test case, you can click the Build icon in the menu bar to call the TPM Compiler to build the executable C file from the test case. When the TPM Compiler has successfully compiled the test case, you can use the TPM
Executor to select the required test job. As soon as you click the Start button, the test job is executed automatically in the ADS2 environ-ment. The test results will be saved automatically to the specified path and according to the layout defined before.
19
SummaryThe following lists some of the highlights of TPM:
All-in-one software tool suite covering the entire test lifecycle of electronic equipment �
High degree of automation in the entire test process �
Easy and intuitive test implementation based on input forms for standard test components �
Full traceability between test requirements, test case(s)/plan(s), and test results �
C-Code interface for complex test cases �
Support for libraries for code-reusability �
SQLite database with GUI for easy import and handling of the system ICD �
Test execution control with debugging capabilities �
Tailored test report generation �
Integrated version control (Clear Case, PVCS, and CVS) �
Support for Windows, Linux, and VxWorks platforms �
1919
Summery
techSAT GmbH Technical Systems for Avionics and Test
Gruber Strasse 46c • 85586 Poing • Germany
Tel +49 (8121) 703-0 Fax +49 (8121) 703-177
techSAT Hamburg GmbH
Hein-Saß-Weg 36 • 21129 Hamburg • Germany Tel +49 (40) 333-979-45 Fax +49 (40) 333-979 54 [email protected]
techSAT North America
Lake Washington Air Harbor • Hangar Three 3849 N.E. 98th Street • Seattle WA 98115 • USA
Tel +1 (206) 910-3908 [email protected]
www.techsat.com
This document describes TechSAT's Test Process Management (TPM) tool suite.The information in this document is subject to change without further notice. TechSAT assumes no responsibility
for any error that may appear in this document.
All product names are (registered) trademarks of the referenced companies.
Copyright © 2010 TechSAT GmbH / Rev-1002. All rights reserved.
