American Institute of Aeronautics and Astronautics1
Optimized Flight Control Component Testing Using TaguchiDesign of Experiments
Yogananda V. Jeppu1 and K. Karunakar2
Aeronautical Development Agency, Bangalore, 560093, India
Prakash R Apte3
Indian Institute of Technology Bombay, Mumbai, 400076, India
Taguchi Design of Experiments (DOE) has been used in manufacturing,marketing, chemistry and biological experimentation. Orthogonal arrays, apart of DOE, has been successfully used in reducing test cases for softwaretesting. Safety critical applications require extensive testing as proof ofrobustness for certification. Component based software development putrestriction on the test cases that can be generated for an end-to-end test. Asample component based safety critical flight control software has beenconsidered as an example for experimenting with the use of DOE. The aim ofthe exercise is to generate a single test case, which can test the scheduled two-dimensional table lookup completely for its data tabulation and interpolationalgorithm. 2D tables of gains and filter time constants are commonly used inflight control systems. A comparison with random test case generation andGenetic Algorithm has been carried to compare the efficacy of the DOEprocess. It is seen that a single test case can be generated by just 38experimentations using DOE as compared to 300 test cases for random and600 test cases in case of Genetic Algorithm.
NomenclatureTd = time for which the aircraft remains at a specific altitudeFz = frequency of the sinusoidal waveform for AltitudeAz = amplitude of the sinusoidal waveform for AltitudeBz = bias of the sinusoidal waveform for AltitudeFm = frequency of the sinusoidal waveform for Mach numberAm = amplitude of the sinusoidal waveform for Mach numberBm = bias of the sinusoidal waveform for Mach number
I. IntroductionOMPONENT based software engineering is a process that emphasizes the design andconstruction of computer based systems using reusable software components. Software, even
safety critical ones, is being developed using components used in other projects or by using
1 Scientist F, IFCS, ADA, PB No 1718, Vimanapura Post, Bangalore, 560017, India2 Group Director, IV&V, ADA, PB No 1718, Vimanapura Post, Bangalore, 560017, India.3 Professor, Reliability Engineering, Dept of Electrical Engineering, IITB, Powai, Mumbai, 400076, India
C
7th AIAA Aviation Technology, Integration and Operations Conference (ATIO)<BR> 2nd Centre of E18 - 20 September 2007, Belfast, Northern Ireland
American Institute of Aeronautics and Astronautics2
commercial off the shelf (COTS) components. Some of these components are qualified i.e.certified for use, in safety critical application. Others, however, may have been used in safetycritical application but not qualified as a component for reuse. Use of components make the taskof the system designer easy but is a huge burden on the tester. Use of certified components isacceptable to the project management but will still have to be tested to ensure that errors havenot been introduced during component adaptation. Component adaptation is a process ofbuilding in wrappers around the component to make it “fit in” the software architecture [1,2].The use of components makes the task still difficult in an end-to-end testing environment wherethe qualified components alter the signal characteristics of the outputs, which in turn are inputs toparts of software, developed in house. In such situations the problem lies in generating efficienttest cases such that the software under test can be adequately tested.
Genetic algorithm has been used to generate test cases automatically [3]. Optimizationtechniques help tester design test cases based on optimizing certain criteria. The most importantcriterion in software testing is the concept of coverage. The DO-178B standard lays out veryspecifically the coverage that has to be ensured while testing safety critical software based on thelevels of criticality [4]. Another optimization technique that has been used by testers is theconcept of orthogonal arrays. Orthogonal arrays reduce the number of test cases drastically whileensuring that pair wise coverage of input combination has been considered in the test casegeneration process [5]. There are several software applications available [6,7] for generatingorthogonal arrays given the test signals and their levels that have to be considered for testing.The Taguchi Design of Experiments (DOE) concept has been used for testing software [8]. Thispaper investigates the possibility of optimizing test cases using Taguchi DOE for a safety criticalapplication.
DOE is a powerful technique giving insight into a complex process and providing an engineerwith a means to optimize the process making it invariant to disturbances. However, testers haveused only a part of the DOE i.e. the orthogonal array for software testing. Here DOE has beensuccessfully applied to generate a single optimal test case for testing a safety critical application.The application is the gain scheduling algorithm for a flight control system. The novelty in thepaper is the application of this technique to design a set of optimal test inputs for an end-to-endtest activity such that adequate coverage is ensured. The test signals are injected into a certifiedsoftware component, which is in series with the software under test. The certified componentconsists of amplitude limiters, rate limiters and digital filters. These elements modify the inputsignal but in this case are treated as a black box with no knowledge of the internals. The softwareunder test is a component, which comes after the certified component but relies on it for its input.The concept of experimentation on software to get an insight into the working of the software isdemonstrated by the simple but realistic safety critical example described here.
II. Problem StatementAircraft avionics measure the speed and altitude of the aircraft based on the data collected
from the air stream [9]. This is carried out using probes and a complex algorithm consisting ofseveral look up tables to provide an accurate estimate of the speed in comparison to the speed ofsound called Mach number and the altitude in comparison with sea level based on theatmospheric pressure called Pressure Altitude. In fly by wire aircrafts this data is used toschedule the control system gains and filter time constants to ensure satisfactory performanceover the complete flight envelope [10]. The aircraft safety and performance depends on thesetwo components of the software namely the air data system and the control law. Needless to say
American Institute of Aeronautics and Astronautics3
CertifiedComponent
Air DataSoftware
SoftwareUnder Test
GainScheduler
for theControl
ControlLaw
Hardware
they are safety critical and warrant a stringent Independent Verification and Validation (IV&V)process for certification based on the standards like DO-178B.
Figure 1 shows the schematic of the system. The hardware provides the signals like the staticpressure of the atmosphere and the dynamic pressure due to the movement of the aircraft in theairstreams. The air data software processes these signals to provide the altitude and Machnumber information to the Gain Scheduler of the Control Law, which in turn computes the gainsand time constants, required for that flight condition for the Control Law. The software undertest is the Gain Scheduler component. The Airdata software is already certified to be working asper its functional requirements. However in an end-to-end test it is not possible to isolate the airdata system from the system under test.
Figure 1. Schematic of the test setup. The hardware component injects the air data sensoroutputs into certified Airdata processing software. The software under test uses the outputs ofthis component to generate gain data for the control law. The sinusoidal waveform injected intothe certified component is changed by the elements of the Air Data System.
The Air Data System provides two values to the Gain Scheduler normalized between 0.0 and1.0. The Mach number 0.0 stands for 0.0 Mach and 1.0 corresponds to Mach number 2.0. The0.0 Altitude value corresponds to 0.0 or sea level altitude and 1.0 corresponds to 15 Km altitude.The gain scheduler has a 22 x 8 array of 2D gain table lookup to be scheduled based on these twoparameters. There could be more than one set of gain tables scheduled based on these parametersin a flight control application. The objective of the test cases is to test the gain table and theinterpolation.
The test setup consists of a hardware emulator, which generates the static and dynamicpressure signals for the given height and mach number of the aircraft. This component ismodeled in software for on ground testing before flight clearance. The output of the air datasystem is two normalized signals used for the table lookup. The tester must be able to test thecomplete grid of 22 x 8 values provided for each set of gains. This will be possible if the testcase can generate at least one point in each cell by generating corresponding Mach number andAltitude values. This is shown in Figure 2. This test point will check for the surrounding 4 points
American Institute of Aeronautics and Astronautics4
and the linear interpolation algorithm. The correctness of the table data and the interpolationalgorithm shall be checked by comparison with a model code. However, the present study islimited to just finding at least one point in the grid. The tables will be completely covered if thereare at least 22 x 8 or 176 points, one in each cell.
M/A 0.0 2.1429 4.2857 6.42860.00.09520.19050.2857
Altitude
Mach TestPoint
Figure 2. Coverage for table lookup. The 2D lookup table is given for Mach number and Altitudevalues. A part of the look up table is shown. The table can be adequately covered if there is atlest one test point in each cell.
III. Taguchi Design of ExperimentsDesign Of Experiments (DOE) is a statistical technique first introduced by R. A. Fisher in
England in the 1920's. His aim was to study the effect of rain, water, fertilizer, sunshine, etc. onthe yield of crop. He designed a set of experiments using orthogonal arrays to limit the numberof experiments. As a researcher in Electronic Control Laboratory in Japan, Dr. Genechi Taguchicarried out significant research with DOE techniques in the late 1940's. He spent considerableeffort to make this experimental technique easy to apply and applied it to improve the quality ofmanufactured products. Dr. Taguchi's standardized version of DOE, popularly known as theTaguchi method or Taguchi approach, was introduced in the USA in the early 1980's. Today it isone of the most effective quality building tools used by engineers in all types of manufacturingactivities.
DOE has been used in testing software and software systems successfully [12,13]. In most ofthe cases where the DOE is used for software testing an orthogonal array is used to reduce thenumber of test cases. It is shown that the reduced number of test cases is efficient enough to finderrors in the software. In fact, DOE is more complex and useful and not just the use oforthogonal array for experimentation. It has been used to study the effects of multiple factors(variables, parameters, ingredients, etc.) on the performance, and solve production problems byobjectively laying out the investigative experiments. It has been used for process optimization inmanufacturing, advertising and experimentation in chemistry and biology [14].
The Taguchi approach lays down a systematic approach to the application of DOE to aproblem. The steps followed in every DOE are detailed in [14] and are described in brief here.
American Institute of Aeronautics and Astronautics5
BrainstormingThis is perhaps the most important step in the design process. The team gets together to
discuss the problem at hand. DOE considers a process to be affected by factors and their levels.In software, this would mean the various inputs to the software and its value or amplitude as thecase may be. The process is also affected by noise. This is the uncertainty. In a manufacturingprocess it would mean the skill of the worker, the weather conditions etc. In software this wouldbe the environment the software operates in, or the randomness on the known inputs etc. Theteam decides on these issues. The objective of the exercise is very important. What are we tryingto achieve? How do we measure it? This is an important issue that is debated in brainstormingsessions.
Designing ExperimentsThe next step is to tabulate the number of factors and the noise and their levels and planning
to formulate it in a way such that it can be put into a standard orthogonal array. Once an arrayhas been selected the experiments are tabulated with each column assigned to a factor and eachrow to a specific combination of factors and their levels forming the experiment. Theexperiments can now be run sequentially or in a random fashion.
Running ExperimentsThis step is physically running the experiment. It could be a computer simulation, a laboratory
experiment, a field study or a manufacturing process. The desired outputs are computed for eachexperiment run and tabulated manually or automatically. Many experiments may have to becarried out in a noisy environment to get a statistical estimate. Repetition and replication is notnecessary but desirable. The collected data is then analyzed to study the effect of the factors,their levels and interactions.
Analyzing ResultsThe analysis can be grouped into two types. In the first simple arithmetic computation provide
an insight into the experiment. The information in this case consists of Average Factor Effects(i.e. main effects), Optimum condition and estimated performance. In the second type statisticalcomputations are carried out to compute the relative influence of factors and the confidenceinterval on optimum performance.
Optimizing and ConfirmingThe optimal condition found during the analysis may not have been included as a test case. In
this case confirmatory tests are carried out with the optimal setting to confirm that the estimatedoptimum is close to the estimate.
The five steps described for the Taguchi DOE were applied to the problem stated above andare detailed in the next section.
American Institute of Aeronautics and Astronautics6
IV. DOE FormulationThe system under test is shown in Figure 1. The hardware elements are modeled in software
and take two inputs viz. the aircraft altitude and Mach number. During the flight-testing ofaircraft certain maneuvers are carried out to test the Airdata system. The simplest of them beingis to trim the aircraft at the lowest possible speed and at a specific altitude. The aircraft is thenspeeded up to the maximum possible speed at that altitude. The aircraft speed is then reduced toits start up speed. This maneuver scans the Mach number range at the specific altitude. The pilottakes the aircraft to a different altitude and repeats the experiment [15].
This procedure is adapted in the design of test cases to test the table lookup. The altitude isheld constant for a certain time Td seconds, at he initial altitude Bz km and then increased by BzKm till the final altitude of 15 Km is reached. This forms a staircase waveform. The Machnumber is a sinusoidal waveform of a specific frequency Fm Hz and amplitude Am about a meanvalue of Bm. This will emulate the scanning of the Mach number at a specific altitude. Anadditional feature in terms of a sinusoidal variation in Altitude over the bias in altitude Bz isadded to generate a dynamically changing scenario. The sinusoidal variation is defined by twoadditional factors namely the frequency Fz Hz and the amplitude Az Km. Three levels arechosen for each factor. Table 1 provides the factors, their units and the three levels L1, L2 andL3. The three levels have been chosen to provide a slowly varying signal with sufficientamplitude and bias to scan the 0.0 to 2.0 Mach number and 0.0 to 15.0 Km altitude range. Theseare tentative values and selected based on an engineering “feel for the problem” similar to theselection of initial conditions in any optimization problem.
LevelsSNo
Factor UnitL1 L2 L3
1 Td Sec 5 10 202 Fm Hz 0.01 0.05 0.13 Bm - 0.5 1.0 1.54 Am - 1 1.5 25 Fz Hz 0.01 0.05 0.16 Bz Km 1 2 57 Az Km 0 1 2
Table 1. Factors and Their Levels - Set I. The seven factors for DOE and their three levels areused as inputs for the experiment.
Designing ExperimentsThe Taguchi approach provides a set of standard orthogonal arrays for designing an
experiment. A L18 Array is selected for the experimentation. L stands for Latin Squares in theDOE terminology and the suffix 18 signifies the number of rows or number of experimentsdefined by the orthogonal array. L18 array can be used for 7 factors with 3 levels and 1 factorwith two levels. In the problem statement the first column is not used and is treated as dummy. Itmust be noted here that a full factorial test for all factors and level would lead to 3^7 or 2187tests. Columns corresponding to factors 2 to 8 are assigned to factors 1-7 of the experiment. TheL18 array used for the test case generation is shown in Table 2.
American Institute of Aeronautics and Astronautics7
The test case is designed with the inputs from the array and the simulation carried out for thetest setup described. The frequency, amplitude and bias of the two signals are changed accordingto the selected value and injected into the hardware emulator. The numerals 1,2 and 3 in thecolumns correspond to the levels L1, L2 and L3. The test case 10 corresponds to Td level 1 or5.0 seconds (Factor 2 or Col 3), Fm level 1 or 0.01 Hz and so on. The output measured is the“coverage” or the number of cells covered in the look up table. This is defined as the number ofcell with at least one point. Another output that has been considered in the experiment is the“wastage”. This is defined as the number of points in a cell greater than 3. All the 18 test caseswere executed and outputs generated. A typical waveform set generated using a test case shownin Figure 3. The sinusoidal Mach waveform and the staircase signal with the superimposedsinusoidal waveform for the Altitude is generated by the test case. The certified Airdata Systemchanges these signals due to the dynamic elements present in it to the corresponding normalizedwaveforms.
Table 2. L18 Orthogonal array. The L18 array can be used for designing test cases with 7 factorsof 3 levels and 1 factor of 2 levels.
American Institute of Aeronautics and Astronautics8
Figure 3. A typical test case signals. The dynamic elements change the input signal to thecorresponding normalized outputs.
Analysis of MeansAn analysis of the mean was carried out to analyze the main effect of the factors with their
levels on the coverage and wastage. The mean effect of a factor for a specific level is computedby computing the average of the outputs for test cases where the particular factor has thatspecific level. For example, the average effect of Td = 5.0 seconds (Factor 1, Level 1) iscomputed as the average of the outputs of test cases 1,2,3,10,11 and 12. This is compared withthe grand average or the average of all the test case outputs. Figure 4 shows this as the maineffect graph. The wastage criterion is divided by 100 to give a similar scale as the coverage. The
0 100 200 300-2
-1
0
1
2
3
Time (sec)
Mac
h
0 100 200 3000
5
10
15
20
Time (sec)
Alt
itu
de
0 100 200 3000.2
0.4
0.6
0.8
1
Time (sec)
M_n
orm
0 100 200 3000
0.2
0.4
0.6
0.8
1
Time (sec)
A_n
orm
Input Signal to Component and its Output
American Institute of Aeronautics and Astronautics9
mean effect of factor 1 at level 1 is 32.5 for coverage and 30.595 for wastage/100. Thecorresponding grand averages are 59 and 73.8583. This shows that level 1 of Td has a lessereffect on the global mean. Increasing Td to level 3 increases the mean effect of the factor for thatlevel to be higher than the grand mean. If one has to increase coverage one has to increase thelevel.
It can be seen from Figure 4 that increasing Td increases the coverage it also increases thewastage. This is not much of an inference, as it is quite obvious that executing the test for a longwill certainly increase the number of points. Fm and Bm show an optimum at level 2. Similar,insights can be gained about all the seven factors. This is an important aspect of Taguchi DOE.The analyst gets a good insight into the system behavior by executing a few test cases.
Figure 4 Analysis of means – main effect graph. The effect of the various factors and theirlevel 1, 2 and 3 are plotted with respect to the grand mean. The coverage and wastage metricsare shown.
OptimizingIt is clear from Fig 4 that factors with different levels affect the system differently. To obtain
an optimal solution DOE provides a parameter called the Quality Characteristics (QC). QC isdifferent from the measured outputs like “coverage” and “wastage” as it provides an aspect ofdesirability. QC is normally defined as “bigger is better”, smaller is better” or “nominal is best”.In this problem the QC or the desired output for coverage is bigger is better and for wastage issmaller is better.
Figure 4 provides this measure visually. Any output parameter, which is higher than the grandmean is desirable for bigger, is better. Higher the difference between the output and the grandmean more is it desired. This is true for “coverage”. The inverse is true for wastage. A lowervalue than the mean is desired for wastage. An optimal output is now computed by selecting thefactors and levels, which provide a higher coverage. Thus a L3, L2, L2, L1, L3, L1 and L3 forthe seven factors Td, Fm, Bm, Am, Fz, Bz and Az respectively would provide an optimal value.This specific test case however is not a part of the 18 experiments designed. A new experiment isdesigned using these levels selected for the factors. The 19th experiment gives coverage of 168
1 2 30
50
100
150Td
1 2 340
60
80
100Fm
1 2 340
60
80
100Bm
1 2 350
60
70
80
90Am
1 2 350
60
70
80
90Fz
1 2 30
50
100
150Bz
1 2 340
60
80
100Az
WasteCoverageMean CMean W
Analysis ofMean
American Institute of Aeronautics and Astronautics10
when compared to the maximum coverage obtained by a single test case in the 18 tests is only125 cells. Thus the coverage has been increased by the optimal test case. Another test case canbe selected from the 18 tests to cover the cells not covered by this optimal test case. Thecoverage of the optimal test case is shown in Fig 5.
The cumulative coverage of the 18 test cases is shown in Fig 6. The optimized test casecovers the regions not covered by the cumulative test case thus the 19 test cases together nowcover the entire array of 176 cells.
Optimised Coverage Set I
Altitude Grid
Mac
hG
rid
1 2 3 4 5 6 7 8
2
4
6
8
10
12
14
16
18
20
22
Cell Not Covered
Figure 5 Optimized table coverage for Set I. The figure shows the number of cells not covered bythe test case. A total of 8 cells are not covered by the Set I values for the 7 factors.
Altitude Grid
Mac
hG
rid
Coverage Cumulative 18 Tests
1 2 3 4 5 6 7 8
2
4
6
8
10
12
14
16
18
20
22
Cell Not Covered
Figure 6. Cumulative coverage of 18 test cases. The total coverage obtained by all the 18 testcases together is shown.
American Institute of Aeronautics and Astronautics11
Levels GAOptimal
SNo
Factor Unit
L1 L2 L3 -1 Td sec 12 15 20 13.02 Fm Hz 0.01 0.04 0.07 0.023 Bm - 0.8 1.0 1.2 1.06254 Am - 0.7 1.0 1.5 2.18755 Fz Hz 0.1 0.3 0.5 0.26256 Bz Km 0.5 1.0 1.5 0.69097 Az Km 0 2 4 3.1892
Table 4. Factors and Their Levels - Set I. The table gives the new levels used as Set II in theexperiment. The optimal values given by the genetic algorithm is provided in the last column.
The tester can further look at the main effect graph in Fig 4 and select another set of levels torun another set of 18 experiments to further improve the performance. This may not be necessaryin this trivial example but in situations where a test case can save considerable time and money alittle experimentation can go a long way. Another set of values was chosen for the test casegeneration. This is tabulated in Table 4. The value of Td has been changed from 5, 10, 20seconds to 12, 15 and 20. A large value of Td say 30 could be selected for the experimentationbut this would considerably increase the test execution time and the wastage. The values of Fmhave been changed from 0.01, 0.05 and 0.1 to 0.01, 0.04 and 0.07 and so on. The logic that hasbeen followed is if L2 factor shows a higher effect then narrow the range, if L3 shows a highereffect then increase the range and if L1 shows a higher effect decrease the range. The 18 testcases were executed with these new values. Figure 7 shows a comparison of the coverage metricobtained for each test from set I and set II levels for the factors. Every test case shows a highervalue of coverage confirming the efficacy of the selection logic. A main effect graph was plottedfor this set of test cases and a 19th test case was generated. The coverage obtained by this singletest case is 175 as seen from Fig 8. Figure 8 shows a single uncovered cell.
American Institute of Aeronautics and Astronautics12
0 2 4 6 8 10 12 14 16 180
20
40
60
80
100
120
140
160
180
Test Case No
Co
vera
ge
-C
ells
Coverage Metric for Each Test Case - Comparison
SET ISET II
Figure 7. Comparison of coverage metrics for Set I and Set II data. Set II data shows a highercoverage value for all the test cases.
Optimised Coverage Set II
Altitude Grid
Mac
hG
rid
1 2 3 4 5 6 7 8
2
4
6
8
10
12
14
16
18
20
22
Cell Not Covered
Figure 8. Optimized table coverage for Set II. The figure shows the number of cells notcovered by the optimized test case. Only a single cell is left uncovered by the optimal test case.
American Institute of Aeronautics and Astronautics13
Altitude Grid
Mac
hG
rid
Coverage Cumulative 300 Random Tests
1 2 3 4 5 6 7 8
2
4
6
8
10
12
14
16
18
20
22
Cell with Single Point
Cell Not Covered
Figure 9. Table coverage for 300 random cases. Figure shows the cumulative coverage achievedafter executing 300 random tests.
Comparison with Random Test Case and Genetic AlgorithmA comparison of the cumulative coverage was carried out by injecting random values of
Altitude and Mach number at the input point. A fixed time of 20 seconds was used for each test.It is seen that about 300 test cases give a cumulative coverage of 174 cells as seen in Fig 9. Thisdemonstrates the second advantage of DOE, especially the use of orthogonal arrays. A fewnumbers of test cases give a very wide coverage thus reducing the number of test casesdrastically.
Table 4 gives the optimal values of the factors obtained for ensuring 100% coverage. Thiswas obtained after running the test cases for 30 generations or after “automatically” analyzing600 test cases! The optimal values are close to the optimal levels selected after 18 experiments.
V. ConclusionThe use of DOE to effectively develop test cases without knowing the internal working of the
software has been demonstrated by the experiment described in this paper. A single test case hasbeen designed after just 38 experiments which provides a coverage comparable to a cumulative300 random test runs or 30 generations of 20 population size (600 test cases). DOE also providesa graphical means of analyzing the effect of varying each factor, which is not available inrandom test cases or in GA. The ease of use of DOE and the simple mathematics is highlyadvantageous to a software tester who may not have the necessary domain expertise inoptimization techniques.
The use of DOE has been applied to Simulink based testing of autopilot control law in anotherstudy. This has also provided excellent preliminary results. There are still certain areas where
American Institute of Aeronautics and Astronautics14
study is required. One such area is the MCDC coverage, essential for certifying any safetycritical software by DO-178B standard.“A fool... is a man who never tried an experiment in his life” - Erasmus Darwin grandfather ofCharles Darwin. Systematically experimenting with software will certainly improve theunderstanding of the software and system behavior.
AcknowledgmentsYogananda Jeppu and Dr K Karunakar are grateful to Director, Aeronautical Development
Agency for permitting them to publish this paper.
References1. Roger S Pressman, Software Engineering - A Practitioners Approach, McGraw Hill
International Edition 6h Ed, 20052. Brown A.W. and K.C. Wallnau, “Engineering of Component based Systems”,
Proceedings of the 2nd IEEE International Conference on Engineering of Complex ComputerSystems (ICECCS '96), 1996, pp 414-422
3. Harmen - Hinrich Sthamer, "The Automatic Generation of Software Test Data UsingGenetic Algorithms", Ph.D. Dissertation, University of Glamorgan, 1995
4. RTCA/DO-178B, "Software Considerations in Airborne Systems and EquipmentCertification", RCTA, December 1992, pp.31, 74.
5. Phadke, M. S., Planning Efficient Software Tests, CrossTalk, Vol. 10, No. 10, pp. 11-15,October 1997.
6. Allpairs - http://www.satisfice.com/tools.shtml7. Telcordia Technologies, Inc’s AETG - http://aetgweb.argreenhouse.com/8. Eldon G. Leaphart, Steve E. Muldoon and Jill N. Irlbeck, “Application of Robust
Engineering Methods to Improve ECU Software Testing”, 2006 SAE World Congress Detroit,Michigan, April 3-6, 2006, SAE 2006-01-1600
9. “Introduction to Avionics”, R.P.G. Collinson, Microwave Technology Series 11,Chapman & Hall, 1996
10. Roger W Pratt, Flight Control Systems, Progress in Astronautics and Aeronautics, Vol.184, AIAA and IEE (2000)
11. Nutek website http://nutek-us.com/12. Mandl, Robert, "Orthogonal Latin Squares: An Application of Experiment Design to
Compiler Testing," Communications of the ACM, Vol. 128, No. 10, October 1985, pp. 1054-1058.
13. Brownlie, Robert, James Prowse, and Madhav S. Phadke, "Robust Testing of AT&TPMX/StarMAIL Using OATS," AT&T Technical Journal, Vol. 71. No. 3, May/June 1992, pp.41- 47.
14. Ranjit K Roy, Design of Experiments using the Taguchi Approach, John Wiley and Sons,2001.
15. NASA Technical Memorandum 104316 Airdata Measurement and Calibration Edward A.Haering, Jr. December 1995.
16 Genetic Algorithm and Direct Search Toolbox For Use with MATLAB® User’s Guide,The MathWorks, Inc.
