3 I-Fusion Technology, Inc, Germantown, MD USA 20878 4 Interlink Sciences Systems, Inc., 1979 Marcus Ave., STE 210, Lake Success, NY 11042
5 Temple University, 1947 N. 12th Street, Philadelphia, PA 19122
ABSTRACT There are hosts of target tracking algorithm approaches, each valued with respect to the scenario operating conditions (e.g. sensors, targets, and environments). Due to the application complexity, no algorithm is general enough to be widely applicable, nor is a tailored algorithm able to meet variations in specific scenarios. Thus, to meet real world goals, multitarget tracking (MTT) algorithms need to undergo performance assessment for (a) bounding performance over various operating conditions, (b) managing expectations and applicability for user acceptance, and (c) understanding the constraints and supporting information for reliable and robust performance. To meet these challenges, performance assessment should strive for three goals: (1) challenge problem scenarios with a rich variety of operating conditions, (2) a standard, but robust, set of metrics for evaluation, and (3) design of experiments for sensitivity analysis over parameter variation of models, uncertainties, and measurements. Keywords: Tracking, Information Fusion, Performance Evaluation, Metrics, Scenarios
1. INTRODUCTION Target tracking applications require the ability to maintain tracks on a large number of targets of varying sizes. Increasingly complex, dynamically changing scenarios have evolved that require intelligent tracking strategies. These intelligent strategies have to be evaluated over various locations, observing changing targets, and detecting unknown threats. [1] Targeting intelligence, surveillance, and reconnaissance (ISR) systems incorporate many research issues in intelligent tracking, sensor development, and information fusion as shown in Figure 1 to engage targets from detections. [2] To assist in the coordination of assets, it is imperative to (1) understand the performance driving each system, (2) determine the metrics for effective collaboration, and (3) proactively utilize the systems relative to the real-world scenario of interest. [1] Performance assessment of multitarget tracking (MTT) for the real world requires (1) pragmatic understanding of the algorithm designs, portability and applicability, (2) detailed testing over various scenarios and operating conditions, and (3) focused assessment of sensitivity analysis and performance metrics [3] through tailored design of experiments. While the MTT community has made progress in algorithm comparisons and metrics through the work of Blair in the Benchmark problems [4], metrics and tailored target tracking performance evaluations from Drummond [5, 6, 7], and algorithms from Blackman [8] (as well as many others in
Figure 1. Sensing and Targeting processes in tracking.
the community e.g. [9, 10, 11]); there still is a need for reliable and designed testing scenarios to increase the understanding of the MTT approaches for the operational use of the techniques. Real world applications of target tracking are embedded in larger operational goals such as surveillance, targeting, and force assessment that includes both entity estimation and situation control. The Information Fusion community has adopted the Data Fusion Information Group (DFIG) model, shown in Figure 2, with emphasis placed on sensors and their estimations from the real world. Target tracking is but a piece of a larger information fusion system including user requirements, registration algorithms, and sensor management. Level 1 fusion, which includes target tracking and identification, is merely a technique to provide situational awareness. Situational awareness in and of itself is a picture of the environment that requires situation assessment (i.e. number of targets, movement of targets, and the identity of targets). This information aggregation supports a user’s real world decision-making goals of security, strike packages, and positioning forces. However, the tracking performance assessment of the situation can also be viewed as a control problem. The control problem requires tools and techniques for decision support such as metrics and analysis to place sensors, waveform diversity to choose operating modes, and situation characterization to relay information to other distributed users. The real world is complex; however, effective methods of tracking performance analysis will aid users to appreciate and trust tracking systems, utilize the tracking results for situational awareness, and adaptively control sensors for effective performance. Three areas of support that are important to the community are (1) scenarios [13], (2) metrics [14, 15, 16] and (3) design of experiments [17]. A “Challenge Problem” would support the community for problem formulation [4], metrics [18], and MTT design of experiments [19]. Building on these developments, the MTT community can support future challenge problems so that continued understanding, testing, and focused assessment can improve MTT research developments. In [20], we overviewed many contributors to both tracking approaches and metrics for tracking performance evaluation (TPE) Highlighted were the contributions from K. C. Chang, and S. Mori, and C. Y. Chong [21, 22, 23] along with X. R. Li [24], of which a series of TPE contributions have been reported. In 2011, tracking metrics were overviewed [25] with fidelity metrics [26, 27]. Fidelity track metrics include the cardinality rankings as many of the fidelity metrics are normalized without units. The fidelity metrics include such issues as track association that we use here. For the analysis, we use the track purity [28] as method for track-to-track association, with the interest of distributed fusion analysis. However, we need to preface the distributed track fusion evaluation concept based on the operational need. Our goal is to summarize (1) transition the techniques, (2) manage expectations for groups incorporating MTT into larger designs, and (3) develop the resources available for the next generation of tracking researchers. In a dynamic targeting scenario, there are hosts of algorithms that affect performance: sensor registration, measurement-to-track (M2T) assignment, track-to-track (T2T) association, sensor management, and ultimately, the user. In many operational contexts, the platform, sensor, and algorithms for target tracking and identification (ID) are designed together which requires novel metrics for distributed tracking [29]. Based on M2T algorithms [30, 31], tracking evaluation [31, 32], T2T developments [33, 34], and simultaneous tracking and ID (STID) approaches [35, 36, 37, 38], we seek a method for distributed tracking evaluation. The goal of target tracking is to associate measurements of moving objects. There are many tracking approaches that we overviewed in previous publications [39] that included linear and nonlinear as well as Gaussian and non-Gaussian approaches [40]. The focus has been on comparative analysis of tracking approaches with interest in metrics and performance. Examples of approaches have been developed for radar GMTI and HRRR tracking [41, 42, 43, 44] and comparative analysis of different methods in the nonlinear-estimation toolbox [45, 46, 47].
Figure 2. DFIG Model [12]
Proc. of SPIE Vol. 8392 839207-2
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
The paper describes brief attributes where MTT performance assessment can be furthered for real world realization of the techniques. Section 2 starts with scenario designs in Challenge Problem development. Section 3 places emphasis on a robust set of metrics that needs to be adopted as a standard for the tracking community. Section 4 describes a design of experiments approach which typical complex systems designs utilize for understanding. Section 5 draws conclusions.
2. SCENARIOS Tracking algorithms for the real world require a pragmatic understanding of the complexities surrounding the application such as the number and types of targets, the sensors including their physical parameters and communication links, and the varying environmental conditions. [48] Utilizing static and dynamic environmental conditions can lead to host of challenges for MTT algorithms. Typically, with weak applicability to the real world, MTT algorithms focus on performance assessment over different target densities, ordered set of detection policies (e.g. Probability of Detection > 0.8), and various forms of clutter (e.g. Gaussian and Poisson). As example of challenging scenario designs, the DARPA Dynamical Tactical Targeting (DTT) program [13] tested a variety of scenarios and evaluated system performance. Figure 3 represents the design space of different scenarios available for a given MIDB (Mission Data Base), Nominated Area of Interest (NAI), and air task order (ATO) of mission objectives. After the selected scenario conditions for each run were determined, a simulator was used to instantiate the truth information, such as actual target behaviors and sensors measurements. The simulator used the mean values and variance bounds associated with selected factors to create a half-day scenario which afforded a longitudinal performance analysis based on ergodic assumptions. While the various MTT performance assessment techniques have been documented in the literature [49, 50, 51], there is a long way ahead for reliable and robust performance assessment of real world systems. Issues of communications [52], measurement uncertainties, and models are important for sensor management. [53, 54, 55] As shown in Figure 4, there are many functions surrounding target tracking and identification over the operating conditions of sensors, targets, and objects that lead to MTT metric evaluation. To instantiate the metrics for real world testing includes data truthing, metric evaluation, and careful sensitivity analysis over operating conditions for model enhancements. To utilize the various parameters in the real world analysis, the MTT community should support challenge problems and performance models to enhance the understanding of real world issues in MTT assessment.
Figure 3: Scenario Parameters. [22]
Figure 4: Operating Conditions of Sensor, Target, and Environment.
Proc. of SPIE Vol. 8392 839207-3
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
2.1. Benchmark Challenge Problems Benchmark challenge problems support the MTT community in design and analysis. [56] A scenario includes various kinematic target movements, possible sensor signals and target signatures, and terrain details. For example, Ross et al. [57] and the MSTAR team created a set of operating conditions over sensor, targets, and environments.
As a quick summary, the following pieces constitute a “Challenge Problem:” [56]
– Problem Definition: The scope and significance
– Data: Applicable data for the defined problem • Tools for reading and processing data • Suggestions on training and test sets • Characterization of the data
– Goals: Research challenge questions and suggested experiments
reproducible minimum performance for the defined problem
Scenarios provide collected data, synthetic models, and support documentation for real world analysis either through analytical, simulated, or empirical results. 2.2. Performance Models Different models affect the MTT evaluations as shown in Figure 5. A sensor model is derived from physical, simulated, or empirical analysis. The physical parameters can be incorporated into physical or analytical models such as the radar equation. A more detailed analysis of the sensor system feeding an MTT system would be an exploitation (behavior) model, such as a SAR performance model [58]. There are a host of other sensors for MTT such as electro-optical sensors [59, 60] that need development for models that are verified and validated. Second, there is a need for target models. In the case of MTT, many stationary target models exist such as the object models. [61] Third, environment models such as terrain, Google Maps, and other data are available. Weather and terrain contour maps could aid in the understanding of target movement. The documentation and model fidelity [62] (for a given scenario) can be included in a benchmark challenge problem for MTT comparison and evaluation.
3. MTT PERFORMANCE METRICS What you measure is what you get. To support performance assessment, there is a need for an authoritative standard, defined calculations, and suggested sensitivity metrics from which to test and evaluate algorithms. For example, MTT is based on a host of metrics that start with the measurement (detection and cueing) and end with (engagement and assessment) as shown in Figure 6. MTT metrics can be viewed as a producer of or consumer of the information. If the user is only interested in target identity, then MTT kinematic behavior assessment would provide additional information for a goal outside traditional MTT processes. Likewise, MTT consumes the uncertainty and detection metrics from the sensors. Together the MTT system uses and consumes the metric information for analysis in conjunction with a sensor, algorithm, and platform manager.
Figure 6: Fusion Processes.
Figure 5: Target Tracking Performance Models.
Proc. of SPIE Vol. 8392 839207-4
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
A host of MTT metrics have been described in the literature. In each case, the metrics are associated with the algorithms themselves and provide a good taxonomy of issues to consider. In future real-world applications, there is a need to relate the metrics to user concerns above and beyond the standard metric of RMS position error for target location. One example is the communications community with a standard set of Quality of Service (QoS) metrics. [63] From the list of metrics, and important metric not currently considered for future real-world MTT analysis, includes Timeliness Variation. Sensor and information fusion systems require that the sensed information be communicated to the tracker in a timely manner with the associated throughput restrictions for hardware limitations. Communication methods transfer data from the sensor to the MTT center through routers, which can be modeled as a queue. For example, wireless systems will require ad hoc configurations of routers to transfer the information directly from the sensors to a mesh network. The mesh network, consisting of these routers, would then enable rapid transfer of the information to a distributed set of MTT sensors. Thus, to afford proactive MTT strategies, we need to consider the communication delays for timely decision making. For future MTT systems the Quality of Information will be important [64] as a sensitivity parameter as well as achieving the high fusion data quality (FDQ) [65] which may not be always commensurate with high QoS. 3.1. Standardizing the MTT Metrics to Include System Metrics To design a complete MTT system, we need to address user information needs in the design and development process. Blasch [66, 67] explored the concepts of situation assessment (SA) by detailing the user needs of attention, workload, and trust which can be mapped into metrics of timeliness, throughput, confidence, cost (utilization), and accuracy. Workload and attention can be reduced if the MTT system cues the operator to a few selected target descriptions (hence workload and timeliness). If the MTT performs better against a known a priori data set, accuracy and confidence increase, enhancing operator trust. Dynamic situation analysis [3] has three components: (1) dynamical responsiveness to changing conditions, (2) situational awareness, and (3) continual analysis to meet throughput and latency performance requirements. The combination of these three entities is instantiated by a tracking and identification system, an interactive display to allow the user to make decisions, and metrics for replanning and sensor management [68]. To afford interactions between future MTT system designs and users information needs, metrics are required. The MTT metrics should include timeliness, accuracy, throughput, confidence, and cost. These metrics are similar to the standard quality of service (QOS) metrics in communication networking [69] and theory and human factors literature, as shown in Table 1 [3].
Table 1: Metrics for Various Disciplines. [3] COMM Human Factors Info Fusion ATR/ID TRACK
Delay Reaction Time Timeliness Acquisition /Run Time Update Rate
The detailed metrics are required for a comparison analysis. For example, it is widely known that the interacting-multiple model (IMM) is best for maneuvering targets. In order to convey the results of tracking algorithm approaches, care must be taken to ensure that the performance analysis of the algorithm versus the data quality. For example, X Rong Li [70] has proposed relative metrics for analysis of trackers to separate the tracker performance from the quality of the measurements. 3.2. System Performance Metrics The goal of any multisensor system intelligent MTT analysis is to have a track gain over improved lifetime, timeliness, accuracy, and throughput. The information fusion gain can be assessed as Measures of Effectiveness (MOE) or Measures of Performance (MOP). MOPs include the standard MTT parameters of number of targets tracked, throughput, time, and PD. MOEs include force protection, situation analysis, and event occurrence. Key attributes of tracking algorithms are accuracy and throughput, whereas the scenario is cost, confidence, and timeliness, as shown in Figure 7 [71], where the tracker acronyms can be found in tracking textbooks (e.g. [72]).
Proc. of SPIE Vol. 8392 839207-5
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
MTT performance metrics includes number of targets tracked; however, the system performance metric is throughput. Throughput can be determined as the average rate, peak rate, and variability of the system to deliver information to the user. The average rate (events processed / time) is the average load that can be sustained by the sensor over an extended period of time. The peak rate tells the network what type of surge traffic must be coped with, either by dedicating data-rate capacity or by allocating sufficient buffer space to smooth out measurement surges. Variability measures the source burstiness and is an indication of the extent to which statistical multiplexing is needed for distributed tracking. Timeliness is QoS metric for system performance; however, system Delay (or latency) is a system-level metric which can be measured with time to assessment or delay variation. Transfer delay measures the delay imposed by the network on data transferring from a source to a destination. Delay variation is an important parameter for real-time applications, in which the data displayed at the destination must appear at a smooth continuous rate matching the rate generated at the source. 3.3. Scalar Performance Metrics A scalar performance measure is sought as the objective function for tracking performance optimization in resource management. The selected performance measure provides a scalar representation of the error covariance P or information matrix M. Alternatively, the quality of the information can be measured as the tightness (or spread) of the error covariance P. In the literature, different quantities have been used. This section considers these different scalar representations, as listed in Table 2, to quantify the size of the M or P matrix.
Table 2: Performance Measures Defined on a Matrix. [17]
Performance Measures A: Covariance or Information Matrix Trace trace(A) = ∑eig(A)
Determinant det(A) = ∏eig(A) Eigenvalue max or min {eig(A)}
Operations Carried out by Sensor Resource Management Minimization
of Performance Measures Defined on Covariance Matrix
Maximization Information Matrix Notations: trace(⋅), det(⋅), and eig(⋅) stand for the operations obtaining the trace, determinant, and eigenvalues of a matrix, respectively. ∑ = Sum, ∏ = Product. The trace and determinant of the covariance matrix are the two most commonly used matrix measures for MTT resource management. These quantities are popular because they provide physical interpretation about the spread of the posterior
Figure 7: Fishbone Diagram of Metrics.
Proc. of SPIE Vol. 8392 839207-6
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
distribution. Under the Gaussian assumption, the covariance matrix physically represents an ellipsoid bounding the likely target state values whose probability exceeds a threshold. The lengths of the axes for the ellipsoid are proportional to the eigenvalues of the covariance matrix. The trace of the covariance matrix is the overall expected mean squared error (MSE) for the state estimator (or the sum of the length of the axes), and the determinant is the volume of the ellipsoid. In both cases, the goal is to reduce the measure, which in turn reduces the size of ellipsoid. A variant of the trace measure is the popular geometric dilution of precision (GDOP) defined as the square root of the trace of P when R = I (identity matrix) and P0 = 0, which only reflects the geometric effect on the root mean squared error. 3.4. Robustness The MTT community is familiar with root-mean square error (RMSE), track purity, and track life. While each of these is based on the scenario, it lends itself to false understanding of the tracker itself. To look at the tracker, many times the covariance information is used as a measure of the relative performance of the track estimation. The MTT community, already accustomed to a system metric of track accuracy and track lifetime, should be acknowledging a system metric of robustness. Robustness includes specifications of designs (e.g. robust design), trusted reliability (e.g. robust decision making and estimation e.g., [73 , 74, 75, 76, 77], and insensitivity to parameter variation (e.g. robust control). The goal for the MTT community is to explore the sensitivity of performance over a variety of conditions to ensure that the tracker designs relate to robust performance [78, 79]. Essentially, a robust assessment, extending from confidence and reliability [80], would lead to performance bounds that give an envelope of performance to support a managed expectation to users of MTT performance. In summary, the choice of metrics, either described in the challenge problem or supported by the tracking community, needs to be standardized, consistently calculated, and allow for sensitivity studies.
4. DESIGN OF EXPERIMENTS DOE supports the valid and verifiable direct, indirect, and interactive effects between the various operating conditions over the changes in these scenarios for a sensitivity analysis. Acknowledging that MTT is in a complex system leads one to seek complex performance analysis for which a standard approach is Design of Experiments. To carry out a DOE, we need to construct scenarios, determine the experiments, compute the metrics, and diagnose performance. A general design of experiments evaluation methodology is shown in Table 3.
Table 3: DOE Methodology Steps.
Method Product Construct Scenarios Acquire or construct terrain and weather products Define fixed facility layout and characteristics Define vehicle mix and missions Define clutter and background traffic characteristics Specify the force structure
Scenario Definition (Lists) Location data Man-made structures Vehicle choice Additional moving objects Grouping of target allegiance
Spiral Test Plans Monte Carlo specifications Parameter selections and data Order of testing
Compute Metrics Set up and supervise runs Compare estimated sensor positions and parameters with ground truth Compare estimated target locations and identities with ground truth Measure delays between key events Statistically analyze results
System Test Reports Number of trials Sensor-to-truth reports Track-to-truth reports (accuracy) Timeliness reports Visualization of plots
Diagnose Performance Identify limits to system performance
Performance Bounds Throughput, Cost reports
Since real-world MTT incorporates a host of interactive effectives in a complex environment, there is no such analytical model to cover all aspects of the research. While the analytical approach is not without merit, only parts of the problem can be modeled. With some models, simulation studies can lead to an understanding of the domain. The MTT
Proc. of SPIE Vol. 8392 839207-7
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
community is rich with papers utilizing kinematic models with simulated data to test out a set of operating conditions. There are even studies that include analysis over an empirical data set. However, the issue of the “Real World” incorporates both the expected and the unexpected events in a scenario. Thus, the MTT community needs to realize that the tracker is a piece of a larger complex system. 4.1. DOE Test Scenarios Design of Experiments (DOE) [81] is a methodical way to control the performance assessment by changing certain parameters and regressing over the outcome. The purpose of using formal DOEs is to achieve the highest level of statistical significance of the tests while requiring the minimum experimentation e.g., [82]. DOE includes a host of techniques used to support the validity and reliability of testing complex systems while systematically understanding the process. Measures of Merit (or Figures of Merit) such as accuracy, confidence, and timeliness; can be studied by changing parameters. To effectively evaluate system performance, three types of parameters are assessed: (1) constants (e.g. NAI); (2) Monte Carlo variables (e.g. sensor variation), and (3) factors that change (e.g. number of targets). Table 4 shows the DOE analysis for the target, sensor, and environment EOCs (experimental operation conditions). Constant parameters were determined from either un-modelable effects, issues that did not affect the outcome, or real-world operating conditions. Constant target behavior includes emissions and on-road travel while variable target behavior includes move-stop-move cycles, paths, and speed. The sensors and platforms are a result of the mission scenario. The sensor variables included exploitation accuracy and sensor factors including an On/Off bias selection. The largest category of scenario OCs is from the environment which includes weather, nominate area of interest (NAI) selected by the user, facilities of target initiation/termination, terrain/road conditions, and ‘no-fly zones’. For each scenario, the NAI was fixed and available terrain and weather data provided.
Table 4: DOE of Test Scenarios. [13]
OC Category Parameter Flat Terrain Mountainous Targets Targets 6, 12, 25 1, 2, 5 Moving Confusers (Dens. / AOI) Low(10), 50 , High (1000) Low (0), Med (10), High (25) Routes-Stop-Move Variable Variable Sensors Initial Start Points Variable Variable Bias On/Off On/Off Environment ‘No Fly’ Zones Variable Area Locations Variable Area Locations
4.2. Sensor Management Metrics for Tracking Sensor resource management for MTT actually shares most of the optimality criteria used in the theory of design of experiments (DOE) [83]. For instance, the A-optimality (average or trace) is defined to minimize the trace of the inverse of the information matrix (i.e., the covariance matrix), which results in minimizing the average variance of the estimates. The D-optimality (determinant) is defined to maximize the determinant of the information matrix, which results in maximizing the differential Shannon information content of the parameter estimates. The E-optimality (eigenvalue) is defined to maximize the minimum eigenvalue of the information matrix. However, the relative merits of the optimality criteria and the dangers of relying on any single numerical criterion were discussed in [84] and the references therein in the context of optimal experimental design. Similar issues are investigated in this paper as to the comparison of the optimality measures. For a single sensor, it has been shown that one should try to position the sensor so that the measurement aligns with the largest eigenvector of P0. For example, if the sensor is a ranging sensor, then the LOS between the target and sensor must align with the largest eigenvector of P0 (or equivalently the smallest eigenvector of M0). For a bearing sensor, the LOS will be orthogonal to the largest eigenvector of P0. This section has shown that all of the matrix measures in Table 2 make sense for P. However, only some measures make sense for M such as the determinant. Note that some quantities defined on a covariance matrix may be redefined for an information matrix to serve the purpose of a scalar performance measure. One example is to use the minimum eigenvalue of the information matrix as the performance measure for E-optimality. This optimality criterion can replace the usual maximum eigenvalue, i.e., the 2-norm of the matrix. In this case, the E-optimality for the information matrix is inversely proportional to the E-optimality for the covariance matrix, similar to the
Proc. of SPIE Vol. 8392 839207-8
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
determinant (i.e., the D-optimality). However, it should be noted that the calculation of the minimum eigenvalue is more computationally expensive than that of the maximum eigenvalue. Overall, Table 5 summarizes the suitability of various matrix measures for sensor management that are important in MTT evaluation as the suggested performance measures determine the ability of a sensor to track a target. By calculating the determinant at each time step, the results can be used in MTT evaluation.
Table 5: Effect of Optimization of Performance Measures on Covariance and Information Matrices. [17]
Performance Measures Covariance Matrix Information Matrix
Trace (A-optimality) Minimizes the sum of lengths of error ellipsoid axes
Ignores angular coordinates of the sensors relative to the target
Determinant (D-optimality) Minimizes the volume of the error ellipsoid
Dual to determinant of the covariance matrix
Eigenvalue (E-optimality)
Minimizes the major axis of the error ellipsoid. Solution is not unique in n-D for n > 2.
Maximizes the smallest eigenvalue of M, which is dual to the E-optimality criteria for the covariance. Again, solution is not unique for n > 2.
Norms
2-norm (Largest eigenvalue)
Equivalent to E-optimality Counter to goals of measures for the covariance
Frobenius norm (~ Trace)
Minimizes the sum of lengths squared of error ellipsoid axes
Counter to goals of measures for the covariance
Notes: Recommend to use, Recommend not to use, Use with caution.
5. CONCLUSIONS Together the scenarios, operating conditions, metrics, and DOE lead to focused challenge problem sets for a performance analysis and repeatable understanding. However, there are common themes associated in MTT sensor management, such as dense targets, roads and infrastructure, and known sensor measurements. Scenarios require many novel MTT applications that are not well researched such as evaluation metrics, proactive control of sensors, distribution of information to users, usefulness of pedigree information, and robust metrics. To determine successful intelligent MTT processing strategies, an evaluation assessment is needed to determine if developments have increased the capabilities of the MTT solutions, shown in Figure 8. [85] From the MTT analytical model studies, towards simulations with generated and empirical data, MTT systems are ready for real-world DOE testing and sensor management selections. The paper overviewed MTT design of experiments performance assessment from the standpoint of years of MTT research in designs, metrics, and simulated studies with the systematic view of transitioning these systems to real world capabilities. We highlighted a case for sensor management selection based on the MTT results. Whether the users are single sensor operators or part of a team, there is a need for system-level understanding of MTT performance. The ideas presented that support the transition of MTT systems to the real world include (1) enhanced
Figure 8: MTT evaluation transition process.
IT&E Innovation Test and Evaluation DT&E: Design Test and Evaluation OT&E: Operational Test and Evaluation
Proc. of SPIE Vol. 8392 839207-9
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
scenarios delivered to the community by way of challenge problems and sensor, target, and environment models, (2) standardized metrics and appropriate methods for calculating the metrics, and (3) systematic testing of the complex system-level analysis through design of experiments to determine the sensitivity of the MTT solutions. With these ideas, the MTT community can manage user expectations, bound performance for sensor management, and determine the research areas of need for the future robust MTT success [86].
REFERENCES [1] Kadar, I., Blasch, E., Blair, W. D., Chong, C-Y., Drummond, O. E., Kirubarajan, T., Li, X. R., Mahler, R. P. S., and Yang, C.,
“Invited Panel Discussion: Issues and Challenges in Model-Based Performance Assessment of Multi-Target Trackers,” Proc. SPIE 6968, (2008).
[2] Blasch, E., Yang, C., Kadar, I., Chen, G. and Bai, L., “Performance Assessment of Multitarget Tracking Algorithms in the Real World,” Proc. of SPIE, Vol. 6968, (2008).
[3] Blasch, E., Pribilski, M., Daughtery, B., Roscoe, B., and Gunsett, J., “Fusion Metrics for Dynamic Situation Analysis,” Proc. SPIE 5429, (2004).
[4] Kirubarajan, T., Bar-Shalom, Y., Blair, W. D., and Watson, G. A., “IMMPDAF for Radar Management and Tracking Benchmark with ECM,” IEEE Trans. Aerospace and Electronic Systems, Vol 34, No. 4, Oct. (1998).
[5] Fridling, B. E., “Performance Evaluation Methods for Multiple Target Tracking Algorithms,” Proc. SPIE, Vol. 1481, (1991). [6] Drummond, O., and Fridling, B. E.,“Ambiguities in Evaluating Performance of Multiple Target Tracking Algorithms,” Proc. SPIE
Vol. 1698, (1992). [7] Drummond, O. E., Blair, W. D., Brown, G. C., Ogle, T. L., Bar-Shalom, Y., Copperman, R. L., and Barker, W. H., “Performance
Assessment and Comparison of Various Tracklet Methods for Maneuvering Targets,” Proc. SPIE, Vol. 5096, (2003). [8] Blackman, S., “Tracking System Performance Prediction and Evaluation,” Ch. 13 in [Modern Tracking Systems], Artech House,
(1999). [9] Mori, S., Chong, C-Y., Tse, E., and Wishner, E. P., “Tracking and Classifying Multiple Targets Without A Priori Identification,”
IEEE Trans. On Automatic Control, Vol. AC-31, No. 5, May (1986). [10] Yang, C., and Blasch, E., “Mutual-Aided Target Tracking and Identification,” Proc. SPIE 5099, (2003). [11] Blasch, E., “Modeling Intent for a Target Tracking and Identification Scenario,” Proc. of SPIE, Vol. 5428, (2004). [12] Blasch, E., Kadar, I., Salerno, J., Kokar, M. M., Das, S., Powell, G. M., Corkill, D. D., and Ruspini, E. H., “Issues and Challenges
of in Situation Assessment (Level 2 Fusion),” J. of Advances in Information Fusion, Vol. 1, No. 2, pp. 122 - 139, Dec. (2006). [13] Hanselman, P., Lawrence, C., Fortunano, E., Tenney, B., and Blasch, E., “Dynamic Tactical Targeting,” Proc. SPIE, Vol. 5441,
(2004). [14] Rothrock, R. L., and Drummond, O. E., “Performance Metrics for Multiple-Sensor, Multiple-Target Tracking,” Proc. SPIE, Vol.
4048, (2000). [15] Hall, D. L., [Mathematical Techniques in Multisensor Fusion], Chapter 7. Artech House, Norwood, MA, 1992. [16] Li, X. R., and Zhao, Z., “Evaluation of Estimation Algorithms Part 1: Incomprehensive Measures of Performance,” IEEE Trans.
Aerospace and Electronic Systems, Vol. 42, No. 4, Oct. (2006). [17] Yang, C., Kaplan, L., and Blasch, E., “Performance Measures of Covariance and Information Matrices in Resource Management
for Target State Estimation,” IEEE Transactions on Aerospace and Electronics, July (2012). [18] Drummond, O. E., Ogle, T. L., and Waugh, S., “Metrics for Evaluating Track Covariance Consistency,” Proc. SPIE Vol. 6699,
2007. [19] Rawat, S., Llinas, J., and Bowman, C., “Design of a Performance Evaluation Methodology for Data Fusion-Based Multiple Target
Tracking Systems,” Proc. SPIE, Vol. 5099, (2003). [20] Blasch, E., and Valin, P., “Track Purity and Current Assignment Ratio for Target Tracking and Identification Evaluation,” Int.
Conf. on Info Fusion, (2011). [21] Mori, S., Chang, K. C., and Chong, C. Y., “Performance Analysis of Optimal Data Association with Application to Multiple
Target Tracking,” in [Multitarget-Multisensor Tracking: Apps. & Advs., Vol. II], Ch. 7, Y. Bar-Shalom (Ed.), Artech House, (1992).
[22] Chong, C. Y., “Problem Characterization in Tracking/Fusion Algorithm Evaluation,” IEEE AES Sys. Mag., July (2001). [23] Chang, K. C., Tian, Z., Mori, S., and Chong, C.-Y., “MAP Track Fusion Performance Evaluation,” Int. Conf. on Info Fusion,
(2002). [24] Nguyen, T., Jilkov, V., and Li, X. R., “Comparison of Sampling-Based Algorithms for Multisensor Distributed Target Tracking,
Int. Conf. on Info Fusion, (2003). [25] Gorji, A. A., Tharmarasa, R., and Kirubarajan, T., “Performance Measures for Multiple Target Tracking Problems,” Int. Conf. on
Info Fusion, (2011). [26] Blasch, E. P., “Fusion Evaluation Tutorial,” Int. Conf. on Info Fusion, (2004). [27] Blasch, E., Lavely, E., and Ross, T., “Fidelity Metric for SAR Performance Modeling,” Proc. of SPIE, Vol. 5808, (2005). [28] Mori, S., Chang, K.-C., Chong, C.-Y., and Dunn, K.P., “Tracking Performance Evaluation: Prediction of Track Purity,” Proc.
SPIE, Vol. 1096, (1989).
Proc. of SPIE Vol. 8392 839207-10
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
[29] Liggins, M. E., Chong, C.-Y., Kadar, I., Alford, M. G., Vannicola, V., and Thomopolous, S., “Distributed Fusion architectures and algorithms for target tracking,” Proc. IEEE, Vol. 85, No. 1, (1997).
[30] Bar-Shalom, Y., and Li, X. R.. [Multitarget-Multisensor Tracking: Principles and Techniques], YBS, New York, (1995). [31] Blackman, S., and Popoli, R., [Design and Analysis of Modern Tracking Systems], Artech House Publisher, Boston, (1999). [32] Li, X. R., Zhao, Z., and Li, X-B., “Evaluation of Estimation Algorithms - Credibility Tests,” IEEE T. Sys. Man., Cyber-A, Vol. 42
Issue 1, (2012). [33] Yuan, T., Bar-Shalom, Y., and Tian, X., “Heterogeneous Track-to-Tack Fusion,” J. of Adv. in Info. Fusion, Vol. 2, No. 6, (2011). [34] Tian, X., Bar-Shalom, Y., Yuan, T., et. al., “A Generalized Information Matrix Fusion Based Heterogeneous Track-to-Track
Fusion Algorithm,” Proc. SPIE, Vol. 8050, (2011). [35] Salmond, D., Fisher, D., and Gordon, N., "Tracking and Identification for closely spaced objects in clutter,” Euro. Contr. Conf.,
(1997). [36] Blasch, E. P., and Hong, L. “Simultaneous Identification and Track Fusion,” IEEE Conf. on Dec. Control, (1998). [37] Blasch, E., “Data Association through Fusion of Target track and Identification Sets,” Int. Conf. on Info Fusion, (2000). [38] Blasch, E., [Derivation of A Belief Filter for High Range Resolution Radar Simultaneous Target Tracking and Identification],
Ph.D. Dissertation, Wright State University, (1999). [39] Blasch, E., Rice, A., Yang, C., and Kadar, I., “Relative Performance Metrics to Determine Model Mismatch,” IEEE NAECON,
(2008). [40] Ristic, B., Arullampalam, S., and Gordon, N., [Beyond the Kalman Filter: Particle filters for tracking Applications], Artech House,
(2004). [41] Blasch, E., and Yang, C., “Ten methods to Fuse GMTI and HRRR Measurements for Joint Tracking and Identification”, Int. Conf.
on Info Fusion, (2004). [42] Yang, C., and Blasch, E., “Pose Angular-Aiding for Maneuvering Target Tracking,” Int. Conf. on Info Fusion, (2005). [43] Yang, C., Blasch, E., Garber, W., and Mitchell, R.,“A Net Track Solution to Pose-Angular Tracking of Maneuvering Targets in
Clutter with HRR Radar,” IEEE. Conf. on Sig, Sys. & Comp, (2007). [44] Yang, C., Garber, W., et. al., “A Simple Maneuver Indicator from Target’s Range-Doppler Image,” Int. Conf. on Info Fusion,
(2007). [45] Straka, O., Flídr, M., Duník, J., and Šimandl, M., “A software framework and tool for nonlinear state estimation,” System ID Conf.,
(2009). [46] Straka, O., Flidr, M., Dunik, J., Šimandl, M., and Blasch, E. P., “Nonlinear Estimation Framework in Target Tracking,” Int. Conf.
on Info Fusion, (2010). [47] Blasch, E. P., Straka, O., Duník, J., and Šimandl, M., “Multitarget Performance Analysis Using the Non-Credibility Index in the
Nonlinear Estimation Framework (NEF) Toolbox,” Proc. IEEE Nat. Aerospace Electronics Conf., (2010). [48] Waltz, E., and Llinas, J., “System Modeling and Performance Evaluation,” Chapter 11 in [Multisensor Data Fusion], Artech
House, (1990). [49] Zhu, Y., Zhoa, J., and Zhang, K., “Performance Analysis for Distributed Track Fusion with Feedback,” Proc. 3rd World Congress
on Intelligent Control and Automation, (2000). [50] Chang, K. C., Zhi, T., and Saha, R. K., “Performance Evaluation of Track Fusion with Information Matrix Filter,” IEEE Trans.
Aerospace and Electronic Systems, Vol. 38, No. 2, April (2002). [51] Chen, H., Chen, G., Blasch, E., and Pham, K., “Comparison of Several Space Target Track Filters,” Proc. of SPIE, Vol. 7330,
(2009). [52] Shen, D., Tian, Z., Chen, G., Pham, K., and Blasch, E., “Information-based awareness model for active sensing and resource
management,” Proc. of SPIE, Vol. 7698, (2010). [53] Yang, C., Blasch, E., and Kadar, I., “Post-Update Compensation (PUC) with maneuver Indicator,” Proc. of SPIE, Vol. 7336,
(2009). [54] Yang, C., Blasch, E., Douville, P., and Kaplan, L., “Evaluation of Covariance and Information Performance Measures for Dynamic
Objects Tracking,” Proc. of SPIE, Vol. 7697, (2010). [55] Yang, C., Kadar, I., Blasch, E., and Bakich, M., “Comparison of Information Theoretic Divergences for Sensor Management,”
Proc. SPIE 8050, (2011). [56] Blasch, E., Deignan,P. B.,, Jr, Dockstader, S. L., Pellechia, M., Palaniappan, K., and Seetharaman, G., “Contemporary Concerns in
Geographical/Geospatial Information Systems (GIS) Processing,” Proc. IEEE Nat. Aerospace Electronics Conf., (2011). [57] Mossing, J. C., and Ross, T. D., “An Evaluation of SAR ATR Algorithm Performance Sensitivity to MSTAR Extended Operating
Conditions”, Proc. of SPIE, Vol. 3370, (1998). [58] Kaufmann, V., Ross, T., Lavely, E., and Blasch, E., “Score-Based SAR ATR Performance Model with Operating Condition
Dependences,” Proc. SPIE, Vol. 6568, (2007). [59] Johnson, J., “Analysis of Image Forming Systems”, Image Intensifier Symposium, Ft Belvoir, Va. Oct 6-7, (1958). [60] Kahler, B., Blasch, E., Pikas, D., and Ross, T., “EO / IR ATR Model to Support Target Identification Fusion,” Proc. SPIE, Vol.
6566, (2007). [61] Ross, T. D., Bradley, J. J., Hudson, L. J., O’Connor, M. P., “SAR ATR – So What’s The Problem? – An MSTAR Perspective,”
Proc. of SPIE, Vol. 3721, (1999). [62] Blasch, E., Lavely, E., and Ross, T., “Fidelity Metric for SAR Performance Modeling,” Proc. SPIE, Vol. 5808, (2005).
Proc. of SPIE Vol. 8392 839207-11
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
[63] Blasch, E. P., Valin, P., and Bossé, E., “Measures of Effectiveness for High-Level Fusion,” Int. Conf. on Info Fusion, (2010). [64] Johnson, M. E, and Chang, K. C., “Quality of Information for Data Fusion in Net Centric Publish and Subscribe Architectures,”
Intl. Conf. on Information Fusion, (2005). [65] Kadar, I., “Research Challenges in Network and Service Management for Distributed Net-Centric Fusion“, Invited Panel Session
on Issues and Challenges in Resource Management with Applications to Real World Problems,” Organizers: Ivan Kadar and Ronald Mahler, Moderators: Ivan Kadar and Thia Kirubarajan, Proc. SPIE, Vol.6235, (2006).
[66] Blasch, E., and Plano, S., “Level 5: User Refinement to Aid the Fusion Process,” Proc. SPIE 5099, (2003). [67] Blasch, E., “Situation, Impact, and User Refinement,” Proc SPIE 5096, (2003). [68] Blasch, E., Kadar, I., Hintz, K., Salerno, J., Chong, C-Y., Salerno, J., and Das, S., “Resource Management Coordination with
Level 2/3 Fusion,” IEEE Aerospace and Electronic Systems Magazine, Vol. 23, No. 3, pp. 32-46, Mar. (2008). [69] Kadar, I., “Distributed Multisensor Fusion with Network Connection Management,” Proc. SPIE, Vol. 5809, (2005). [70] Li, X. R., and Zhao, Z., “Relative Error Measures for Evaluation of Estimation Algorithms,” Intl. Conf. on Information Fusion,
(2005). [71] Blasch, E. P., Straka, O., Yang, C., Qiu, D., Šimandl, M., and Ajgl, J., “Distributed Tracking Fidelity-Metric Performance Analysis
Using Confusion Matrices,” Int. Conf. on Info Fusion, (2012). [72] Bar-Shalom, Y., Willett, P. K., and Tian, X., [Tracking and Data Fusion : A Handbook of Algorithms], YBS Publications, (2011). [73] Kadar, I., and Kurz, L., "Robustized Scalar Form of Gladyshev's Theorem with Applications to Non-Linear Systems," Proceedings
of the Fourteenth Princeton Conference on Information Sciences and Systems, (1980). [74] Kadar, I., and Kurz, L., “A Robustized Vector Recursive Stabilizer Algorithm for Image Restoration," Information and Control,
Vol. 44, No. 3, (1980). [75] Kadar, I., "A Robustized Vector Recursive Algorithm in Estimation and Image Processing," Abstracts of papers, 1981
International Symposium on Information Theory, (1981). [76] Kadar, I., "A Recursive Algorithm in Estimation and Image Processing," IBM Research Report, T. J. Watson Research Center,
Yorktown Heights, NY, March (1981). [77] Kadar, I., "Robust Tracking Novelty Filters Based on Linear Models," IEEE 1st Int. Conf. on Neural Networks, (1987). [78] Arnborg, S., “Robust Bayesianism: Imprecise and Paradoxical Reasoning,” Int. Conf. on Information Fusion, (2005). [79] Arnborg, S., “Robust Bayesianism: Relation to Evidence Theory,” Journal for Advances in Information Fusion, Vol. 1, July
(2006). [80] Bai, L, and Blasch, E., “Two-Way Handshaking Circular Sequential k-out-of-n Congestion System,” IEEE Journal of Reliability,
Vol. 57, No. 1, pp. 59-70, Mar. (2008). [81] http://en.wikipedia.org/wiki/Design_of_Experiments. [82] Kadar, I., and Kurz, L., "A Class of Robust Edge Detectors Based on Latin Squares," Pattern Recognition, Vol. 11, No. 8/9,
(1979). [83] Blasch, E., “Assisted Target Recognition through User-Algorithm Fusion,” National Symp. On Sensor and Data Fusion Conf.,
(2007). [84] Montgomery, D.C., [Design and Analysis of Experiments] (6th Ed.), John Wiley & Sons, Inc., (2005). [85] Hardin, R.H., and Sloane, N.J.A., “A New Approach to the Construction of Optimal Designs,” Journal of Statistical Planning and
Inference, Vol. 37, 339-369, (1993), [86] Blasch, E. P., Bosse, E., and Lambert, D. A., [High-Level Information Fusion Management and Systems Design], Artech House,
Norwood, MA, (2012).
Proc. of SPIE Vol. 8392 839207-12
Downloaded From: http://spiedigitallibrary.org/ on 11/26/2013 Terms of Use: http://spiedl.org/terms
