The URREF Ontology for Semantic Wide Area Motion Imagery Exploitation

Abstract

- Today's information fusion systems (IFSs) require common ontologies for collection, storage, and access to multi intelligence information. For example, ontologies are needed to represent the connections between physics-based (e.g. video) and text-based (e.g. reports) describing the same situation. Situation, user, and mission awareness are enabled through a common ontology. In this paper, we utilize the uncertainty representation and reasoning evaluation framework (URREF) ontology as a basis for describing wide-area motion imagery (WAMI) analysis to determine uncertainty attributes. As part of the Evaluation of Technologies for Uncertainty Representation Working Group (ETURWG), both the URREF and a WAMI challenge problem are available for research purposes. We provide an exemplar schema to link physics-based and text-based uncertainty representations to explore a common uncertainty demonstration.

Keywords

• Hard-soft Information Fusion
• Performance Evaluation
• Uncertainty Reasoning
• Knowledge Representation
• Ontology
• Measures of Effectiveness

-

INTRODUCTION

A fundamental goal of information fusion is to reduce uncertainty by combining information from multiple sources. When inputs come from disparate, heterogeneous sources, there is a need for a unified, common, and standardized semantic understanding of the information being fused, and also of the associated uncertainty. Ontologies [ 1 ] provide a means for such shared semantic understanding, thus enabling interoperability among systems in application domains such as command and control, emergency response, and information sharing [ 2 ]. In this work, we focus specifically on the need for interoperable representations of uncertainty. Figure 1, taken from [ 3 ], depicts the transformation of evidence from sensors through a fusion system to produce outputs reported to users. The fusion system employs uncertainty representation and uncertainty for machine processing and user interaction, refinement, and understanding [ 3, 4, 5, 6 ].

The evaluation of how uncertainty is processed is dependent on system-level metrics such as timeliness, accuracy, confidence, throughput, and cost [ 7 ], which also are information fusion quality of service (QoS) metrics [ 8 ]. Future large complex information fusion systems will require performance evaluation [ 9 ] and understanding of the connections between various metrics [ 10 ]. It is a goal of the Evaluation of Technologies for Uncertainty Representation Working Group (ETURWG) to formulate, test, and evaluate different approaches to uncertainty representation and reasoning. The URREF ontology provides a common semantic understanding to support evaluation of the uncertainty aspects of IF systems.

Information fusion system-level metrics include timeliness (how quickly the system can come to a conclusion within a specified precision level), accuracy (where can an object be Figure 1 - Boundaries of the Uncertainty Representation and Reasoning Evaluation Framework [ 3 ]. found for a specified localization level), and confidence (what level of a probability match for a defined recall level). Clearly, different choices in uncertainty representation approaches will affect the achievable timeliness, accuracy, and confidence of a system, and therefore must be considered when evaluating both the system’s performance as a whole [ 11 ] and the specific impact of the uncertainty handling approach. Yet, when evaluating timeliness (or any other system-level metrics), one will likely find some factors not directly related to the handling of uncertainty itself, such as object tracking and identification report updates (i.e., Level 1 fusion) [ 12, 13, 14 ], situation and threat assessment relative to scenario constraints (i.e., Level 2/3 fusion) [ 15 ], overall system architectures (e.g. centralized, distributed, etc.) [ 12 ], data management processes and feedback / input control processes (i.e., Level 4 fusion considerations) [ 16 ], and user-machine coordination based on operating systems (i.e., Level 5 fusion) [ 17 ], and others. In other words, evaluation of the uncertainty handling aspect of a fusion system is closely related to, yet distinct from, evaluation of the performance of the system overall.

Key to the various Data Fusion Information Group (DFIG) [ 18 ] levels of information fusion is evaluation. For example, there have been efforts in comprehensive tracking [ 19, 20 ], object classification [ 21 ], and situation awareness evaluation [ 22 ], which focus on measures of performance (MOPs). Future evaluations will include high-level information Measures of Effectiveness (MOEs) [ 23 ] that include uncertainty characterization [ 24 ].

Along with the URREF ontology, the ETURWG has also developed a series of use cases. The purpose of the use cases is to provide concrete realizations of the range of problems to which the URREF is intended to apply, to help ensure that the framework can address this range of problems. One use case is the use of Wide-Area Motion Imagery (WAMI) for Level 1 fusion [ 25, 26, 27, 28 ]. Other computer vision working groups [ 29 ] are exploring semantic technology with datasets that are not necessary focused on uncertainty, but have a rich set of ontologies and datasets for collaboration and comparisons.

The paper investigates the use of URREF for WAMI tracking. Section II explores the issues of uncertainty characterization and Section III, the uncertainty evaluation framework. Section IV presents a WAMI tracking use case using the URREF for timeliness, accuracy, and confidence. Section V provides and discussion and Section VI conclusions.

II.

THE UNCERTAINTY REPRESENTATION PROBLEM

The Information Fusion community envisions effortless interaction between humans and computers, seamless interoperability and information exchange among applications, and rapid and accurate identification and invocation of appropriate services. As work with semantics and services grows more ambitious, there is increasing appreciation of the need for principled approaches to representing and reasoning under uncertainty. Here, the term "uncertainty" is intended to encompass a variety of aspects of imperfect knowledge, including incompleteness, inconclusiveness, vagueness, ambiguity, and others. The term "uncertainty reasoning" is meant to denote the full range of methods designed for representing and reasoning with knowledge when Boolean truth-values are unknown, unknowable, or inapplicable. Commonly applied approaches to uncertainty reasoning include probability theory [ 30 ], expert systems [ 31 ], fuzzy logic, subjective logic [ 32, 33 ], Dempster-Shafer theory, DSmT [ 34 ], and numerous other techniques.

To illustrate the challenges of evaluating uncertainty representation and reasoning in information systems, we consider below a few reasoning challenges faced within the World Wide Web domain that could be addressed by reasoning under uncertainty [ 1 ]. Uncertainty is an intrinsic feature of many of the required tasks, and a full realization of the World Wide Web as a source of processable data and information management services [ 3 ] demands formalisms capable of representing and reasoning under uncertainty such as:

Automated agents (e.g., to exchange Web information); Uncertainty-laden data. (e.g., terrain information); Non-sensory collected information (e.g., human sources);

Dynamic composability (e.g., Web Services); or

Information extraction (e.g., indexing from large databases) These problems are all related to information fusion, involve both text-based [ 35 ] and physics-based [ 36 ] data, and can be easily extrapolated to represent the more general classes of problems found in the sensor, data, and information fusion. A recent example of hard-soft fusion uses a controlled natural language (CNL) for data-to-decisions [ 37 ].

III.

THE UNCERTAINTY EVALUATION FRAMEWORK

The uncertainty representation and reasoning evaluation framework (URREF) includes both hard (e.g. imaging, radar, video, etc.) and soft (e.g., human reports, software alerts, etc.) sources, which require integration for uncertainty MOEs.

Effectiveness relates to a system’s capability to produce an effect. Benefits of fusion include providing locations of events, extending coverage, and reducing ambiguity and false alarms. The goal of the IFS is to support users in their tasks to provide refined information, reduce time and workload, or enable complete, accurate, and quality task completion. Effectiveness includes efficiency: doing things in the most economical way (good input to output ratio), efficacy: getting things done, (i.e., meeting objectives), and correctness: doing "right" things, (i.e., setting right thresholds to achieve an overall goal - the effect). The MOEs support system-level management and design verification, validation, testing, and evaluation. The URREF output step involves the assessment of how information on uncertainty is presented to the users and, therefore, how it impacts the quality of their decision-making process.

Key aspects of effectiveness include quality of service (QoS) and quality of information, also known as information quality (IQ). QoS relates to the ability of a system to provide timely and dependable data transmission. QI relates to the fitness for purpose of the content. QoS and QI metrics can be utilized for hard-soft semantic information fusion [ 38, 39, 40, 41 ]. Representing and measuring QI typically requires addressing the semantics of the domain and the problem. Thus, ontologies are an indispensible tool for measuring QI [ 42 ]. Because QI is inherently focused on uncertainty, probabilistic ontologies [ 43 ] are useful for representing QI metrics.

The URREF ontology, whose main concepts are depicted in Figure 2, is a first step towards a common framework for evaluating uncertainty in fusion systems. These core classes are subclasses of the top level class, which in OWL is called Thing. The core of the ontology is the Criteria class, which drives the development of the elements of the subclasses (Section II.B). The Uncertainty Classes were either taken or adapted from the Uncertainty Ontology developed by the W3C’s URW3-XG [ 1 ]. The ontology must also be used as a high-level reference for defining the actual evaluation criteria items that will comprise a comprehensive uncertainty evaluation framework. Other main class definitions include:

A source class is the origin of the information. A physical sensor is one important example of a source; where natural language inputs from a human is another.

A Sentence class captures an expression in some logical language that evaluates to a truth-value (e.g., formula, axiom, assertion).

A Uncertainty Derivation class refers to the way it can be assessed which is decomposed into: 1) Objective Subclass: (e.g., factual and repeatable derivation process). 2) Subjective Subclass: (e.g., a subject matter expert's (SME’s) estimation).

A Uncertainty Model class contains information on the mathematical theories for the representing and reasoning with the uncertainty types.

A. Uncertainty Type Class

Uncertainty Type is a concept that focuses on underlying characteristics of the information that make it uncertain. Its subclasses are Ambiguity, Incompleteness, Vagueness, Randomness, and Inconsistency, all depicted in Figure 3. These subclasses were based on the large body of work on evidential reasoning by David Schum [ 31 ].

B. Criteria Class

The Criteria Class is the main class of the URREF ontology, and it is meant to encompass all the different aspects that must be considered when evaluating information uncertainty handling in multi-sensor fusion systems. Figure 4 depicts the Criteria Class and its subclasses: 1) Input Criteria: encompasses the criteria that directly affect the way evidence is input to the system. It focuses on the source of input data or evidence, which can be tangible (sensing or physical), testimonial (human), documentary, or known missing.

Relevance to Problem assesses how a given uncertainty representation is able to capture why a given input is relevant to the problem and what was the source of the data request.

Weight or Force of Evidence measures how a given uncertainty representation is able to capture the degree to which a given input can affect the processing and output of the fusion system. Ideally, the weight should be an objective assessment and the representation approach must provide a means to measure the degree of impact of an evidence item with a numerical scale such as value of information [ 24 ].

Credibility, also known as believability, comprises the aspects that directly affect a sensor (soft or hard) in its ability to capture evidence. Its subclasses are Veracity, Objectivity, Observational Sensitivity, and SelfConfidence. 2) Representation Criteria: encompasses the criteria that directly affect the way information is captured by and transmitted through the system. These criteria can also be called interfacing or transport criteria, as they relate to how the representational model transfers, passes, and routes information within the system.

Evidence Handling: is a subclass of representation criteria that apply particularly to the ability of a given representation of uncertainty to capture specific characteristics of incomplete evidence that are available to or produced by the system. The main focus is on measuring the quality of the evidence by assessing how well this evidence is able to support the development of a conclusion. It has subclasses Conclusiveness, Ambiguousness, Completeness, Reliability, and Dissonance.

Knowledge Handling: includes criteria intended to measure the ability of a given uncertainty representation technique to convey knowledge. Its subclasses are Compatibility and Expressiveness (which is further divided into the subclasses Assessment, Adaptability, and Simplicity) 3) Reasoning Criteria: contains criteria that directly affect the way the system transforms its data into knowledge. These can also be called process or inference criteria, as they deal with how the uncertainty model performs operations with information. It has the following subclasses: Correctness measures of the ability of the inferential process to produce results close to the truth. In cases where there is no ground truth to establish a correct answer (including a simulated ground truth), the representation technique can still be evaluated in terms of how its answers align with what is expected from a gold standard (e.g. subject matter experts, etc.).

Consistency assesses of the ability of the inferential process to produce the same results when given the same data under the same conditions.

Scalability evaluates how a representational technique performs on a class of problems as the amount of data or the problem size grows very large. Scalability could be broken down into additional sub-criteria.

Computational Cost computes the number of resources required by a given representational technique to produce its results.

Performance includes metrics to assess the contribution of the representational model toward meeting the functional requirements of an information fusion system. Other system architecture factors also affect these metrics. This criterion is divided into subclasses Timeliness and Throughput. 4) Output Criteria relates to the system’s results and its ability to communicate it to its users in a clear fashion. It has the following subclasses:

Quality serves to assess the informational assessment of the system’s output. It includes Accuracy and Precision as subclasses. It is common to see in the literature the same concepts with different names. For example, accuracy sometimes is used as a synonym of precision; and sometimes precision is a refinement of accuracy. As one makes the granularity coarser, one can expect that the system will have a better accuracy. Precision can also be used to determine bounds on the certainty of the reported result.

Interpretation refers to the degree to which the uncertainty representation and reasoning can be used to guide assessment, to understand the conclusions of the system and use them as a basis for action, and to support the rules for combining and updating measures.

The above concepts are being explored within the ETURWG, which is making use of this ontology (shown in Figure 4) to support the development of uncertainty evaluation criteria over a set of information fusion use cases. The interested reader should refer to the group’s website for more specific details (http://eturwg.gmu.edu). Note that the URREF ontology is not supposed to be a definitive reference for evaluation criteria, but simply an established baseline that is coherent and sufficient for its purposes. This approach privileges the pragmatism of having a good solution against having an “ideal” but usually unattainable solution. For instance, a definitive reference would involve having universally accepted definitions and usage for terms such as "Precision." This is clearly infeasible. The approach also takes into consideration that more important than naming a concept is to ensure that it is represented clearly and distinctly within the ontology so as to ensure the consistency for such applications as hard-soft fusion.

To assure utility and acceptability of the URREF ontology, most of its concepts have been drawn from seminal work in related areas such as uncertainty representation, evidential reasoning, and performance evaluation. The ontology has built on the URW3 uncertainty ontology [ 1 ]. Also, the structure and viewpoint adopted in the ontology development have been tuned to addressing the uncertainty evaluation problem and its associated perspective (e.g. how information is handled within a fusion system). Next, we present simultaneous tracking and identification (ID) application using the URREF.

IV.

EXAMPLE – WAMI Wide area motion imagery (WAMI) systems provide imagery and video surveillance of large areas.

A. Schema

A schema for image processing is shown in Figure 5 for the Cursor on Target (CoT) program [ 44 ]. As detailed, the schema provides target type and identification (ID) allegiance, time stamps, and coordinate locations (much as the DFIG level 1 object assessment information of target track and ID information). While the schema is simple, and worked well [ 45 ], for purposes of information transmission, processing, exploitation, and dissemination, future developments could include uncertainty fields from the URREF ontology. It is important to assess which semantic content is most relevant for operational information fusion management and systems design.

In order to determine what uncertainty attributes can be added to such a message passing schema, there are three issues (1) what, (2) how much, and (3) which ones. For the case of physics-based (video) and textual-based reports, we need to determine what semantic content could be useful. One simple case is that either a human analyst can report a “friendly” in the uid field, or a machine tracker could extract the information from the video to update the uid field of “friendly”. One example of “friendly” could be from extracted text and video exploitation of a blue vehicle. What is obviously missing from the CoT schema is some notion of uncertainty with the measurements and information as to the confidence, timeliness, and position accuracy. While the entire URREF cannot, and should not, be considered for the schema updates, as a message passing service for the ontology, the first issue is to calculate possible uncertainty metrics that could go into the schema.

B. Metrics to Support the URREF Ontology

For the metrics available in the Cursor on Target Schema, we seek measures of confidence, accuracy, and timeliness, as related to uid, time, and point; respectively.

Credibility / Confidence: evaluates the ability to discern an object based on a known target. Classification is the target match, while identity is target allegiance. If targets are of known entities, it can be assumed that the targets not classified could pose an ID uncertainty. Using a Bayesian approach for this example, we determine the relative probability from the likelihood values of the object, versus of target clutter ℓO | c , where c j is for j = 1, ..., n clutter types:

PrO | c =

[ ℓ O | C ] Σ c j ∈ C [ ℓ O | c j ] Using plausibility, uncertainty is everything unknown

UL = 1 - PrO | c Timeliness: evaluates when the system knows enough information to make a decision versus when it was collected. For the purpose of this analysis we simulate the deadtime for an input time delay (TDi) for a decision i, as related to the user achieving a control decision [46]. Likewise, in the action selection requires time as modelled as an output time delay (TOi). The updated state-space representation is: (1) (2) (3) (4) y(t) = C x(t − TOi) + D u(t) To determine the estimation parameters of A and B, as well as the output analysis of C and D, we model the importance of the information processing as related to the cognitive observe-orient-decide-act (OODA) functions. Uncertainty is defined as the decision time difference of arrival:

UT = x(t − TOi) - x(t − TDi) Accuracy: evaluates how the real world track estimates from the measurements compare to the ground truth. For the purpose of this analysis, the real world is reduced to a specified track estimate xM, as related to ground truth xT. Using a root-mean square error, we have: UL = (xM - xT)2 + (yM - yT)2 (5)

Accuracy can be determined versus the ability to track a target exactly: 1 - UL. Other aspects could include track purity for track-to-track association [46] for situation awareness including: Specificity: evaluates how much of the real world clutter is reduced such as reducing the false alarms. While we do not simulate, we can deduce from the track confidence. Situation Completeness: evaluates how much of the real world the system knows. For the purpose of this analysis the real world is reduced to a specified region of space (the volume of interest, VOI) during a given time interval (the time interval of interest, TOI).

C. Wide Area Motion Imagery Example

WAMI has gained in popularity as it affords advanced capabilities in persistence, increased track life, and situation awareness, but it also poses new challenges such as low frame updates (timeliness) [ 47, 48 ]. Leveraging developments from computer vision [ 49, 50, 51, 52, 53 ], methods are being applied as part of the ETUWG [ 29 ]. The persistence coverage affords such methods as multiple object and group tracking [ 54, 55, 56 ], road assessment and tracking [ 57, 58 ], contextual tracking [ 59, 60 ], and advances in particle filtering [ 59, 61 ]. Because of the numerous objects and their movements, there are opportunities for linear road tracking, but also there is a need for nonlinear track evaluation [ 62 ] such as the randomized unscented transform (RUT) filter [ 63 ] for accuracy assessment. These issues will be important for future work.

We utilize the results from a WAMI tracker for track location accuracy, the pixels on target for classification for target identity (e.g. credibility), and the timeliness to make a decision. We are tracking four targets with an on-road analysis with a nominated target of interest, as shown in Figure 6. Vehicles turning off road are not considered as part of the user defined targets of interest. Note: the entire Columbus Large Image Format (CLIF) WAMI image collection has been presented in previous papers with discussions with the entire video data set [ 27 ] (see the ETURWG website). 0.9 0.8 0.7 1 0.9 0.8 0.7 e c timely 0.6 ien A-ac fnd B-ac 0.5 oC C-ac ID D-ac 0.4 treg A-cf 0.3 aT B-cf C-cf 0.2 D-cf 0.1 200 00 2 4 6 8 10 12 14

Time Figure 7 – Target Accuracy 16 18 20

Figure 7 plots the target accuracy (which is the inverse of the typical plots that show the target tracking error). Figure 8 combines the track accuracy in a unified display plot showing the target confidence (uid) and the accuracy. The confidence is shown as solid lines and the timeliness presented as the black humps where the time intervals are shown as: orient (t = 2.55), observe (t = 5-10), decide (t = 10-13) and act (t =13-18) time steps.

Confidence-Accuracy-Timeliness Plot f n o

C tID e g r a

T 2 4 6 8 10 12 Time Response 14 16 18

Figure 8 – Confidence-Accuracy-Timeliness Results.

Using the above information, we combine the credibility /confidence, accuracy, and timeliness (CAT) for a semantic notion of fused uncertainty in Figure 9 (where the normalized values are UT = UC + UT + UA). Together, the combined uncertainty could be a ontology field in an updated CoT schema to give the user a quality assessment of a machine processed semantic representation of uncertainty.

V.

DISCUSSION

Figure 9 shows a case for unified uncertainty estimation and is meant for discussion. Given the choice to utilize the URREF ontology, there are issues associated with choosing an ontology representation that can work within a message passing schema. If only one field was available, say ut, then is it appropriate to normalize the uncertainty and combine for purposes of the schema? For this case, only one target was nominated (like the CoT program), from which we see that the combined evidence supports a reduction in uncertainty; namely decreased track error, increased plausibility and hence ruling out the uid error, and the timeliness in decision making.

CAT Uncetainty Plot 1 0.9 0.8 0.7 y 0.6 it n tra 0.5 e c nU0.4 0.3 0.2 0.1

CONCLUSIONS

Characterizing the uncertainty is important in information fusion (IF) processes. Evaluation of IF systems presents various challenges due to the complexity of fusion systems and the sheer number of variables influencing their performance. Developing the operational semantics will include issues of representation, reasoning, and policy which need to be considered for command and control [ 64 ]. Representing uncertainty has an overall impact on system performance that is hard to quantify or even to assess from a qualitative viewpoint. The ETURWG technical considerations unearthed many issues that demand a common understanding that is only achievable by a formal specification of the semantics involved [ 65, 66 ].

In the paper, we utilized the current URREF ontology in relation to an established schema (Cursor on Target) to support the development of a specific use case for wide-area motion imagery (WAMI) simultaneous tracking and identification. We also presented a visual analytic method for uncertainty metrics and analytics. Future work includes group tracking, activity analysis, hard-soft fusion, and contextual understanding.

More specific requirements to evaluate a set of use cases and associated data sets designed by the ETURWG are accessible through our webpage [http://eturwg.c4i.gmu.edu]. Although it is clear that the URREF ontology is not a definitive reference for all types of information fusion activities, it has proven to be a discussion towards a common framework.

ACKNOWLEDGMENT

The authors recognize the contributions of the members of the International Society of Information Fusion’s ETURWG in formulating the URREF ontology, which was discussed in several of its meetings and includes the WAMI use case.

References

[1] K.B. Laskey , K.J. Laskey , P.C.G. Costa , M. Kokar , T. Martin , and T. Lukasiewicz (eds.) “ Uncertainty Reasoning for the World Wide Web: W3C Incubator Group Report.” World Wide Web Consortium . Available at http://www.w3.org/2005/Incubator/urw3/XGR-urw3/.

[2] E. P. Blasch , E. Bosse, and D. A. Lambert , High-Level Information Fusion Management and Systems Design , Artech House, 2012 .

[3] P. C. G. Costa , K. B. Laskey , E. Blasch and A-L. Jousselme , “ Towards Unbiased Evaluation of Uncertainty Reasoning: The URREF Ontology,” Int. Conf. on Info Fusion , 2012 .

[4] E. P. Blasch and P. Hanselman , "Information Fusion for Information Superiority," IEEE Nat. Aerospace and Electronics Conference , 2000 .

[5] E. Blasch , “ Level 5 (User Refinement) issues supporting Information Fusion Management” Int . Conf. on Info Fusion , 2006 .

[6] E. Blasch , "User refinement in Information Fusion" , Chapter 19 in Handbook of Multisensor Data Fusion 2nd Ed , (Eds.) D. Hall, and J. Llinas , CRC Press, 2008 .

[7] D. L. Hall and J. M. Jordan , Human Centered Information Fusion, Artech House, 2010 .

[8] E. Blasch , M. Pribilski , B. Roscoe , et. al., “Fusion Metrics for Dynamic Situation Analysis , ” Proc SPIE , Vol. 5429 , Aug 2004 .

[9] E. Blasch , I. Kadar , J. Salerno , M. M. Kokar , S. Das , G. M. Powell , D. D. Corkill , and E. H. Ruspini , “ Issues and Challenges in Situation Assessment (Level 2 Fusion) , ” J. of Advances in Information Fusion , Vol. 1 , No. 2 , pp. 122 - 139 , Dec. 2006 .

[10] E. Waltz and J. Llinas , “ System Modeling and Performance Evaluation,” Ch 11 in Multisensor Data Fusion Systems , Artech House 1990 .

[11] J. Llinas , “ Assessing the Performance of Multisensor Fusion Processes,” Ch 20 in Handbook of Multisensor Data Fusion , (Eds.) D. Hall and J. Llinas , CRC Press, 2001 .

[12] P. Hanselman , C. Lawrence , E. Fortunano , B. Tenney , and E. Blasch, “Dynamic Tactical Targeting, ” Proc. of SPIE , Vol. 5441 , 2004 .

[13] K. C. Chang and Robert M. Fung , “ Target Identification with Bayesian Networks in a Multiple Hypothesis Tracking System” , Optical Engineering , Vol. 36 , No. 3 , pp. 684 - 691 , March, 1997 .

[14] E. Blasch , Derivation of A Belief Filter for High Range Resolution Radar Simultaneous Target Tracking and Identification , Ph.D. Dissertation , Wright State University, 1999 .

[15] C. Yang and E. Blasch , “ Pose Angular-Aiding for Maneuvering Target Tracking” , Int. Conf. on Info Fusion , July 2005 .

[16] G. Chen , D. Shen , C. Kwan , J. Cruz , M. Kruger , and E. Blasch, “Game Theoretic Approach to Threat Prediction and Situation Awareness,” J. of Advances in Information Fusion , Vol. 2 , No. 1 , 1 - 14 , June 2007 .

[17] E. Blasch , I. Kadar, K. Hintz , J. Biermann , C. Chong , and S. Das , “ Resource Management Coordination with Level 2/3 Fusion Issues and Challenges, ” IEEE Aerospace and Electronic Systems Magazine , Vol. 23 , No. 3 , pp. 32 - 46 , Mar. 2008 .

[18] E. Blasch , “ Sensor, User, Mission (SUM) Resource Management and their interaction with Level 2/3 fusion” Int. Conf. on Info Fusion , 2006 .

[19] E. Blasch and S. Plano , “ DFIG Level 5 (User Refinement) issues supporting Situational Assessment Reasoning,” Int. Conf. on Info Fusion , 2005 .

[20] K. C. Chang , Z. Tian , S. Mori , and C-Y. Chong , “ MAP Track Fusion Performance Evaluation,” Int. Conf. on Information Fusion , 2002 .

[21] E. P. Blasch , O. Straka , J. Duník , and M. Šimandl , “ Multitarget Performance Analysis Using the Non-Credibility Index in the Nonlinear Estimation Framework (NEF) Toolbox,” Proc. IEEE Nat. Aerospace Electronics Conf (NAECON) , 2010 .

[22] R. Carvalho and KC Chang , “A Performance Evaluation Tool for MultiSensor Classification Systems,” Int'l Conf. on Information Fusion , 2009 .

[23] J. Salerno , E. Blasch, M. Hinman , and D. Boulware , “ Evaluating algorithmic techniques in supporting situation awareness , ” Proc. of SPIE , Vol. 5813 , April 2005 .

[24] E. Blasch , P. Valin , E. Bossé, “ Measures of Effectiveness for HighLevel Fusion,” Int. Conference on Information Fusion , 2010 .

[25] P.C.G. Costa , R.N. Carvalho , K.B. Laskey , and C.Y.Park, “ Evaluating Uncertainty Representation and Reasoning in HLF systems” , Int. Conference on Information Fusion , 2011 .

[26] O. Mendoza-Schrock , J. A. Patrick , and E. P. Blasch , “ Video image registration evaluation for a layered sensing environment .” in IEEE Nat. Aerospace Electronics Conf. (NAECON) , 2009 .

[27] H. Ling , Y. Wu , E. Blasch, G. Chen, H. Lang , and L. Bai, “ Evaluation of visual tracking in extremely low frame rate wide area motion imagery , ” Int. Conf. on Information Fusion , 2011 .

[28] R. Pelapur , S. Candemir , F. Bunyak , M. Poostchi , G. Seetharaman, and K. Palaniappan , “ Persistent Target Tracking Using Likelihood Fusion in Wide-Area and Full Motion Video Sequences,” Int. Conf. on Info. Fusion , 2012 .

[29] P. Liang , G , Teodoro, H. Ling , E. Blasch, G. Chen, and L. Bai, “ Multiple Kernel Learning for Vehicle Detection in Wide Area Motion Imagery,” Int. Conf. on Info Fusion , 2012 .

[30] E. Blasch , G. Seetharaman, M. Talbert , K. Palaniappan , and H. Ling , “Key Elements to Support Layered Sensing Dismount Tracking, ” Proc. NATO SET 178-RWS 017 Workshop on Detection of Dismounted Combatants, Sept . 2011 .

[31] D. Schum , “ The Evidencial Foundations of Probabilistic Reasoning ,” Northwestern University Press, 1994 .

[32] P. Walley , “ Measures of Uncertainty In Expert Systems,” Artificial Intelligence , 83 ( 1 ), May 1996 , pp. 1 - 58 .

[33] L. M. Kaplan , S. Chakraborty , and C. Bisdikian , “ Subjective Logic with Uncertain Partial Observations,” Int. Conf. on Information Fusion , 2012 .

[34] A. J ∅sang and R. Hankin, “ Interpretation and Fusion of Hyper Opinions in Subjective Logic,” Int. Conf. on Information Fusion , 2012 .

[35] J. Dezert , D. Han , Z -g. L, and J-M. Tacnet , “ Hierarchical DSmP transformation for decision-making under uncertainty , ” Int. Conf. on Information Fusion , 2012 .

[36] A. Auger and J. Roy , “ Expression of Uncertainty in Linguistic Data,” Int. Conference on Information Fusion , 2008 .

[37] H. Ling , L. Bai , E. Blasch, and X. Mei , “ Robust infrared vehicle tracking across target pose change using L1 regularization,” Int'l Conf. on Information Fusion , 2010 .

[38] A. Preece , D. Pizzocaro , D. Braines , D. Mott , G. de Mel, and T. Pham, “ Integrating hard and soft Information Sources for D2D Using Controlled Natural Language,” Int. Conf. on Information Fusion , 2012 .

[39] W. Perry , D. Signori , and J. Boon , “ Exploring the Information Superiority: A Methodology for Measuring the Quality of Information and its Impact on Shared Awareness,” RAND Corporation , 2004 .

[40] M. E. Johnson and K. C. Chang , “ Quality of Information for Data Fusion in Net Centric Publish and Subscribe Architectures,” Int. Conf. on Information Fusion , 2005 .

[41] M. A. Hossain , P. K. Atrey , and A. El Saddik , “ Modeling Quality of Information in Multi-Sensor Surveillance Systems , ” IEEE Int. Conf. on Data Engineering Workshop , 2007 .

[42] C. Bisdikian and L. M. Kaplan , M. B. Srivastava , D. J. Thornley , D. Verma , and R. I. Young , “ Building principles for a quality of information specification for sensor fusion , ” Int. Conf. on Information Fusion , 2009 .

[43] E. P. Blasch , “ Ontological Issues in Higher Levels of Information Fusion: User Refinement of the Fusion Process,” Int. Conf. on Info Fusion , 2003 .

[44] R. Carvalho , P. Costa , K. Laskey , and KC Chang, “PROGNOS: Predictive Situational Awareness with Probabilistic Ontologies,” Int'l Conf. on Information Fusion , 2010 .

[45] M J. Kristan , J. T. Hamalainen , D. P. Robbins , and P. J. Newer , , “ Cursor-on-Target Message Router User's Guide,” MITRE Product - MP090284 , 2009 . (accessed on line ) [46] R. A. Shulstad , “Cursor on Target: Inspiring Innovation to Revolutionizing Air Force Command and Control,” Air and Space Power Journal . Vol. 4 , Dec . , 2011 .

[47] E. Blasch , R. Breton , P. Valin , and E. Bosse, “ User Information Fusion Decision Making Analysis with the C-OODA Model,” Int. Conf. on Info Fusion , 2011 .

[48] E. Blasch and P. Valin , “ Track Purity and Current Assignment Ratio for Target Tracking and Identification Evaluation,” Int. Conf. on Info Fusion , 2011 .

[49] K. Palaniappan , F. Bunyak , P. Kumar , et al., “ Efficient feature extraction and likelihood fusion for vehicle tracking in low frame rate airborne video , ” Int. Conf. on Information Fusion , 2010 .

[50] K. Palaniappan , R. Rao , G. Seetharaman , "Wide-area persistent airborne video: Architecture and challenges," Distributed Video Sensor Networks: Research Challenges and Future Directions , Springer, Part V , Chapter 24 , pp. 349 - 371 , 2011 .

[51] Y. Wu , H. Ling , E. Blasch, G. Chen, and L. Bai, “ Visual Tracking based on Log-Euclidean Riemannian Sparse Representation,” Int. Symp. on Adv. in Visual Computing - Lecture Notes in Computer Science , 2011 .

[52] X. Mei , H. Ling , Y. Wu , E. P. Blasch , and L. Bai, “ Minimum Error Bounded Efficient L1 Tracker with Occlusion Detection,” IEEE Computer Vision and Pattern Recognition, 2011 .

[53] Y. Wu , E. Blasch, G. Chen , L. Bai , and H. Ling , “ Multiple Source Data Fusion via Sparse Representation for Robust Visual Tracking,” Int. Conf. on Info Fusion , 2011 .

[54] Y. Wu , J. Wang , J. Cheng, H. Lu , E. Blasch , L. Bai , and H. Ling , “ RealTime Probabilistic Covariance Tracking with Efficient Model Update , ” IEEE Trans. on Image Processing , 21 ( 5 ): 2824 - 2837 , 2012 .

[55] X. Zhang , W. Li , W. Hu , H. Ling , and S. Maybank , “ Block covariance based L1 tracker with a subtle template dictionary , ” Pattern Recognition , 2012 .

[56] E. P. Blasch and T. Connare , “Improving Track maintenance Through Group Tracking, ” Proc of the Workshop on Estimation , Tracking, and Fusion ; A Tribute to Yaakov Bar Shalom, 360 - 371 , May 2001 .

[57] T. Connare , E. Blasch , J. Schmitz , F. Salvatore , and F. Scarpino , “ Group IMM tracking utilizing Track and Identification Fusion,” Proc . of the Workshop on Estimation, Tracking, and Fusion; A Tribute to Yaakov Bar Shalom , Monterey, CA, 205 - 220 , May 2001 .

[58] M. Wieneke and W. Koch , “ A PMHT Approach for extended Objects and Object Groups,” IEEE T. Aerospace and Electronic Systems , vol. 48 , no. 3 , July 2012 .

[59] C. Yang and E. Blasch , “ Fusion of Tracks with Road Constraints,” J. of. Advances in Information Fusion , Vol. 3 , No. 1 , 14 - 32 , June 2008 .

[60] X. Shi , H. Ling , E. Blasch, and W. Hu, “ Context-Driven Moving Vehicle Detection in Wide Area Motion Imagery,” Int'l Conf on Pattern Recognition (ICPR) , 2012 .

[61] E. D. Marti , J. Garcia , and J. L Crassidis , “Improving Multiple-Model Context-Aided Tracking through an Autocorrelation Approach,” Int. Conf. on Info Fusion , 2012 .

[62] L. Snidaro , I. Visentini, K. Bryan , and G. L. Foresti , “ Markov Logic Network for context integration and situation assessment in a maritime domain , ” Int. Conf. on Info Fusion , 2012 .

[63] F. Papi. M. Podt , Y. Boers , G. Battistello, and M. Ulmka , “ Constraints Exploitation for Particle Filtering based Tagret Tracking,” Int. Conf. on Info Fusion , 2012 .

[64] Z. Duan , X. Rong Li , and J. Ru , “ Design and Analysis of Linear Equality Constrained Dynamic Systems,” Int. Conf. on Information Fusion , 2012 .

[65] O. Straka , J. Duník , and M. Šimandl , “Randomized Unscented Kalman Filter in Target Tracking, ” Int. Conf. on Information Fusion , 2012 .

[66] L. Scholz , D. Lambert , D. Gossink , and G. Smith , “ A Blueprint for Command and Control: Automation and Interface,” Int. Conf. on Info Fusion , 2012 .

[67] E. Blasch , P. C. G. Costa , K. B. Laskey , D. Stampouli , G. W. Ng , J. Schubert , R. Nagi , and P Valin, “ Issues of Uncertainty Analysis in HighLevel Information Fusion - Fusion2012 Panel Discussion,” Int. Conf. on Info Fusion , 2012 .

[68] P. C. G. Costa , E. P. Blasch , K. B. Laskey , S. Andler , J. Dezert , A-L. Jousselme , and G. Powell, “ Uncertainty Evaluation: Current Status and Major Challenges - Fusion2012 Panel Discussion,” Int. Conf. on Info Fusion , 2012 .