From Wall Street to the streets of Baghdad, information drives action. Confounding this edict for the military is not only the unprecedented increase in the types and amount of information available, but the ability to separate the important information from the routine. Termed the value of information (VOI), the modern military commander and his staff require improved methodologies for assessing the applicability and relevance of information to a particular operation. This paper presents the approach used to elicit the knowledge necessary to value information for military analysis and enable the construction of a fuzzy-based prototype system for automating this valuation.
Today’s military operations require information from an
unprecedented number of sources which results in an
overload of information. With the requirement that
relevant information be consistently available to troops as
they conduct operations, a primary challenge for military
commanders and their staff is separating the important
information from the routine
Currently the VOI assigned a piece of information is ascertained via a multiple step process requiring intelligence collectors and analysts to judge its value within a host of differing operational situations. For example, the types and immediacy of mission information needs will influence the amount of data reviewed and the value that an analyst will ascribe. While there is doctrine that describes a process of assigning value, it is sufficiently vague to allow multiple interpretations. As such, the Research was sponsored by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-11-2-0092. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. cognitive processes behind these conclusions resist codification with exact precision and offer an excellent opportunity to leverage a computational intelligent solution using fuzzy inference.
This paper presents the approach used for gathering parameters necessary to value information for military analysis. Section 2 reviews the background information on the military domain with respect to VOI. Section 3 is an overview of knowledge elicitation techniques and the knowledge elicitation process utilized to capture values for fuzzy VOI rules. Section 4 presents a brief overview of the resulting prototype system. The conclusions and next steps are presented in Section 5.
On today’s battlefield, information drives action. Commanders must know details about important persons, places and events within their area of operations to address issues ranging from kinetic fights to adjudicating legal disputes to revitalizing a depleted economy. From sophisticated unmanned ground acoustic sensors to opensource RSS news feeds, military commanders are inundated with an unprecedented opportunity for information. Table 1 depicts military information volume. As unit echelon increases, the scope of military operations and number of information reports grows tremendously. Intelligence analysts examine this information to determine the impact of trends, important human networks, and threat tactics, techniques, and procedures on current and future plans.
As shown in Figure 1, accurate VOI estimation is
essential to the intelligence analysis process, promoting
improved situational understanding and effective
decisionmaking. The entire process is designed to produce and
make available relevant intelligence information. For all
military data, intelligence collectors are responsible for the
initial estimation of information value. While there are
guidelines for VOI determination, these are subject to
collector/analyst interpretation. In point of fact, a recent
US Army Intelligence Center of Excellence study
considered “Information Validation (Data Pedigree,
Corroboration and Cross Validation) and Stance Analysis
(Elimination of Bias and Use of Multiple Analysis
Perspectives)” as major issues
Proper VOI is integral to battlefield success. VOI is essential in the collect-assess portion of the intelligence process. At higher echelons, VOI is a metric useful in determining the degree of situational estimate accuracy amidst the uncertainty of combat. Additionally, VOI is a focusing element as a searchable criterion, enabling analysts to find relevant information quickly. At lower echelons, analysts can use VOI to create an optimum course of action for immediate mission execution.
The procedure for alphanumerically rating the
“confidence” or “applicability” assigned a piece of
information is essentially described in the annex to NATO
STANAG (Standard Agreement) 2022 as well as in
Appendix B of US Army FM-2-22.3
As depicted in Table 2 and Table 3, respectively, the
alphabetic Reliability scale ranges from A (Completely
Reliable) to E (Unreliable) while the numeric Content scale
ranges from 1 (Confirmed by other sources) to 5
(Improbable)
So as an example, a piece of information that was received by a source that has in the past provided valid information would be scored a Reliability Rating of either B or C; depending on the degree of doubt in authenticity. That same piece of information, if not confirmed, but seeming logical, would receive a Content Rating of either 2 or 3; again depending on the degree the information was consistent with other information. It quickly becomes obvious the subjective nature of the ratings (B2 vs. C3) can quickly lead to ambiguity.
A B C D E F 1 2 3 4 5 6
Reliable Usually Reliable Fairly
Reliable Not Usually
Reliable
Unreliable Cannot Judge Confirmed Probably
True Possibly
True Doubtfully
True Improbable
Cannot Judge
No doubt of authenticity, trustworthiness, or competency; has a history of complete reliability Minor doubt about authenticity, trustworthiness, or competency; has a history of valid information most
of the time
Doubt of authenticity, trustworthiness, or competency but has provided valid information in
the past
Significant doubt about authenticity, trustworthiness, or competency but has provided
valid information in the past Lacking in authenticity, trustworthiness, and competency; history of invalid information No basis exists for evaluating the reliability of the
source Confirmed by other independent sources; logical in itself; Consistent with other information on the
subject Not confirmed; logical in itself; consistent with
other information on the subject Not confirmed; reasonably logical in itself; agrees
with some other information on the subject Not confirmed; possible but not logical; no other
information on the subject Not confirmed; not logical in itself; contradicted by
other information on the subject No basis exists for evaluating the validity of the
information
In an attempt to guide the application of composite
ratings (i.e., B2 vs. C3) to varied operational situations,
organizations have generalized the usefulness of data by
developing charts similar to the one shown in Figure 2
Knowledge elicitation is generally the first step in the
construction of a system that seeks to use expert
knowledge to solve a problem. In the process of building
such a system, the knowledge engineer must interact with
one or more Subject Matter Experts (SMEs) to gather,
organize, and codify the appropriate domain specific
problem-solving expertise
While knowledge elicitation and knowledge acquisition
are occasionally used interchangeably in the literature,
most researchers draw a clear distinction between the two;
additionally, knowledge engineering is a third concept that
appears in the literature
The knowledge elicitation process is much more complex
than just arranging a meeting or meetings with SMEs. One
important but perhaps subtle aspect of the process is the
need to choose experienced and available experts that have
excellent communication skills as well as at least some
commitment to the project at hand
There are a myriad of assessments of knowledge elicitation
methods and numerous representations for how to classify
them. For our purposes here, we will present and briefly
describe the four categories of knowledge elicitation
methods identified by
Observation. This process consists of watching an SME perform the task or tasks in question. Typically, great care should be taken to avoid disrupting the SME during the reasoning process. The observations are recorded somehow (video, photographs, audio, notes, and the like). This method can be particularly useful as a beginning technique to allow the knowledge engineer to understand enough to develop more structured knowledge elicitation sessions.
Interviews. Interviews are used to simply ask SMEs what they know. The interviews may be structured, unstructured, or a combination. Unstructured interviews are free-form and use open-ended questions; they may be useful in the beginning knowledge elicitation efforts to get a preliminary understanding of the domain. Structured interviews set up an artificial scenario to impose constraints on the SME’s responses. Interview methods are often specifically tailored to the particular domain or problem so that some precise type of knowledge may be obtained. The Critical Decision Method falls into this category of techniques.
Process Tracing. This method is used for gathering information that is procedural in nature; it looks at behavioral events that are sequential in form. It is useful to ascertain conditional rules or note the order in which cues are used by the decision maker. The “think-aloud” technique is included in this category.
Conceptual Methods. This process attempts to gather conceptual structures present within the domain that are derived as concepts and interrelations. Steps include: 1) discovering relevant concepts, perhaps through interviews; 2) gathering opinions from one or more SMEs as to how the concepts relate; 3) representing the relationships; and 4) interpreting the result. One method of obtaining the SME’s beliefs as to how the concepts relate is by using a grid approach. In this method, concepts are rated across a set of dimensions, and then the similarity among concepts can be determined in some way.
In general, military operations are defined by their associated operation tempo; that is, the time it takes to plan, prepare and execute an exercise. High-tempo operations typically require the decision cycle to be measured in minutes to hours. Slower tempo operations will generally allow the decision cycle to be measured in months or longer. Absent from the model presented in Section III is the application of the information applicability rating to a specific operation type. Without the specific framework of a given operation type the associated impact of information latency (or information timeliness) requirements are lost. Restated, the true VOI is dependent upon the type of military operation to which the information is being applied. For instance, in a high-tempo operation, where decisions are made in short timeframes, added emphasis is assigned to information that has high applicability and was more recently received than others.
In order to capture the cognitive requirements necessary to refine our model and build the fuzzy association rules, the team applied the Conceptual Method posed by Cooke. A review of the military intelligence process revealed several relevant concepts such as operational tempo mentioned above. The team and the SMEs then discussed the relationships between data age, operational tempo, and information applicability. These relationships were developed into a two-part Likert survey instrument and the final product presented to the SMEs to gather specific values; the process of using the two surveys is detailed further in the rest of this section. The initial interpretation of the results led to the fuzzy rules that were codified in the prototype. Of course, any sort of “validation” of the system actually implies that the SMEs must corroborate the fuzzy rules, which basically requires other iterations of knowledge elicitation to ensure that the resulting system is accurate and precisely reflects the meaning and relationships the SMEs intended to convey. These efforts are briefly discussed in later sections.
The first survey was used to capture the generic information applicability rating from the doctrinal model described in Section III; that is, how to define the potential importance of a piece of information given a specific type of operation. The second survey was used to calculate the actual VOI based on the temporal latency of the information and a particular operational tempo. In this case the temporal latency was defined as either: recent, somewhat recent or old. It is particularly noteworthy that the cognitive concept of temporal latency was purposefully left as a subjective construct for the SME. In general, the surveys provided contextual structure for the structured interview. Additionally, the matrices proved useful in physically recording SME VOI determination responses to the questions of information applicability and the value within military mission execution context.
For the first part of the survey, a Likert instrument was
developed that incorporated the military doctrinal
information rating system. This system features a
combination of information content and source reliability.
Information content is rated on a scale of one thru five with
one (best case) being termed as, “Confirmed by other
independent sources” and five (worst case) being termed
as, “Not confirmed.” Likewise, source reliability is also
rated on a scale of one thru five; with one being termed as,
“No doubt of authenticity, trustworthiness, or
competency”, and five being termed as, “Lacking in
authenticity, trustworthiness, an competency”
With the generic information applicability ratings completed, the second step involved applying those ratings against the aspects associated with a specific mission type. While many different aspect possibilities exist, the focus of this pilot survey was on the two primary military aspects of operational tempo and the temporal latency of the information. In this case the operational tempo was defined as either ‘tactical’, ‘operational’ or ‘strategic’, where the differences between the operational tempos is defined by the immediacy of the mission and is measured in the amount of time it takes to plan, prepare and execute a mission. The temporal latency of the information, on the other hand, was measured as a degree to which the information was either recently collected, somewhat recently collected or old.
The resulting VOI matrix that would be used for one of the specific operational tempos is shown in Figure 5. Here the composite VOI rating is expressed on a Likert scale of zero thru ten with ten being extremely valuable and zero equally no value to the mission.
The SMEs used three individual surveys to gauge the
VOI for military mission immediacy of data use, namely
one for use within a short time, one for use within a
moderate time and one for use within a long time. The VOI
results gained for data use in a short amount of time are
shown in Figure 6.
The Fuzzy Associative Memory (FAM) model was chosen
to construct the prototype fuzzy system. A FAM is a
kdimensional table where each dimension corresponds to
one of the input universes of the rules. The ith dimension
of the table is indexed by the fuzzy sets that compromise
the decomposition of the ith input domain. For the
prototype system, three inputs are used to make the VOI
decision (source reliability, information content, and
timeliness); with three input domains, a 3-dimensional
FAM could be used. However, the decision was made to
use two, 2-dimensional FAMs connected “in series” to
produce the overall VOI result for several pragmatic
reasons
The overall architecture of the prototype fuzzy system is shown in Figure 7. Two inputs feed into the Applicability FAM: source reliability and information content; the output of the FAM is the information applicability decision. Likewise, two inputs feed into the VOI FAM: one of these (information applicability) is the output of the first FAM; the other input is the information timeliness rating. The output of the second FAM, and the overall system output, is the VOI metric.
The rules elicited from the SMEs are represented in the appropriate FAMs and form the fuzzy rule bases. The number of fuzzy sets, and thus the “language” of the rules, was defined in the knowledge elicitation phase using the two surveys described above. That is, the decomposition of the domains is as shown in Figures 3 and 5. The two inputs to the Applicability FAM are divided into five fuzzy sets; the output domain is divided into nine fuzzy sets. Likewise, for the VOI FAM, the input domain for information applicability is divided into nine fuzzy sets (as just mentioned), the timeliness input domain three fuzzy sets, and the output domain eleven fuzzy sets.
Figures 4 and 6 actually represent the fuzzy rules bases for the two FAMs resulting from the knowledge elicitation process. For example, Figure 4 demonstrates that one rule in the Applicability FAM is “If source reliability is reliable (A) and information content is possibly true (3), then information applicability is highly applicable (7)”. Note that the VOI FAM shown in Figure 6 applies only to the fast operational tempo (tactical) mission context, while the Applicability FAM is constant across all three mission contexts.
Triangular membership functions are used within the
system, wherein the triangles are isosceles with evenly
spaced midpoints. The output from each FAM is
determined by the standard centroid defuzzification
strategy. More detailed description of the FAMs, the fuzzy
rule bases, the domain decompositions, and other
implementation aspects of the prototype system can be
found in
The prototype system has been exercised across numerous scenarios (that is, various combinations of input values) to produce VOI determinations. These preliminary system results have been demonstrated to the SMEs and the system performance has been validated in principal and concept. That is, the system output has been judged to be consistent with what the SMEs would expect, and the prototype has demonstrated the feasibility to both elicit rules from experts in this domain as well as to use the extracted knowledge in a meaningful way.
Note that there is no current system against which the results can be compared. As such, the system has not been tested comprehensively due to the human-centric, contextbased nature of the problem and usage of the system. Thus, the system performance will need to be validated by providing the SMEs with various scenario-based VOI results for their examination and feedback. In some cases the output of the system is an exact application of the rules provided by the SMEs which should permit easy judgment; in other instances, the system output is less clear and will require more detailed examination.
Information drives action and for the military that is facing an unprecedented increase in the types and amount of information available, the ability to separate the important information from the routine is paramount. This paper presented an approach used for gathering the parameters to calculate the VOI for military analysis and allow the subsequent development of a fuzzy-based prototype system.
The obvious next step for this effort is to seek validation of the system from the SMEs by producing a comprehensive, well-designed set of scenario-based VOI results for their examination and feedback. It is entirely possible that the concepts and relationships captured through the conceptual method of knowledge elicitation would require modification. If so, further iterations of the knowledge elicitation process will occur. As the program matures, the capability to accommodate inconsistent or contradictory information will be investigated. For the military, the ability to efficiently and effectively calculate VOI and separate the wheat from the chaff is paramount. This program is an important step towards that goal.