- Multiple lines of research in cognitive science have brought insight on the role that internal (cognitive) representations of situational context play in framing decision making and in differentiating expert versus novice decision performance. However, no single framework has emerged to integrate these lines of research, particularly the views from narrative reasoning research and those from situation awareness and recognition-primed decision research. The integrative framework presented here focuses on the cognitive processes involved in developing and maintaining context understanding, rather than on the content of the context representation at any given moment. The Narratively-Integrated Multilevel (NIM) framework views context development as an on-going and selforganizing process in which a set of knowledge elements, rooted in individual experience and expertise, construct and maintain a declarative, hierarchical representation of the situational context. The context representation that arises from this process is then shown to be the central point of both situational interpretation and decision-making processes at multiple levels, from achieving specific local goals to pursuing broad motives in a domain or theater of action.
INTRODUCTION
The current scientific understanding of the role of context
in decision-making has evolved in multiple steps over the last
forty years. Cognitive science research has long shown that
while human actions and decisions are based on the person’s
environmental context, the decision-making process relies on
an internal (cognitive) representation of the context, not
directly on the context as sensed (see [
mental models in cognition, e.g.,[
While this thread of cognitive research was building an
understanding of the role of context from the bottom-up (i.e.,
building from fundamental insights on human information
processing mechanisms), a separate thread of ‘top-down’
cognitive research unfolded from the 1980s forward. This
thread explored how people understand and reason about
sequences of action and interaction in which the main source of
variability is human behavior. (This aspect is particularly
germane to military decision-making, in that it typically
involves situations with both adversaries and non-combatants).
This research focused on narrative reasoning processes in
which the observer/participant constructs, analyzes, and
explains complex situations through a narrative (story-telling)
process. Specifically, it found that people almost universally
use story narratives to represent, reason about, and make sense
of contexts involving multiple interacting agents, using
(general) motivations and (local) goals to explain both
observed and possible future actions. In other words, people
were found to generally make sense of their human contexts by
either integrating them into a novel narrative or, more
commonly, by recounting them as an instance of a
commonlyknown or culturally based narrative [
Despite their convergent directions, the bottom-up SA/RPD theories and the top down narrative reasoning theories have not yet met. This paper presents a framework in which such an integration can occur, and explores its benefits for decision support and human-machine integration.
II.
CONTEXT AS INTEGRATED PROCESS
This failure of the two theories to integrate immediately
points out several unmet challenges for decision support. For
example, changing patterns within SA do not, by themselves,
present the DM with any easy way to see alternative narrative
interpretations for the context dynamics (making DMs more
vulnerable to deception). SA theory and RPD theory have
worked best in contexts that involve well-defined
problemsolving in bounded problem domains, such as putting out fires
[
The authors and colleagues have conducted a line of
research to develop and apply computational models of expert
cognition in various domains, both to test and refine cognitive
theory and to develop support for decision making and decision
training. That research initially focused on operationalizing the
SA/RPD body of theory, and resulted in a computational
architecture called COGNET [
A. Framework for Integration
The main idea underlying this integration is that what SA/RPD and narrative reasoning theories implicitly or explicitly refer to as the understanding or awareness of context is really a momentary “snapshot” of fundamental processes integrating multiple sources of information about the natural and human (i.e., social) aspects of the environment. This process of context development is constructive, self-organizing, operates at multiple discrete levels of abstraction which generally involves increasing time-scales across levels. These four key features are defined as follows: • • •
Constructive -- consists of constituent elements that, through their interaction, build a symbolic representation, the momentary content of which we may consciously recognize as the current context.
Self-organizing -- the constituent elements operate independently but follow principles or rules of operations that are organic to the human information processing design, such that a consistent and self-regulating process (of context development) emerges.
Operates at multiple-discrete levels of abstraction -- the symbolic representation which is built and maintained has distinct layers of structure which reflect levels of understanding that each incorporate a broader scope of information about the environment but in correspondingly increasingly abstract terms that include salient and diagnostic attributes, with links to lower levels of abstraction where more detailed (but less integrated) information is maintained. These levels equally organize the constituent processing elements that build the context representation as much as they organize the representation itself. In this initial formulation of the framework, there are four levels corresponding to the three hierarchical levels of Situation Awareness (Perception, Comprehension, Projection) and one higher level of Narrative Understanding which integrates the other three. We thus call the framework the NIM (Narratively-Integrated Multilevel framework.
Involves increasing time-scales across levels -- each increasing level of abstraction deals with a broader scope of events (from perceptual events at the lowest level to narrative units at the highest level). As that scope increases, the general time-scale of events similarly increases. For example, perceptual events, such as those tracking locations of a (single) moving object, are very dense in time and result in repeated updates to perceptual level information in the context representation. At higher levels, updates typically occur less frequently, as many lower level changes are needed to create a significant or meaningful update. Narrative pacing, the highest level, typically is the slowest, as a great deal of action in the environment is typically aggregated into a single narrative unit. This relationship of increasing time scale and increasing scope is very similar to the concepts presented in Newell’s timescale of human action [19: Figure 3-3]. Thus, the amount of processing would tend to be much greater at lower levels, though the scope and usefulness of the information in the representation would tend to be much broader at higher levels. However, because of the constructiveness feature, the highest level cannot be constructed without all the processing involved in building and maintaining the lower levels.
The dynamics of the process are moved forward both by sensory information (on the external world), physical actions (taken in the external world), and internal sources of information that can be termed knowledge elements. In the NIM framework, the context representation is constantly being manipulated in different ways by knowledge elements (KEs) that themselves are activated by externalities (in the form of sensations and/or physical actions), or by internalities (in the form of patterns of information within the declarative representation or associations to past experiences). Thus, the various knowledge elements construct and maintain the context representation in a self-organizing way, without any explicit starting or stopping (or other control) mechanism.
B. Computational View of the NIM Framework
As a process, context development is an example of, and
can be computationally modeled using, Selfridge’s
Pandemonium architecture [
It can be argued that a background process that develops and maintains an understanding of context is a highly adaptive characteristic of human beings, because it provides the individual a constantly available basis for interacting with the environment. The representation-building knowledge elements that construct the context representation reflect both individually acquired expertise and culturally-transmitted understanding of the local or domain-specific environment, so the context representation is not only always available, but also encodes information that experience (individual and collective) has shown to be useful in those environmental interactions. Ultimately, it is through its ability to support effective actions and interactions in the natural and social environment that the value of the context process is realized.
Representation-building
Knowledge Elements
Research into decision-making has explored some of the ways in which the context representation supports decisionmaking. The RPD model, most specifically, has demonstrated that expert DMs are in many cases able to select an action or adapt a pre-existing action plan to a specific situation based on the patterns of information in the context model. The patterns of information prime a specific decision (course of action) without requiring intervening deliberative processes. More analytical decision processes, in contrast, involved multiple reasoning steps that manipulate the context representation to construct, rather than derive, a plan or specific action. Across this full continuum of analytical to automatized decision making, (often called the Cognitive Continuum, see [38]) the same process is occurring. Knowledge elements derive or construct decision options and courses of action by manipulating and operating on the information in the context representation. These can be called decision-development KEs.
In light of the above discussion on context development, the decision-development KEs can be seen as are analogous, to representation building KEs. Both use the information in the context representation, but the representation-building KEs use it to create changes to the context representation, while the decision development KEs instead use it to reason toward actions to be taken in the external environment.
To some degree, the preceding begs the question “what is decision-making?” For purposes here, decision is used broadly to refer to the processes by which purposive actions are selected or constructed, whether or not there is a conscious awareness at the time that a decision is being made. This is broadly in line with RPD theory which notes that the RPD process typically renders what appears, to a novice or outsider, to be a difficult decision, as simply an obvious or automatic action to the expert.
One additional feature needs to be added to the NIM framework to describe or model the relationship of the contextdevelopment process to the decision-making process. That is the notion of hypothesizing – constructing and manipulating alternative descriptions or relationship sets for part or all of a context representation, typically by creating hypothesized representations of future contexts that might result from contemplated decisions or actions. For context to support decision making, there needs to be proxy representations of context, in which decision-development KEs can use to construct and assess potential decisions and actions. This space, unlike the context representation, is not an internal model of the external situation, but is rather a hypothesized representation of it as it might be, if potential decisions and actions were taken. This allows such decision-development KEs to maintain alternative multi-level representations of an evolving situation, or project forward possible decisions or actions based on a narrative interpretation or course of action being considered. Figure 2 expands Figure 1 to show how decision-development KEs and hypothetical context representations extend the context development process to support dynamic decision-making of all kinds.
The cognitive process of context development and maintenance is common to all human adults, just as is the process by which context understanding is used to make decisions and construct actions in the external environment. The environments in which these human capacities evolved were relatively bounded and unfolded in time scales generally in line with human information processing. However, this began to change in historical times, as social and technological complexity rapidly increased. Since the start of the electronics and computer age, human DMs find themselves increasingly embedded in complex environments in which the speed and complexity of events greatly outstrip human cognitive abilities. Real-time decision-making domains such as military command and control or management of large-scale industrial processes bring environments in which it is essentially impossible for an unaided DM to fully understand the context in which actions must be taken.
The preceding half century has seen increasingly sophisticated efforts to support and augment human decisionmaking. Research to understand human cognition has been stimulated by the need for more effective decision-support, and has driven the evolution of decision support. In particular, it has resulted in an approach (termed cognitive engineering) to designing decision support systems, based on designing the systems to integrate well with the ways in which humans perceive, think, and act.
The NIM context-development view offers a new basis for cognitive engineering of decision support systems. The framework shows how multiple levels of context understanding are simultaneously developed and maintained, and are also simultaneously used to identify opportunities for action and for action options. This suggests a way to design decision support, in which the support system develops its own context representation (based on a model of the human contextdevelopment process), and applies this model to develop decision/action information at multiple levels of abstraction. Further, such a system can both provide its context representation to the DM as representational support, and provide its decision/action information to the DM as decision support. Because it is expressed in fundamentally computational terms, the NIM framework suggests a way to develop the context and decision models that such a support system would require.
Before providing a brief example of how this might work,
we note two other interesting characteristics of the NIM
framework with regard to the application areas of interest to
this conference. The first is in the area of human-machine
integration. Substantial research and engineering effort has
been devoted to automating the process by which a human
operates a continuous-control system, such as a vehicle or
power plant. In between manual control and full automation,
however, are many approaches to partial automation that
structure the engineering space. All generally fall under the
concept of supervisory control (originated by Sheridan and
Johannsen, [
The second is an interesting correspondence between the
context development NIM view of context development and
military models of decision making, particularly the military
decision making model known as the
Observe-Orient-DecideAct or OODA Loop, first created by Boyd in the 1980s [
A notional example is provided below to demonstrate the potential application of the NIM framework. The example focuses on the management and control of multiple uninhabited vehicles (UxVs). Such groups of vehicles can be used in diverse missions ranging from post-disaster search and rescue, to battlefield intelligence collection and tactical interdiction. The framework was used to map out the context process in this domain, and to link it to support for both the Observe/Orient stages of the OODA loop and the Decide/Act stages. The result is pictured in Figure 3.
The figure is organized top-to-bottom into four stacked bands that represent the four levels of context representation. The figure has a left-to-right structure as well. In the center of the figure is a box that represents the dynamic context development process, as it would be performed by a computational model. That box is divided into two columns, with the left depicting the various levels of context representation, and the right representing the corresponding representations constructed to develop decision and action plans from the context representation. These two columns correspond to the Observe/Orient and Decide/Act phases of the OODA loop.
On the immediate left of the context development box is a column that represents the representation-building KEs. These KEs both dynamically build/maintain the context representation, and push information to the next (on the left) column as support for the human DM’s understanding of the context. On the immediate right of the context-development box is a column that represents decision-development KEs that dynamically build/maintain representation of decisions and actions based on current context dynamics, and that push information to the next (on the right) column as support for the human DM’s selection and instantiation of action options. Thus, the entire left side of the figure represents support for the OO parts of the OODA loop, while the entire right side represents the support for the DA parts.
Below the lowest level of context is a black bar that represents the environmental interfaces decision system (human augmented by context-driven support). In the case of multi-UxV command and control, these environmental interfaces would be with various sensors and information streams from the UxVs being controlled.
In Figure 3, the context-development process builds upward from perceiving basic situational information (Level 1) through identifying the significance of the elements (Level 2) and projecting the capabilities of key elements forward into the future (Level 3). From that, the lower level information is fit into stories and understood in the context of the narrative of the current mission (Level 4). The right-most column of Figure 3 then depicts the reasoning activities that the context-based decision support model is performing to take action in the environment and accomplish the mission. At the highest level, the model may revise or refine the current story narrative, and update it in terms of his/her evolving lower level context understanding. As the action proceeds to the point that a choice must be made between possible ‘next’ narrative units, the model makes use of the current context to choose a possible path forward (through the current narrative space), and conveys it to the human DM. If the DM concurs, the model could translate that general narrative step into specific local action plans (e.g., creating new waypoints, altitude, sensor-settings, etc.).
Additional detail can be seen by more closely examining the two columns labeled “Observe/Orient” and “Representation Building KEs” from bottom to top. Figure 3 shows that the: •
Object representations of information from sensors and/or data streams are created as the lowest levels of context information, using sensory KEs (e.g., monitoring sensor feeds looking for new data, which are then processed to create a new track object or update an existing one). STIDS 2013 Proceedings Page 53 • • • •
Declarative context representation is built and updated from the primitive object representations by perceptual KEs that construct a multi-level structure with built-in semantic significance regarding the levels; information is created and modified as elements of meaning are inferred or created for them. Initially, the perceptual KEs look for information with specific kinds of meaning (e.g. waypoints, vehicle locations) and place them in the context structure.
Context representation updates happen continuously as situational KEs combine information from multiple places in the context representation. For example the appearance of a hostile radar emission (created by a perceptual KE) might trigger a situational KE to examine all UxV tracks and infer which one(s) may have been detected, and to add a ‘likely detection by hostile’ annotation to that UAVs information on the context representation. That changemay, in turn, trigger another KE to add a ‘need to evaluate’ annotation on the track to stimulate examination of its altitude or flight path.
Narrative updates happen as changes in the dynamic content trigger Dynamic Narrative KEs to offer evidence on whether the narrative may have changed from one narrative state to another. For example, the preceding hostile radar may, if expected, activate a Dynamic KE to post evidence that the narrative may have moved from an ‘ingress’ phase to an ‘in hostile airspace’ narrative state. Finally, Narrative Space Update and Narrative Interpretation are made as Narrative Update KEs weigh evidence for and against a transition across narrative units. If posted evidence outweighs posted counter-evidence above a threshold, then a Narrative Update KE may be triggered to update the story narrative to reflect that narrative-state transition. Other Narrative Update KEs can be triggered by very anomalous information that may activate narrative re-examination. For example, if the story narrative were about a reconnaissance in a demilitarized area, the presence of the sudden hostile radar detection may trigger a Narrative Update KE that would look for other narratives that might incorporate this fact which does not ‘make sense’ in the baseline narrative. That KE might suggest re-examination of the data against stories of outbreak of hostilities or new insurgent activity as alternative stories.
This example is intended to point out how the NIM view of context development as an ongoing and core cognitive process can act as an integrating element for advanced decision support systems. Moreover, the example suggests how the framework can be further applied to integrate the design of human-systems integration and to translate the cognitive and technological issues into widely accepted military concepts such as OODA that can support the transition of such advanced decision support systems into operational use.
VI.
DISCUSSION This NIM framework presented here is built on the premise that human decision makers approach and resolve a decision based on their understanding of the situational context of that decision. When the decision maker is operating within a class of situations whose structure he/she understands very well, her or his internal context model will be rich and organized at multiple interconnected levels of abstraction. Such a NIM context representation provides insights at each level of abstraction – from low-level immediate details to long-term high-level story-structures – and enables mechanisms that allow situational interpretations and decision options to be considered at each level in an integrated way. A key implication of this research is that any externally provided (i.e., computational) decision support information will be evaluated and considered by the decision maker in terms of his/her own internal context understanding. Thus, from a cognitive engineering perspective, any and all decision support components, algorithms, etc., should present their results in terms of the decisions makers’ context model, and should ideally be designed to be presented in such terms from the start. As implied here, one way in which this can be done is for the computational decision support system to build and maintain its own context representation, strongly modeled to mimic the context representations created and maintained by expert decision-makers in the domain.
In conclusion, we offer thoughts on the validation of the NIM framework, and the ways in which semantic technologies can be used to implement the NIM framework.
Validation. The difficulties of validating models of
cognitive processes, which are inherently unobservable, are
well discussed in the literature. Validation in cognitive science
is, in philosophy of science terms, typically limited to
standards of sufficiency (i.e., can a model explain all the data)
rather than necessity (i.e., only that model do so). Prolonged
validation studies for very fundamental constructs (such as
short term and working memory, see [
Semantic Technologies and NIM Implementation. Semantic technologies (the topic of this conference) can form the core of a computational system that implements a domainspecific model using the NIM framework. In fact, initial efforts to date have made increasing use of these, particularly the Resource Description Framework (RDF) semantic representation. While the earlier COGNET software used a custom-coded blackboard representation to create the lower three levels of the NIM declarative context representation, the most recent versions of the PAC software have moved toward a implementing the declarative context representation fully in RDF. Current research to integrate these two computational models is also focusing on RDF for all levels of context representation. The semantic RDF representation is then manipulated by KEs implemented as production mechanisms, sometime gathered into more complex require structures that chunk multiple reasoning elements into a unitary NIM KEs. [11] Hair, D. C., & Pickslay, K. (1993). Explanation-based reasoning in decision support systems. Technical Report. NAVAL COMMAND CONTROL AND OCEAN SURVEILLANCE CENTER RDT & E DIV.
SAN DIEGO CA.