Creation of machine-usable, high-quality knowledge-bases is a critical prerequisite for many important applications that rely on availability of high-level of autonomous decision-making and reasoning capabilities. Manual construction of knowledge-bases for complex applications is a time-consuming and expensive process. In such application domains, however, a vast amount of knowledge is available in human-readable format, and it could be leveraged to build knowledge-bases automatically. Natural Language Processing (NLP)-based techniques provide an attractive option for this process. The field of NLP has made rapid strides in last several years and resulted in increased usage across a variety of consumer-facing applications. However, their usage for knowledge-base construction in the aviation industry remains rather limited to date. We present our assessment of using various NLP-based tools for the creation of aviation-focused, high-quality, machineprocessable, and human-legible knowledge bases (KBs) for various applications. We identify several gaps, both at the application and fundamental levels, and also identify potential directions for future research that could help overcome the challenges.
Fueled by the fast-paced advances in the field of Artificial Intelligence (AI), in particular Machine
Learning (ML), autonomous and intelligent systems are becoming pervasive in a broad range of
applications, from tailored advertisements and suggestions, shaping the on-line user experience, to
complex cyber-physical systems such as autonomous cars. These systems interact with the environment
and the humans, understand the current situation and plan actions to achieve goals as commanded
by other agents or predefined at design time. While no industrial sector seems to be able to resist the
allure of delivering “intelligent” products and services, and realizing the potential economic benefits
that AI may bring to their businesses, aerospace and defense have been slow adopters, playing the
role of observers, still hovering around research, prototyping and small-scale demonstrations. We
believe that the main reasons are operational complexity and stringent safety requirements. Here,
we specifically refer to operational autonomy, namely the ability of an embedded system to replace
humans in the execution of a mission that involves several cyber-physical platforms and a mix of
physical and non-physical actions. For example, replacing pilots to bring passengers from one airport
to another is a complex mission that involves executing pre-flight operations, pushing of the gate,
reaching the runway, taking of, climbing, cruising, descending, landing, and taxiing to the
destination gate. Currently, pilots deal with hundreds of tasks [
Pilots operating commercial flights benefit from automation functions implemented in sub-systems such as autopilots, instrument landing, and low-level controls for electrical systems, air-management systems, and vehicle health assessment. However, not only these systems may fail or disengage at any time, but during of-nominal situations, their signals need to be integrated to understand the rootcause of a contingency and take appropriate actions. An intelligent system capable of replacing a pilot needs to have the same level of proficiency in both nominal and of-nominal situations. Considering that pilots have trained or flown for thousands of hours, and have studied the inner workings of the machine they are operating, such intelligent system must be equipped with a vast amount of background knowledge that reaches as far as physics. Furthermore, some common-sense knowledge should be taken into account to deal with other situations that are not directly related to flying such as keeping the hundreds of passengers happy and relaxed, or dealing with medical emergencies.
Knowledge base construction (KBC) is the process of populating a database with information from
data such as text, tables, images, or video [
Automated knowledge base construction (AKBC) is another approach where tools are employed
to process a potentially much larger set of input sources, at a remarkably higher throughput rate
compared to trained domain experts, and generate more comprehensive knowledge bases. As an
example, PaleoDeepDive, a DeepDive-based approach for construction of entities and relations from
PDF documents was able to process roughly 10x the number of documents, with per-document recall
roughly 2.5x that of human annotators [
The aviation domain poses a unique set of challenges to automated knowledge base construction due to its stringent safety requirements. The constructed knowledge base cannot only contain correlation data among facts, but for critical decisions, the deduction process from inputs to conclusions must lead to actions that are safe. It is therefore important to revisit the architecture of an automated knowledge base construction pipeline to exploit its ability to be comprehensive while assuring correctness.
AKBC starts from sources that have been created by humans for human consumption including text
and tables, and generates knowledge bases that can be used by machines in a variety of applications.
The techniques that are typically used in this process include Natural Language Processing (NLP) [
However, many use-cases for the constructed knowledge-bases require a deeper understanding of
the world and the rules under which the system and processes that are the subject of the knowledge
model operate. For many complex applications, the knowledge acquisition and reasoning processes
need to understand and manipulate rich models of the world evoked by textual sources such as causal
relations or adherence to given procedures and standards [
Automated knowledge-base construction for aviation applications presents a set of unique challenges that are not necessarily present in consumer-facing applications. Consider the example of building an intelligent cyber-agent capable of safely and successfully operating an aircraft during a typical flight from the departure to the arrival gate. Such cyber-pilot (CP) (Figure 1(right)) must be knowledgeable about the vehicle that it is operating, the mission to be performed, the load that the vehicle is carrying (cargo and/or passengers, together with their properties such as value and health conditions), and aviation rules and regulations. The sources of aviation knowledge are vast and varied including flight manuals, maintenance manuals, accident reports, system models encoded as simulators, certification requirements documents, textbooks on flight dynamics, structural mechanics, aerodynamics, etc. (see Figure 1(left) for some examples).
To deploy a CP on commercial passenger vehicles, two other requirements must be fulfilled. First, it must be shown that safety will meet or exceed current standards, with a CP delivering a lower rate of mishaps than human-piloted aircraft in a variety of scenarios. Such stringent certification requirements are not considered in consumer-facing applications since errors don’t typically lead to catastrophic failures, and the decision loop is never closed without human supervision (human-inthe-loop). Certification is a process used to expose risks, and to evaluate whether they are acceptable. Current standards rely on analysis tools, but also heavily on human revisions and inspection. Thus, the second requirement for a CP is transparency or explainability in decision-making. Explainability as a feature is crucial at run-time where a CP may be required to communicate with Air Trafic Control (ATC), technicians, engineers, or other vehicles, and it is not only required to inspect and revise the model that the CP has learned, but also as a mean to identify, isolate, and manage contingencies, which is key in redundant architectures where alternative actions may guarantee safety. The ability to generate explanations goes beyond having a human-understandable model, but requires a knowledgebase that enables abductive reasoning.
To add to the complexity of knowledge acquisition and reasoning, significant amount of
knowledge is actually background knowledge that may have been gradually assimilated over a long period
of time (e.g., basic laws of mechanics, thermodynamics, laws about cause-efects). Consider for
example operational and procedural documents such as [
Automatic Knowledge-Base Construction System technical knowledge (e.g., fuel systems) General background knowledge
This documentdoesnotcontainanyexportcontrolledtechnicaldata.
RTX Proprietary. This material contains proprietaryinformation ofRaytheon Technologies Corporation. Anycopying, distribution, or dissemination of the contents of this material isstrictlyprohibitedand maybe unlawful withouttheexpresswrittenpermissionofRTX.Ifyouhaveobtainedthismaterialinerror,pleasenotifyRTXResearchCenter Counselat(860)610-7000 immediately. target knowledge representation languages that are able to capture a typical ontology of classes and logical facts expressed in a formal language such as First Order Logic, and also operators that change the state of the system which could be expressed as actions with preconditions and efects. Extracting such structures is challenging and requires integrating, extending, and developing new AKBC tools.
In this section, we present some challenges faced by current NLP-based tools in the automatic construction of aviation-focused, machine-understandable, high-quality knowledge-bases. We also discuss some potential approaches for overcoming the challenges.
Most of the existing state-of-the-art NLP tools (e.g., Torch-T5 text summarization model [
A domain specific corpus, or other filters and customization inputs to extraction tools alone would
perhaps not be suficient. Many relations used in aviation documents have a precise meaning that
is captured by domain specific models, reasoning and simulation engines. Learning these models
from a potentially large set of text sources or even data sets seems unnecessary, ineficient, and
suboptimal. Consider the following example taken from the emergency descent non-nominal checklist
of Boeing 737 [
Recommendations. The creation of a dedicated corpus for aviation is needed: it should contain sources with a mix of procedural and descriptive knowledge, and a variety of formats (see Section 2.2 for some considerations on how structure carries information). New techniques are also needed to leverage domain specific models and reasoning engines that precisely capture large knowledge fragments. After disambiguating and grounding facts into aviation-specific contexts, predicates and relations need to be mapped to available models. This is not a one-to-one mapping because the same model (or even a combination of models) could be used as interpretation of many predicates. Domain specific tools typically require a set of inputs and parameters that will need to be synthesized from qualitative statements, or computed by other models. The quantitative results from these models will then need to be lifted back into the knowledge base. Finally, the execution of the various reasoning engines and models will also need to be orchestrated (see Section 2.4).
Human-readable documents are typically written following a set of conventions, relying extensively
on formatting and highlighting. In many cases, such conventions are also typically explained at the
beginning of a document. Humans rely critically on this type of structure for eficiency and to avoid
ambiguities. Removing this structure from the document would certainly reduce its legibility. More
importantly (and relevant to the knowledge extraction process), the semantics of relations between
entities depends on the structure. For example, the relation between a component and a numeric quantity
in [
Another example of structure can be found in the procedures for stall recovery in [
Among existing tools for AKBC, Fonduer [
Once a document is processed from the formatting standpoint, the information needs to be
organized such that one can reason over the encoded knowledge. We refer to this as Structure of
Information. One view of such structure is parts of speech (POS) which provides a grammatical understanding
of paragraphs, sentences and words [42]. Another view is the logical and ontological structure
contained in the text [43, 44, 45, 46]. POS tagging is relatively well-developed, and mature tools exist
today for such task. However, extracting logical structures from raw sources such as text is less
mature [43, 44, 45, 46]. Having a logical representation enables reasoning by deduction and abduction
which are both important in situational assessment, decision-making, and root-cause analysis.
Moreover, POS tagging alone cannot be used to resolve semantic ambiguity, where the same word may
have two completely diferent roles and meaning in a sentence. A logical representation would
instead enable disambiguation by reasoning and elimination of hypotheses. We also believe that the
types of structures to be extracted are not only logical connectives such as And-Or patterns, but also
include causal and temporal dependencies among events, hidden states, action models, and
performance curves. As an example, the non-normal checklists, flight patterns and maneuvers described
in [
Recommendations. As a first step, we recommend performance evaluation of Fonduer and similar frameworks in identifying the structure present in aviation corpora discussed in Section 2.1. As a next step, we suggest extension of existing AKBC approaches to move beyond document-level structure and towards aviation-level hierarchical organization of concepts - for example, Aircraft → Boeing → 737-800. In efect, we are essentially proposing a path forward for extending AKBC approaches to go from Information in Structure towards Structure of Information. Creation of a large-scale aviation ontology would help in defining and organizing the structure of information and ontology learning techniques from text could help here [51]. We also recommend further research to extend current tools with capabilities to extract events, causal and temporal relationships, and action models to be used in decision-making, plan verification, and impact analysis.
Humans are incredibly good at attaching context to text and overcoming ambiguities in text through a deep understanding of rules of the world. However, it is highly non-trivial to replicate and automate this process [52]. The challenge in resolving ambiguity points to the underlying issue of background (unstated) vs. foreground (explicitly stated) knowledge. We as human readers make use of an incredible amount of background knowledge when understanding and attaching context to the meaning of words, sentences and paragraphs.
Common problems related to the ambiguous use of words in natural languages are less prevalent in the aviation domain as manuals and reports are written in such a way to make sure that the messages cannot be misinterpreted by humans. Clearly, a first step has to be taken towards removing some common potential ambiguity that may arise. For example, when processing a set of sentences such as Every airplane has two wings; Boeing 747 has four engines; Every wing is a part; Every engine is a part, it is important to identify “four” as the same as the number 4. Also, “Boeing 747” should be identified as an airplane (perhaps by processing other documents that mention such a fact). Semantically, it is also important to establish whether the closed or open world assumption [53, 54] is used since a fact such as Boeing 747 has 6 parts may or may not be inconsistent with the description above.
A more serious issue, however, is the semantic ambiguity of words and relations that, when
interpreted by a reader with enough background knowledge, have instead an unambiguous meaning.
Consider for example a fragment of the Emergency Descent non-normal checklist for the Boeing
747 [
Currently, no mature techniques exist to guide the separation between foreground and background knowledge [55]. Accompanying this challenge is the issue of innate knowledge (e.g., intuition about basic laws of physics) [56, 57], and it is not clear how to incorporate such knowledge within the AKBC process.
Recommendations: We recommend investigating the possibility of combining ideas from Controlled Natural Languages (CNLs) with existing AKBC frameworks. CNLs [52, 58, 59] help to partially overcome ambiguity present in textual sources written for human consumption. CNLs are subset of natural languages (e.g., English) and have well-defined formal semantics. However, the use of CNLs requires expert understanding of the CNL itself and hence encoding is still non-trivial, and challenges remain [58]. Hybrid CNLs that combine ideas from formal logic with CNL, e.g., Knowledge Authoring Logic Machine (KALM) [60, 61] hold more promise.
On the issue of discovering missing background knowledge, or to reduce semantic uncertainty, we recommend investigating approaches for combining abductive, deductive and inductive reasoning techniques with iterative learning [62, 63, 55].
Knowledge-base creation is an iterative process. The ability to encode knowledge eficiently depends not only on the framework being used to encode knowledge facts, but also on availability of solvers that can test the encoded knowledge fragments for errors. It is rarely the case that a first attempt results in a correct and useful knowledge-base. Our experience indicates that the processes of encoding knowledge fragments and reasoning over them are in fact tightly interlinked.
There are a multitude of encoding and reasoning approaches, each designed or natural for
processing a specific type of knowledge. We can distinguish two orthogonal ways of specializing
encoding and reasoning: algorithmic domains and application domains. A variety of languages exist to
model knowledge within algorithmic domains. For example, the Plan Domain Definition Language
(PDDL) [64] focuses on encoding decision-making problems, while temporal logics [65] are more
suitable for modeling and reasoning about the temporal relations among events. Similarly, knowledge
graphs [
Domain independent reasoning, however can become ineficient. In many cases, both the encoding and the reasoning algorithms can be specialized to particular domains where only certain queries are of interest. For example, the language, concepts and machinery used to model and solve computational fluid mechanics problems are diferent from the ones used in structural mechanics, which are in turn diferent from the ones used in dynamics and control.
No principled technique exist currently for a systematic orchestration of knowledge elicitation,
encoding, fusion and testing of such multitude of algorithmic and applications domains [67]. We
note here that frameworks such as DeepDive [
Recommendations: There have been some recent advances in the area of knowledge graphs and semantic web that try to achieve knowledge fusion by leveraging ideas used for data fusion [68]. Datafusion inspired knowledge-fusion techniques [68], together with specialized domain solvers (e.g., [69] for combinatorial reasoning) could provide a useful starting point towards solving the knowledge fusion problem in general for aviation and autonomy applications. More advanced techniques could be borrowed from the theorem proving community that has been very active in defining general ways of combining theories and solvers as in the case of Satisfiability Modulo Theories [70]. In addition, development of interface languages, translators, and reasoners to orchestrate diferent solvers and cross the boundaries of diferent domains would also be needed.
As we mentioned in Section 2.1, a crucial gap in existing AKBC frameworks is the limited ability to
define large-scale automated tests that can be used during the AKBC process. Existing benchmarks,
for example those that are used in assessing performance of various NLP tools fall well short of testing
real-world understanding [
Recommendations: We recommend borrowing ideas from formal methods and software testing
communities [75] for expanding the current suite of tests used in various NLP and AKBC
frameworks [
The introduction of autonomy in the management of complex systems and their operations requires developing knowledge-bases that can be used for reasoning and decision-making. Manual construction of these knowledge-bases is ineficient, requiring multi-year eforts and hundreds of contributors. Thus, Automatic Knowledge Base Construction (AKBC) techniques are very much sought in related applications domains. Recent advances in Machine Learning have enabled the development of tools for Natural Language Processing, Natural Language Understanding, and Machine Reading Comprehension that show promise in ingesting raw sources such as text and tables to create knowledge-bases. The aviation industry could benefit from these new technologies, but the resulting knowledge-bases must satisfy stringent assurance requirements that support certification processes. We believe that an AKBC system for these kinds of applications requires the integration and enhancement of several techniques and that it is yet to be developed. The aviation context needs to be captured by a corpus of relevant documents and additional inputs such as extraction rules. Additional context should be injected as interpretation of relations provided by analytic and simulation models. The structure of aviation documents should be exploited in the extraction process to provide the right semantics to parts of speech in diferent paragraphs. The extracted information should be encoded in languages that enable formal reasoning to further reduce ambiguity and noise. The encoded knowledge should be fused with background knowledge and a variety of domain-specific reasoning engines should be harmonized to lead to a high-quality knowledge-base. Finally, the constructed knowledge-base should undergo rigorous proficiency testing to provide assurance and refine its content.
While several gaps exist, we believe that the current set of technologies are promising and that a concrete research roadmap could be developed to construct high-assurance knowledge bases for safety critical applications. arXiv:1910.10683 (2019). [38] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [39] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics 36 (2020) 1234–1240. [40] I. Beltagy, K. Lo, A. Cohan, SciBERT: A pretrained language model for scientific text, arXiv preprint arXiv:1903.10676 (2019). [41] E. Alsentzer, J. R. Murphy, W. Boag, W.-H. Weng, D. Jin, T. Naumann, M. McDermott, Publicly available clinical BERT embeddings, arXiv preprint arXiv:1904.03323 (2019). [42] C. Manning, H. Schutze, Foundations of statistical natural language processing, MIT press, 1999. [43] J. Berant, A. Chou, R. Frostig, P. Liang, Semantic parsing on freebase from question-answer pairs, in: Proceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1533–1544. [44] P. Yin, G. Neubig, TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation, arXiv preprint arXiv:1810.02720 (2018). [45] V. Basile, E. Cabrio, C. Schon, KNEWS: Using logical and lexical semantics to extract knowledge from natural language, in: Proceedings of the European conference on artificial intelligence (ECAI) 2016 conference, 2016. [46] H. Singh, M. Aggrawal, B. Krishnamurthy, Exploring neural models for parsing natural language into first-order logic, arXiv preprint arXiv:2002.06544 (2020). [47] F. Maria, C. A. Aguirre, B. M. Anshutz, W. H. Hsu, MATESC: Metadata-analytic text extractor and section classifier for scientific publications., in: KDIR, 2018, pp. 259–265. [48] H. Yang, C. A. Aguirre, F. Maria, D. Christensen, L. Bobadilla, E. Davich, J. Roth, L. Luo, Y. Theis, A. Lam, et al., Pipelines for procedural information extraction from scientific literature: Towards recipes using machine learning and data science, in: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, IEEE, 2019, pp. 41–46. [49] M. Chang, L. V. Guillain, H. Jung, V. M. Hare, J. Kim, M. Agrawala, Recipescape: An interactive tool for analyzing cooking instructions at scale, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–12. [50] C. X. Chu, G. Weikum, N. Tandon, J. Vreeken, Mining How-to Task Knowledge From Online
Communities, Ph.D. thesis, Universität des Saarlandes Saarbrücken, 2016. [51] R. Lourdusamy, S. Abraham, A survey on methods of ontology learning from text, in: International Conference on Information, Communication and Computing Technology, Springer, 2019, pp. 113–123. [52] R. Schwitter, Controlled natural languages for knowledge representation, in: Coling 2010:
Posters, 2010, pp. 1113–1121. [53] R. Reiter, On closed world data bases, in: Readings in artificial intelligence, Elsevier, 1981, pp.
119–140. [54] G. Bossu, P. Siegel, Saturation, nonmonotonic reasoning and the closed-world assumption, Artificial Intelligence 25 (1985) 13–63. [55] G. Marcus, Deep learning: A critical appraisal, arXiv preprint arXiv:1801.00631 (2018). [56] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, S. J. Gershman, Building machines that learn and think like people, Behavioral and brain sciences 40 (2017). [57] G. Marcus, Innateness, alphazero, and artificial intelligence, arXiv preprint arXiv:1801.05667 (2018). [58] T. Kuhn, A survey and classification of controlled natural languages, Computational linguistics 40 (2014) 121–170. [59] T. Gao, Controlled natural languages for knowledge representation and reasoning, in: Technical Communications of the 32nd International Conference on Logic Programming (ICLP 2016), Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2016. [60] T. Gao, P. Fodor, M. Kifer, High accuracy question answering via hybrid controlled natural language, in: 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI), IEEE, 2018, pp. 17–24. [61] T. Gao, P. Fodor, M. Kifer, Knowledge authoring for rule-based reasoning, in: OTM Confederated International Conferences" On the Move to Meaningful Internet Systems", Springer, 2018, pp. 461–480. [62] A. Weichselbraun, P. Kuntschik, A. M. Braşoveanu, Mining and leveraging background knowledge for improving named entity linking, in: Proceedings of the 8th International Conference on Web Intelligence, Mining and Semantics, 2018, pp. 1–11. [63] J. B. Tenenbaum, C. Kemp, T. L. Grifiths, N. D. Goodman, How to grow a mind: Statistics, structure, and abstraction, science 331 (2011) 1279–1285. [64] P. Haslum, N. Lipovetzky, D. Magazzeni, C. Muise, An introduction to the planning domain definition language, Synthesis Lectures on Artificial Intelligence and Machine Learning 13 (2019) 1–187. [65] E. A. Emerson, Temporal and modal logic, in: Formal Models and Semantics, Elsevier, 1990, pp.
995–1072. [66] M. Richardson, P. Domingos, Markov logic networks, Machine learning 62 (2006) 107–136. [67] X. L. Dong, D. Srivastava, Knowledge curation and knowledge fusion: challenges, models and applications, in: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, 2015, pp. 2063–2066. [68] X. L. Dong, E. Gabrilovich, G. Heitz, W. Horn, K. Murphy, S. Sun, W. Zhang, From data fusion to knowledge fusion, arXiv preprint arXiv:1503.00302 (2015). [69] L. De Moura, N. Bjørner, Z3: An eficient SMT solver, in: International conference on Tools and
Algorithms for the Construction and Analysis of Systems, Springer, 2008, pp. 337–340. [70] C. Barrett, C. Tinelli, Satisfiability modulo theories, in: Handbook of Model Checking, Springer, 2018, pp. 305–343. [71] E. E. Alves, D. Bhatt, B. Hall, K. Driscoll, A. Murugesan, J. Rushby, Considerations in assuring safety of increasingly autonomous systems (2018). [72] M. Cummings, Adaptation of human licensing examinations to the certification of autonomous systems, in: Safe, autonomous and intelligent vehicles, Springer, 2019, pp. 145–162. [73] R. A. Clothier, B. I. Williams, T. Perez, et al., Autonomy from a safety certification perspective, in: AIAC18: 18th Australian International Aerospace Congress (2019): HUMS-11th Defence Science and Technology (DST) International Conference on Health and Usage Monitoring (HUMS 2019): ISSFD-27th International Symposium on Space Flight Dynamics (ISSFD), Engineers Australia, Royal Aeronautical Society., 2019, p. 278. [74] L. Chiticariu, Y. Li, F. Reiss, Transparent machine learning for information extraction: state-ofthe-art and the future, EMNLP (tutorial) (2015). [75] M. Kassab, J. F. DeFranco, P. A. Laplante, Software testing: The state of the practice, IEEE
Software 34 (2017) 46–52. [76] G. Marcus, The next decade in AI: Four steps towards robust artificial intelligence, arXiv preprint arXiv:2002.06177 (2020). [77] V. Kocijan, T. Lukasiewicz, E. Davis, G. Marcus, L. Morgenstern, A Review of Winograd Schema Challenge Datasets and Approaches, arXiv preprint arXiv:2004.13831 (2020).