We build a common-knowledge concept recognition system for a Systems Engineer's Virtual Assistant (SEVA) which can be used for downstream tasks such as relation extraction, knowledge graph construction, and question-answering. The problem is formulated as a token classification task similar to named entity extraction. With the help of a domain expert and text processing methods, we construct a dataset annotated at the word-level by carefully defining a labelling scheme to train a sequence model to recognize systems engineering concepts. We use a pre-trained language model and fine-tune it with the labeled dataset of concepts. In addition, we also create some essential datasets for information such as abbreviations and definitions from the systems engineering domain. Finally, we construct a simple knowledge graph using these extracted concepts along with some hyponym relations.
The Systems Engineer’s Virtual Assistant (SEVA)
SE commonsense comes from years of experience and learning which involves background knowledge that goes beyond any handbook. Although constructing an assistant like SEVA system is the overarching objective, a key problem to first address is to extract elementary commonknowledge concepts using the SE handbook and domain experts. We use the term ‘common-knowledge’ as the ‘commonsense’ knowledge of a specific domain. This knowledge can be seen as a pivot that can be used later to collect ‘commonsense’ knowledge for the SE domain. We propose a preliminary research study that can pave a path towards a comprehensive commonsense knowledge acquisition for an effective Artificial Intelligence (AI) application for the SE domain. Overall structure of this work is summarized in Figure 1. Implementation with demo and dataset is available at: https://github.com/jitinkrishnan/NASA-SE .
Creating commonsense AI still remains an important and
challenging task in AI research today. Some of the
inspiring works are the CYC project
Concept Recognition (CR) is a task identical to the
traditional Named Entity Recognition (NER) problem. A
typical NER task seeks to identify entities like name of a
person such as ‘Shakespeare’, a geographical location such
as ‘London’, or name of an organisation such as ‘NASA’
from unstructured text. A supervised NER dataset consists
of the above mentioned entities annotated at the word-token
level using labelling schemes such as BIO which provides
beginning (B), continuation or inside (I), and outside (O)
representation for each word of an entity.
Relation extraction tasks extract semantic relationships
from text. These extractors aim to connect named entities
such as ‘Shakespeare’ and ‘England’ using relations such
as ‘born-in’. Relations can be as simple as using
handbuilt patterns or as challenging as using unsupervised
methods like Open IE
SE concepts are less ambiguous as compared to generic
natural language text. A word usually means one concept.
For example, the word ‘system’ usually means the same
when referring to a ‘complex system’, ‘system structure’, or
‘management system’ in the SE domain. In generic text, the
meaning of terms like ‘evaluation’, ‘requirement’, or
‘analysis’ may contextually differ. We would like domain specific
phrases such as ‘system evaluation’, ‘performance
requirement’, or ‘system analysis’ to be single entities. Based on
the operational and system concepts described in
Technology Readiness Level. 2. grp: represents a group of people or an individual such as Electrical Engineers, Systems Engineers or a Project Manager. 3. syscon: represents any system concepts such as engineering unit, product, hardware, software, etc. They mostly represent physical concepts. 4. opcon: represents operational concepts such as decision analysis process, technology maturity assessment, system requirements review, etc. 5. seterm: represents generic terms that are frequently used in SE text and those that do not fall under syscon or opcon such as project, mission, key performance parameter, audit etc. 6. event: represents event-like information in SE text such as
Pre-Phase A, Phase A, Phase B, etc. 7. org: represents an organization such as ‘NASA’, ‘aerospace industry’, etc. 8. art: represents names of artifacts or instruments such as ‘AS1300’ 9. cardinal: represents numerical values such as ‘1’, ‘100’, ’one’ etc. 10. loc: represents location-like entities such as component facilities or centralized facility. 11. mea: represents measures, features, or behaviors such as cost, risk, or feasibility.
Abbreviations are used frequently in SE text. We automatically extract abbreviations using simple pattern-matching around parentheses. Given below is a sample regex that matches most abbreviations in the SE handbook. r"\([ ]*[A-Z][A-Za-z]*[ ]*\)" An iterative regex matching procedure using this pattern over the preceding words will produce the full phrase of the abbreviation. ‘A process to determine a system’s technological maturity based on Technology Readiness Levels (TRLs)’ produces the abbreviation TRL which stands for Technology Readiness Levels. ‘Define one or more initial Concept of Operations (ConOps) scenarios’ produces the abbreviation ConOps which stands for Concept of Operations. We prelabel these abbreviations as concept entities. Many of these abbreviations are also provided in the Appendix section of the handbook which is also extracted and used as concepts.
Various locations of the handbook and the glossary provide definitions of several SE concepts. We collect these and compile a comprehensive definitions document which is also used for the concept recognition task. An example definition and its description is shown below:
Definition: Acceptable Risk
Description: The risk that is understood and agreed to by the program/project, governing authority, mission directorate, and other customer(s) such that no further specific mitigating action is required. Using python tools such as PyPDF2, NLTK, and RegEx we build a pipeline to convert PDF to raw text along with extensive pre-processing which includes joining sentences that are split, removing URLs, shortening duplicate non-alpha characters, and replacing full forms of abbreviations with their shortened forms. We assume that the SE text is free of spelling errors. For the CR dataset, we select coherent paragraphs and full sentences by avoiding headers and short blurbs. Using domain keywords and a domain expert, we annotate roughly 3700 sentences at the word-token level. An example is shown in Figure 2 and the unique tag count is shown in Table 1.
Any language model can be used for the purpose of
customizing an NER problem to CR. We choose to go with
BERT
In the hand-labelled dataset, each word gets a label. The
idea is to perform multi-class classification using BERT’s
pre-trained cased language model. We use pytorch
transformers and hugging face as per the tutorial by
In this work, for relation extraction, we focus on hyponyms and verb phrase chunking. Hyponyms are more specific concepts such as earth to planet or rose to flower. Verb phrase syscon opcon seterm mea grp org cardinal event abb art loc micro/macro-avg The definition document consists of 241 SE definitions and their descriptions. We iteratively construct entities in increasing order of number of words in the definitions with the help of their parts-of-speech tags. This helps in creating subset-of relation between a lower-word entity and a higher-word entity. Each root entity is lemmatized such that entities like processes and process appear only once.
Using the words (especially nouns) that surround an already identified named entity, more specific entities can be identified. This is performed on a few selected entity tags such as opcon and syscon. For example, consider the sentence ‘SE functions should be performed’. ‘SE’ has tag NNP and ‘functions’ has tag NNS. We create a relation called subset-of between ‘SE functions’ and ‘SE’.
Relations from abbreviations are simple direct connections between the abbreviation and its full form described in the abbreviations dataset. Figure 3 shows a snippet of knowledge graph constructed using stands-for and subset-of relationships. Larger graphs are shown in the demo.
Finally, we explore creating contextual triples from sentences using all the entities extracted using the CR model and entities from definitions. Only those phrases that connect two entities are selected for verb phrase extraction. Using NLTK’s regex parser and chunker, a grammar such as VP: {(<MD>|<R.*>|<I.*>|<VB.*>|<JJ.*>| <TO>)*<VB.*>+(<MD>|<R.*>|<I.*>|<VB.*>| <JJ.*>|<TO>)*} with at least one verb, can extract relation-like phrases from the phrase that links two concepts. An example is shown in Figure 4. Further investigation of relation extraction from SE handbook is left as future work.
We presented a common-knowledge concept extractor for the Systems Engineer’s Virtual Assistant (SEVA) system and showed how it can be beneficial for downstream tasks such as relation extraction and knowledge graph construction. We construct a word-level annotated dataset with the help of a domain expert by carefully defining a labelling scheme to train a sequence labelling task to recognize SE concepts. Further, we also construct some essential datasets from the SE domain which can be used for future research. Future directions include constructing a comprehensive common-knowledge relation extractor from SE handbook and incorporating such human knowledge into a more comprehensive machine-processable commonsense knowledge base for the SE domain.