=Paper=
{{Paper
|id=Vol-3894/paper21
|storemode=property
|title=Prompt-Time Ontology-Driven Symbolic Knowledge Capture with Large Language Models
|pdfUrl=https://ceur-ws.org/Vol-3894/paper21.pdf
|volume=Vol-3894
|authors=Tolga Çöplü,Arto Bendiken,Andrii Skomorokhov,Eduard Bateiko,Stephen Cobb
}}
==Prompt-Time Ontology-Driven Symbolic Knowledge Capture with Large Language Models==
<pdf width="1500px">https://ceur-ws.org/Vol-3894/paper21.pdf</pdf>
<pre>
                                Prompt-Time Ontology-Driven Symbolic Knowledge
                                Capture with Large Language Models
                                Tolga Çöplü* , Arto Bendiken, Andrii Skomorokhov, Eduard Bateiko and Stephen Cobb
                                Haltia, Inc.


                                                Abstract
                                                In applications such as personal assistants, large language models (LLMs) must consider the user’s personal information and
                                                preferences. However, LLMs lack the inherent ability to learn from user interactions. This paper explores capturing personal
                                                information from user prompts using ontology and knowledge-graph approaches. We use a subset of the KNOW ontology,
                                                which models personal information, to train the language model on these concepts. We then evaluate the success of knowledge
                                                capture using a specially constructed dataset. Our code and datasets are publicly available at https://github.com/HaltiaAI/paper-
                                                PTODSKC

                                                Keywords
                                                ontology-driven symbolic knowledge capture, KNOW ontology, symbolic representation, knowledge graphs, large language
                                                models, fine-tuning



                                1. Introduction                                                                                                    termine which personal information will be cap-
                                                                                                                                                   tured and how it will be associated with other
                                Currently, many generative artificial intelligence (AI) ap-                                                        captured knowledge.
                                plications, particularly personal assistants, strive to offer                                                    • Ontology rules can help identify inconsistencies
                                users personalized experiences. To achieve this, AI appli-                                                         in the captured knowledge, allowing for valida-
                                cations must learn personal information and preferences                                                            tion before storage.
                                from user interactions (knowledge capture) and use this                                                          • Ontology relationships allow the extraction of
                                learned knowledge in future conversations (knowledge                                                               implicit information from captured knowledge,
                                utilization). Implementing this fundamental personal AI                                                            effectively enabling automatic inference that ex-
                                approach depends on addressing several complex sub-                                                                pands the knowledge graph.
                                problems, such as discerning which user prompt infor-
                                                                                                                                                 • A robust, personalized knowledge graph forms a
                                mation is personal, extracting it, determining whether
                                                                                                                                                   reliable foundation for facilitating personalized
                                the extracted information is duplicate, and associating it
                                                                                                                                                   interactions with the application through lan-
                                with other personal data.
                                                                                                                                                   guage models.
                                   These challenges have been the focus of extensive re-
                                search within the AI field for many years. However, the              In this paper, we address a specific aspect of the AI
                                emergence of neurosymbolic approaches through the col-            personalization challenge by focusing on prompt-time,
                                laboration between large language models (LLMs) and               ontology-driven symbolic knowledge capture using lan-
                                symbolic AI has provided researchers with new perspec-            guage models. We explore the extraction from user
                                tives [1, 2, 3, 4]. LLMs’ capabilities in natural language        prompts of subject-predicate-object triples1 that conform
                                processing can be integrated with the representational            to a specified ontology. We have investigated various
                                and factual reasoning abilities of knowledge graphs, en-          methods to enable the underlying language model to
                                hanced by the structure, rules, and inference mechanisms          comprehend a pre-defined ontology, ensuring effective
                                offered by an ontology. For targeted personal AI applica-         symbolic knowledge capture. By utilizing a specially
                                tions, this ontology approach presents several benefits:          designed dataset, we evaluate the effectiveness of these
                                                                                                  methods, emphasizing their strengths and identifying
                                      • Ontology schemas enable language models to de-
                                                                                                  potential areas for improvement.
                                                                                                     The structure of this paper is as follows: Section 2
                                KiL’24: Workshop on Knowledge-infused Learning co-located with
                                30th ACM KDD Conference, August 26, 2024, Barcelona, Spain        discusses  in-context learning and fine-tuning approaches
                                *
                                  Corresponding author.                                           for ontology-driven symbolic knowledge capture and
                                $ tolga@haltia.ai (T. Çöplü); arto@haltia.ai (A. Bendiken);       focuses on the details of the fine-tuning approach. Sec-
                                andriy@haltia.ai (A. Skomorokhov); eduard@haltia.ai (E. Bateiko); tion 3 describes the experimental setup by presenting the
                                steve@haltia.ai (S. Cobb)
                                                                                                  development framework, the language model selection,
                                 0009-0004-9414-0588 (T. Çöplü); 0009-0002-0725-4874
                                (A. Bendiken); 0000-0002-5696-6723 (A. Skomorokhov);              and the ontology and dataset creation process. Section 4
                                0000-0002-6729-5611 (E. Bateiko); 0009-0004-0476-6000 (S. Cobb)
                                          © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License   1
                                          Attribution 4.0 International (CC BY 4.0).                                                         https://www.w3.org/TR/rdf12-concepts/




CEUR
                  ceur-ws.org
Workshop      ISSN 1613-0073
Proceedings
outlines our performance evaluation framework and the            The following points highlight the key aspects of on-
test results. Finally, Section 5 concludes the paper and      tology fine-tuning:
suggests future directions.
                                                                   • The training dataset’s coverage and diversity are
                                                                     vital for successful fine-tuning. These character-
2. Ontology-Driven Symbolic                                          istics greatly influence the LLM’s ability to ef-
                                                                     fectively capture knowledge. Details about the
   Knowledge Capture                                                 dataset and how it is constructed are discussed in
In the literature, language models have demonstrated                 Section 3.4.
their capability to transform unstructured text into a             • The training dataset must include a variety of
knowledge graph [5, 6, 7, 8, 9]. However, the process                examples for each element of the predefined on-
of populating a knowledge graph from user prompts                    tology. This approach avoids scalability issues
in alignment with a pre-defined ontology has been ex-                typically associated with in-context learning and
plored only marginally [10, 11, 12, 13, 14]. Research                ensures comprehensive learning coverage.
typically centers on in-context learning, which heavily            • If the LLM encounters a user prompt that is not
relies on prompt engineering. A significant limitation               relevant to the predefined ontology concepts, it
of this approach is the requirement to incorporate the               should not attempt to capture knowledge. There-
entire custom ontology into the prompt. This neces-                  fore, the dataset should also contain sufficient
sity not only slows down the knowledge capture pro-                  out-of-context samples to enable the LLM to dis-
cess, because of the high token overhead but also re-                tinguish between relevant and irrelevant infor-
stricts the use of larger ontologies due to the constraint           mation for capture.
on context-window length. Given these constraints, in-
context learning methods do not provide a scalable solu-
tion for ontology-driven symbolic knowledge capture.          3. Experimental Setup
   An alternative approach involves training a language
                                                              This section explores the components of our experimen-
model with a pre-defined ontology, so that the model
                                                              tal setup.
internalizes it. There are two strategies to consider: pre-
training the LLM on the ontology or fine-tuning it. This
paper does not explore pre-training due to its extensive      3.1. Development Framework
data, computational resources, energy, and time require-      The methods suggested in this paper have been imple-
ments. Additionally, pre-training does not offer a flexible   mented using the Apple MLX framework [15]. MLX is a
response to ongoing changes or expansions in the ontol-       specialized array framework designed for machine learn-
ogy. Therefore, this paper will focus on fine-tuning as a     ing applications, akin to NumPy, PyTorch, or JAX, with
method to train language models on personal ontologies,       the distinction of being exclusive to Apple silicon.
highlighting advantages in feasibility and maintainabil-         Ontology fine-tuning has been conducted using the
ity.                                                          parameter-efficient QLoRA approach [16] on our custom
                                                              dataset, comprising randomly selected, non-overlapping
2.1. Ontology-Driven Knowledge Capture                        sets of training, validation, and test samples.
     with Fine-Tuning
Fine-tuning is a process whereby a pre-trained language 3.2. Language Model
model is further trained on a specific dataset to tailor The methods we have developed here do not have a struc-
its capabilities to a particular task. In our study, the tural dependency on a particular underlying foundation
language model is expected to learn the classes, object model. The key factors guiding our language model se-
properties, and data properties defined in an ontology, lection were its proven effectiveness across diverse do-
and to use them to populate a knowledge graph from user mains in community benchmarks and its prevalence in
prompts. The first step involves preparing a fine-tuning the field. Owing to its performance in the Hugging Face
dataset, which includes user prompts, system prompts, Open LLM Leaderboard [17] and its robust ecosystem,
and expected model responses for each concept in the the Mistral-7B-Instruct-v0.2 [18], based on the Llama 2
ontology. This dataset is used to fine-tune the language [19] architecture, was selected for our research. We ran
model, which is then evaluated by testing it with new all examples, tests, and benchmarks on the MLX 4-bit
prompts to assess the effectiveness of the knowledge quantized version of this model to be able to run the tests
capture operation. We define a system prompt for this on personal laptops.
task with the requirement of maintaining the model’s
generality across other tasks.
3.3. Predefined Ontology                                   we included 32 generic user prompts in the dataset. The
                                                           composition of this dataset, which consists of 175 user
Our study is inspired by KNOW[20]–the Knowledge Nav-
                                                           prompts, is illustrated in Figure 2. Concepts not associ-
igator Ontology for the World–and utilizes it for repre-
                                                           ated with the ontology are labeled as the ’none’ legend
senting personal information. KNOW is introduced as
                                                           in the figure. As each sample prompt typically contains
a pioneering framework designed to capture everyday
                                                           multiple modeled concepts, the chart shows a total num-
knowledge to enhance language models in real-world
                                                           ber of concept occurrences greater than the number of
generative AI applications such as personal AI assistants.
                                                           prompts.
The ontology focuses on human life, encompassing ev-
eryday concerns and significant milestones, and limits its
initial scope to established human universals, including                     Occurrences of Ontology Concepts

spacetime (places, events) and social dimensions (people,    Person
                                                               Name
groups, organizations). This pragmatic approach empha-           Sex

sizes universality and utility, contrasting with previous
                                                                Child
                                                              Father

works like Schema.org[21] and Cyc[22] by building on          Mother
                                                              Sibling

language models’ inherent encoding of salient common-          Sister
                                                             Brother
sense knowledge.                                             Spouse

   Due to the requirement that each element in the on-
                                                             Partner
                                                              Knows

tology be associated with a diverse set of prompt and re-      None


sponse samples within the training dataset, our research
                                                                      0            100                       200   300


focuses on a specific subset of the KNOW ontology. This                                Number of Occurrences


subset concentrates on core family relationships with
                                                           Figure 2: Occurrences of ontology concepts in the prepared
four ontology classes, eleven object properties, and one dataset.
data property. A visual depiction of this subset is pre-
sented in Figure 1.
                                                              The Turtle format was chosen for serializing the ontol-
                                                           ogy population in our research because of its straightfor-
                        spouse         child        parent
                                                           ward structure, readability, and prevalent use in existing
                                                                 partner     knows     sister     mother       sibling     brother      father
                      {<partner{}}   {<knows{}}
                                                           pre-training datasets for LLMs.
                                                   {<knows{}}   {<knows{}}           {<sibling{}} {<parent{}} {<knows{}} {<sibling{}} {<parent{}}



                                                                        Person
             name:string

                           sex

                                                  Female
                      Sex
             =Female{} or Male{}
                                                  Male
                                                                                                                                                    4. Performance Evaluation
                  mother
             =sex{} value Female{}
                                                                                                                                                    Our research focuses on fine-tuning a language model
 Thing

                   father
                                                                                                                                                    with predefined ontology concepts and capturing knowl-
             =sex{} value Male{}
                                                                                                                                                    edge from user prompts that fits the ontology. This sec-
                    sister                                                                                                                          tion will detail the performance evaluations associated
                                                                                                                                                    with these efforts.
             =sex{} value Female{}




                  brother
             =sex{} value Male{}


                                                                                                                                                                    Occurrences of Ontology Concepts
Figure 1: A visual representation of the ontology design used                                                                                         Person
in this paper.                                                                                                                                         Name
                                                                                                                                                         Sex
                                                                                                                                                       Child
                                                                                                                                                      Father
                                                                                                                                                      Mother
                                                                                                                                                      Sibling
                                                                                                                                                       Sister
3.4. Dataset                                                                                                                                          Brother
                                                                                                                                                     Spouse

For a language model to effectively learn a predefined                Partner
                                                                      Knows
ontology and use it to perform knowledge extraction                     None


and capture, a robust and diverse training dataset is es-                     0             20                       40  60


sential. Our paper focuses on a subset of the KNOW                                             Number of Occurrences

ontology that includes the concepts of ‘person’, ‘name’,
‘sex’, ‘child’, ‘father’, ‘mother’, ‘sibling’, ‘sister’, ‘brother’, Figure 3: Occurrences of ontology concepts in the test
                                                                    dataset.
‘spouse’, ‘partner’ and ‘knows’. We created 143 manually
crafted user prompts along with their respective ontol-
ogy responses for training and tests. Additionally, to                 Initially, we investigated how many training samples
manage inputs that fall outside these ontology concepts, each ontology concept required to effectively teach the
core family ontology to the selected Mistral-7B-Instruct-                                        In the subsequent phase, we explored the optimal num-
v0.2 model. From the dataset described in Section3.4, 41                                      ber of training epochs required to achieve maximum
random samples were selected and reserved for method                                          performance for the training set. For this analysis, we
evaluation. The distribution of ontology concepts within                                      continued using the default MLX QLoRA hyperparame-
this test set is shown in Figure 3.                                                           ters with the 8 samples per ontology concept, but trained
   From the remaining 134 samples, we created three                                           the QLoRA adapter over various epoch lengths. We then
different training datasets. In these three datasets, we                                      conducted evaluations on the test set using each trained
have ensured the inclusion of 2, 4, and 8 sample prompts                                      adapter, and the findings are presented in Figure 5.
for the ontology concepts ‘child’, ‘father’, ‘mother’, ‘sib-
ling’, ‘sister’, ‘brother’, ‘spouse’, ‘partner’, and ‘knows’.                                                             Performance vs. Epochs of Training
Subsequent evaluations using the test set measured the                                                                       Precision       Recall      F1-Score

precision, recall, and f1-scores for each fine-tuning ses-                                     1.00

sion. During these evaluations, the generated prompt
responses were processed triple by triple and compared                                         0.75

against the ground truth established for the test set. The
findings are displayed in Figure 4.                                                            0.50




                                                                                               0.25
                 Performance vs. Samples per Ontology Concept
                            Precision      Recall     F1-Score

 1.00                                                                                          0.00
                                                                                                      9 training epochs      18 training epochs   27 training epochs   36 training epochs



 0.75
                                                                                              Figure 5: Ontology population performance (precision, recall,
                                                                                              and f1-score) of 8 samples per ontology concept for various
 0.50                                                                                         epochs.

 0.25
                                                                                                As depicted in Figure 5, the success rate of the ontol-
 0.00
                                                                                              ogy population increases with longer training. However,
        2 samples/concept   4 samples/concept   8 samples/concept   entire training dataset
                                                                                              considering the resource usage and energy consumption,
                                                                                              we observe that 18 epochs are sufficient for fine-tuning.
Figure 4: Ontology population performance (precision, recall,
and f1-score) for various fine-tuning dataset sizes.
                                                                                              5. Conclusion
   The tests were conducted using the default QLoRA
                                                             In this paper, we first explored prompt-driven, ontology-
hyperparameters specified in the MLX framework. To
                                                             based symbolic knowledge capture and its importance
ensure consistent test results, each training session was
                                                             in the generative AI domain. We then discussed the on-
set to run for 18 epochs.
                                                             tology approach and how to teach ontology concepts
     • Layer keys to apply:             ["self_attn.q_proj", to the language model through in-context learning and
        "self_attn.v_proj"]                                  training. The language model was fine-tuned using a cus-
     • Rank: 8                                               tom dataset focused on core family relationships, and we
     • Alpha: 16                                             evaluated the model’s ability to learn personal ontology
     • Scale: 10                                             concepts.
     • Optimizer: Adam                                          Our findings indicate that fine-tuning is particularly
                                                             effective for teaching ontology concepts to a language
     • Learning rate: 1𝑥10    −5
                                                             model for prompt-time knowledge capture. In our fu-
     • Number of layers to fine-tune: 16
                                                             ture work, we aim to integrate the generated knowledge
     • Minibatch size: 4                                     graph with the language model for knowledge utilization,
  As illustrated in Figure 4, our study, which encom- combining the strengths of the neural and symbolic AI
passes twelve key ontology concepts, demonstrates that approaches.
providing eight diverse examples for each concept yields        Please refer to the paper’s corresponding GitHub repos-
acceptable success rates. Although our training and test itory at https://github.com/HaltiaAI/paper-PTODSKC
datasets are not sufficiently diverse or large enough to
generalize the results to real user scenarios, the high suc-
cess achieved with a small number of training samples is
promising for the feasibility of the proposed approach.
References                                                           org/abs/2307.16699, arXiv:2307.16699 [cs].
                                                                [12] K. Lu, X. Pan, K. Song, H. Zhang, D. Yu, J. Chen,
 [1] A. Sheth, K. Roy, M. Gaur, Neurosymbolic AI – Why,              PIVOINE: Instruction Tuning for Open-world Infor-
     What, and How, 2023. URL: http://arxiv.org/abs/                 mation Extraction, 2023. URL: http://arxiv.org/abs/
     2305.00813, arXiv:2305.00813 [cs].                              2305.14898. doi:10.48550/arXiv.2305.14898,
 [2] L. N. DeLong, R. F. Mir, J. D. Fleuriot, Neurosym-              arXiv:2305.14898 [cs].
     bolic AI for Reasoning over Knowledge Graphs: A            [13] N. Mihindukulasooriya, S. Tiwari, C. F. En-
     Survey, 2024. URL: http://arxiv.org/abs/2302.07200,             guix, K. Lata, Text2KGBench: A Benchmark
     arXiv:2302.07200 [cs, stat].                                    for Ontology-Driven Knowledge Graph Genera-
 [3] F. J. Ekaputra, M. Llugiqi, M. Sabou, A. Ekelhart,              tion from Text, 2023. URL: http://arxiv.org/abs/
     H. Paulheim, A. Breit, A. Revenko, L. Waltersdorfer,            2308.02357. doi:10.48550/arXiv.2308.02357,
     K. E. Farfar, S. Auer, Describing and Organizing                arXiv:2308.02357 [cs].
     Semantic Web and Machine Learning Systems in               [14] M. Funk, S. Hosemann, J. C. Jung, C. Lutz, Towards
     the SWeMLS-KG, 2023. URL: http://arxiv.org/abs/                 Ontology Construction with Language Models,
     2303.15113, arXiv:2303.15113 [cs].                              2023.      URL:     http://arxiv.org/abs/2309.09898.
 [4] L.-P. Meyer, C. Stadler, J. Frey, N. Radtke, K. Jung-           doi:10.48550/arXiv.2309.09898,
     hanns, R. Meissner, G. Dziwis, K. Bulert, M. Martin,            arXiv:2309.09898 [cs].
     LLM-assisted Knowledge Graph Engineering: Ex-              [15] A. Hannun, J. Digani, A. Katharopoulos, R. Col-
     periments with ChatGPT, 2023. URL: http://arxiv.                lobert, MLX: Efficient and flexible machine learn-
     org/abs/2307.06917, arXiv:2307.06917 [cs].                      ing on Apple silicon, https://github.com/ml-explore,
 [5] Q. Guo, Z. Jin, X. Qiu, W. Zhang, D. Wipf,                      2023.
     Z. Zhang, CycleGT: Unsupervised Graph-to-                  [16] T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettle-
     Text and Text-to-Graph Generation via Cy-                       moyer, QLoRA: Efficient Finetuning of Quantized
     cle Training, 2020. URL: http://arxiv.org/abs/                  LLMs, 2023. URL: http://arxiv.org/abs/2305.14314.
     2006.04702. doi:10.48550/arXiv.2006.04702,                      doi:10.48550/arXiv.2305.14314,
     arXiv:2006.04702 [cs].                                          arXiv:2305.14314 [cs].
 [6] X. Li, F. Yin, Z. Sun, X. Li, A. Yuan, D. Chai, M. Zhou,   [17] E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lam-
     J. Li, Entity-Relation Extraction as Multi-Turn Ques-           bert, N. Rajani, O. Sanseviero, L. Tunstall, T. Wolf,
     tion Answering, 2019. URL: http://arxiv.org/abs/                Open LLM Leaderboard, https://huggingface.co/
     1905.05529. doi:10.48550/arXiv.1905.05529,                      spaces/HuggingFaceH4/open_llm_leaderboard,
     arXiv:1905.05529 [cs].                                          2023.
 [7] Y. Xu, L. Fu, Z. Lin, J. Qi, X. Wang, INFINITY: A          [18] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bam-
     Simple Yet Effective Unsupervised Framework for                 ford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
     Graph-Text Mutual Conversion, 2022. URL: http:                  G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-
     //arxiv.org/abs/2209.10754. doi:10.48550/arXiv.                 A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang,
     2209.10754, arXiv:2209.10754 [cs].                              T. Lacroix, W. E. Sayed, Mistral 7B, 2023. URL: http:
 [8] R. Anantharangachar, S. Ramani, S. Rajagopalan,                 //arxiv.org/abs/2310.06825. doi:10.48550/arXiv.
     Ontology Guided Information Extraction from Un-                 2310.06825, arXiv:2310.06825 [cs].
     structured Text, International journal of Web &            [19] H. Touvron, L. Martin, K. Stone, P. Albert, A. Alma-
     Semantic Technology 4 (2013) 19–36. URL: http:                  hairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhar-
     //arxiv.org/abs/1302.1335. doi:10.5121/ijwest.                  gava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer,
     2013.4102, arXiv:1302.1335 [cs].                                M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu,
 [9] T. Çöplü, A. Bendiken, A. Skomorokhov, E. Bateiko,              W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal,
     S. Cobb, J. J. Bouw, Prompt-Time Symbolic                       A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kar-
     Knowledge Capture with Large Language Models,                   das, V. Kerkez, M. Khabsa, I. Kloumann, A. Ko-
     2024.       URL:     http://arxiv.org/abs/2402.00414.           renev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee,
     doi:10.48550/arXiv.2402.00414,                                  D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov,
     arXiv:2402.00414 [cs].                                          P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizen-
[10] H. B. Giglou, J. D’Souza, S. Auer, LLMs4OL:                     stein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.
     Large Language Models for Ontology Learning,                    Smith, R. Subramanian, X. E. Tan, B. Tang, R. Tay-
     2023.       URL:     http://arxiv.org/abs/2307.16648.           lor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov,
     doi:10.48550/arXiv.2307.16648,                                  Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Ro-
     arXiv:2307.16648 [cs, math].                                    driguez, R. Stojnic, S. Edunov, T. Scialom, Llama 2:
[11] P. Mateiu, A. Groza, Ontology engineering with                  Open foundation and fine-tuned chat models, 2023.
     Large Language Models, 2023. URL: http://arxiv.                 arXiv:2307.09288.
[20] A. Bendiken, KNOW: A Real-World Ontology for
     Knowledge Capture with Large Language Mod-
     els, 2024. URL: https://arxiv.org/abs/2405.19877.
     arXiv:2405.19877.
[21] R. V. Guha, D. Brickley, S. Macbeth, Schema.org:
     evolution of structured data on the web, Commu-
     nications of the ACM 59 (2016) 44–51. URL: https://
     doi.org/10.1145/2844544. doi:10.1145/2844544.
[22] D. B. Lenat, R. V. Guha, Building Large Knowledge-
     Based Systems; Representation and Inference in
     the Cyc Project, 1st ed., Addison-Wesley Longman
     Publishing Co., Inc., USA, 1989.

</pre>