The most of the tasks of event extraction from textual data are solved using neural networks. One of the ways to train neural networks on texts is the use of n-grams. In this paper, we consider n-grams in the form of subgraphs of conceptual graphs that have a certain semantics. Each such n-gram can be treated as an event. The paper proposes to use an additional information resource in the form of a hierarchy of multimodal clusters based on a multidimensional formal context, the n+1 dimensional tensor elements of which consist of n-grams and elements selected as objects. The task of event extraction is solved by the joint use of a neural network and a hierarchy of multimodal clusters.
1. Introduction information resource in the form of a hierarchy of multimodal clusters based on a multidimensional formal context, the n+ 1-tensor elements of which are n-grams and the elements selected as objects. The task of event extraction is solved by the joint use of a neural network and a hierarchy of multimodal clusters. The neural network solves the classification problem and returns the class objects corresponding to the query text upon request. Then these objects are searched in multimodal clusters. These clusters contain combinations of events related to the classes of objects found by the network.
Using conceptual n-grams for learning allows one to apply neural network with a simple architecture. The use of an additional resource in the form of a hierarchy of clusters built on n-grams presents additional information to obtain a solution of event extraction task.
The paper is organized as follows. In the Section 2, there are brief descriptions of event extraction task and multimodal clustering problem in FCA with mentioning the main related works in these areas. In the Section 3, the proposed approach is outlined together with its functionality. The Section 4 contains conclusion. The paper ends with acknowledgements and references sections.
This work is related to two principles. The first principle of query refining has long been known in
FCA community [
The second principle of concept-based explanations [
2.1.
The main definitions in the event extraction are the following [
There are two directions in event extraction. They are Closed-domain event extraction and Opendomain event extraction. Closed-domain event extraction uses predefined event schema to discover and extract desired events of particular type from text. An event schema contains several event types and their corresponding event structures. Open-domain event extraction aims at detecting events from texts and in most cases, also clustering similar events via extracted event keywords. Event keywords refer to those words/phrases mostly describing an event, and sometimes keywords are further divided into triggers and arguments.
There are a large number of works devoted to event extraction from the text. The review [
In the work of [
The work of [
Based on the mentioned works and other works in this research area, the following conclusions can be drawn.
1. Graph models of event representation are used in the works. 2. Neural networks used in event extraction have a complex architecture. 3. To extract events, external resources are needed in the form of text corpora, thesauri, and databases. 2.2.
In FCA, multimodal clustering is formulated as follows.
If R D D2 ... Dn is a relation on data domains D1, D2, ..., Dn then formal context is an n +1 set: 1
=<K1, K2 , ..., Kn , R >
where Ki ⊆ Di . Multimodal clusters on the context (1) are n – sets
which have the closure property [
X1, X 2 , ..., X n u (x1, x2 ,..., xn ) X1 X 2, ..., X n , u R (1) (2) (3) and
j 1, 2,..., n, x j D j \ X j X1,..., X j {x j}, ..., X n does not satisfy (3).
A multimodal cluster is a subset in the form of combinations of elements from different sets Ki. It is also defined as a closed n-set [9] since the closure property (3) provides its “self-sufficiency”: it cannot be enlarged without violating (2).
Formal concepts on multimodal formal context are those multimodal clusters where for all u (x1, x2 ,..., xk ) X1, X 2 , ..., X k , u R and k is maximally possible. In other words, they are the largest possible k-dimensional hypercubes completely filled with units. The concept of the density of a multimodal cluster is introduced in FCA and formal concepts are interpreted as absolutely dense clusters [9].
The considered approach is implemented by performing the following steps.
1. Conceptual n-grams are built on the text under processing. We consider every such n-gram as a graph in the form of a tree. For a given sentence, it may be dependency parse tree, AMR-graph, conceptual graph or any other n-gram model. Depending on the type of n-grams, their construction can be performed by an appropriate software tool. All the currently most well known variants of n-grams are supported by parser programs. It is necessary to ensure the storage of n-grams, for example, in a database. Some text corpora store n-grams as tagging.
2. A multidimensional formal context is formed as an n+1-dimensional tensor. Every point of this tensor contains an object and a n-gram it corresponding. Objects can be documents containing texts, terms, entity names, text topics, etc. Multimodal clusters are built on this formal context.
3. The task of event extraction is solved by the joint use of a neural network and a hierarchy of multimodal clusters. The neural network solves the classification problem and, upon request, returns objects-classes corresponding to the query text. Then the clusters corresponding to these objects are determined. They contain combinations of events related to the query text.
3.1.
Consider implementation of the outlined methodology and its experimental study.
Data sets. We explored our methodology on two datasets. The first dataset is The Stanford Question Answering Dataset (SQuAD) [10] . In this dataset there are topics, corresponding texts, questions and answers. This dataset is often used to train neural networks applied in question-answering systems. Extracting events in SQuAD corresponding to the semantic template “who, what, with whom and where and when” will allow one to form answers to questions related to the template elements.
The second dataset (KaggleTDC) is a collection of approximately 1000 newsgroup documents from 10 different newsgroups [11]. This dataset differs from SQuAD in that its newsgroups more overlap by words. Therefore, we expected greater uncertainty in determining the newsgroup from the text by the neural network.
Conceptual n-grams. The sequence of n tokens in the text is called an n-gram. The token can be a phoneme, a syllable, a letter, a word or base pairs according to the application. We construct n-grams by acquiring conceptual graphs [12] from text and selecting their subgraphs having three, four or five concepts connected by agent, patient or attribute relations. Accordingly, we have trigrams, 4-grams and 5-grams.
Figure 1 shows an example of conceptual graphs corresponding to the sentence "Dioxins are highly toxic, and health effects on humans include reproductive, developmental, immune and hormonal problems" related to the topic of the Immune system.
In Figure 1, two disconnected conceptual graphs correspond to the sentence above. By selecting the agent and patient relationships in graphs, we have the trigrams “effect – include - problem” and “dioxin – be - toxic". When adding attribute relations, we get five 5-grams based on the first trigram with the concepts of health, hormonal, immune, developmental, reproductive. For example, it will be a 5-gram “health – effect - include – problem – hormonal”, which corresponds to the event template given above. All these conceptual n-grams may be considered as events.
Conceptual graphs used in our work constitute a special case of AMR graphs. Conceptual graphs represent the semantics of individual sentences by acquiring from text known semantic roles as agent, patient, attribute, etc. We do not use external information resources and train neural network on ngrams extracted from conceptual graphs.
Neural Network. We use a compact chain recurrent neural network of standard architecture. An example of a structure of such network for 5 classes is shown in Figure 2.
The network is trained on n-grams generated from the texts of the dataset. When training the network, several optimization methods were used: ADAM, RMSProp, SGD, Signed.
The input data in the network are phrases in natural language, usually in the form of a question. Conceptual n-grams are constructed corresponding to input phrases. If no n-grams correspond to a phrase, the first n words of the phrase come to the input layer. If several n-grams correspond to a phrase, then they consistently arrive at the input layer. The output of the network is the object-class corresponding to the text query. Training and network operation using conceptual n-grams were compared with the training method in which n-grams formed by using a sliding context window (SCW). The results are presented in the tables 1, 2 and on Figure 3.
The tables 1 and 2 contain some results of training and network operations for conceptual n-grams and for SCW n-grams constructed on the SQuAD dataset and on the KaggleTDC dataset.
The table 3 contains information about the five most voluminous topics-classes from the SQuAD data set.
In general, from the most general positions, it can be argued that the more words are used when training a neural network, the better it works. The latter means, among other things, that the network responds to requests more meaningfully. Here we do not take into account the effect of overfitting.
Among others, there are two ways to train neural networks on textual data. The first one uses a bag of words – all the words of the text without stop words. The second uses n-grams obtained by applying a sliding context window. Both methods use a lot of words, but they clearly do not reflect the semantics of the text, since they are simple sequences of words. Figure 4 shows how the volumes of n-grams and sentences of the text are correlated.
Conceptual n-grams that we form consist of a small number of words and their number is more than 10 times less than the number of n-grams formed by using SCW as can be seen in Figure 4. Based on this, it can be assumed that the use of conceptual n-grams will reduce the performance of the neural network. In reality, this does not happen, as can be seen from Table 2, Table 3 and Figure 3. In our opinion, this is due precisely to the presence of semantics in conceptual n-grams.
As the value of n in conceptual n-grams increases, the quality of the network increases too, as can be seen from Table 4.
Joint use of neural net and multimodal clusters. The neural network is trained on n-grams, which are interpreted as events. The texts at the network input are also converted into n-grams. As a result, the network works only with n-grams. When solving a classification task, in response to an input query, the network returns a set of class names for the SQuAD dataset and a set of newsgroup names for the KaggleTDC dataset. It was found in experiments that in these sets there are often objects-classes with similar probabilities, which makes it difficult to choose between them. For these purposes, multimodal clusters are used. There are such clusters-formal concepts, where subsets of objects include objectsclasses with similar probabilities. In such clusters, all n-grams are combined with all objects. As a result, we get a set of events corresponding to the request to the neural network.
Multimodal clustering is performed using an evolutionary algorithm that allows obtaining Paretooptimal solutions for several criteria of cluster optimality. The framework [13] is used for multimodal clustering. The problem of multimodal clustering is formulated as a problem of multiobjective optimization. In the experiments, two clustering optimality criteria were used, the cluster volume and its density. These criteria contradict each other, so compromise Pareto-optimal solutions are needed. First of all, we were interested in absolutely dense clusters being formal concepts. In them, all objects are combined with all elements of n-grams. Large and dense clusters are interesting because combinations of elements of its subsets set a property that manifests itself on a large number of elements and, possibly, means a regularity. However, often the clustered data is sparse and the existence of large and dense clusters on them is unlikely. Therefore, when selecting clusters, a trade-off between density and volume is provided by the algorithm.
The quality characteristics that are usually used in the classification problem (accuracy and F1score) are integral and do not reflect the quality of individual solutions, their semantics. In the task of event extraction, it is important to detect events as facts and present the composition of events. In our experiments, we still detect events as facts. In multimodal clusters, we count the number of n-grams and fix their composition. Table 5 shows the content of several multimodal clusters containing pairs of class objects. For every pair there are corresponding number of n-grams shown in the table.
In the Table 5, University of Chicago and Warsaw classes are linked by eight n-grams having mutual words in corresponding texts. Immune_system and European_Union_law classes are practically unlinked since their mutual words in two n-grams are commonly used. It is evident that European_Union_law and Scottish Parliament classes are closely linked by 28 n-grams.
As a result, in clusters we find events that have common class objects and class objects that have common events.
Obviously, in the task of event extraction, only n-grams are not enough for the end user and additional information is needed. The n-grams found in multimodal clusters correspond to text fragments that can be presented to the user in response to a request. In this case, events take the form of descriptions understandable to the user. Such a solution may be designed in the form of a user interface focused on a specific subject area.
In this paper, an approach to training neural networks on n-grams in the form of conceptual graphs is proposed, which allows using a recurrent network with a simple architecture. Another contribution is the use of multimodal clusters as an additional information resource in the task of event extraction. The results of the application of these solutions are the following.
1. The neural network learns on n-grams no worse than in words, but it works faster. 2. The use of an additional resource in the form of a multidimensional formal context and multimodal clusters makes it possible to improve the interpretability of the results of the neural network.
In the future, an analytical explanation of the experimental results obtained in this work is necessary.
The reported study was funded by Russian Foundation of Basic Research, the research project № 19-07-01178 and RFBR and Tula Region research project № 19-47-710007.
[9] Ignatov D. I., Gnatyshak D. V., Kuznetsov, S. O., Mirkin, B., G.: Triadic Formal Concept
Analysis and triclustering: searching for optimal patterns. Mach. Learn. 101:271–302. (2015). [10] The Stanford Question Answering Dataset. URL: https://rajpurkar.github.io/SQuAD-explorer/ [11] The Kaggle Dataset Text Document Classification. URL: https://www.kaggle.com/datasets/jensenbaxter/10dataset-text-document-classification. [12] Sowa J.: Conceptual Graphs: Draft Proposed American National Standard, International
Conference on Conceptual Structures ICCS-99, Springer (1999). [13] Mikhail Bogatyrev, Dmitry Orlov. Framework for Pareto-Optimal Multimodal Clustering. In Sergei O. Kuznetsov, Amedeo Napoli, Sebastian Rudolph (Eds.): The 10th International Workshop "What can FCA do for Artificial Intelligence?" FCA4AI 2022, co-located with IJCAI-ECAI 2022, July 23 2022, Vienna, Austria. CEUR Workshop Proceedings. Vol. 3233. Pp. 51–62.