English. In this paper we present a system called Text Frame Detector (TFD) which aims at populating a frame-based ontology in a graph-based structure. Our system organizes textual information into frames, according to a predefined set of semantically informed patterns linking pre-coded information such as named entities, simple and complex terms. Given the semiautomatic expansion of such information with word embeddings, the system can be easily adapted to new domains.
Textual data are still the most widespread content
around the Web
Relation extraction techniques have been
widely applied to populate semantic frames
Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
with the consequent high cost of long annotation
time. On the other hand, unsupervised approaches
do not need any training data, but mapping
extraction results onto predefined relations or
ontologies is often quite challenging with this
kind of methods
Moreover, semi-supervised methods exploit
bootstrap learning, so that any new relation
requires a small set of labelled data to be extracted
Finally, another kind of approach has been
proposed, which relies on knowledge bases (KBs) to
produce training data. Introduced by Mintz et al.
(2009), distant supervision detects relations on
semantically annotated texts where entities which
co-occur in the same sentence match with
entitypairs contained in the KB. Then a classifier is
trained using features extracted from the annotated
relations
In this paper, we present a system to populate a
frame-based ontology, whose values are stored in a
graph-based structure. Our method exploits some
aspects of distant supervision, leveraging on
domain specific KB to infer the relations, and
populates the frames with specific information (i.e., the
participants) as well as the portions of text (i.e.,
the snippets) which contain them. Thus, the
output of the system for a single frame is a set of
snippets, one for each of its slots. Each snippet is
also associated with a weight encoding how likely
it is expected to contain the information about
a certain relation. Such a weight is calculated
with a scoring function based on similarity
measures and textual distance information. The
system has been tested on the administrative domain,
with the goal of gathering information related to
taxes and agenda events. Indeed, since the KB can
be semi-automatically enriched with Named
Entities (NEs) and vocabularies of simple and
complex terms, our approach can be easily adapted
to different domains. Furthermore, system recall
can be increased by expanding the frame and
attribute vocabulary by exploiting word embeddings
Our approach differs from existing systems like
PIKES
This paper is structured as follows: in section 2 we describe the general methodology of the system, we define terminology and notation and we describe the main features of the proposed approach. The system implementation is illustrated in section 3, which shows the extraction algorithm as well as the indexing methods in the knowledge graph. Evaluation and results are reported in section 4. 2
Following Riedel et al. (2010), we assume that “if two entities he1; e2i participate in a relation hri, then there is at least one sentence hsi in the text expressing such relation”. We adopt this hypothesis for both simple and complex relations (cf. infra), by considering the sentence hsi itself and the [hs ki; : : : ; hs + ki] adjacent ones, where k is a system parameter.
In order to identify sentences where one or more relations are expressed, we developed a system called Text Frame Detector (TFD).
Given a KB where domain terms are
associated to a given set of frames, TFD populates
them, by making explicit the semantic relation
between terms and named entities (NEs). In
particular, TFD exploits linguistic analysis and IE
algorithms: texts are processed up to part of speech
tagging, then NEs
In order to increase the system recall of relevant information, we also used the semantic neighbors of the terms defining the frames. For example, if a text contains the word “versamento” (‘deposit’) but the KB only contains the word “pagamento” (‘payment’), the term “versamento” may be extracted because it is a semantic neighbor of the latter (see Table 1).
Neighbor rimborso (‘refund’) versamento (‘deposit’) versare (‘to deposit’)
Cosine Similarity
0.89
0.86
0.78
We trained fastText word embeddings
Currently, KB terms are expanded with their 10 nearest semantic neighbors in terms of cosine similarity, which can be filtered through a parametric threshold. Frame: Terms and entities contained in the KB are organized in frames. Frames allow to structure the implicit knowledge contained in texts around concepts that define the relevant semantic categories in a domain. For instance, the frame EVENT corresponds to entities like concerts, shows, etc. Each frame is defined by its frame triggers and attributes. Frame trigger: It corresponds to an instance of the semantic class described by the frame (e.g., in the administrative domain, the frame TAX is expressed by its instances: “TARI” (‘Garbage tax’), “IMU” (‘Municipal tax’)). Frame triggers suggest the presence of frame attributes in the text.
Attribute: A frame is composed by a set of slots,
which must be filled by specific instances or
data
Simple attribute: Their values correspond to simple and complex terms, NEs or Temporal Expressions (TEs) identified during the IE step. The EVENT frame attributes are considered simple because they usually appear right near the frame trigger (cfr. Figure 1).
Complex attribute: The values of these attributes do not correspond to a single entity, but are expressed by whole text segments. Concerning the TAX frame, the deadline attribute cannot be filled by simply extracting the due dates from the text, because the reported information would be incomplete if taken out of context (cfr. Figure 2). Therefore, it is necessary to return the entire text snippet, which includes the attribute triggers that allow to identify the complex attribute. Attribute trigger: They represent the linguistic cues of an attribute instance. They are manually selected by domain experts and stored in the KB with a standard form t and a small number of orthographic and morphosyntactic variants v. Attribute triggers can be: (i) single and multiword terms, like “bollettino postale” (‘postal order’), “saldo” (‘balance’), NEs, such as “Firenze” (‘Florence’) or TEs, like “18 giugno” (‘18th June’); (ii) complex patterns, such as “non inferiore a” (‘not lower than’). 3
In order to fill the frame slots, textual data are analyzed by TFD in various steps. After linguistic annotation, NER, and term extraction, TFD looks for frame triggers and for its attribute triggers, in the same sentence or in the sentences around it. More specifically, given a snippet , a frame instance F is expressed by a frame trigger Ft, and a set of attributes A, containing both simple (As) and complex (Ac) attributes, so that F = fFt; Ag where ai 2 As [ Ac. 3.1
Since both simple and complex attributes of a frame are expressed by means of the set T of their attribute triggers, we can say that F is instantiated in a text by the joint occurrence of a frame trigger Ft and a set of attribute triggers T related to one or more of its attributes, namely F = fFt; T g where T = ft1; :::; tng.
In order to retrieve a frame F in a portion of text, first of all we look for its frame triggers. Once a Ft has been detected, we search for its potential attributes. Given such F , its potential instances in the text consist of the co-occurrence of Ft and a subset of T . To guarantee a certain degree of flexibility, we decided to provide each of the elements in T with a binary feature that can be set to 1 if the attribute trigger ti is mandatory to extract the F , and to 0 if the attribute trigger is optional. A further implementation could consider to convert these features in continuous weights. In this way the TFD would be able to consider some triggers as more relevant than others to populate the frame.
Moreover, the attribute triggers of F belonging to T are selected within terms and entities used to express its attribute instances. Such triggers are then exploited by the attribute retrieval system of the TFD. Concerning the retrieval of simple attributes, see the extraction of the EVENT frame from the sentence in Figure 1.
The trigger for the EVENT frame (“spettacolo di Roger Waters”) in Figure 1 is a clue for the presence of its attributes which populate the frame instance showed in Table 2.
Moreover, the TFD stores the raw text in Figure 1 as the relevant snippet for both the attributes Lo [spettacolo di Roger Waters]nome_evento si terrà il [26 giugno]data allo [stadio di Firenze]luogo. when and where.
Il [versamento]pagamento dell’[IMU]tassa deve essere effettuato con [bonifico bancario]mod_pagamento o [bollettino postale]mod_pagamento in due [rate]somma: l’[acconto]somma entro il [18 giugno]data e il [saldo]somma entro il [17 dicembre]data.
Examples of complex attributes can be found in the TAX frame, namely deadline, indicating the due date of the tax payment, and methods of payment, indicating how it is possible to pay it. For example, the triggers detected for the attribute deadline in Figure 2 are “somma” (‘sum’), “pagamento” (‘payment’) and two TEs, namely “18 giugno” (‘June 18th’) and “17 dicembre” (‘December 17th’). The snippet contains also the attribute methods of payment, which is expressed by the triggers “pagamento” (‘payment’) and “mod_pagamento” (‘methods_payment’), expressed by “bonifico bancario” (‘wire transfer’) and “bollettino postale” (’postal order’). Table 3 shows the TAX frame instantiated with the extracted attributes. Also in this case, the full snippet (the raw text in Figure 2) is stored for both the attributes deadline and methods of payment. 3.2
The binary features associated to each attribute trigger in a frame instance lead also the snippet TAX deadline methods of payment
IMU 18 giugno, 17 dicembre bonifico bancario, bollettino postale selection and ranking system. Given a potential instance of a frame, its attribute triggers are associated with a binary feature indicating their compulsory presence in order associate the attribute with a certain snippet. On the basis of how many features are set to 1, the TFD will be more or less strict in the selection phase. For example, given the following sentences, where the frame triggers appear in bold and attribute triggers are underlined (the standard form for “pagata” is “pagamento” and “17 giugno” is marked as “data”), Table 4 shows which snippets are extracted according to the binary values associated to each attribute trigger.
A “L’IMU va pagata entro il 17 giugno” (‘The Municipality tax must be paid before June 17th’) B “La scadenza dell’IMU è fissata al 17 giugno” (‘The deadline for the Municipality tax payment is on June 17th’) Line ID 1 2 3 4 5 6 7 8 pagamento (‘payment’) 0 0 0 0 1 1 1 1 scadenza (‘deadline’) 0 0 1 1 0 0 1 1
data (‘date’) 0 1 0 1 0 1 0 1 snippet extracted
A,B A,B B B A A
Each line of the table represents a potential combination of attribute triggers, with the respective mandatoriness. According to these features, the absence of mandatory attribute triggers (line 1) allows the retrieval of both the snippets A and B. Otherwise, if the system is expected to find all the attribute triggers (line 8), none of the two snippets is extracted because “pagamento” and “scadenza” never appear in the same sentence. This system is useful in order to balance the extraction flexibility based on the domain. For example, in administrative documents, where the language is bounded to stereotyped phrases (Brunato, 2015) a more strict approach is preferable, whereas in general domain ones it might be better to work with a higher number of optional triggers.
Moreover, a second objective of the TFD is to rank the extracted snippets according to their relevance with respect to a given attribute. Such relevance is calculated through a co-occurrence analysis, which employs measures based on semantic and distance features. One of these measures is the Sentence score, defined as:
SS = jtj jvj (1) where t is the number of attribute triggers (standard forms) and v is the total of their variants.
This formula takes into account the ratio between the number of attribute triggers and their variants. In particular, the TFD favours the snippets containing the highest number of distinct attribute triggers, namely their standard forms. In the case of simple attributes, t represents the number of entity types and v the number of NEs.
Furthermore, although different frame triggers may be found all over a given document, they may refer to the same domain entity, hence to the same frame instance. For example, we observed that Italian municipality web pages dedicate entire articles to a single tax, which can be mentioned in different ways, such as their full names and their acronyms (e.g., the Italian Tax “Imposta Municipale Propria” (‘Municipality tax’) can be mentioned also with the acronym, “IMU”). In order to avoid that attributes belonging to the same frame are associated to different ones and affect the scoring process, our system can be set to apply a “fuzzy normalization” strategy that is able to associate all the triggers of a document to a frame referring to the same entity. For example, the snippets extracted from a municipality web page and associated to the deadline attribute of the TAX frame can be ranked together, regardless the frame triggers they contain, such as “Imposta Municipale Propria” (‘Muncipality tax’) or its acronym, “IMU”.
At a document level, the snippet selected is simply the one with the highest Sentence Score, but we provide an additional level of analysis, which is applied when the snippet has to be chosen within a group of documents, instead of a single one. In that case, TFD selects the snippet with the highest Document score (DS), which encodes how likely the document contains a relevant information about a certain attribute. The Document score is calculated as follows:
DS = Pin=l1 T S (2) where l is the sentence length in terms of tokens, and T S is the Trigger score of a given variant v. T S is defined as:
T S = d1 cos (3) where d is the distance between the attribute trigger (or NEs) and the frame trigger, and cos is the cosine similarity between the trigger variant contained in the KB and the neighbor found in the text (the cosine is equal to 1 for the KB terms). 3.3
Extracted frame instances are stored in a Neo4j GDB. The Knowledge Graph (KG) contains several root nodes, one for each of the frames detected in the document or in the collection of documents (Figure 3).
For instance, there are two root-nodes corresponding to the EVENT and TAX frames. If we consider the frame TAX (the node “Frame” in Figure 3), the nodes “Frame Trigger” can be populated with instances like “Imposta Municipale Propria” (‘Muncipality tax’) or its acronym, “IMU”. Each frame trigger node is linked to the corresponding frame attributes (“Attribute” node in Figure 3) which can be populated with information like “scadenza” (‘deadline’) and “modalità di pagamento” (‘methods of payment’). Documentnodes (“Document” node in Figure 3), labelled by document names, are placed between attributenodes and attribute-trigger-nodes in order to facilitate the retrieval phase. Each document node is associated with the snippet having the highest Sentence score for the connected attributenode (e.g., ‘deadline’), along with its Document score. In the retrieval phase, unless the information is extracted from a single document, the snippet with the higher Document score is selected and returned (see Section 3.2). The other levels of the graph contain information extracted from each document. Every attribute-trigger-node (“Attribute Trigger” node in Figure 3) is labelled by the standard form of the attribute trigger extracted from the connected document-node (e.g., ‘sum’). Then, each attribute-trigger-node is connected to one or more nodes representing the trigger variants (“Attribute Variant” node in Figure 3). Continuing with this example, attribute variants can consist in ‘installments’, ‘balance’ and ‘down payment’. Finally, the last node of the graph consists of the snippet-node (“Doc. snippet” node in Figure 3), storing the snippet containing the information extracted. For example, the node can be populated with a snippet like the one reported in Figure 2: “Il versamento dell’IMU deve essere effettuato con bonifico bancario o bollettino postale in due rate: l’acconto entro il 18 giugno e il saldo entro il 17 dicembre” (‘The Municipality tax disbursement must be made through wire transfer or postal order in two installments: down payment by June 18th and balance by December 17th’). 4
The extraction of attributes related to TAX and EVENT frames were evaluated on Italian language texts by an administrative domain expert. We decided to evaluate these frames because the first one is very specific of the administrative domain, whereas the second one can be seen as a general purpose one. The gold standard includes both administrative documents as well as social media texts and news published on the municipalities websites. Both frames were evaluated on 50 texts, including information about taxes (municipality online guidelines), events (administrative acts, press releases, Facebook statuses and tweets) and other topics (municipality web pages). For municipality guidelines web pages, the “fuzzy normalization” strategy has been applied (see Section 3.2). The results of the TFD are shown in Table 5.
Since simple attribute values consist mostly of NEs, these results are strictly dependent on the generalization capability of the models used to extract those entities. In other cases, a wrong snippet is selected as relevant for an attribute, although triggers and NEs are correctly annotated and extracted. Moreover, additional errors depend on the absence of attribute triggers variants in the Knowledge Graph.
More specifically, errors are mainly related to a wrong NE annotation (35%). In the 22:8% of cases, a wrong sentence is selected as relevant for a certain attribute, although triggers and NEs are correctly annotated and extracted. False negative errors are caused by relevant information spread in several sentences (8:8%), whereas each extracted snippet consists of a single sentence, by unknown triggers describing an attribute (7:5%), by partial information contained in the extracted sentence (5%), by wrong lemmatization (1:75%) or by the overlapping of named entities and events (1:75%) (e.g., ‘Roger Waters’ show’ is not annotated as an event, however ‘Roger Waters’ is extracted as a named entity). In other cases (3:5%), attribute triggers are too distant from their frame trigger to be extracted. Although this span is customizable, an excessive distance between frame and attribute triggers could produce noise in the retrieval phase. Finally, the application of the “fuzzy normalization” strategy (see Section 3.2) led to errors in the ranking phase (14:3%). One of the municipality web pages in which the strategy has been applied contained information on more than one tax, but only one frame instance has been returned. This kind of errors can be limited by automatically checking the frame triggers cited on the text, and deciding whether applying or not the normalization according to external lexical resources, such as gazetteers or dictionaries. 5
In this paper we presented a domain independent system for slot filling that exploits a graph to populate a frame-based ontology. The Text Frame Detector extracts a relevant snippet for each frame attribute from textual information with good results in terms of F1 score (0:796). Nonetheless, the evaluation showed that there is room for improvement in some of the TFD modules. For example, the annotation of the semantic neighborhood of single and multiword terms, which are particularly relevant in technical domains, should led to further improve recall performances for complex attributes.
Moreover, although we did not adopted Fillmore’s semantic frames in the present work, we would like to explore the possibility of integrating our domain frames with FrameNet ones, which might contribute to enhance the system flexibility.
Finally, in the near future, we plan to finetune parameters and to implement additional features such as to associate multiple snippets to the same attribute. Furhermore, we intend to convert the binary features used in the snippet selection system into continuous weights. These weights, along with the collected data about frame population, would be also employed to train a supervised model for slot filling, in order to test TFD across new domains.
This research has been funded by the Project “SEM il Chattadino” (SEM), funded by Regione Toscana (POR CreO Fesr 2014-2020). The project brings together the CoLing Lab and the companies ETI3 s.r.l. (coordinator), BNova s.r.l. and Rigel Engineering s.r.l.