This paper presents a semi-automatic method for identifying terms that evoke semantic frames (Fillmore, 1982). The method is tested as a means of identifying lexical units that can be added to existing frames or to new, related frames, using a large corpus on the environment. It is hypothesized that a method based on distributional semantics, which exploits the assumption that words that appear in similar contexts have similar meanings, can help unveil lexical units that evoke the same frame or related frames. The method employs a distributional neighbourhood graph, in which each word is connected to its nearest neighbours according to a distributional semantic model. Results show that most lexical units identified using this method can in fact be assigned to frames related to the field of the environment.
Recent work has shown that Frame
Semantics
1The work reported in this paper is carried out within a larger project entitled “Understanding the environment linguistically and textually”, whose objective is to develop methods for characterizing the contents of texts on two different levels: 1. textual (using methods and techniques derived from corpus linguistics and text mining); and 2. linguistic (based on lexical semantic models).
Frame Semantics has proved especially useful
to represent predicative units (verbs such as
deforest, recycle, warm; predicative nouns such as
impact, pollution, salinization; adjectives such as
clean, green, sustainable), units that are often
ignored in terminological resources. L’Homme
et al. (2014) showed that the framework and
more specifically the methodology devised within
the FrameNet Project
This paper is structured as follows. Section 2 explains how semantic frames help reveal part of the lexical structure of a specialized field of knowledge. Section 3 describes the graph-based method used to identify frame-relevant lexical units. Section 4 discusses how the model used in the manual evaluation of this method was selected. Section 5 presents the evaluation methodology and the results of the evaluation. 2
In a specialized field such as the environment,
many concepts correspond to processes, events
and properties which are typically expressed
linguistically by predicative terms (verbs, predicative
nouns and adjectives). However, traditional
terminological models (and even less traditional ones,
such as ontologies) are not properly equipped
to describe the terms that denote these concepts
and account for their specific linguistic
properties, namely the fact that they require arguments
(X changes Y; impact of X on Y). Frame
Semantics
L’Homme et al. (2014) describe a method to discover semantic frames based on an existing terminological resource called DiCoEnviro2, that contains English and French terms related to the field of the environment. Each entry in DiCoEnviro is devoted to a lexical unit (LU), i.e. a lexical item that conveys a specific meaning, and states the argument structure of the LU, as in the following examples: warm1a, vi: climate[Patient] warms warm1b, vt: gas[Agent] or change[Cause] warms climate[Patient] warm, adj.: warm climate[Patient]
Argument structures state the number of obligatory participants, and two different systems are used to label them: the first one accounts for the semantic roles of arguments (Agent, Patient, Cause); the second one gives a typical term, i.e. a term that is representative of what can appear in that position.
Many entries – especially entries that describe predicative terms – come with annotated contexts that show how arguments3 are realized in sentences extracted from an environmental corpus. For example, annotated contexts for warm1b are shown in Table 1.
2See http://olst.ling.umontreal.ca/ cgi-bin/dicoenviro/search_enviro.cgi.
3Non-obligatory participants are also annotated, as shown in the last sentence in Table 1, in which the phrase since 1750, which expresses Time, is annotated.
The primary radiative effect of CO2 and water vapour[CAUSE] is to WARM the surface climate[PATIENT] but cool the stratosphere. As increases in other greenhouse gases[CAUSE] WARM the atmosphere and surface[PATIENT], the amount of water vapour also increases, amplifying the initial warming effect of the other greenhouse gases.
The simulations of this assessment report (for example, Figure 5) indicate that the estimated net effect of these perturbations[CAUSE] is to HAVE WARMED the global climate[PATIENT] since 1750[TIME].
Argument structures and annotations were used
to discover frames using two different methods.
A semantic frame is a knowledge structure that
represents specific situations (e.g. a teaching
situation, a selling situation, a driving situation). A
frame includes participants (called frame elements
or FEs), some of which are obligatory (core FEs)
and some of which are optional (non-core FEs).
For instance, the Operate vehicle frame describes
a situation in which a Vehicle is set in motion by a
Driver and includes the following core FEs: Area,
Driver, Goal, Path, Source, and Vehicle. Lexical
units such as cycle, cruise, drive, pedal, and ride
evoke this frame
The first method consisted in comparing the argument structures and non-obligatory participants of terms already encoded in the terminological resource. This method shows that the verbs cool1a, warm1a and the nouns cooling1 and warming1 share many features. They all have a single argument (a Patient) and share some non-obligatory participants (Degree, Duration, Location).
The second method – which was applied only to the English terms – consisted in comparing the contents of the terminological resource to that of FrameNet.4 Relevant data were extracted from the FrameNet database for terms that were recorded in the terminological resource, as shown in Figure 1. This figure shows an example in which a correspondence between FrameNet and the terminological database could be established. However, in many instances, matches could not be made as nicely. In various cases, specific frames needed to be defined for the environmental terms (for instance, a new frame was created to capture adjectives such as clean, environmental and green, whose meaning can be loosely described as “that does not harm the environment”). In other cases, existing frames in FrameNet needed to be adapted to the data extracted from the terminological database for different reasons (slightly more specific meanings, different number of arguments, etc.). 2.2
It soon became obvious that some of the frames identified based on the methods described in Section 2.1 could be linked. For instance, all processes related to changes affecting the environment appeared to be somehow related.
Again, using
The methods described above allowed us to define a first subset of frames that are relevant for the field of the environment, link part of these frames and assign lexical units (LUs) to them. Based on this preliminary data, we explored the potential of a semi-automatic method to enrich our resource by adding new LUs to existing frames or discovering new frames. This method exploited distributional information obtained from a much larger corpus than the one used in the work described above.
The method we tested to discover related LUs
is based on the neighbourhood graph induced by a
distributional model of semantics. Distributional
semantic models are commonly used to estimate
semantic similarity, the underlying hypothesis
be4The FrameNet team releases an XML version of the
database
5See http://olst.ling.umontreal.ca/ dicoenviro/framed/index.php (in development). Cause temperature change Change of temperature Cause change of position on a scale Cause balance Ambient temperature
Weather event
Change natural feature
Change of phase
Change position on a scale
Balancing Cause change natural feature
Ceasing to be
Undergo change of state
Water emanating
Change of impact Damaging
Being at risk
Progress Cause change of impact
Cause change into organized society
ing that words that appear in similar contexts tend
to be semantically related
Various kinds of graphs could be used to
compute and visualize the distributional
neighbourhood of a particular word or set of words. We use
a k-nearest-neighbour (k-NN) graph, two
examples of which are the symmetric k-NN graph and
the mutual k-NN graph
The graph construction procedure can be sum6We also tested the symmetric k-NN graph, but we only report results obtained with the mutual graph. We achieved higher F-scores using the mutual graph. marized as follows. Given a distributional semantic model, we compute the pairwise similarity between all words. For each word, we compute its k NNs by sorting all other words in decreasing order of similarity to that word and keeping the k most similar. Then, for each word wi and each neighbour wj in the k NNs of wi, we add an edge in the graph between wi and wj if wi is also among the k NNs of wj . The resulting graph can be used to visualize the distributional neighbourhood of a term or set of terms. 4
Any model that allows us to estimate the semantic similarity of two words can be used to build a semantic neighbourhood graph such as the one described in Section 3. We tested two different distributional semantic models for this purpose. Both models have several parameters which must be set and which can have a significant impact on the accuracy of the model in a given application. We therefore used an automatic evaluation procedure to tune the models’ parameters and select a model for manual evaluation. 4.1
The corpus used to build the models is the
PANACEA Environment English monolingual
corpus (Catalog Reference ELRA-W0063), a
corpus containing 28071 web pages related to the
environment (approximately 50 million tokens).
The corpus was compiled automatically using
a focused web crawler developed within the
PANACEA project, and is freely distributed by
ELDA for research purposes.7 The corpus was
7See http://catalog.elra.info/product_
info.php?products_id=1184.
converted from XML to raw text and lemmatized
using TreeTagger
Reference data were extracted from the Framed DiCoEnviro.8 The reference data are sets of LUs that evoke the same semantic frame. The list of English LUs was extracted from each of the frames included in the Change of temperature (COT) scenario9 (cf. Figure 2). Two LUs (thawing and thinning) were excluded because they were not in the vocabulary used to construct the models, which contains the 10,000 most frequent lemmatized words in the corpus, excluding stop words. We obtained 13 sets containing a total of 53 LUs, each frame containing between 2 and 7 LUs. The number of unique LUs is 45, several LUs evoking more than one frame.10 4.2
Two different distributional semantic models were
tested. The first is a bag-of-words (BOW)
model
The models’ parameters were tuned by testing various combinations of parameter values, building neighbourhood graphs from each resulting model, and computing evaluation metrics on these graphs based on the reference data described in Section 4.1.
Some of the main choices that must be made when training a model using word2vec pertain to the architecture of the model (continuous skipgram or continuous bag-of-words), the training algorithm (hierarchical softmax or negative sampling), the use of subsampling of frequent words, the size (dimensionality) of the word vectors and 8Data extracted on 2015-05-22. Data has been added since then, as the resource is in development.
9Frames related to the scenario only through a See also relation were excluded.
10Polysemous LUs evoke different frames. For instance, warm1a (intransitive verb) evokes the Change of temperature frame; warm1b (transitive verb) evokes the Cause temperature change frame; and warm2 (adjective) evokes the Ambient temperature frame. the size of the context window. We tested various values for each of these parameters, including the recommended values11 when available. A total of 160 models were tested. In the case of the BOW model, important parameters12 include the type, shape and size of the context window, the weighting scheme applied to the cooccurrence frequencies, and the use of dimensionality reduction. Again, we tested different values for these parameters. Each model was tested with and without dimensionality reduction, for which we used singular value decomposition (SVD). A total of 320 BOW models were built and evaluated (160 unreduced and 160 reduced using SVD).
For both models, we used the cosine similarity to estimate the similarity between words. 4.3
For each model tested, we constructed multiple kNN graphs, using different values of k. For each of these graphs, we computed evaluation metrics using the reference data described in Section 4.1. We used precision and recall to check to what extent LUs belonging to the same frame were connected in the graph. These metrics are computed for each of the 45 unique LUs in the reference data. Let wi be an LU, R(wi) the set of related LUs that evoke at least one of the frames evoked by wi, and NN (wi) the set of words that are adjacent to wi in the graph. Furthermore, let TP i (true positives) be the number of words in NN (wi) that are one of the related LUs in R(wi), FP i (false positives) the number of words in NN (wi) that are not in R(wi) and FN i (false negatives) the number of words in R(wi) that are not in NN (wi). The evaluation metrics are then calculated as usual: precisioni = recalli =
TP i TP i + FP i
TP i
TP i + FN i F-scorei = 2
precisioni recalli precisioni + recalli 11See https://code.google.com/p/ word2vec/#Performance.
12Several studies have assessed the influence of this model’s parameters. The relative importance of several parameters was quantified using analysis of variance by Lapesa et al. (2014). 0.2120 0.1681 0.1429 0.1253 0.1108
By analyzing how precision and recall varied with respect to the BOW model’s parameters, we determined the optimal parameter values for this application. For example, the optimal window size was determined to be 3 words. The corresponding graph was then evaluated manually. 5
Once the model had been selected, the corresponding neighbourhood graph was evaluated manually. The evaluation was carried out by one of the co-authors of this paper, who is responsible for the development of the Framed DiCoEnviro. The 45 unique LUs in the reference data had 137 unique neighbours (adjacent nodes in the graph). These 137 words were evaluated manually in order to determine to what extent the graph can serve to discover frame-evoking LUs that can be added to the database.
The evaluation was carried out one frame at a time by observing the subgraph corresponding to that frame’s LUs and their neighbours (adjacent nodes in the neighbourhood graph). For example, the subgraph for the frame Cause change of impact is shown in Figure 3. In each subgraph, the LUs already encoded in that frame were highlighted in green, and those encoded in other frames in the COT scenario were highlighted in blue. One or more numbers were appended to the label of each LU to indicate which frame(s) it evokes.
For each word that was not already encoded as an LU in the COT scenario (i.e. for each white node), the evaluator was asked to choose one of the following categories: 1. The word should be encoded as an LU in the COT scenario (a) in an existing frame; (b) in a new frame. 2. The word should be encoded as an LU in another scenario (a) in an existing frame that is related to the
COT scenario (by a See also relation); (b) in an existing frame that is not related to the COT scenario; (c) in a new frame. 3. The word should not be encoded as an LU in the database, but it is the realization of a core FE of one of the frames in the COT scenario. 4. The word should not be encoded as an LU in the database, nor is it the realization of a core FE of one of the frames in the COT scenario.
Table 4 shows the results of this evaluation. As these results show, most lexical items identified by the method (105 out of 137) can be encoded in a relevant frame in the field of the environment and should be added to our resource. Among these, 88 would be frame-evoking LUs (categories 1 and 2) and 17 would be encoded as realizations of FEs (category 3). Interestingly, 48 lexical items are related to the COT scenario (categories 1a and 1b). The method allowed us to identify: 1. new frame-evoking LUs (such as amplification, drop, and scarcity) that had not been encoded in existing frames (category 1a); 2. LUs (such as alteration and eliminate) that evoke frames that had not been created (category 1b); and 3. variants (such as cooler for cool and stabilise for stabilize).
Category
Nb cases 1a 1b 1 (total) 2a 2b 2c 2 (total) 3 4 Total
The method also identified 40 items that would be encoded in environmentally relevant frames, but in a different scenario (category 2). It is worth pointing out that among these, 7 items correspond to LUs that would evoke a frame that is linked to the COT scenario (category 2a).
Finally, although 32 lexical items identified by the method would not be encoded in the resource and are thus considered false positives from the point of view of our application, further explanations are required. Some lexical items could evoke more general frames. For instance, rapid and slow would appear in the same frame if the general lexicon were considered. Other items identified are acronyms. GW, for instance, is the acronym for global warming. Technically, it could be defined as an LU evoking the COT frame, but multi-word terms and acronyms are not considered in the resource. 6
All in all the results obtained are quite interesting and show that the method can be used to assist lexicographers when defining frames and their lexical content, as the distributional neighbourhood of frame-evoking LUs often contain LUs that evoke the same frame or related frames. Distributional neighbourhood graphs provide information about the content of a specialized corpus that would be impossible to extract manually from such a large corpus. They are a very useful complement to other corpus tools, such as term extractors and concordancers, as they help lexicographers save time and locate relevant lexical units (near synonyms, variants) that they would otherwise miss.
In future work, we plan to integrate this methodology to assist lexicographers when defining new frames related to the field of the environment. It could be particularly useful to obtain a view on corpora that deal with new or more specific topics and unveil the lexical units used to convey the knowledge related to these topics. It would also be interesting to test the potential of the method in other fields of knowledge. Extensions of this work could also involve using a graph-based clustering method to discover sets of lexical units that evoke the same frame without using existing frames.
This work was supported by the Social Sciences and Humanities Research Council (SSHRC) of Canada.