English. The great majority of compositional models in distributional semantics present methods to compose distributional vectors or tensors in a representation of the sentence. Here we propose to enrich the best performing method (vector addition, which we take as a baseline) with distributional knowledge about events, outperforming our baseline.
Composing word representations into larger
phrases and sentences notoriously represents a
big challenge for distributional semantics
Among all proposed compositional functions,
vector addition still shows the best performances
on various tasks
The majority of available models in literature assumes the meaning of complex expressions like sentences to be a vector (i.e., an embedding) projected from the vectors representing the content of its lexical parts. However, as pointed out by Erk and Pado´ (2008), while vectors serve well the cause of capturing the semantic relatedness among lexemes, this might not be the best choice for more complex linguistic expressions, because of the limited and fixed amount of information that can be encoded. Moreover events and situations, expressed through sentences, are by definition inherently complex and structured semantic objects. Actually, assuming the equation “meaning is vector” is eventually too limited even at the lexical level.
Psycholinguistic evidence shows that lexical
items activate a great amount of generalized event
knowledge (GEK)
In this paper, we introduce MEDEA, a compositional distributional model of sentence meaning which integrates vector addition with GEK activated by lexical items. MEDEA is directly inspired by the model in Chersoni et al. (2017a) and relies on two major assumptions: lexical items are represented with embeddings within a network of syntagmatic relations encoding prototypical knowledge about events; the semantic representation of a sentence is a structured object incrementally integrating the semantic information cued by lexical items.
We test MEDEA on two datasets for compositional distributional semantics in which addition has proven to be very hard to beat. At least, before meeting MEDEA.
MEDEA consists of two main components: i.) a
fragment of semantic memory activated by lexical units (Section 2.1); ii.) a Meaning Composition Function that dynamically integrates information activated from DEG to build a sentence semantic representation (Section 2.2). 2.1
We assume a broad notion of event, corresponding to any configuration of entities, actions, properties, and relationships. Accordingly, an event can be a complex relationship between entities, as the one expressed by the sentence The student read a book, but also the association between an individual and a property, as expressed by the noun phrase heavy book.
In order to represent the GEK cued by lexical items during sentence comprehension, we explored a graph based implementation of a distributional model, for both theoretical and methodological reasons: in graphs, structural-syntactic information and lexical information can naturally coexist and be related, moreover vectorial distributional models often struggle with the modeling of dynamic phenomena, as it is often difficult to update the recorded information, while graphs are more suitable for situations where relations among items change overtime. The data structure would ideally keep track of each event automatically retrieved from corpora, thus indirectly containing information about schematic or underspecified events, by abstracting over one or more participants from each recorded instance. Events are cued by all the potential participants to the event.
The nodes of DEG are lexical embeddings, and edges link lexical items participating to the same events (i.e., its syntagmatic neighbors). Edges are weighted with respect to the statistical salience of the event given the item. Weights, expressed in terms of a statistical association measure such as Local Mutual Information, determine the event activation strength by linguistic cues.
In order to build DEG, we automatically harvested events from corpora, using syntactic relations as an approximation of semantic roles of event participants. From a dependency parsed sentence we identified an event by selecting a semantic head (verb or noun) and grouping all its syntactic dependents together (Figure 1). Since we expect each participant to be able to trigger the event and consequently any of the other participants, a relation can be created and added to the graph from each subset of each group extracted from sentence. The resulting structure is therefore a weighted hypergraph, as it contains relations holding among groups of nodes, and a labeled multigraph, since each edge or hyperedge is labeled in order to represent the syntactic pattern holding in the group.
As graph nodes are embeddings, given a lexical cue w, DEG can be queried in two modes: retrieving the most similar nodes to w (i.e., its paradigmatic neighbors), using a standard vector similarity measure like the cosine (Table 1, top row); retrieving the closest associates of w (i.e., its syntagmatic neighbors), using the weights on the graph edges (Table 1, bottom row).
essay/N, anthology/N, novel/N, author/N, para. neighbors publish/N, biography/N, autobiography/N, nonfiction/N, story/N, novella/N publish/V, write/V, read/V, synt. neighbors include/V, child/N, series/N,
have/V, buy/V, author/N, contain/V In MEDEA, we model sentence comprehension as the creation of a semantic representation SR, which includes two different yet interacting information tiers that are equally relevant in the overall representation of sentence meaning: i.) the lexical meaning component (LM), which is a context-independent tier of sentence meaning that accumulates the lexical content of the sentence, as traditional models do; ii.) an active context (AC), which aims at representing the most probable event, in terms of its participants, that can be reconstructed from DEG portions cued by lexical items. This latter component corresponds to the GEK activated by the single lexemes (or by other contextual elements) and integrated into a semantically coherent structure representing the sentence interpretation. It is incrementally updated during processing, when a new input is integrated into existing information.
Each lexical item in the input activates a portion of GEK that is integrated into the current AC through a process of mutual re-weighting that aims at maximizing the overall semantic coherence of the SR.
At the outset, no information is contained in the AC of the sentence. When new lexeme - syntactic role pair hwi; rii (e.g., student - nsbj) are encountered, expectations about the set of upcoming roles in the sentences are generated from DEG (figure 2). These include: i.) expectations about the role filled by the lexeme itself, which consists of its vector (and possibly its p-neighbours); ii.) expectations about sentence structure and other participants, which are collected in weighted list of vectors of its s-neighbours.
These expectations are then weighted with respect to what is already in the AC, and the AC is similarly adapted to the ewly retrieved information: each weighted list is represented with the weighted centroid of its top elements, and each element of a weighted lists is re-ranked according to its cosine similarity with the correspondent centroid (e.g., the newly retrieved weighted list of subjects is ranked according to the cosine similarity of each item in the list with the weighted centroid of subjects available in AC).
The final semantic representation of a sentence
consists of two vectors, the lexical meaning
vector (LM!) and the event knowledge vector (A!C),
which is obtained by composing the weighted
centroids of each role in AC.
We wanted to evaluate the contribution of
activated event knowledge in a sentence
comprehension task. For this reason, among the many
existing datasets concerning entailment or
paraphrase detection, we chose RELPRON
RELPRON consists of 1,087 pairs, split in development and test set, made up by a target noun labeled with a syntactic role (either subject or direct object) and a property expressed as [head noun] that [verb] [argument]. For instance, here are some example properties for the target noun treaty: (1) a. OBJ treaty/N: document/N that delegation/N negotiate/V b. SBJ treaty/N: document/N that grant/V in
dependence/N
of 108 pairs of transitive sentences, each annotated with human similarity judgments collected through the Amazon Mechanical Turk platform. Each transitive sentence in composed by a triplet subject verb object. Here are two pairs with high (2) and low (3) similarity scores respectively: (2) a. government use power
b. authority exercise influence (3) a. team win match
b. design reduce amount 3.2
We tailored the construction of the DEG to this
kind of simple syntactic structures, restricting it
to the case of relations among pairs of event
participants. Relations were automatically
extracted from a 2018 dump of Wikipedia, BNC,
and ukWaC corpora, parsed with the Stanford
CoreNLP Pipeline
Each h(word1; word2); (r1; r2)i pair was then weighted with a smoothed version of Local Mutual Information1:
LMI (w1; w2; r1; r2) = f(w1; w2; r1; r2)log(P^(wP^1()wP^1;(ww22;)rP1^;(rr21);r2)) where: f (x)
P^ (x) = Px f (x) Each lexical node in DEG was then represented with its embedding. We used the same training parameters as in Rimell et al. (2016),2, since we wanted our model to be directly comparable with their results on the dataset. While Rimell et al. (2016) built the vectors from a 2015 download of Wikpedia, we needed to cover all the lexemes contained in the graph and therefore we used the same corpora from which the DEG was extracted.
We represented each property in RELPRON as
a triplet ((hn; r); (w1; r1); (w2; r2)) where hn is
the head noun, w1 and w2 are the lexemes that
1The smoothed version (with = 0:75) was chosen in
order to alleviate PMI’s bias towards rare words
2lemmatized 100-dim vectors with skip-gram with
negative sampling (SGNS
In MEDEA, the SR is composed of two vectors: LM!, as the sum of the word embeddings (as this was the best performing model in literature, on the chosen datasets); A!C, obtained by summing up all the weighted centroids of triggered participants. Each lexeme - syntactic role pair is used to retrieve its 50 top s-neighbors from the graph. The top 20 re-ranked elements were used to build each weighted centroid. These threshold were choosen empirically, after a few trials with different (i.e., higher) thresholds (as in Chersoni et al. (2017b)).
We provide an example of the re-weighting process with the property document that store maintains, whose target is inventory: i.) at first the head noun document is encountered: its vector is activated as event knowledge for the object role of the sentence and constitutes the contextual information in AC against which GEK is re-weighted; ii.) store as a subject triggers some direct object participants, such as product, range, item, technology, etc. If the centroid were built from the top of this list, the cosine similarity with the target would be around 0:62; iii.) s-neighbours of store are reweighted according to the fact that AC contains some information about the target already, (i.e., the fact that it is a document). The re-weighting process has the effect of placing on top of the list elements that are more similar to document. Thus, now we find collection, copy, book, item, name, trading, location, etc., improving the cosine similarity with the target, that goes up to 0:68; iv.) the same happens for maintain: its s-neighbors are retrieved and weighted against the complete AC, improving their cosine similarity with inventory, from 0:55 to 0:61. 3.4
We evaluated our model on RELPRON development set using Mean Average Precision (MAP), as 3The relation for the head noun is assumed to be the same as the target relation (either subject of direct object of the relative clause). in Rimell et al. (2016). We produced the compositional representation of each property in terms of SR, and then ranked for each target all the 518 properties of the dataset portion, according to their similarity to the target. Our main goal was to evaluate the contribution of event knowledge, therefore the similarity between the target vector and the property SR was measured as the sum of the cosine similarity of the target vector with the LM! of the property, and the cosine similarity of the target vector with the A!C cued by each property. As shown in Table 2, the full MEDEA model (last column) achieves top performance, above the simple additive model LM.
verb arg hn+verb hn+arg verb+arg hn+verb+arg
LM 0,18 0,34 0,27 0,47 0,42 0,51
RELPRON
AC LM+AC 0,18 0,20 0,34 0,36 0,28 0,29 0,45 0,49 0,28 0,39 0,47 0,55
For the transitive sentences dataset, we evaluated the correlation of our scores with human ratings with Spearman’s . The similarity between a pair of sentences s1; s2 is defined as the cosine between their LM vectors plus the cosine between their EK vectors. MEDEA is in the last column of Table 3 and again outperforms simple addition. sbj root obj sbj+root sbj+obj root+obj sbj+root+obj
We provided a basic implementation of a meaning composition model, which aims at being incremental and cognitively plausible. While still relying on vector addition, our results suggest that distributional vectors do not encode sufficient information about event knowledge, and that, in line with psycholinguistic results, activated GEK plays an important role in building semantic representations during online sentence processing.
Our ongoing work focuses on refining the way in which this event knowledge takes part in the processing phase and testing its performance on more complex datasets: while both RELPRON and the transitive sentences dataset provided a straight forward mapping between syntactic label and semantic roles, more naturalistic datasets show a much wider range of syntactic phenomena that would allow us to test how expectations jointly work on syntactic structure and semantic roles.