As highlighted by Pustejovsky (1995, 2002), the semantics of each verb is determined by the totality of its complementation patterns. Arguments play in fact a fundamental role in verb meaning and verbal polysemy, thanks to the sense co-composition principle between verb and argument. For this reason, clustering of lexical items filling the Object slot of a verb is believed to bring to surface relevant information about verbal meaning and the verb-Objects relation. The paper presents the results of an experiment comparing the automatic clustering of direct Objects operated by the agglomerative hierarchical algorithm of the Sketch Engine corpus tool with the manual clustering of direct Objects carried out in the T-PAS resource. Cluster analysis is here used to improve the semantic quality of automatic clusters against expert human intuition and as an investigation tool of phenomena intrinsic to semantic selection of verbs and the construction of verb senses in context.
Clustering techniques have been used extensively
in recent decades in Linguistics and NLP,
especially in Word Sense related tasks. As a matter of
fact, partitioning data sets on the basis of their
similarity at a distributional level clarifies the
meaning of lexical elements
Clustering lexical items filling the argument positions of a verb is then believed to bring to surface relevant information about verbal meaning and the verb-arguments relation. Clustering them, and especially direct Objects in pro-drop languages such as Italian, allows hence to investigate how to better induce, discriminate and disambiguate verb senses. Because argument fillers share the same semantic nature, they can be grouped and generalized with respect to their content and be associated with semantic types, i.e. empirically identified semantic classes representing selectional properties and preferences of verbs.
Clustering of Objects can therefore be used as a
survey tool for the intrinsic phenomena of
semantic classes and, at the same time, as an object of
investigation to improve the clustering automatic
models themselves against human partitioning.
This paper presents the results of an experiment
comparing manual and automatic clustering of
Italian Object fillers to be used in verb-sense
identification and, along with it, it describes the
linguistic phenomena that emerged from the
semantic analysis of non-supervised clusters. The
comparison concerns the agglomerative hierarchical
clustering algorithm of the Sketch Engine corpus
tool1
Clustering is a Data Mining task (Kotu &
Deshpande, 2014) in which a grouping process of a set
of objects is carried out, obtaining clusters of
elements which are similar to each other but
dissimilar from the objects of other groups
As for automatic clustering, in our comparison we
used the built-in clustering function
pilotare_clust3: {caccia (fighter aircraft)}5 The main difference between T-PAS and SkE clustering procedures are the semantic-distributional criteria on which they are based. T-PAS approach can be defined as verb-oriented: Objects are primarily clustered on the basis of their verbal distributional behaviour and ability to activate a given verbal sense as direct objects. Since all fillers occupying a given slot for a given sense share the same relation with the verb, they can be ontologically and semantically generalized with an ST on the basis of their common semantic traits. This generalization allows to make the verbal selectional constraints visible. On the contrary, SkE 4 We also conducted a similarity value manipulation experiment, which confirmed what discussed in detail in section 3. 5 Clusters consist at least of 1 word and up to 1000. performs noun-based clustering: it takes into account the general distributional behaviour of fillers, not merely the verbal one. In the process of creating sets, each filler behaviour is weighed against the entire reference corpus and with respect to the frequencies of appearance in different contexts. The elements clustered together in SkE are therefore not only similar in their sense and behaviour as direct objects, but also respect to the whole nominal class they belong to.
As mentioned above, in T-PAS argument slots are
linked to ST labels, semantic classes able to
generalize over the sets of lexical items in argument
positions found in the corpus (
The research has been developed through a
pipeline organized according to the following steps:
1. Data extraction: Data for both clusterings are
extracted from the web crawled corpus ItWac
reduced
2. Data intersection: The obtained clusters are intersected with each other in order to obtain a database in which there are sets for the same verbs and containing the same fillers, to focus on how the two models carried out the partition.
3. Data filtering: In this step the database is cleared from: a) verbs with structures recognized as complex and non-compositional, i.e. idiomatic constructions; b) verbs with the ST [Anything] (top node in Fig. 1) in the object slot, as it does not entail selection restrictions within the T-PAS clusters; c) verbs with Object clusters with more than 29 internal elements.
At the end of the filtering process the clusters of the two models are aligned with respect to the STs, i.e. all possible STs signaled in T-PAS for the Object of a verb are treated as a single set of semantic conditions, in order to analyze the internal quality of SkE clusters through them. The aligned structure of the verb acquisire (to acquire) in Figure 2 is given here as an example. tpas_acquisire ske_acquisire clust_1 clust_2 clust_3 [Institution], [Artifact] The final database comes to a total of 397 verbs and 3938 clusters, including both T-PAS and SkE clusters. We provide an illustrative table (Table 1) showing the first and last verb among those analyzed and the information on their respective clusters: abbagliare (to dazzle) and votare (to vote).
To verify the compatibility between the two clusterings, the similarity between the two partitions has been evaluated through different metrics able to offer an external evaluation of the unsupervised model. To account for both the presence of common pairings, as well as the homogeneity and completeness of the clustering, the following metrics were considered: Fowlkes & Mallow Index (F&M), Adjusted Rand Index (ARI), Homogeneity, Completeness.
F&M, as the geometric mean between precision
and recall, was used to verify the similarity
between the two models from how many partition
pairings are in common. This index also allows to
better balance the possible noise or unrelatedness
between clustering
As reported by their respective creators, all
metrics have an optimal result range between 0 and 1.
The possible results between these two limits can
be classified with respect to the greater or lesser
proximity to the optimal limit: the results closer to
1 denote greater similarity of output between the
two models, the results closer to 0 instead less
similarity
In this sense, we can define three bands of possibilities, coherently with the approach the higher the better generally used in cluster analysis: from 0.01 to 0.399, the clustering compared to the golden standard is highly different, from 0.4 to 0.699 the result and the correspondence is medium-good, while the results above 0.7 and up to 0.999 are the ideal ones, which indicate a marked correspondence between the compared models. However, since metrics such as F&W and ARI have shown the lack of partitions in higher ranges (the first beyond 0.82, the latter beyond 0.7), we choose to consider the whole group of medium good results between 0.4 and 1. The absence of the higher ranges stands for low compatibility between the two models.
Metric Adjusted Rand Score Fowlkes & Mallow Homogeneity Completeness Clusters in the [0.4-1] range 11.08% 36.18% 94.96% 41.31% As we see in Table 2, what we find in fact is a situation of only limited correspondence between the two clustering, with a rather low overlap and similarity as indicated by the ARI and the F&M, even with the internal noise balance. At least two reasons may be behind the scarce similarity: the tendency of SkE to create small fine-grained clusters populated by few elements that give more weight to specificity than to generalization capacity; the fact that in T-PAS for a given verb sense the Object slot can be compatible with more STs (see (1)), and such STs can also be hierarchically distant in the general system of labels. This leads to clusters containing fillers able to activate a given verbal sense but which are quite heterogeneous among themselves and semantically dissimilar, with respect to the rest of the distributional relations between the fillers. An example can be the verb trasportare (to transport), which has as T-PAS cluster for the first sense a set of 18 Objects (see (4)); such fillers belong to three different STs: [Inanimate], [Animate] and [Energy]. The latter ST, [Energy], is hierarchically distant to the others since it has a different parent node than [Animate] and [Inanimate], which both pertain to a lower level in the hierarchy. (3) trasportare 1. [Human] | [Vehicle] | [Watercourse] trasportare [Inanimate] | [Animate] | [Energy] (4) trasportare_clust1: {acqua (water), alimento (nourishment), animale (animal), arma (weapon), bene (asset/good), bicicletta (bicicle), cadavere (corpse), cibo (food), gas (gas), gommone (inflatable raft), macchina (machine), oggetto (object), peso (weight), student (student), terra (soil), traffic (traffic), viaggiatore (traveller), visitatore (visitor)} It is clear that a fine-grained algorithm, not able to generalize at a higher level as in T-PAS, will divide fillers labelled with [Animate] or [Inanimate] from those labelled with [Energy]. In fact, SkE for the same verb creates 12 clusters.
As shown in Table 2, the results of the Completeness are in line with what has just been discussed for ARI and F&M: only in 40% of the cases all members of a T-PAS cluster are members of a single SkE cluster. These are generally small or medium sized clusters with only one associated ST or with hierarchically close alternative ST structures. Homogeneity highlights the primary characteristic of SkE clusters and the algorithm: it is preferable to create smaller but internally purer clusters, rather than larger sets with members of other classes. This implies the creation in SkE of semantically specific clusters, that privilege the inter-relation between Object fillers but not the higher semantic level between Object fillers and verb.
From a different perspective, we can say that the noun-oriented criteria of clustering and the verboriented ones tend to converge when we consider small clusters, in which the elements belonging to a set in SkE generally belong to the same set in TPAS.
As for wide clusters, they are particularly rare in SkE and tend to be smaller in size than T-PAS anyway. Their content also seems to be dependent on various factors on which the linguistic analysis has shed light.
To verify the nature of the diversity between the two clusterings measured with the metrics reported in 3.1, a detailed analysis of the lexical-semantic phenomena visible internally to the clusters was carried out considering: - The consistency, for automatic clusters, with one and only one of the aligned T-PAS STs, i.e. the precision and purity at the semantic level of clusters compared to the generalization of the ST; - Internal homogeneity, i.e. whether the clusters meet verb-sense oriented or noun-sense oriented criteria and, if the latter, whether the cluster items are linked by syntagmatic relationships and there is some kind of affinity or implication between them. Thus, the types of semantic relations present between the words are identified; - The overlap between clusters with respect to the ARI, and in relation to cluster size and clustering difficulty depending on several STs possible for the same slot; - The problem of incorrect mapping as Objects of postverbal Subjects, subjects of inaccusative verbs, structures with si particle (e.g. reflexive, impersonal), i.e. the clusters' internal noise. The research has shown that SkE clusters tend to be small-medium sized, semantically homogeneous, often able to isolate very specific semantic relations. They are generally not consistent per se with the verb sense identified by T-PAS but create partitions: a) usually of medium size and consistent with only one parallel ST, b) single element groups that generally belong to a higher level of specificity or to a different semantic domain, and c) groups that are inconsistent with the sense of the verb but cluster words on the basis of the following criteria: - Belonging to the same domain (e.g. informatics for distribuire {software, applicazione}); - Being part of the same ST, but as very specific instances, not separated by the T-PAS hierarchy (e.g. {abbazia, monastero, santuario} for saccheggiare and the type [Location]); - The possibility of a conceptual association or affinity (e.g. {seminario, incontro, seduta} for organizzare); - Purely distributional parameters and undefined semantic relations (e.g. in gestire {contenuto, caso}); - A relationship of synonymy or meronymy (e.g. {spinta, propensione} for frenare or for fratturare {dito, mano, braccio}); antonymy, hyponymy, hyponymy are generally represented by different clusters.
The parameters of consistency, internal homogeneity and overlapping between the models seem to relate to the same factors: first, the size of the clusters, i.e. how many clustered elements are part of the set; second, the structure of STs possible for the Object (see (5)), i.e. if for the same slot only one ST is possible, if several alternatives are available or, also, if a lexical set is signaled in the T-PAS annotation - that is, if among the fillers a set of lexical elements is present that has high frequency or has the typical behaviour of a collocation (e.g. {messaggio | ricordo} in (5)). This is relevant since the computation of SkE starts precisely from the frequency and collocational behaviour of a word. (5) cancellare (sense description: to eliminate, to make inexistent): 1. [Human] | [Inanimate1] | [Abstract Entity1] | [Eventuality1] cancellare [Inanimate2] | [Abstract Entity2 {messaggio | ricordo}] | [Eventuality2] The third relevant factor is hierarchical proximity, i.e. if STs possible for a slot are sisters of the same parent node between the types present in the hierarchy (see (6)). (6) no proximity: [Animate] vs. [Institution] in proximity: [Command] vs [Request] The clusters of SkE, even if not corresponding to those of T-PAS, are rated totally consistent or at 70-80% consistent with one of the aligned STs 62% of the times; in the remaining 38% of cases, there is a significant number of clusters that can be generalized with a ST. Consistency is more difficult to reach if a given sense is annotated in T-PAS with several alternative STs or STs and lexical sets co-presenting; SkE can produce new combinations in which fillers corresponding to different STs are included in the same cluster. Very frequently SkE atomizes the set of fillers in nuclear clusters, made up of only one or two elements that are necessarily consistent with one ST but are not of much help to the study of semantic relations.
As said, hierarchical proximity of STs and the size of the cluster can influence its handling: if the possible STs for the Obj-slot are hierarchically distant and the T-PAS cluster is small, the SkE outcome will tend to be heterogeneous and inconsistent. Consider the verb affogare (to drown), which has [Animate] and [Emotion] as possible STs for the same slot but different senses. T-PAS clusters are small, clust_1 counts 3 elements and clust_2 counts 7, in which the two possible types are clearly distinguished. One would expect to find the two senses separated in the SkE clusters as well, since [Emotion] belongs to lower levels of the hierarchy and has a different parent node of [Animate]. However, for SkE we find clusters such as: (7) affogare_clust3: {figlio (son), bimbo (child), pensiero (thought)} If, on the contrary, we consider closer ST, the clustering will be homogeneous, even if not verbsense-oriented because too fine-grained. Medium-sized clusters (more than 10 clustered elements) seem to perform quite well both with hierarchically close and distant STs. A useful example can be smarrire (to lose) in T-PAS (8), for which SkE presents the clusters in (9): (8) smarrire 1. [Human] smarrire [Artifact] 2. [Human] | [Human group] smarrire [Concept] | [Property] (9) smarrire_clust1: {borsello (man bag)} smarrire_clust2: {significato (meaning), ragione (reason), memoria (memory), pensiero (thought)}
smarrire_clust3: {capacità (capacity), consapevolezza (awareness)}
smarrire_clust4: {nozione (notion), certezza (certainty)} smarrire_clust5: {senno (sense)} smarrire_clust6: {fiducia (trust), voglia (will)} smarrire_clust7: {documento (document)} smarrire_clust8: {cellulare (mobile phone)} Considering that for smarrire T-PAS creates only two clusters (see (10)), the sets in (9) also highlight the general tendency of SkE to create smaller, semantically highly fine-grained clusters. (10) smarrire_clust1: {borsello (man bag), cellulare (mobile phone), documento (document)} smarrire_clust2: {consapevolezza (awareness), capacità (capacity), certezza (certainty), fiducia (trust), memoria (memory), nozione (notion), pensiero (thought), ragione (reason), senno (sense), significato (meaning), voglia (will)} Large clusters are generally difficult to handle because ideally portionable in more distributionally cohesive groups. As regards the problem of internal noise, due to the PostV_N relation, we can note that the phenomenon is pervasive and important, since it affects the results of internal coherence and homogeneity. It is, however, a phenomenon that can be curbed with a revision of the extraction function. What emerges from the analysis is a distance in the general structure of the two clustering results but a good compatibility from the internal semantic point of view. T-PAS privileges rather more complex semantic groupings on a level of co-composition between verb and meaning, linked to conceptual operations of generalization. On the contrary, SkE creates complex and homogeneous structures of relations inside data, even if sometimes this implies clusters that are too fragmented and not always optimal also from a noun-oriented perspective. T-PAS seems to pertain to a higher level of granularity respect to SkE, whose clusters can be considered as possible subpartitions of the STs. 4
The paper presented the statistical and linguistic results of a comparison between SkE unsupervised clustering model and the manual and verbsense oriented clustering of T-PAS. It highlighted how the noun-oriented model and the verb-oriented one are not overlapping if not partially. The SkE clustering, even if not overlapping, can still be considered as internally compatible with the TPAS partition, since the homogeneity metric reaches good results. The internal linguistic analysis allowed to identify the semantic quality through the consistency with a semantic type, the internal homogeneity, the adherence with the verb-oriented approach of T-PAS. The reasons that regulate the fragmentation of clusters in SkE, i.e. motivations that follow a fine-grained logic, were then presented. The analysis made possible to shed a light on the semantic compatibility between the two approaches, which seem to pertain to different levels of granularity.
The difference in the partition output and the parallel semantic compatibility allows us to claim that the SkE automatic clustering is more useful for the internal investigation of STs than to investigate the verb-Object co-composition relation. It would be interesting to conduct further comparisons between other automatic clustering techniques and that of T-PAS, to investigate additional semantic implications of clustering through nounbased and verb-based approaches.