English. This paper describes a preliminary expansion and assessment of the Latin WordNet for the purposes of the LiLa: Linking Latin project. The objective of this study is to better understand the implications of expanding and evaluating the sense coverage of the Latin WordNet, with a view to identifying the most effective method for its refinement and inclusion in the LiLa Knowledge Base of Latin resources. Our test empirically demonstrates the inadequacy for Latin of a common semi-automated approach of expansion and informs potential lines of improvement for the resource.1
WordNets are among the most used lexicosemantic resources in Natural Language Processing (NLP). Indeed, their value is such as to warrant the annual Global WordNet Conference, which is now in its tenth edition.2 In the words of Fellbaum (1998, p. 52):
widely used electronic dictionary of English and serves as the lexicon for a varity [sic] of different NLP applications including Information Retrieval (IR), Word Sense Disambiguation (WSD), and Machine Translation (MT).
Since the release of the Princeton WordNet
(hereafter PWN) in the mid 1980s
2http://globalwordnet.org/
languages has far exceeded that for historical
languages. With the exception of the Historical
Thesaurus of English, whose purpose is not dissimilar
to that of a WordNet but whose distinct structure
sets it apart from this type of resource,3 the only
two historical language WordNets in existence
today are the Latin
The study described here falls within the scope
of the LiLa: Linking Latin project
This paper describes a preliminary assessment of the LWN with a view to better understanding how to approach its expansion and evaluation: Sections 2 and 3 briefly outline existing research in WordNet evaluation and the structure of the LWN, respectively; Section 4 details our evaluation method; Section 5 discusses our preliminary results; finally, Section 6 summarises our contribution and focusses on directions for future research. 2
Evaluation. To evaluate a WordNet is to evaluate its coverage of a specific linguistic domain or of 3https://ht.ac.uk/ 4Most recently Franzini et al. (2018). 5https://lila-erc.eu (2018-2023). an entire language (period), be that qualitative (accuracy) or quantitative (inclusivity). Among others, Bodenreider et al. (2003) conducted a quantitative evaluation of the bio-genetic domain in the PWN by mapping a list of relevant terms against manually-established semantic classes of nominal synsets, and proved PWN’s coverage to be satisfactory. A study by Hajicˇ et al. (2004) sought to manually evaluate and improve the Czech WordNet using the lexico-semantic annotation of the Prague Dependency Treebank. In spite of achieving poor inter-annotator agreement, their outcome can inform future improvements of the resource.
The first automated, qualitative evaluation of a WordNet was performed by Nadig et al. (2008) on the PWN. Using dictionary definitions, the authors applied different extraction and matching algorithms to automatically validate 38,840 nominal synsets (corresponding to 103,620 lemmas) and 56,203 hypernym-hyponym noun pairs, reaching accuracy rates of 70% and 70.88%, respectively. These high rates are hardly surprising, given that the PWN is a handmade resource; nevertheless, they give us an indication as to what might be expected from a similar evaluation performed on automatically-generated WordNets.
Extension. Researchers looking to extend
WordNets in languages other than English
typically do so by semi-automatically comparing
lemmas and synsets in their target language against
the contents of the PWN with the help of
bilingual dictionaries and linguistic resources. This
is the case of the Arabic WordNet (AWN),
extended through semi-automated comparison with
a lexicon of modern standard Arabic and the
PWN
The LWN was first created in 2004 following
the Expand Method
latin relation:sql (lemma-to-lemma) type n % Antonymy 4,538 32.95% Pertainymy/Derivation* 9,233 67.04% common relation:sql (synset-to-synset) type n % Hyper/hyponymy 3,900 85.00% Meronymy, part of 292 6.36% Entailment (v) 90 1.96% Attribute (n) 80 1.74% Value of (a) 80 1.74% Similar to (a) 54 1.17% Cause (v) 34 0.74% Meronymy, substance of 32 0.69%
Meronymy, member of 26 0.56%
The criteria behind the selection of LWN
lemmas remain unclear, and there are some
noticeable gaps, both lexical (amo, amare ‘to love’) and
relational (the adjectives inaequabilis ‘unequal’
and aequabilis ‘equal’ are placed in a relation of
derivation only but could also count as antonyms).
Examples of erroneous, modern senses inherited
by the LWN from MWNE are shown in Table 2.
In point of fact, in his most recent publication, the
creator of LWN states that the lexical coverage and
the results of his automatic assignments need
further evaluation and verification
Data. Firstly, we formatted LWN and all necessary Machine Readable Dictionaries for the task as relational SQL tables: these included WW, L&S, MWNE and MWNI.
Machine-recommended senses. Next, inspired by the work of Abouenour et al. (2013), we formalised a rudimentary algorithm in bash script to automatically extend the sense coverage of the LWN by proposing new synsets taken from the MWNE. While aware that this method would introduce some noise, the neither exact nor approximate amounts could not be quantified a priori. Figure 1 exemplifies the algorithmic process: for the LWN adverb velociter ‘swiftly, quickly’, the algorithm 1) searched for joint lemma and PoS overlaps between LWN and WW; 2) where there was a match, it then looked for overlaps between the single-word WW glosses and MWNE lemmas; 3) where these also matched, it checked the lemma’s corresponding synset(s) in MWNE for that PoS against existing LWN synsets to 4) label machine recommendations as NEW (machinesuggested and not already present in LWN) or COM (for “common”, i.e., machine-suggested but already present in LWN). Table 3 lists the results of the recommender system for velociter.
synset id definition label r#00051957 in a swift manner; she moved NEW swiftly r#00082992 with rapid movements; he works COM quickly r#00102338 with little or no delay; [. . . ] COM r#00285860 without taking pains; [. . . ] COM
The recommender system produced 121,098 lemma-synset entries for the whole LWN: 93,479 synset assignments (77.19%) were classified as NEW, 25,613 (21.15%) as COM and 2,006 (1.65%) as OLD (synsets present in the LWN only). Given the algorithm’s optimisation on recall, we expected these large numbers to include many false positives and homography, e.g., the verbs edo, edere ‘to eat’ (3rd conjugation) and edo, edare ‘to publish’ (1st conjugation) or volo, velle ‘to want’ (irregular conjugation) and volo, volare ‘to fly’ (1st conjugation).
Lemma selection. Next, for our test evaluation, we randomly selected 100 LWN-WW matched lemmas, 25 per PoS, featuring both NEW and COM synset assignments. This selection resulted in 3,746 lemma-synset entries to be evaluated.8
Manual evaluation. Of the five raters recruited
for the task, four were in possession of
intermediate Latin proficiency and one had expert
(including spoken) knowledge of the language.9 Using a
custom web annotation environment designed to
facilitate the task and with Latin dictionaries at
hand
Unsurprisingly, our synset recommender generated irrelevant assignments, as shown in Table 4.
The evaluation was performed over a period of approximately two months and informed the formulation of guidelines to enforce consistency. Among other directives, the guidelines demanded that raters accept an assignment even if specific 8Of the 100 selected lemmas, 36 had multiple homographic entries with the same PoS.
9Those with intermediate Latin knowledge were pursuing a Master’s degree in Theoretical and Applied Linguistics, while the expert rater completed a Master’s in Modern Philology (“Lettere” and Semantics). to an idiomatic use of the lemma (e.g., edo, edere ‘to eat/consume/devour’ but edere voces ‘utter’); accept an assignment even if its specificity is not mirrored in the reference dictionary (e.g., while the specific sense ‘to sodomize’ for caedo is not explicitly mentioned in Castiglioni and Mariotti (2007), the verb is said to have sexual connotations as well);10 reject an assignment if the corresponding sense is not included in their reference dictionary; and reject an assignment should there be any other strong uncertainty not covered by the guidelines. The assessment of the relations, if any, between OLD synset assignments in our evaluation set was ignored at this stage.
Missing senses. Where applicable, raters were also instructed to make a note of missing senses, be those from the Classical, Medieval or Late periods of Latinity.11 Inclusion of these missing senses in the LWN is not described here but is planned future work (see Section 6). Examples are:
10IV ed., s.v.,“caedo,” Def. fig. “in senso osceno, sbattere, Catull. 56, 7 e a.”
11We do not consider contemporary Latin (19th and 20th centuries). prudenter (r): skillfully; puto, putare (v): to clean; to prune, trim radix (n): radish; liquorice tener (a): erotic, amorous; adaptable (style); soft (soil)
Inter-rater reliability agreement. Next,
we measured inter-rater reliability (IRR) using
percentage agreement without chance
correction
Ao(r) = abs(NA(r) NR(r))
NV (r) where the observed agreement Ao on each lemmasynset relation (r) is calculated by dividing the absolute difference of accepted NA and rejected NR assignments by the total number of evaluations NV . Agreement values range between 0.0 and 1.0, where 0.0 means no inter-rater agreement and 1.0 means perfect inter-rater agreement. 5
In this section we assess IRR agreement rates against the table proposed by McHugh (2012, p. 279). As previously observed in related studies, lower agreements are not a reflection of raters’ inability to distinguish word meanings but, rather, of their difficulty in selecting the synsets that best fit their subjective opinion (Hajicˇ et al., 2004, p. 28). Table 5 provides minimum (m v), maximum (M v) and average values of agreement (A v) per type of synset assignment as well as standard deviations (S v). The A v values all fall within the strong tier of McHugh agreement (64-81%, corresponding to a square k agreement of .80-.90), but reveal that almost 13 of all synsets was not reliably
type OLD COM NEW
IRR agreement is a measure of both actual agreement but also of disagreement among raters. So, for a better understanding of the quality of both Minozzi’s and our own synset assignment, we calculated the acceptance rates of OLD, COM and NEW assignments. As Table 6 shows, the acceptance rates on all three types of assignment is very low, with an average 77% of all assignments being rejected by all raters and a tenuous average of 0.02% of unanimous acceptance. These results are particularly worrying for OLD and COM assignments, as they give us a first indication of the quality, and hence usability, of LWN.
Acceptance in % type OLD COM NEW
As far as Part of Speech (PoS) is concerned,
the most prolific syntactic category in terms of
machine-proposed synset assignments were verbs,
followed by nouns, adjectives and adverbs. Table
7 shows IRR agreement rates per PoS relative to
the number of synset assignments; x indicates the
average or arithmetic mean of synsets per lemma
(25 in total) per category. Nouns and verbs fared
the best, with strong agreement on large
percentages of assignments (84% and 60%, respectively);
adjectives, on the other hand, appear to have been
more challenging, as the percentages of
assignments on which the raters moderately and strongly
agreed are roughly the same (44% and 48%,
respectively). Finally, against our expectations,
despite the comparatively lower number of synsets,
48% of adverbial assignments were met with
moderate agreement. Low agreement values might
be caused by incorrect assignments (as was the
case of the NEW assignment ‘with honesty; he was
rightly considered the greatest singer of his time’
to proprie, approved by only one rater) or, more
problematically, differences of opinion on subtle
semantic differences. A close examination of the
data, and, specifically, of the adverbs with
agreement values below 60% (6 out of 25), points to the
latter. A clear example is given by the adverb
breviter, whose lowest rated assignment ‘with rapid
movements; he works quickly’ (COM) was
approved by two raters only. The adverb’s primary
sense is ‘shortly, in a brief space of time’, and
while ‘rapidly’ might, in some cases, reasonably
be equated to ‘shortly’, three raters discarded the
assignment as the senses conveyed by the terms
‘rapid’ and ‘quickly’ are better expressed by the
Latin adverb celeriter. Similarly, in the case of
subtus ‘below, underneath, in a lower position,
beneath’, the NEW assignment ‘at a later place; see
below’ was also rejected by three raters, despite
it being a potentially valid sense. It is worth
noting that in these and other arguable cases, synsets
carrying temporal meanings tend to show lower
agreement rates than those associated with space
(i.e., ‘rapidly’ and ‘later’ are temporal equivalents
of ‘short’ and ‘below’). The higher agreement rate
on the spatial dimension resonates with cognitive
linguistic theories on spatial semantics, according
to which “Space is at the heart of all
conceptualization”
Our synset recommender system produced many false positives, with only 0.18% (7) machine suggestions approved by all five raters. Even if the precision of the synset-recommendation algorithm were to be improved, recall would likely still be high due to the unavoidable assignment of modern senses to a historical resource. If applied to the entire LWN, the evaluation method described here, coupled with the additional evaluation of the relations between synsets, would turn this process of revision into an unsustainable effort or, at the very least, one that is not achievable within the scope and duration of LiLa.12
Moving forward, our plan for the improvement
of the LWN will develop into various tasks. The
first, ongoing effort is the manual removal of the
modern senses originally inherited by the LWN.
Next, once cleaned, we will extend the sense
coverage of the LWN by manually adding the
missing senses recorded by the raters for the 100
evaluated lemmas, careful not to introduce too much
granularity (i.e., too many senses with only
subtle semantic differences); extract hypernyms,
synonyms and bags of words from dictionary
definitions
13Available from: https://nikita-moor. github.io/dictionaries/dictionaries.html
Thirdly, connect a graph version of the LWN
to textual resources in LiLa to acquire lexical
knowledge, and explore the possibility of
extracting hypernym/hyponym pairs using syntactic
patterns
The data and code repository for this paper are available at: https://github.com/ CIRCSE/latinWordnet-evaluation
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme Grant Agreement No. 769994. The authors also wish to thank William Michael Short of the University of Exeter for the productive discussions leading up to this article, and the anonymous reviewers for their helpful suggestions.