English. This article describes a computational text reuse study on Latin texts designed to evaluate the performance of TRACER, a language-agnostic text reuse detection engine. As a case study, we use the Index Thomisticus as a gold standard to measure the performance of the tool in identifying text reuse between Thomas Aquinas' Summa contra Gentiles and his sources.
Thomas Aquinas (1225-1274) was a prolific
medieval author from Italy: his 118 works, known
as the Corpus Thomisticum, amount to 8,767,883
words
1. Our English translation reads: ‘It will be even harder to automatically compare all of Thomas against all of the texts of one or multiple authors to check for the presence of implicit sources. Such a task would only be possible under two conditions: firstly, the texts of the authors quoted by Thomas would have to be digitised and searchable in their entirety; secondly, one would need very powerful and sophisticated software’. 2.1
Text reuse (TR) can be summarily described as
the written repetition or borrowing of text and can
take different forms. Bu¨chler et al. (2014)
separate syntactic TR, such as (near-)verbatim
quotations or idiomatic expressions, from semantic TR,
which can manifest itself as a paraphrase, an
allusion or other loose reproduction. The study of
quotation is key to any philological examination
of a text, as it is not only indicative of the
intellectual and cultural endowment of an author, but
may shed light on the sources used, the relation
between works and literary influence. Crucially,
quotations may also preserve text that is now lost,
thus facilitating efforts of textual reconstruction. 2
Owing to the magnitude of the task, the
publication of a work’s complete index of references,
conventionally known as Apparatus fontium or
Index scriptorum, is rare
Like many of his Christian predecessors,
Aquinas’ body of work teems with references to secular
and Christian literature alike. In the ScG
(12591265) Aquinas cites 170 works both explicitly and
implicitly
3. For problems with implicit quotations, see
4. “Da Thomas die Schriften des Aristoteles [. . .] gewo¨hnlich nur dem Gedanken nach, nicht wo¨rtlich anfu¨hrt.” In English: ‘Since Thomas usually quotes paraphrastically, not literally.’
Roberto Busa’s effort in the late 1940s
resulted in the creation of the Index Thomisticus, a
manually-lemmatised version of Thomas
Aquinas’ opera omnia
The (QL) portion of this example contains the literal quote, while the second (QR) portion provides the reference. 2.3
HTRD is a Natural Language Processing (NLP)
task aimed at identifying syntactic and semantic
TR in historical sources. The computational
analysis of historical languages is particularly
challenging as tools at our disposal are often trained
on a synchronic rather than diachronic state of
a language 6 and on controlled textual corpora.
Eger et al. (2015) and Passarotti (2010) tested
the performance of seven different taggers,
including TreeTagger
The extent to which the limitations of these
resources and taggers (e.g., correct resolution of
homographs) affect HTRD tools, including
Tesserae
6. See Janda and Joseph (2005) for the dichotomy.
7. https://github.com/dasmiq/passim
fully understood. Reasons for this are the
field’s lack of progress caused by “inconsistent
standards and the scattering of insights across
publications”
To facilitate the classification of automaticallydetected reuse, all QL-, QS- and QR-annotated tokens were extracted from the Index Thomisticus. Of the total 24,416 sentences constituting the ScG, the 7,396 (30:29%) containing any combination of QL, QS and QR were stored in a tabular file, which we define as the Index Thomisticus Gold Standard of TR (hereafter IT-GS). The number of sentences containing only QL tokens (1,139) compared to that of sentences containing only QS tokens (2,270) corroborates expert assertions about Aquinas’ paraphrastic style of TR. 3.2
For the sake of processing efficiency, out of the
ScG’s 170 source works we began with a set of
five readily available texts. These are Philosophiae
Consolationis and De Trinitate of Boethius, De
Deo Socratis of Apuleius, Cicero’s De Divinatione
and the Moerbeke Latin translation of Aristotle’s
Metaphysica. The texts were acquired from
different sources and cleaned of all paratextual
information. The clean texts were then
segmentised by sentence, PoS-tagged and lemmatised with
the TreeTagger Brandolini parameter file (with an
average accuracy of 93:72%), whose tag-set
provides the degree of granularity needed in this
experiment. 8 Finally, a script was used to format
sen8. The Brandolini tag-set was manually mapped against
that of Morpheus
The HTRD on this corpus was performed (server-side) with TRACER, a language-agnostic framework comprising hundreds of information retrieval (IR) algorithms designed to work with historical and modern languages alike. 9 TRACER is a Java command-line tool driven by an XML configuration file, which users can modify to fit their detection needs. TRACER follows a sixstep architecture, 10 which demystifies the detection process by storing the computed output of each step on the disk so that users can more easily follow and locate errors in the processing chain, if any. TRACER is resilient to OCR-noise and capable of detecting both (near-)verbatim quotations and looser forms of TR. The detection of paraphrase requires the use of linguistic resources to help TRACER match a word against its synsets and an inflected form against its base-form. For synonym detection, we extracted synonymous relations from the Latin WordNet. TR identified with TRACER was manually compared against the ITGS to separate the True (TP) from the False Positives (FP), and to identify False Negatives (FN). 4 4.1
To detect both verbatim quotations and paraphrase, TRACER was optimised for recall over precision and configured to work with single words as features, to ignore the top 20% most frequent words, 11 to link text pairs with a minimum overlap of 5 features, 12 to expand the query to synonyms, and to return only those aligned text pairs presenting an overall sentence similarity of at least 50%. 13 Of the eight reuses indicated in the Editio Leonina, we were unable to precisely locate one as it alludes to four paragraphs of text; 14 of the remaining seven, as shown in Figure 1, TRACER identified three (42%). Upon close inspection, two FNs were affected by the 20% threshold of feature removal, for example: Boethius 1.4.105 Unde haud iniuria tuorum quidam familiarium quaesivit: “Si quidem deus”, inquit, “est, unde mala ? 15 Aquinas 3.71.10 , introducit quendam philosophum quaerentem: si deus est, unde malum ? 16
Here, the tokens si, est and unde were ignored as they fell within the pool of the 20% most frequent words removed.
One reuse was successfully identified on the basis of feature overlap but did not amount to a 50% sentence similarity; and the fourth reuse could not be identified because of a missing synonymous relation in the Latin WordNet (i.e., gaudiumbeatitudo) 17 and its insufficient feature overlap. The resulting F1-score is 4; 6 10 3. 4.2
Given the results of the previous analysis, for
this second investigation the feature removal and
the sentence similarity values were lowered to
10% and 40% respectively, thus optimising for
even higher recall (10,349 total sentences aligned).
Of the four known reuses, TRACER identified
three. The 40% similarity threshold was essential
to the identification of one reuse (where the score
is 0.4375); the FN, which was indeed found on the
basis of an eight-word overlap but did not meet
the minimum sentence similarity threshold,
revealed another missing synonymous relation in the
WordNet (i.e., disciplinatus-eruditus) 18 and a
failed alignment of the variants temptare (Boethius)
and tentare (Aquinas) owing to inconsistent
TreeTagger lemmatisation (tempto and tento,
respeclength
14. This reuse would have doubtless been overlooked by TRACER too owing to the absence of features to compare.
15. Our English translation reads: ‘It is not wrong that a certain acquaintance of yours has questioned: ‘If in fact God exists,’ he asks, ‘where is evil from ?”
16. Our English translation reads: ‘(Boethius) introduces a certain philosopher who asks: ‘If God exists, where is evil from ?’.’
17. Incidentally, this relation is also not mapped in BabelNet (bn:00042905n) nor in ConceptNet (http:// conceptnet.io/c/la/gaudium) (as of 8 June 2018). 18. Also not present in neither BabelNet nor ConceptNet. tively). The F1-score for this analysis was 5; 6 10 4. 4.3
This work of Apuleius is quoted twice in the ScG. Of the two reuses, TRACER was able to detect one in full and only parts of the second. The second reuse spans three sentences and is mostly paraphrastic, with only three words annotated in the Index Thomisticus as QL (sunt animo passiva). 19 To capture the fullest range of reuse diversity, TRACER’s feature removal was set to 10%, the overlap to 3 and the overall similarity to 20%. However, as sunt (form of the verb sum ‘to be’) is the most frequent word across the texts, TRACER’s inbuilt feature removal prevented the detection of the short QL portion of the reuse; the QR+QS portions, on the other hand, were successfully detected. We counted both results as TPs, resulting in an F1-score of 2; 6 10 5. 4.4
The only recorded reuse that Aquinas makes of Cicero’s text is implicit and alludes to a block of text, making it difficult to manually pinpoint with precision. To detect as loose a similarity as possible, the TRACER search was cast with the same configuration used in the previous analysis. No reuse, however, was found. 4.5
The Editio Leonina lists 97 reuses of Aristotle’s Metaphysica. As previously mentioned, Pelster describes Aquinas’ reuse of the Latin translation of the Metaphysica as more paraphrastic than literal. Our manual examination of the texts and the results of TRACER confirmed this observation, in that we could not manually locate seven reuses (due to their strong allusiveness) and a fault-tolerant TRACER configuration (removal of the top 10% most frequent words, overlap of 3 features and an overall sentence similarity of 40%) yielded 19 TPs only (6 out of 15 QL 20 and 13 out of 75 QR+QS). The F1-score resulting from this analysis is 3; 8 10 4.
19. [daemones] [. . .] sunt animo passiva or ‘demons are
emotional in mind’
20. The QL quotations in the ScG seem to refer to a different Latin translation than that available to us, which would explain why some instances of QL went undetected.
Our results show that the FNs emerging from the computational analyses were largely caused by Aquinas’ paraphrastic and allusive TR style, which at times challenged our own ability to spot similarities, even with the help of the critical edition. The allusions that we could identify generally retain the semantics of the alluded-to texts, thus confirming Durantel’s insights. While a number of these negative results were also directly tied to lacunae in the Latin WordNet and to inconsistent lemmatisation, the flexibility and methodological transparency of TRACER allowed us to locate error sources and accordingly tune configurations to work around these issues (e.g., by increasing the feature overlap and/or lowering the sentence similarity scoring thresholds). Notwithstanding, TRACER’s panlingual feature removal parameter affected the retrieval of shorter instances of reuse, particularly those containing forms of the highly frequent verb sum.
The manual evaluation of TRACER results against the IT-GS for the creation of an Index fontium computatus was time-consuming, not least because of a number of reference inaccuracies in the critical edition itself (in one case, the reference is off by ten lines). Nevertheless, the creation of the index is proving essential to the assessment of TRACER’s fitness for purpose on Latin texts.
As far as the usability of the tool is concerned,
TRACER’s detection power is offset by its
cumbersome setup, which is unfriendly to those who
are not familiar with the command line, NLP
basics and/or Java (stack traces). This issue is being
addressed with the development of a user manual
This article describes a computational text reuse
study on Latin texts designed to evaluate the
performance of TRACER, a language-agnostic IR
text reuse detection engine. The results obtained
were manually evaluated against a gold standard
and are contributing to the creation of an Index
fontium computatus to both assess TRACER’s
efficacy and to provide a test-bed against which
analogous IR systems can be measured and thus
compared to TRACER. Our study shows that despite
the known limitations of existing linguistic
resources for Latin, the diverse spectrum of
paraphrastic reuse encountered and its own
languageagnosticism, TRACER is equipped to detect a
wide range of explicit text reuse in the ScG, be
that short or long, verbatim or paraphrastic, and
implicit reuse only if coupled with explicit. To
increase the detection accuracy, we are
implementing a black/white list to give users the power
to control words or multi-word expressions to be
ignored or retained in the detection; furthermore,
we plan on re-running these analyses with the
disambiguated linguistic annotation currently being
added to the text of the ScG
The data used and generated in the current study is available from: https://github. com/CIRCSE/text-reuse-aquinas.
The authors would like to thank Eleonora Litta for proofreading this article and the anonymous reviewers for their valuable comments. This research was funded by the German Federal Ministry of Education and Research (No. 01UG1409). David Bamman and Gregory Crane. 2008. The Logic and Discovery of Textual Allusion. In Proceedings of the ACL Workshop LaTeCH - Language Technology for Cultural Heritage Data. ACL. http: //hdl.handle.net/10427/42685.
Clovis Gladstone and Charles Cooney. forthcoming.
Opening New Paths for Scholarship: Algorithms to Track Text Reuse in ECCO. Digitizing Enlightenment.