Large-scale synthetic research in ancient history is often hindered by the incompatibility of taxonomies used by diferent digital datasets. Using the example of enriching the Latin Inscriptions from the Roman Empire dataset (LIRE), we demonstrate that machine-learning classification models can bridge the gap between two distinct classification systems and make comparative study possible. We report on training, testing and application of a machine learning classification model using inscription categories from the Epigraphic Database Heidelberg (EDH) to label inscriptions from the Epigraphic Database Claus-Slaby (EDCS). The model is trained on a labeled set of records included in both sources (N =46,171). Several diferent classification algorithms and parametrizations are explored. The final model is based on Extremely Randomized Trees algorithm (ET) and employs 10,055 features, based on several attributes. The final model classifies two thirds of a test dataset with 98% accuracy and 85% of it with 95% accuracy. After model selection and evaluation, we apply the model on inscriptions covered exclusively by EDCS (N =83,482) in an attempt to adopt one consistent system of classification for all records within the LIRE dataset.
A principal goal of digital scholarship is to produce new insights through the aggregation and
synthesis of many sources [cf. 31, 3, 24, 28, 26, 25, 10]. Our ability to address fundamental
historical questions, such as the waxing and waning of cities and civilizations, depends on our
capacity to efectively reuse and integrate large evidentiary datasets [
2.1. EDH
The Epigraphic Database Heidelberg (EDH) represents a flagship resource for the field of Latin
digital epigraphy. It has been in development for 35 years, providing meticulously curated
content and built with the consideration of open-research needs such as easy accessibility
and reuse [
The EDH dataset is programmatically accessible via the public API [
The LIRE dataset represents an aggregate of the streamlined and enriched EDH and EDCS datasets (as published on Zenodo). The process of aggregation and filtering included several steps: • mapping and deduplication of the records included in both sources, using the EDH-ID listed in the ‘links’ attribute of EDCS and linked data in the Trismegistos TexRelations API; • filtering for records with valid geospatial data, represented by a pair of coordinates • filtering for records that fall within the boundaries of the Roman Empire at its largest extent (under Trajan in AD 117; as delimited by the Pleiades project shapefile ); • filtering for records containing temporal information in the form of a temporal interval of creation, expressed in years and stored in attributes ‘not_before’ and ‘not_after’; • filtering for records whose temporal interval of creation intersects with the timespan of the Roman Empire (arbitrarily set to 50 BC through AD 350).
The deduplication and filtering reduced the number of records in the aggregate substantially:
the initial 500,000+ records in EDCS and 81,000+ records in EDH produced 137,305 records
that had a valid date and location within the boundaries of the Roman Empire. The resulting
LIRE dataset contained 49,916 inscriptions shared by the EDH and EDCS, inheriting attributes
from both parent collections. In addition, there were 3,907 inscriptions recorded exclusively in
EDH and 83,482 inscriptions originating solely from EDCS, containing set of attributes only
from the parent dataset. After combining the valid unique records from the two sources and
appending their attributes, we sought to integrate the attributes. The following attributes were
shared and used consistently across the two sources, and could be combined in a straightforward
way, with data originating in EDH taking precedence over EDCS whenever the attributes
overlapped (cf. [
Some of the attributes shared by both sources, however, could not be easily integrated such as the EDH attribute ‘type_of_inscription_clean’. Type of inscription is an interpretive category that labels the function of an inscription following one of established domain typologies. Epigraphers classify inscription function after evaluating its content, context, and physical form during the first publication. As type definitions are broad and ambiguous, label assignment is subjective and may fluctuate through time.
The ‘type_of_inscription_clean’ column in EDH implements the controlled vocabularies of
the EAGLE Europeana Project, a standardized list of inscription types created in 2013-2015
to tackle the vagueness of existing typologies [
EDCS stores the type of inscription information in an attribute ‘status_list’ together with other information extracted from the text of inscriptions, such as the social status of persons named in the text, e.g. slaves, priests, or high-ranking officials. We extracted the type values into a separate ‘inscr_type’ attribute. The resulting column, however, was often multivalued and followed a diferent typology than EDH, relying on Latin labels and referring to non-overlapping categories. The category ‘owner/artist inscription’ in EDH, for example, corresponds both to ‘tituli possessionis’ and ‘tituli fabricationis’ in EDCS, hampering dataset-wide comparison. To overcome such limitations and have a consistent typology applied across the entire dataset, we reclassified the inscriptions from EDCS using the inscription type categories from EDH.
All scripts used for aggregation, filtering and enriching of the LIRE dataset are available
on GitHub. The repository includes training, evaluation, selection, and application of the
classification model introduced below. The final version of the dataset is also published via
Zenodo [
We trained a machine-learning classification model using the EDH-labeled subset of inscriptions with attributes from both sources. From the 49,916 inscriptions which are shared by both the EDH and EDCS dataset, 46,171 records are properly labeled, i.e. classified using the ‘type_of_inscription_clean’ attribute in EDH. These were used for training of the classification model. The remaining 3,745 inscriptions were classified as ‘NULL’ and thus were excluded from the training.
There are 22 unique classification categories among EDH labels. Their distribution is highly imbalanced (see Table 4). The three most common categories in the training set are ‘epitaph’ (N =21,520), ‘votive inscription’ (N =11,728), and ‘owner/artist inscription’ (N =3,340). The three least common are ‘assignation inscription’ (N =15), ‘calendar’ (N =10), and ‘adnuntiatio’ (N =1).
For a preliminary model selection, we compared outcomes of several supervised machine
learning algorithms commonly used for document classification [
For training of the models, we combined features extracted from several diferent EDCS
attributes, namely:
• ‘status_list’, including information about the inscription category according to EDCS
combined with other metadata (‘status/titulorum distributio’ in EDCS)
• ‘Material’, containing information about the predominant material or medium on which
the inscription is found
• ‘clean_text_interpretive_word’, text of the inscription without the Leiden Conventions
for editorial markup of texts, see [
The content of the three attributes was extracted independently, preprocessed, and then
combined together into a ‘bag-of-words’ model to feed a tfidf vectorizer [
To evaluate performance of each model variant, we relied mainly on weighted variant of the F1 score, referred below as F1(w). The F1(w) score is the harmonic mean of Precision (proportion of every observation predicted to be positive that is actually positive) and Recall (proportion of every positive observation that is truly positive). The weighted variant of the F1(w) score metric, which is consistently reported below, takes into account label imbalances: for each label, the average value of both metrics is weighted by the number of true instances. In some cases, we also report accuracy, which equals to the proportion of correctly classified records, what makes it very intuitive. But it has to be taken with reservation here, since it does not take into account class imbalances.
For a preliminary model evaluation, we explored diferent combinations of the above mentioned feature groups and trained and tested diferent models on a subset of 5000 labeled records (80 % for training and 20 % for testing, using stratified k-fold cross-validation method). For instance, using the 37 features based on the ‘status_list’ attribute only, LR resulted in F1(w)=0.724. The performance of the model slightly improved when we added 18 features based on the ‘Material’ attribute (F1(w)=0.734); but the model improved substantially when we included the bigrams, from F1(w)=0.786 for the 100 most frequent bigrams up to F1(w)=0.832 for the 10,000 most frequent bigrams (the reported F1(w) is an average based on 5 stratified cross-fold validation tests). Thus, for the subsequent model selection we employed the features set including 10,000 bigrams.
Table 1 shows the diferences in performance of the above introduced classification algorithms trained on a subset of 4,000 randomly chosen inscriptions and the above specified features set. In the case of LR and SVM, we tested diferent settings of the C parameter, which stands for inverse regularization strength. In the case of RF and ET, we explored several diferent values for the number of estimators. We see that the best results are achieved by LR (C =1000) and ET (n_estimators=100).
Drawing on these results, we continued with LR and ET only and trained them on the full dataset. In this setting, ET significantly outperforms LR, with F1(w)=0.878 over F1(w)=0.867 (the reported F1(w) is an average based on 10 stratified cross-fold validation tests). On the basis of these results, we continued with the ET model which we also saved for future reuse.
Each prediction of the model is accompanied by probability on scale 0-1, expressing a level of certainty concerning the predicted classification category. The resulting probabilities might be used to formulate thresholds under which the classification will not be accepted. In Table 2, we see that 96 % of inscriptions in the test dataset were classified with probability equal toor higher than 0.4 and that this classification was correct in more than 90 % of cases (see the ‘accuracy’ score column). Further, approximately 85 % of inscriptions have been classified with probability equal to or higher than 0.6. From these more than 95 % were classified correctly. These are important observations, which might be used later on when we apply the model upon unlabeled data, where we can expect comparable ratios between threshold values, proportions of covered inscriptions, and the extent of correct classifications.
However, before we proceed to apply the model on unlabeled data, we have to evaluate the performance of the model with respect to individual categories. For that purpose, we generated a precision table in Figure 1, where we see the model’s accuracy with respect to 10 most common categories. We see that the accuracy difers from category to category. In case of 4 categories (‘epitaph’, ‘votive inscription’, ‘mile-/leaguestone’, and ‘defixio’) the model correctly classifies 98 % or more records. For instance, it correctly classifies all 18 instances of ‘defixio’ in the test set. In the case of other categories, the performance is much worse: e.g. from 4 instances of ‘list’, 3 are incorrectly classified as ‘epitaph’. The ambiguity of ‘list’ definition in the EAGLE vocabularies likely causes the inconsistent manual markup in EDH and the poor reclassification performance.
After tuning, training, and testing the model on labeled data, we proceeded to apply the model on 83,482 inscriptions which are recorded exclusively in EDCS. Further, employing the 0.6 probability threshold, we accepted the automatic labels for 82 % of inscriptions in the dataset. To estimate the proportion of correctly classified inscriptions within this subset, we generated a random sample of 100 inscriptions. The sample was labeled manually by two domain experts. The first expert was drawing on the same attributes as the automatic classifier. In this case, the manually and automatically assigned labels were in agreement in 94% of cases. Another domain expert manually labeled the data without taking into consideration the ‘status_list’ attribute. In this case, the agreement with the automatically assigned labels was approximately 88%. Combining this with the results from the test set, we estimate between 90 and 95 % of When we look at the LIRE dataset as a whole, we see that from the 137,305 inscriptions, 117,710 (85 %) are classified in ‘type_of_inscription_auto’ with probability equal to- or higher than 0.6. In the following overview of LIRE, we use this probability as a cut-of threshold, under which the automatically assigned categories are not accepted and the corresponding records are excluded from the cross-category comparison.
blocks. The width of the resulting curve, which changes from time block to time block, reflects
the extent of temporal uncertainty in the underlying data. Adopting this approach, we are
able to assess temporal trends in the datasets and to compare the diferences in temporal
distributions of individual inscription types. For instance, we see that the temporal distribution
of epitaphs changes substantially if we use the EDH dataset versus the LIRE dataset, which
contains a larger number of epitaphs inherited from EDCS. It reveals that the spike in the
production of epitaphs in EDH (a) in the second half of the second century AD disappears
once we include the epitaphs from EDCS (c). This trend is only apparent after the automatic
reclassification, as the original classification systems were mutually incompatible and not yet
mapped onto a single ontological system [
Many data collections in small-science disciplines are fragmented among numerous content
silos. Scholars wishing to synthesise these fragments need to ensure their analytical
comparability, specifically column- and value-level consistency. Such consistency has been achieved
in the past through semi-automatic mapping to relevant ontologies (see tDAR example in
[
Despite the relatively high accuracy rates reported above, there is still the probability that
every 20th automatically classified inscription is classified erroneously. Imperfections in the
training dataset due to the ambiguity of inscriptions likely contribute. We can also look to
other studies for guidance. Survey pottery specialists, for example, point to ambiguity
surrounding the interpretation of type and chronology in artefacts that sufer from high wear and
fragmentation [
Finally, when integrating two typologies, the choice will always entail a compromise that carries with it some limitations. Inspecting the LIRE inscriptions by type across the longterm in Figure 2(c), most major trends from EDH (Figure 2a) and EDCS (Figure 2b) are preserved (e.g. dominance of epitaphs and votives) with subtle alterations to trajectories due to the combined data. Detailed labels from EDCS ‘tituli fabricationis’ and ‘tituli possesionis’ have been combined into one overarching category ‘owner/artist inscription’, which might be a problem if the originals have special value for the next researcher. The one category that disappears is the ‘tituli christianae’, utilized by EDCS but absent from EDH. It was relabelled as ‘epitaph’ in the LIRE dataset, causing a secondary rise associated with this category around AD 300. This may be a loss for scholars of Early Christianity, but represents a move towards greater consistency from a cultural label towards a functional description. While LIRE accomplishes our needs of comparability, the approach is flexible and the selection of classification can be flipped around should other scholars need it.
This research was funded by the Aarhus University Forskningsfond Starting grant no. AUFFE-2018-7-22 awarded to the ‘Social Complexity in the Ancient Mediterranean’ (SDAM) project.