Page layout analysis is a fundamental step in document processing which enables to segment a page into regions of interest. With highly complex layouts and mixed scripts, scholarly commentaries are text-heavy documents which remain challenging for state-of-the-art models. Their layout considerably varies across editions and their most important regions are mainly defined by semantic rather than graphical characteristics such as position or appearance. This setting calls for a comparison between textual, visual and hybrid approaches. We therefore assess the performances of two transformers (LayoutLMv3 and RoBERTa) and an objectiondetection network (YOLOv5). If results show a clear advantage in favor of the latter, we also list several caveats to this finding. In addition to our experiments, we release a dataset of ca. 300 annotated pages sampled from 19th century commentaries.
contents [
Visual approaches are often considered the standard way to go. This trend is probably
encouraged by the recent progress of pre-trained convolutional neural networks (CNN)
and by their ability to deal with non-textual regions. These approaches unsurprisingly
show their best performances in distinguishing regions with highly contrasting graphical
attributes, such as tables, illustrations and drop capitals. Yet, regions are often
characterised by semantic rather than graphical features. In this case, it makes sense to opt for
textual or hybrid approaches. If purely textual approaches prove their usefulness when a
page’s image is not aligned to its text or not available at all (e.g. [
In this paper, we focus on historical classical commentaries. We place them in the broader category of text-heavy documents as they mostly contain text, as opposed to more visual documents like illuminated manuscripts. The research project Ajax Multi-Commentary2 serves as the context for this work. It aims to create an automated pipeline to convert digitized commentaries into a body of structured information to aid in their comparative analysis. Within this pipeline, page layout analysis plays a crucial role as it can enable the (semi-)automatic markup of information contained within commentaries.
Together with critical editions and translations, commentaries are one of the main genres of publication in literary and textual scholarship. Providing in-depth analyses often side by side with a critical edition of the chosen text, commentaries can have very sophisticated layouts with considerable variations between editions (Fig. 1). A common layout type has the commentary section as a single or double column footnote section positioned below the primary text or its translation. In other layout types, however, commentary sections span over the entire page. Conversely, regions with similar placement and appearance can have different functions. Besides the complexity of their layout, commentaries also feature a specific prose style, clearly recognizable by its intertwining of multiple scripts and its pervasive use of abbreviations. Comments generally follow a determined pattern such as line number - commented word or excerpt - comment, for instance "1 (line number). ... (Excerpt): cp. Tr. 689-691. The passage in Aesch. Ag. 587-598 is scarcely a true parallel [...] (comment)".
As our project’s pipeline starts with commentary images and ends with text mining, we value page layout analysis as a crucial step in which primary text, margin notes, line numbers and commentary sections ought to be precisely segmented. This task remains a challenging one given the characteristics listed above. If information about layout is mainly conveyed by semantic rather than by graphical clues, visual features are not irrelevant. Commentary regions are generally written in a smaller font and are often punctuated by bold line numbers anchors. Besides, for Greek commentaries, the script lends to a good visual feature to differentiate between the primary text and its translation. 1.3. Goals In this challenging and mostly uncharted setting, our primary goal is to assess the performances of textual, visual and hybrid approaches. For each of these approaches, we ask the following questions: • RQ1: How well do state-of-the art models perform over commentaries of works written in different scripts (e.g. Latin or Polytonic Greek), belonging to different literary genres and having different layout types? Which of the textual, visual and hybrid approaches is best suited for the task? • RQ2: What is the impact of the quantity of training data? • RQ3: How well do models generalize on layout type they have not seen during training? For textual and hybrid approaches only, we address two additional questions: • RQ4: How do hybrid models perform on languages they have not been pre-trained on? • RQ5: To which extent do textual and visual features separately account for the model’s decision?
Generic approaches to page layout analysis have known considerable progress in recent
years. Overtaking CNNs, image transformers such as DiT [
Simistra et al. [
Mehri et al. [
Finally, Yang et al. [21] proposed a multimodal CNN to extract semantic structure
from documents. The principle is to build a text embedding map which is accessible
by the last layer of the model. Building on this idea, Barman et al. [
While ground truth datasets already exist for layout analysis of historical documents
such as manuscripts and early printed books [
To perform layout annotation we devise a content-based region taxonomy geared towards commentaries and critical editions (Fig. 2). It consists of 18 fine-grained classes, which are mapped to 8 coarse-grained classes in order to reduce the class number and class sparsity. Mapping is achieved by grouping region types with similar visual characteristics (e.g. numbers). The list of classes defined by our taxonomy is given in Table 1. For the experiments reported below, we exclusively consider coarse-grained classes.
This taxonomy distinguishes between the original Greek text of the work being commented upon (Primary text), the commentary sections (Commentary), the commentator’s translation of the commented text (Translation), the section containing information about manuscript readings and editorial conjectures (Critical apparatus), the paratextual elements — e.g. table of contents, appendices, indices, footnotes, introductory and prefatory materials — (Paratext) and finally page and line numbers (Number).
In order to make the published dataset
as widely reusable as possible, we mapped
our classes to the SegmOnto controlled
vocabulary [
As a sampling strategy, we started with ca. 40 pages of annotation per commentary. We made sure that all page layout types (see Fig. 3 for selective examples) of any given commentary are included in the sample because page layout can vary quite substantially throughout a commentary depending on the section contents.
The data used for experiments consist of an internal and an external dataset. The internal dataset comprises of commentaries to Sophocles’ Ajax, published from the beginning of the 19th century to date. Of these 12 commentaries, slightly less than a half are in the public domain, while the remaining are still under copyright. The external dataset, instead, consists of commentaries to both Latin and Greek classical works, sampled to include works both in prose and poetry. It contains an English commentary to Tacitus’ Annals (Latin prose), a German commentary to book 6 of Vergil’s Aeneid (Latin poetry), and a German commentary to book 7 of Thucydides’ History of the Peloponnesian War (Ancient Greek prose). The specific purpose of this external dataset is to evaluate with which accuracy layout analysis models trained on data from one specific genre and literature (i.e. Greek poetry, in the case of the Ajax) can be applied to commentaries about works from a wider variety of literary genres (see RQ6).
Given this important distinction, the ground-truth dataset we release contains the public domain portion of the internal dataset, as well as the entire external dataset (as it consists exclusively of out of copyright documents). Detailed statistics about these datasets can be found in Table 2.
LayoutLMv3 For hybrid experiments, we use LayoutLMv3 [
RoBERTa Provided the multilingualism of commentaries, it could have been relevant
to use a multilingual transformer for text-only experiments. However, as LayoutLMv3
uses RoBERTa [
YOLO +LayoutLM/RoBERTa This model combines the two approaches, using YOLO to detect regions and LayoutLM/RoBERTa to classify them. Words contained within predicted regions are fed to LayoutLM/RoBERTa. The majority class among words is then used to label the regions.
We implement our experiments using HuggingFace transformers6 and YOLO’s API7. Training was performed on two NVIDIA GeForce GTX TITAN X GPUS, each with 12.2GO of memory. The code is made publicly available on GitHub8.
As LayoutLM and RoBERTa are used for token classification, they should be evaluated on entity or word basis. However, in order to enable a meaningful comparison with YOLO, we group consecutive words with identical labels and build up a region from their bounding rectangle. Notice that LayoutLM and RoBERTa are severely disadvantaged by this evaluation procedure, as a single incorrectly labelled word among an actual region disrupts its unity. This problem is illustrated Figure 4 in the appendix and is not straightforward to mitigate without visual operations or carefully tailored rules. Indeed, homogenising long strands of tokens could result in the absorption of tiny regions like line numbers. We therefore evaluate the results without post-processing them and compute all mean average precision (mAP) scores at a 0.5 IoU threshold9. It is worth noting 4https://github.com/ultralytics/yolov5 5Despite multiple attempts, we couldn’t get Kraken’s (https://github.com/mittagessen/kraken) segmentation training to work on our infrastructures and therefore removed it from our experimental procedure. 6https://github.com/huggingface/transformers 7https://github.com/ultralytics/yolov5 8https://github.com/AjaxMultiCommentary/ajmc/tree/main/ajmc/olr 9We used the Python package mean-average-precision, https://github.com/bes-dev/mean_average_ precision, version 2021.4.26.0
Jebb - Base Kamerbeek - Base Jebb - Half trainset Jebb - Token ablation Jebb, Kamerbeck - base Jebb - Text only Campbell, Jebb - Transfer Kamerbeek, Jebb - Transfer Garvie, Jebb - Transfer Paduano - Base Wecklein - Base Omnibus (internal) Omnibus (external) Omnibus - Transfer
RQ
RQ1 Jebb Jebb RQ1 Kamerbeek Kamerbeek RQ2 Jebb Jebb RQ5 Jebb Jebb RQ3 Jebb, Kamerbeek Jebb RQ1&5 Jebb Jebb RQ3 Campbell Jebb RQ3 Kamerbeek Jebb RQ3 Garvie Jebb RQ4 Paduano Paduano RQ4 Wecklein Wecklein RQ1 All (internal) All (internal) RQ1 All (external) All (external) RQ3 All (internal) All (external)
Languages
en, gr
en, gr
en, gr
en, gr
en, gr
en, gr
en, gr
en, gr
it, gr
de, gr
en, de, it, lat, gr
en, de, lat
en, de, lat
that the obtained scores are approximately .10 mAP points below the scores produced
by YOLO’s built-in evaluation tool, a discrepancy already mentioned by [
We divide our experiments according to our research questions and list them in Table 3.
As using only textual features is consistently reported to yield lower results [
scribed in Section 1.3, our primary goal is to assess the performances of state-of-the-art models on classical commentaries and to investigate which of the three named approaches is the most appropriate for this kind of data.
Experimental design. As LayoutLM is pre-trained on English data, we first test its performances on two English commentaries: Jebb’s (baseline, experiment 0A) and Kamerbeek’s (0B). Besides its scholarly resonance, we chose Jebb’s commentary as a baseline because it contains regions of all coarse classes. As for Kamerbeek’s commentary, it presents an utterly different layout in which the commentary sections span over an entire page. 10See also https://github.com/bes-dev/mean_average_precision/issues/1
Additionally, we also train and test our models on a diverse set of commentaries on Sophocles’ Ajax (experiment 4A) and on other Greek and Latin prose and poetry works (4B). We then test visual approaches by running the same experiments with YOLO . Finally, we test textual approaches with RoBERTa using the same data as 0A (experiment 1E).
Results and Discussion. Results show a net advantage in favor of YOLO , which overtakes LayoutLM by an average of .27 points over experiments 0A, 0B, 4A and 4B. Interestingly and on the contrary to LayoutLM, YOLO completely misses footnotes in Jebb (N=8) and systematically incorporates them within the main paratext region. As for RoBERTa, its poor results are inline with previously mentioned studies showing the inferiority of text-only approaches. This first series of experiments shows that imagebased approaches can perform well even on region with few distinctive graphical features if they have seen similar layouts in training.
setting of experiment 0A, only changing the size of the training set by sampling half of it randomly (experiment 1A).
Results and Discussion. If both YOLO and LayoutLM show a performance drop in comparison with 0A, it is worth noting that depriving the former of half its training data only leads to a .05 decrease in mAP. The latter’s case is more concerning and deserves a more thorough inquiry. First, the model did not seem to be penalised by the number of epochs, as its maximum score is already attained at epoch 33/50. Secondly, the difference in mAP does not reflect the difference in word-based F1-score, which only decreases of .10. In-depth analyses revealed predictions to be much more scattered, which drastically hampers homogeneous region building and accounts for the plunge of mAP scores. The takeaway of this experiment is that 15 to 20 pages of ground-truth data already opens the way to satisfactory results, whereas doubling this amount only accounts for an improve of .05 mAP.
RQ3: How well do models generalize on layout types they have not encountered during training? We address the question of generalization in three ways. We first train a model on two commentaries and evaluate it on Jebb (baseline) to see whether mixing layout types can confuse the model. For this sub-experiment (1D) we use the commentaries by Jebb and Kamerbeek, two English commentaries with different layouts which we already have individual baselines for (cf. 0A and 0B). We then train three models on three English commentaries and evaluate them, again, on Jebb. We choose one commentary with an almost identical layout (Campbell, experiment 2A), and two commentaries with a completely different layout, Kamerbeek (2B) and Garvie (2C). In these two items, commentary sections cover the main zone of the page. Finally, we train a model on all internal commentaries and test it on external commentaries (4C).
Results and discussion. First, it seems that mixing two types of layout in training did not confuse the model. On the contrary, YOLO shows a .15 increase in mAP between 0A and 1D. This improvement is probably due to the quantity of available data, as regions such as running headers, paratext, numbers and footnotes see their number of training instances doubled and their scores consistently improved. Interestingly, this correlation is not present for commentary sections. Its AP remains at .90 despite a rise in from 40 to 66. This plateau can be explained by the important change in the region’s morphology between Kamerbeek and Jebb. More generally, this result suggests that using a single model with more data yields better results than individual models.
For experiments 2A, 2B and 2C, we generally observe a net decrease of performance when compared to the baseline. This being said, results are still better when generalizing to a similar layout type. Experiment 2A is therefore above 2B and 2C for both LayoutLM and YOLO . These results also hint at the fact that LayoutLM only seems to gain little information from the textual channel, a trend to be confirmed below. Performances also decrease in experiment 4C, despite the broadness of the training set. This result shows compelling evidence about the model’s struggles with completely unseen data. Indeed, if many of the layout types are present in the training, one must not understate the importance of other image features like the quality of the scan, the binarization threshold and so forth. To circumvent this problem, it is maybe sufficient to add very few images from the target data in training or fine-tuning. We keep this hypothesis to be tested in future works.
on? We then measure the impact of the text’s language by training two models on an Italian (Paduano) and a German commentary (Wecklein) respectively (experiments 3A and 3B).
Results and discussion. As it appears, LayoutLM does not seem to be impacted by the commentary’s main language. If results on German data fail to equate those of Jebb, Italian data gets the best results for a single commentary overall. These results can be explained by the domain-specific prose style of commentary writing. As mentioned in Section 1.2, the text often patches Greek scripts, abbreviations, rare words and proper nouns together. To circumvent this unusual distribution, LayoutLM’s tokenizer has to chunk words into extremely tiny pieces to match them to its vocabulary. It is therefore very frequent to see words fed to the model as sequences of single-character embeddings. This setting lessens the model capacity to rely on the knowledge acquired during pretraining and hence degrades its overall performances.
outLM’s decision? To measure this last statement more precisely, we run LayoutLM in token ablation mode (1C), feeding the model only with null tokens, thereby constraining its weights to rely solely on coordinates and images.
Results and discussion. RoBERTa’s poor results (1E) already indicate that LayoutLM is highly dependant on coordinates and images. Experiment 1C confirms this intuition and contributes to explaining the model’s indifference towards language. As a matter of fact, blanking textual inputs only diminishes the models performances by .01 mAP. In some regions with consistent positioning, textual inputs are even worsening the results: this is the case with commentary, critical apparatus and footnotes. However, the textual contents of page regions such as running headers and numbers do contain straightforward meaningful information. The former always contains identical words and the latter almost only consist of Arabic numerals. This could explain why token ablation deteriorates the model’s results in these two cases.
YOLO and YOLO +LayoutLM. With a single class to predict, YOLO unsurprisingly surpasses YOLO and displays very encouraging results. The model is above .9 in experiments 0B, 3A and 4B, generalizes better than its rivals and even reaches perfect mAP@0.5 for experiment 3B. Though these results can already be useful for other downstream tasks like pre-OCR region detection, we leverage YOLO ’s predictions and use them as a basis for LayoutLM, thereby addressing the problem of rebuilding regions. With an intriguing exception in experiment 0B, this method consistently improves LayoutLM. This result is coherent with caveats that come with our evaluation system (cf. Section 4.3). Rebuilding regions from labelled sequence can indeed lead to unwanted patchwork like schemes, as small nested clusters divide regions and build new ones. However, taking the majority class among labelled tokens in an already predicted region alleviates the harshness of region-based evaluation. This methods conveys two other remarks. First, though we applied token classification to enable a fair comparison with baseline settings, the fact that regions are predefined allows for implementing a sequence classification model, which could improve the results. Indeed, it may be tough for the model to correctly label a single page number token lost in a long sequence of text. However, classifying an isolated line or page number in a pre-defined region could be an easier task. As a second remark, it is worth recalling that if this approach remains inferior to YOLO in our case, it could prove to be more efficient with less domainspecific and noisy texts. Lifting this barrier could perhaps be achieved by the use of multilingual models such as LayoutXLM or by continuing the transformers pre-training on domain-specific data, an investigation we plan to pursue in future works. Inter- and intra-experiment variances For experiments 0A, 0B, 3A and 3B, we train a single YOLO model per commentary. Despite similar training parameters comparable amounts of data, we witness a strong variance between commentaries, with a gap of .18 between 0A and 0B. If this variance might be explained by layout particularities, we are also aware that it can be caused by the sparsity of the evaluation set. To acknowledge this limitation, we indicate the number of training and evaluation instance in Table 4. This sparsity also correlates with intra-experiment variance, i.e. differences between each
Our main contributions lie in our experiments and in the release of an annotated dataset. As part of our annotated data is still under copyright, we also release YOLO models trained on the entirety of our data in the hope that they might constitute a useful starting point for similar research 11. The key takeaways of this research are listed below: • We show that an object detection model such as YOLO succeeds in classifying semantic regions of text-heavy documents even if they feature little obvious graphical differences. • Hybrid models like LayoutLM may be of help to researchers working with clean and generic English data. However, in a highly noisy, multi-lingual, domain-specific and historic setting, they tend to make little use of the textual channel and mainly base their decision on coordinates and images. • With 8 classes in total, we show that annotating 15 to 20 pages of ground truth data already yields satisfactory results. Doubling this amount amends results by .05% in average.
Furthermore, because historical classical commentaries and critical editions have a
significant similarity in terms of layout, our approach establishes the groundwork for
developing a robust, generic model for page layout analysis of these publications in the
near future. Such a model, in combination with existing open source tools for annotation
that can be chained into a seamless pipeline (e.g. eScriptorium for annotation and
Kraken for OCR), has the potential to open up new perspectives to researchers for
exploiting openly available digitized editions and commentaries. Similarly, such a model
could be useful to projects aimed at the creation of large-scale corpora of marked up texts
such the Free First Thousands Years of Greek (FF1KG) project [
As for future works, we think two strands of possible improvement are worth investigating. First, and as mentioned in Section 5, we would like to explore the effect of adding minimal in-domain data when fine-tuning a generic model. Based on our experiments (notably 1A), our hypothesis is that providing only a few pages should improve the results on unseen commentaries. Secondly, we think our hybrid would yield better results if they could use more meaningful representation of the text. It could therefore be worth testing with a multilingual model such as LayoutXLM or with a domain-specific language modelling pre-training.
This research has been supported by the Swiss National Science Foundation under an 11https://github.com/AjaxMultiCommentary/layout-yolo-models Ambizione grant PZ00P1_186033. We thank Carla Amaya who helped in the ground truth annotation process. [21] X. Yang, E. Yumer, P. Asente, M. Kraley, D. Kifer, and C. L. Giles. Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network. 2017. url: http://arxiv.org/abs/1706.02337.