English. This paper describes our contribution to the EVALITA 2020 shared task DaDoEval - Dating Document Evaluation. The solution we present is based on a linear multi-class Support Vector Machine classifier trained on a combination of character and word n-grams, as well as number of word tokens per document. Despite its simplicity, the system ranked first both in the coarse-grained classification task on same-genre data and in the one on cross-genre data, achieving a macroaverage F1 score of 0.934 and 0.413, respectively. The system implementation is available at https://github.com/ matteobrv/DaDoEval.
Temporal information, such as the publication date
of a document, is of major relevance in a number
of domains, like historical linguistics and digital
humanities
Such information, however, is not always
readily available and must therefore be inferred,
relying either on qualitative or quantitative methods,
if not both
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). is most likely bound to change as the increasing availability of large-scale, time-annotated digital resources, such as Google n-grams1, is promoting research in this direction. Two recent examples of this new trend, in line with the present task, are the Diachronic Text Evaluation shared task organised by Popescu et al. (2015) at SemEval 2015 and the RetroC Challenge presented by Gralin´ski et al. (2017).
In this work we propose a simple, yet effective, approach for automatic document dating based on a linear multi-class Support Vector Machine classifier, trained on a combination of character and word n-grams, as well as document length in word tokens.
The solution is evaluated in the context of
the DaDoEval – Dating Document Evaluation –
shared task at EVALITA 2020
The proposed solution tackles the first two subtasks, coarse-grained classification on same-genre and cross-genre data. Both sub-tasks require to correctly assign document samples to one of the main five time periods identified in De Gasperi’s political life, spanning a range of over fifty years from 1901 to 1954.
The paper is structured as follows: in section 2 we provide a brief overview of the training data set, in section 3 we go over the system setup and describe the feature space, section 4 is dedicated to results analysis and discussion, in section 5 we 1http://books.google.com/ngrams consider possible improvements while section 6 is reserved for final remarks. 2
The training data set released for the shared task
includes 2,210 document samples extracted from
the Alcide De Gasperi’s corpus of public
documents, a multi-genre collection of 2,759 texts
written or transcribed between 1901 and 1954
With respect to the coarse-grained classification sub-tasks, the given samples are organised into five classes (see Table 1) corresponding to the main time periods historians identified in De Gasperi’s political life: Habsburg years 19011918, Beginning of political activity 1919-1926, Internal exile 1927-1942, From fascism to the Italian Republic 1943-1947, Building the Italian Republic 1948-1954.
A preliminary analysis of the data set reveals an imbalanced class distribution, with a significantly lower number of samples in the third class, corresponding to the 1927-1942 interval. This, however, is partially mitigated by the markedly higher average number of word tokens per sample observed in this class compared to the other ones. 3
The proposed solution is based on a Support
Vector Machine (SVM) classifier implemented using
the Scikit-learn library
To account for the rather imbalanced data set, the SVM is tuned in such a way that classes are assigned weights inversely proportional to their frequency in the input data.
Following the assumption that most text
categorisation problems are linearly
separable
The system relies solely on the data provided by the task organisers and is split into training set (80%) and development set (20%). No preprocessing is applied, as measures such as case normalisation and punctuation removal do not seem to improve the classification result on the development set, but rather to worsen it.
Each document in the data set is represented
using three sets of features: document length in
terms of word tokens as well as character and word
n-grams. In this respect, we explore the idea that
SVMs trained on combinations of character and
word n-grams are particularly effective in tackling
text classification tasks
Character n-grams are extracted for n 2 f3; 4; 5g and span across word boundaries, thus capturing punctuation and space characters occurring at the beginning and at the end of each word token. Word n-grams, on the other hand, are extracted for n 2 f1; 2g. Both feature sets are weighted using term-frequency, inverse-document frequency (TF-IDF) to scale down the impact of the most frequent n-grams.
The number of word tokens per document is
computed in a naive way, splitting each sample at
every white space. Similarly to n-gram features,
tokens count are scaled down to a 0-1 range in an
attempt to avoid numerical problems and prevent
features in higher numeric ranges from
dominating those in smaller ones
The system hyper-parameters are optimised to obtain the best F1 score on the development set.
A subset of the hyper-parameters is tuned empirically through several experiments or on the basis of existing literature. This is the case for kernel type, decision strategy, class balancing, tolerance for stopping criterion (tol) and n-grams size.
The remaining hyper-parameters considered during optimisation are the regularisation parameter (C) together with the maximum and minimum document frequency (max df, min df), which in the present approach are used to set an acceptance threshold for high and low frequency ngrams.
max df min df ngram range lowercase
max df min df ngram range lowercase SVM kernel decision function tol C class weight word 0.9 0.004 (1, 2) False char 0.3 0.001 (3, 5) False linear ovo 1e-12 0.881 balanced
These hyper-parameters are tuned through
the BayesSearchCV algorithm implemented
in the scikit-optimize library
Table 2 summarises the best hyper-parameters setup obtained from the tuning process. 4
In this section we present the results for the two sub-tasks the system participated to. Results are summarised in Table 3 and reported in terms of macro-average F1 score.
The system ranked first both in the same-genre and in the cross-genre coarse-grained classification task, obtaining a macro-average F1 score of 0.934 and 0.413, respectively.
same-genre
matteo-brv cross-genre matteo-brv team 1 baseline team 1 baseline team 1 The runs submitted for the first sub-task are based on test samples of the same genre as the ones in the training set. The system scored well above the baseline, which was computed with a Logistic Regression model trained on TF-IDF-weighted word unigrams, without performing any preprocessing.
Overall, the results registered on the test set are in line with those observed during training. This is confirmed by the data summarised in Table 4 and by the confusion matrix in Figure 1.
The confusion matrix depicts a run on the development set which achieved a macro-average F1 score of 0.95, while Table 4 reports the perclass results of the best test run submitted for the sub-task. In both cases 1919-1926, 1943-1947 and 1948-1954 are the classes showing the highest number of misclassifications and, incidentally, are also the ones corresponding to the shortest time periods.
1919-1926 1927-1942 1943-1947
Predicted label
The runs submitted for the second sub-task are
based on samples coming from a cross-genre,
outof-domain test data set. These samples are a
subset of the documents collected for the Epistolario
project
Results for the same-genre task are quite
encouraging and in line with those obtained on the
development set, where the F1 score ranges between
0.92 and 0.96. However, with the current data
and setup, there might not be much room for
further improvement. Nonetheless, additional
features like richness measures and linguistically
motivated features (e.g. POS tags) are explored in
other contributions
On the other hand, results for the second
subtask suggest a lack of generalisation on
crossgenre, out-of-domain data. In this respect, even
though SVM-based systems for text classification
should be able to perform well and take
advantage of high dimensional feature spaces
In this paper we describe a simple, yet effective, approach for automatic document dating implemented for the DaDoEval shared task at EVALITA 2020. The system is based on a linear Support Vector Machine and is trained on a small set of stylistic and lexical features, resulting in a fast and efficient classification model.
In particular, the approach achieves top scores in both coarse-grained classification sub-tasks, thus confirming that SVM-based systems trained on character and word n-grams are indeed well suited to tackle text classification problems.
Nonetheless, results observed in the second task suggest that the model does not generalise well on cross-genre data, leaving room for further improvements.
As expected, despite scoring above the baseline, cross-genre results are significantly lower than those obtained in the same-genre task. Perclass results summarised in Table 5 show how promising system performances registered in the same-genre task do not transfer to the cross-genre one, suggesting a poor ability of the model to generalise. Particularly interesting and worth investigating are the results registered for the third class, corresponding to the 1927-1942 interval. With respect to this class precision and recall values are equal to 0, indicating that model did not recognise any sample as belonging to this time period.
We thank Dr. C¸ ag˘ rı C¸ o¨ ltekin for his patient encouragement and valuable suggestions throughout this project.