The recent growing interest in low-resource languages has been significantly bolstered by transformer-based models. By ifne-tuning three such models, two based on BERT and the other on RoBERTa, I aim at geolocating sequences exhibiting non-standard language varieties relying solely on linguistic content. I find that, given that the information contained in the embeddings is all we need to carry out this complex task, a model architecture with less task-specific layers leads to better results. Furthermore, models pre-trained on miscellaneous corpora generalize better than those trained exclusively on tweets. The work also shows that the greater availability of resources of a certain regional variety positively afects the capacity of the model.
1.1. Task
of longitude and latitude coordinates (fine-grain
geolocation) of tweets exhibiting non-standard language, based
solely on linguistic content. This is a (double) regression
task. In contrast to previous geolocation shared tasks on
other areas ([
Recognizing varieties and forming an opinion about
where the speaker comes from is something so ingrained
in our experience as speakers that it seems innate, and
even a little magical. The question that drives this work
is: can Large Language Models (LLMs) do what we do
and if so, how well can they do it? do they do that in
a way that is operationally similar to ours? The Italian
scenario is a good testing ground as despite his limited 1.2. Dataset
geographical extent, it is one of the most
linguisticallydiverse in Europe. In their work, Ramponi and Casula say GeoLingIT task data comprises 15K geotagged tweets
that current transformer-based models are rather limited that exhibit non-standard Italian language use (the
confor modeling language variation over space in highly mul- tent may be fully written in local language varieties or
tilingual areas such as Italy [
This document describes the model I submitted to the gual BERT-based [
EVALITA 2023: 8th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, Sep 7 – 8, Parma, IT $ i.salogni@studenti.unipi.it (I. Salogni)
© 2023 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org) 2.1. Comparing diferent models
perform on the given dataset, I fine-tuned the
RoBERTabased umberto-commoncrawl-cased-v1 [
To explore the potential benefits of introducing additional The second best MAE results were achieved using the complexity to the model, I designed a new task-specific 300 neurons single hidden layer task-specific architecture, architecture adding a single hidden layer right after the and then followed by the 5 neurons single hidden layer pooling layer, testing diferent sizes (5 neurons and then architecture, as shown in Table 2. This can be explained 300 neurons) followed by an activation function (Iden- by thinking that adding a small hidden layer after the tity, Sigmoid or ReLU), and finally, a two-neuron output pooling layer leads to an initial drastic reduction in the layer. To reduce the computational cost, only umberto- size of the model output. commoncrawl-cased-v1 was tested using this and the The worst results, on the other hand, were all obtained next architectures. For the same reason batch size 50 was with the 3 hidden layers architecture and ReLU as acmaintained. tivation function. The accuracy dropped possibly because of excessive feature compression: when several 2.3. Adding more hidden layers hidden layers are stacked, this reduction is followed by another further reduction of the size of the input vecStill with the rationale of knowing whether adding fur- tor, and the linear activation function was of no use in ther complexity would enhance the model’s learning ca- this case. Therefore, further complicating the architecpacity, I tested a task-specific setting with 3 hidden layers ture requires an additional regularization efort, which with neurons in combination (5, 5, 10) (10, 5, 5) and (300, the results achieved with only one hidden layer or even
There are no specific areas where the inputs have a
larger error. In contrast, inputs from areas in
PiedmontLombardy-Veneto and Lazio-Campania have lower error
than the others. In fact, two or three marked clusters
can be observed in the scatterplots of the outputs (Figure
2), depending on the model configuration, the most
persistent of which is between Lazio and Campania, then
a cluster that follows the Alpine arc and finally less
frequently by a cluster on Sicily. Excluding that this can be
attributed to an imbalance in our fine-tuning dataset, this
result must come from the representation of the
embeddings of each model. Ramponi and Casula [
However, it is very complex to reconnect these observations to one or more linguistic facts concerning the Italian regional varieties. The question then is how did
The behavior shown by our models (need for regularization in the presence of numerous layers, better results with a single bigger hidden layer) is what we can expect from a simple neural network. However, it is astonishing that such a simple architecture manages to obtain non-disastrous results in an complex NLP task. The success of this regression task is undoubtedly attributable
ian language varieties (such Piedmontese), but the oficial umbertocommoncrawl-cased-v1 model card says that it was pre-trained only on the Italian subcorpus, deduplicated.
Figure 2: Scatterplots of the target (in black) and output (in red) coordinates for each configuration to the high-level representations of the input data, together with BERT’s ability to understand the linguistic context. Therefore, less is more: a simple setup, using even just two output neurons, seems to work better than a more complex one for BERT fine-tuned models on this task. Furthermore, in this work the models pre-trained on a miscellaneous corpus provides embeddings that performed better on tweets than a corpus specifically of the same genre. In conclusion, it is dificult to say how close we came to the goal, if the goal was to adequately map the diatopic variation of contemporary Italian, trying to automatically extract regional and dialectal patterns.
Even if in this work we were unable to further probe the
linguistic information used to carry out our task, the
studies converge in believing that BERT’s structure is, however,
linguistically founded, although perhaps in a way that is
more nuanced than can be explained by layers alone [