This paper introduces the LyS A Coruña proposal for the Guarani-Spanish Code Switching Analysis task at IberLEF2023. The shared task proposes to analyze Guarani-Spanish code-switched texts, focusing on language identification, named entity recognition (NER), and a novel classification task for Spanish spans in a code-switched Guarani-Spanish context. We propose three multi-task learning systems that have common encoders based on two language models and diferent decoders in a multi-task learning setup. The encoders use the contextual embeddings by: (i) a large language model (LLM) pretrained on bidirectional machine translation on 200 languages (including Spanish and Guarani) from the No Language Left Behind project, and (ii) a BERT-based model pretrained in Spanish and finetuned in around 800k Guarani tokens. The decoders are: (i) a softmax output layer for Task 1, and (ii) conditional random ifelds (CRF) output layers for Tasks 2 and 3. According to oficial results, we ranked third in the three tasks.
Indigenous and European languages have coexisted in South America for centuries, causing
lexical and cultural interchanges to diferent degrees. In Paraguay, where both Guarani and
Spanish are recognized as oficial languages and the population is mostly bilingual,
codeswitching has emerged in this bilingual environment. This bilingual environment has favored
the apparition of code-switching. Code-switching is a linguistic phenomenon where people
alternate between two or more languages in the same conversation, which is common in places
where more than one language interact [
Code-switching is not a new topic in natural language processing (NLP), including language
identification [
The main challenge of this task is the disparity of resources of the two languages involved.
While Spanish is one of the most rich-resourced languages in the world, resources in Guarani
are very scarce due to the lack of labeled and unlabeled data [
Our approach We follow a multi-task learning approach using MaChAmp [
The shared task aims to analyze code-switched texts in Guarani and Spanish through three token-level tasks: language identification in code-switched data, named entity classification, and Spanish code classification. The specific tasks are: • Task 1: Language identification in code-switched data. Given a sentence, this task consists in identifying the language of the token or a named entity. The possible labels are gn (Guarani word), es (Spanish word), ne (part of a named entity), mix (mixture between Guarani and Spanish), foreign (words from neither Guarani and Spanish) and other. • Task 2: Named entity classification. The task goal is to identify the tokens that span named entities in the text and classify them into three diferent categories. It follows the BIO schema (first token of an entity is marked with B- plus the label, and the following starting with I-, and lastly the tokens that are not part of an entity are marked with an O). The diferent labels of this tasks are: ne-b-loc and ne-i-loc for location, ne-b-org and ne-i-org for organization, and ne-b-per and ne-i-per for person. • Task 3: Spanish code classification. The task goal is label the spans of text in Spanish depending on its form: es-b-cc and es-i-cc represent change in code, i.e. Spanish tokens that keep all its original characteristics of Spanish, while es-b-ul and es-i-ul represent not-adapted loans, i.e. Spanish words that have been partial adapted to Guarani syntax.
Metrics For Task 1, the metrics used are accuracy, weighted precision, weighted recall, and the main metric is weighted F1. For Tasks 2 and 3, the metrics are precision, recall and F1, unlabeled and labeled, being the latter the main one. The criterion used for considering a named entity valid is exact match.
Data We only used the dataset provided by the competition to train our models. The dataset consists in sentences annotated at token level with the previously introduced labels. There are 19 003 training tokens in 1 140 training sentences, 2 989 development tokens in 180 development sentences and 2 857 test tokens in 180 test sentences.
We rely on MaChAmp [
Our models uses a pre-trained large language model as a shared encoder for all tasks, and independent decoders for each of the tasks. For Task 1, we use a softmax output layer on the output of the encoder’s embeddings, while for Tasks 2 and 3 we use a masked conditional random field [ 14] that force the output labels to follow the BIO schema. During training, the layers of the encoder are updated for all of the tasks, while the decoders’ layers are only trained with the loss of the corresponding task.
The final results of the shared task are chosen independently of the systems, that means that if one submission had a better result in Task 1 and another in Task 2, these two will appear in the final result. What we are reporting here are the independent results of each model in each task. In our case, each category was topped by a diferent system, all of them using the same approach described, but with diferences in encoders and decoders. We trained three diferent models that were tested independently, one trained only on Task 1 and two trained on the three tasks: 1. single-nllb. The model that performed the best in Task 1 uses facebook/nllb-200-distilled-600M [15] as the encoder, a large language model with 600 million parameters, pretrained in machine translation across 200 languages (including Spanish and Guarani), for a total of 40 602 total translation directions, and a softmax output layer as a decoder for Task 1. This model has not been trained on Tasks 2 and 3. 2. mtl-nllb. With the same encoder as before, the model that obtained the best results in Task 3 is trained on the three tasks instead of just one, using a softmax output layer for Task 1 and CRFs output layers for Tasks 2 and 3. 3. mtl-beto-gn. The model that obtained the highest results in Task 2 uses the same decoders as mtl-nllb, but uses the contextual embeddings generated by mmaguero/beto-gb-base-cased [16]. This LLM is a BERT-based model pretrained in Spanish and finetuned on approximately 800K Guarani tokens.
In the oficial Shared Task description, Tasks 2 and 3 are represented as a single label together with ne and es labels respectively. However, as we predict each label with an independent decoder , we represent them independently: tagsets are {gn, es, ne, mix, foreign, other} for Task 1, {B-loc, I-loc, B-org, I-org, B-per, I-per, O} for Task 2 and {B-cc, I-cc, B-ul, I-ul, O} for Task 3, so for example an original label es-b-cc would be represented as (es, O, B-cc) in our data.
As the three tasks are predicted independently, we introduced a post-processing step with a heuristic in order to ensure the correctness of the output labels. As Task 2 can only go together with named entities in Task 1 and Task 3 with Spanish tokens, we had to modify the output labels that did not match these requirements with a simple heuristic: 1. If the token is tagged for Task 2 and the label for Task 1 is not ne, we changed label from Task 1 to ne. The same happens for Task 3, if the label for Task 1 is not es, we change label from Task 1 to es. For example, (foreign, O, B-cc) would be change to (es, O, B-cc) 2. If the token is both labeled for Task 2 and 3, which should not happen, we decide which one remains depending on Task 1: if label 1 is ne, we delete label 3, and if label 2 is es, we delete label 2. For example, (es, B-org, B-cc) would be changed to (es, O, B-cc). 3. If label 1 is neither ne nor es and the token is labeled for Task 2 and 3, we delete randomly Task 2 or Task 3, and change Task 1 to the correspondent label for the remaining task. For example, (foreign, B-org, B-cc) could be transformed into (es, O, B-cc) or (ne, B-org, O).
For an illustrative example, see Table 1.
Input token Horacio Melanio
Output Post-processed output (es, B-per, O) (ne, B-per, O) (gn, I-per, O) (ne, I-per, O)
Our three models have been trained using the same hyperparameters during 100 epochs on a Tesla A100 GPU. Hyperparameters are listed in Table 2.
Hyperparameter "batch_size" "diverse" "max_tokens" "sampling_smoothing" "shufle" "sort_by_size" "default_dec_dataset_embeds_dim" "encoder_dropout" "max_input_length" "update_weights_encoder" "keep_top_n" "cut_frac" "decay_factor" "discriminative_fine_tuning" "gradual_unfreezing" "lr" "num_epochs"
The overall results of the competition are shown in Table 3. Detailed results of the runs for our three models are shown in Tables 4, 5 and 6.
Task 1
Accuracy 0.8530
W. Precision 0.8502
W. Recall 0.8530
W. F1 0.8500
M. Precision 0.6699
M. Recall 0.6056
M. F1 0.6294
As we can see in Table 3, our model is only able to surpass the baseline for Task 1, while being a bit under it for Tasks 2 and 3. We believe our models are unable to generalize well for these tasks, as they were able to learn the training test easily but they could not obtain good results on the development and test sets. There are two main reasons that can make this happen: Task 2 Task 3 Task 1 Task 2
Task 3
• The small quantity of data makes it challenging to obtain a representation that generalizes well over the task. The modest amount of data for Guarani in the pretrained representations does not allow the model to learn better representations that would have helped on Tasks 2 and 3 and avoided overfitting. • Second, the selected models were the only models on HuggingFace that had data on Guarani and Spanish. However, their big number of parameters could lead to more overfitting. We tried to control this by using diferent rates of dropout in both the encoder and the decoders, but we could not obtain a satisfactory result. We also tried some smaller models with data in Guarani, but the lack of Spanish data in the contextual embeddings harmed the final results.
Our paper describes LyS A Coruña contribution to the GUA-SPA Code Switching Analysis at IberLEF2023. We participated in the three tasks, applying a multi-task learning setup with hard-sharing and an independent decoder for each task. As encoders, we used a linear decoder for Task 1 and a CRF both for Tasks 1 and 2 to make sure the predicted labels follow the BIO schema. According to oficial results, we ranked third in the three categories, not being able to surpass the baselines for Tasks 2 and 3, due to the lack of generalization of our proposed model.
This paper has received funding from grant SCANNER-UDC (PID2020-113230RB-C21) funded by MCIN/AEI/10.13039/501100011033, grant FPI 2021 (PID2020-113230RB-C21) funded by MCIN/AEI/10.13039/501100011033, the European Research Council (ERC), which has supported this research under the European Union’s Horizon Europe research and innovation programme (SALSA, grant agreement No 101100615), Xunta de Galicia (ED431C 2020/11), and Centro de Investigación de Galicia “CITIC”, funded by Xunta de Galicia and the European Union (ERDF Galicia 2014-2020 Program), by grant ED431G 2019/01. M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://aclanthology.org/2020.emnlp-demos.6. doi:10.18653/v1/2020.emnlp-demos.6. [11] R. Caruana, Multitask learning, Machine learning 28 (1997) 41–75. URL: https://link.
springer.com/article/10.1023/A:1007379606734. [12] S. Ruder, An overview of multi-task learning in deep neural networks, arXiv preprint arXiv:1706.05098 (2017). URL: https://arxiv.org/abs/1706.05098. [13] R. Caruana, Multitask learning: A knowledge-based source of inductive bias1, in: Proceedings of the Tenth International Conference on Machine Learning, Citeseer, 1993, pp. 41–48. [14] J. Laferty, A. McCallum, F. C. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data (2001). [15] N. Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Hefernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hofman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzmán, P. Koehn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, J. Wang, No language left behind: Scaling human-centered machine translation, 2022. arXiv:2207.04672. [16] L.-H. A. G. Agüero-Torales, Marvin Matías, D. Vilares, Multidimensional afective analysis for low-resource languages: A use case with guarani-spanish code-switching language, Cognitive Computation (2023).