In this paper, we give a description of the systems submitted to the three tracks of FIRE 2020 - Hate Speech and Ofensive Content Identification in Indo-European Languages (HASOC), Sentiment Analysis of Dravidian Languages in Code-Mixed Text and Event Detection from News in Indian Languages (EDNIL). While the first two tasks were binary and multi-class text classification problems, EDNIL was a sequence classification problem. For all the three tracks, we jointly fine-tuned mBERT, DistilBERT, RoBERTa and XLM-R using the dataset from all the languages for the given task.
• There are only a handful of languages for which the pre-trained models are available.
Since pre-training is quite expensive - both in terms of computational resources and infrastructure required as well as linguistic resources needed for building reasonably good models - these pre-trained models are not available for a large number of languages.
The lack of suficient resources for most of the world’s languages has been a classic problem
in NLP, resulting in a hugely imbalanced progress in the development of language technologies
for diferent languages (see [
In this paper, we discuss our experiments using multilingual joint training methods for three diferent kinds of tasks - sentiment analysis, ofensive language identification and event detection - and report their results. All these three tasks were organised as shared tasks under the aegis of the Forum of Information Retrieval Evaluation 2020 (FIRE 2020).
We participated in the following tasks at FIRE 2020.
• Sentiment Analysis of Dravidian Languages in Code-Mixed Text [
For each of the language in each of the task, we fine-tuned the following models • cased multilingual BERT
Two sets of models were trained for each of these - one was a separate model for each language and the other was a joint multilingual model using the dataset of all the languages available for each task. Thus for the sentiment analysis task, the multilingual model was trained using Malayalam and Tamil datasets, for EDNIL task, Bangla, Hindi and English datasets were Language
Sub-task A
Sub-task B TOTAL
NOT
HOF
TOTAL
HATE
OFFN
PRFN Hindi English German
Train Dev Test Train Dev Test Train Dev Test 2599 354 663 3300 408 814 2000 373 526 1851 265 466 1653 199 391 1431 269 392 748 99 197 1647 209 423 569 104 134 748 99 197 1647 209 423 569 104 134 • cased multilingual DistilBERT • XLM-R (cross-lingual, multilingual model of RoBERTa) used and for HASOC task, Hindi, English and German datasets were used.
The training was carried out using the Simple Transformers library. We used a batch size of 12, maximum sequence length of 512 and a learning rate of 6e-5 for fine-tuning each of the models. The models were trained for 15 epochs. For all the other hyperparameters, the default setting of the library was retained. The results of each of the model for each of the task is summarised in Table 4. In this table we report the weighted F-score of all the systems that we submitted for diferent tasks.
In addition to this, the Dravidian sentiment analysis task involved an additional language classification task. We used an SVM classifier with character 5-gram and word unigram as features for this task. We experimented with diferent combinations of character bigram to 5-gram and word unigram to trigram feature set to find the optimum features for the language identification. In this task, the first step was that of language identification. If the language was either Tamil or Malayalam then the test instance was classified for its sentiment polarity. The scores reported here is the final score of the classification pipeline and includes errors made by the language identification system.
For EDNIL, we did not submit the models trained on individual languages because there was no significant diference between the multilingual and the monolingual models. One reason for this apparent failure of joint multilingual training could be the kind of data released for EDNIL. The dataset is taken from the newspapers, which are heavily edited (and so kind of not very natural) and removes all the characteristics of a natural multilingual communication such as borrowings and code-mixing. Moreover the three languages that we jointly trained on - Hindi, Bangla and English - all three did not share the script. Hence it is quite natural that knowledge (or more appropriately patterns) from one language could not be transferred to the other languages, hence, no improvement is noticed with joint training.
However, in case of Dravidian sentiment analysis tasks, the dataset was taken from social media which are more naturalistic and depicts the properties of multilingual communication such as code-mixing and switching as well as the use of the same script (Roman) for writing all languages including Malayalam and Tamil. As such we witness knowledge transfer from one language to the other in case of joint multilingual training in these tasks.
For HASOC, the complete test set is not yet released and the organisers have also not released the scores for each model submitted to the task separately. However, only 15% of the test set was supposed to be unseen - the labels for rest of the instances used for testing has been released by the organisers. So the scores reported here are those obtained on the released test set (and so do not exactly match those on the leaderboard since those scores are based on the additional test set).
In general, we see that the joint training across diferent tasks and languages have yielded marginal improvements in performance. One of the reasons could be the small dataset size for all the languages. Generally in cases of multilingual training at least one of the languages have a relatively large dataset and the knowledge from that dataset is transferred to other languages. However, because of lack of this, the improvement is marginal but nevertheless joint training seems to give some improvement.
In this paper, we have presented a description of our system submitted to three tracks at FIRE 2020 - Sentiment Analysis for Dravidian Languages, EDNIL and HASOC. We have explored multilingual joint training of three transformers-based models - mBERT, mDistilBERT and XLM-R - and evaluated their performance vis-a-vis monolingual models in each of the tasks. The results demonstrate that multilingual joint training provides significant advantage in processing what has been traditionally termed ’dificult’ (or noisy) datasets - the naturalistic, multilingual, multi-scriptal social media conversations - while there is no significant gain when the datasets are from what has been traditionally termed ’clean’, carefully edited datasets from newspapers (in case of EDNIL). Of course, since these results are obtained on diferent tasks, in order to validate these results, they also need to be tested on the same task with diferent kinds of datasets (or even better with the two versions of the same dataset). However, notwithstanding this extra variable, it is quite apparent that multilingual methods are quite efective for classification and processing of multilingual documents while not so for the monolingual documents. [3] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae, Corpus creation for sentiment analysis in code-mixed Tamil-English text, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. URL: https://www.aclweb.org/ anthology/2020.sltu-1.28. [4] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae, A sentiment analysis dataset for code-mixed Malayalam-English, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URL: https://www.aclweb.org/anthology/ 2020.sltu-1.25. [5] B. Dave, S. Gangopadhyay, P. Majumder, P. Bhattacharya, S. Sarkar, S. L. Devi, Overview of the FIRE 2020 EDNIL track: Event Detection from News in Indian Languages, in: P. Mehta, T. Mandl, P. Majumder, M. Mitra (Eds.), Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, CEUR Workshop Proceedings, CEUR-WS.org, 2020. [6] T. Mandl, S. Modha, G. K. Shahi, A. K. Jaiswal, D. Nandini, D. Patel, P. Majumder, J. Schäfer, Overview of the HASOC track at FIRE 2020: Hate Speech and Ofensive Content Identification in Multiple Languages), in: Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, CEUR, 2020. URL: http://ceur-ws.org/.