A code-mixed language is a type of language that involves the combination of two or more language varieties in its script or speech. Analysis of code-text is dificult to tackle because the language present is not consistent and does not work with existing monolingual approaches. We propose a novel approach to improve performance in Transformers by introducing an additional step called "Siamese Pre-Training", which allows pre-trained monolingual Transformers to adapt language representations for code-mixed languages with a few examples of code-mixed data. The proposed architectures beat the state of the art F1-score on the Sentiment Analysis for Indian Languages (SAIL) dataset, with the highest possible improvement being 5.1 points, while also achieving the state-of-the-art accuracy on the IndicGLUE Product Reviews dataset by beating the benchmark by 0.4 points.
performance on code-mixed natural language tasks due to inherent inconsistencies and large variability in data.
Our novel approach generates Code-mixed Adaptive Language representations using
Bidirectional Encoder Representations from Transformers (CalBERT) by adding an additional
pre-training step called "Siamese Pre-Training" where a pre-trained monolingual Transformer
[
BERT [
The methodology proposed in this work seeks to adapt existing representations for monolingual text into representations that can be fine-tuned for code-mixed tasks. This allows for representations of code-mixed language to be generated without having to pre-train a Transformer model from scratch on large quantities of code-mixed data, which is a time consuming and computationally intensive process.
Mikolov, in his paper titled “Eficient Estimation of Word Representations in Vector Space” [
Reimers et al, in their work titled “Sentence-BERT: Sentence Embeddings using Siamese
BERT-Networks” [
“A Passage to India: Pre-trained Word Embeddings for Indian Languages" [11] by Kumar et al, shows that models trained on subword representations perform better as Indian languages are morphologically rich. Their multilingual embeddings when evaluated on next sentence prediction using pre-trained BERT gives 67.9% accuracy. Another paper titled “Towards SubWord Level Compositions for Sentiment Analysis of Hindi-English Code Mixed Text” [12] by Joshi et al, discusses the advantage of using sub-word representations compared to character level tokens to deal with inconsistencies in the code-mixed text. This method was evaluated on a custom dataset and yields an 1-score of 65.8%.
Choudhary et al [13], in their paper “Sentiment Analysis of Code-Mixed Languages leveraging Resource Rich Languages” propose a novel method that uses contrastive learning. They use a Siamese model to map code-mixed and monolingual language to the same space by clustering based on the similarity between their skip-gram vectors. This works on the hypothesis that similar words have similar contexts. The model achieves 75.9% F-score on the HECM dataset, an improvement over existing approaches.
We propose a novel approach to adapt Transformer representations for a code-mixed language from existing representations that exist in another language by introducing an additional pre-training step using a Siamese network architecture [14]. We attempt to minimise the distance between the two semantic spaces of the corresponding languages and obtain a joint or shared representation for equivalent sentences in both languages. This additionally enables the generation of code-mixed language representations from an existing language’s representations, without the need to pre-train a Transformer from scratch.
Our novel approach is implemented by using a shared pre-trained Transformer layer in each
branch of the Siamese network and using an appropriate loss function to bring the embeddings
closer between each pair of sentences. To Siamese pre-train CalBERT, the same sentence needs
to be obtained in both the language that the Transformer was pre-trained in, along with the
language for which the Transformer is trying to adapt representations.
5. Methodology
5.1. Terminologies
• Base Language: The single language that was used to pre-train the Transformer
architecture using any task like Masked Language Modelling [
A Siamese network is a network consisting of two or more branches, wherein each branch receives an input. The network learns to distinguish between the examples provided through each branch. Siamese networks have been used extensively in computer vision for image classification.
A pre-trained monolingual Transformer being trained on only a single language is capable of generating language representations for that language only, thus performing poorly when the language is code-mixed. Pre-training a Transformer from scratch on code-mixed data is a dificult task since it is computationally expensive and also time-consuming. Since the Transformer has learnt language representations for one of the languages, it is far more optimal to adapt these existing representations for new and similar words belonging to the other language.
A pre-trained monolingual Transformer is provided with diferent representations of the same sentence or phrase and learns to minimize the distance between their embeddings (Fig. 1). The two representations, in this case, are the transliterated version of the code-mixed sentence in the target language and the translated version in the base language. Since the two versions have the same semantic meaning, their similarity should ideally be as high as possible and thus the distance between their representations should be minimum. Since the Transformer already knows representations for the base language, it only needs to adapt representations to map the target language to the same space.
The Siamese pre-training is called thus because of its use of the Siamese network architecture, wherein the two arms of the network are fed with two semantically equivalent input sentences in the base and the target language respectively. The network learns the embeddings for both sentences by comparing their similarity (normally, this involves the use of a labelled dataset with sentence pairs and the corresponding similarities, but here all the sentence pairs have maximum similarity). The loss function (Eqn. 1) used here is used to bring the representations from the two branches closer. In our work, we use the cosine distance as the loss function, although similar loss functions such as contrastive loss can also be used. Minimizing the cosine distance implies that the similarity is maximized between the sentence embeddings, which is the desired outcome.
The Siamese pre-training is performed as the last pre-training step (after the usual pretraining strategies used in a Transformer such as masked language modelling or next sentence prediction) since it needs existing base language representations to learn the target language efectively. Additionally, our work shows that a significant amount of data is not required to efectively perform Siamese pre-training and that the model can learn with a fraction of the data size that was used for pre-training, thus showing that our approach does not require vast computational resources to improve performance.
A variety of BERT-based architectures were pre-trained and fine-tuned for this work. The models that were evaluated were BERT, RoBERTa, DistilBERT and XLM-RoBERTa. These architectures are used either pre-trained on an English corpus (as available publicly on the HuggingFace Hub) or pre-trained on the corpus of code-mixed data that was collected as part of this experiment. 6. Workflow 1 We focus our eforts on Transformer architectures based on the Bidirectional Encoder Representations from Transformers (BERT) architecture, since they are bidirectional models capable of learning representations from a given sequence using both forward as well as backward contexts. We demonstrate our novel approach on the Hindi-English (Hinglish) code-mixed language and data for the same was obtained from social media and news articles to maintain a good balance of well structured as well as informal code-mixed language usage.
The objective is to minimize the distance between the representation in the base language and the representation in the target language. The loss function, hence, needs to reflect this minimization of the distance between the two representations.
The cosine distance loss function in (1) represents the angle between two vectors. Minimizing the cosine distance implies that the two vectors are highly similar as a smaller angle between the two vectors creates a smaller cosine distance.
→(− →,− ) = 1 − →− →.− →‖− ‖→‖− ‖ (1) 1https://github.com/aditeyabaral/calbert
Since code-mixed Hinglish data is abundantly present on platforms like social media, we choose to use social media as our source of data. There are several online archives [15] available that host code-mixed data from platforms like Twitter and Facebook for several tasks. Additionally, Hinglish code-mixed data is already available on IndicCorp, which is one of the largest publiclyavailable corpora for Indian languages.
After compiling all the sources of data together, they need to be converted into a suitable pairwise format to be used along with CalBERT.
Although a large amount of data is obtained (Table 1), most of it is not in a format that can be directly used to train CalBERT. Since Hinglish code-mixed language can exist in both the Hindi (Devanagari) script as well as the Roman script, we need to transliterate all of them to the same single script language (this is achieved using the indic-transliteration pip package [16]). For our work, we choose the script language to be Roman since most popular Transformer architectures have been pre-trained in this script language, and hence have representations for the base language, in our case English.
For code-mixed data that already exists in the Roman script, we need to obtain the translated version of the same for the base language representation. This has been performed by using software automation tools such as Selenium combined with Google Translate. Alternatively, for code-mixed data that exists in the Devanagari script, we employ the use of a transliteration library and convert them into the ITRANS ASCII transliteration scheme.
The input to CalBERT consists of sentence pairs (Table 2). Each pair consists of the transliteration of the code-mixed sentence in the target language and the translation of the same code-mixed sentence in the base language. Both sentences are represented in the Roman script in our work. Since both sentences have the same semantic meaning, the objective is to reduce the distance between the corresponding sentence representations. The shared Transformer layer in CalBERT thus allows it to learn joint representations for both languages and adapts base language representations to the target language.
Due to computational limitations, we did not Siamese pre-train our models on the data
obtained from IndicNLP [
Since CalBERT is trained in a task-agnostic manner, it can be fine-tuned on any downstream
natural language understanding task like sentiment analysis and question answering. We
evaluate CalBERT on the Sentiment Analysis for Indian Languages (SAIL) 2017 dataset [
We also evaluate CalBERT on the IndicGLUE Product Reviews dataset released by IIT Patna, which serves as another benchmark dataset for sentiment analysis of Hinglish text, but in the Hindi (Devanagari) script. All models evaluated on this script have either been trained on English scripted text or Devanagari scripted text. We however propose to evaluate CalBERT on a code-mixed version of the dataset by transliterating the text from Devanagari to Roman.
The dataset is popularly used for evaluating the performance of Transformers built for Indian
languages such as IndicBERT [
To Siamese pre-train a Transformer (Table 3), we first initialise a Siamese network with a shared layer containing the Transformer whose representations we intend to adapt (BERT, RoBERTa or DistilBERT). This can be done efectively using a sentence-transformer architecture. We add a single pooling layer to each branch, and then finally combine the pieces by adding a suitable loss function to reduce the distance between the representations. In our preliminary experiments, we observe that the contrastive and cosine distance loss functions perform nearly the same and hence use the cosine distance loss function for all the experiments performed. This shared Transformer layer can then be extracted and fine-tuned on other downstream tasks.
CalBERT is meant to be fine-tuned for downstream tasks involving code-mixed data. Due to the abundance of code-mixed Hinglish data available on social media and the much need for code-mixed Transformer architectures, we evaluate performance on the popular downstream task of sentiment analysis. 7.1. SAIL 2017 The Sentiment Analysis of Indian Languages (SAIL) 2017 dataset is a collection of sentences in two popular Indian code-mixed languages – Hindi-English and Bengali-English. The datasets are composed of sentences from various sources like news articles as well as social media and are in the Roman script. It consists of 9945 train examples, 1238 test examples and 1240 validation examples. There is also a high degree of variability in the data, with multiple forms existing for the same word and diferent styles of writing. The sentences are classified into 3 polarities – positive, neutral and negative. The SAIL 2017 task is a challenging benchmark and is widely regarded as one of the benchmark datasets for sentiment analysis of the Hinglish language. The highest documented 1-score on the benchmark is 56.9% (achieved by IIIT Hyderabad)
The SAIL 2017 dataset is already partitioned into the train, test and validation splits. All comparisons are made using the 1 score on the validation split. For our experiments, we finetune existing pre-trained models with and without CalBERT’s additional Siamese pretraining step and compare the score obtained by each model. The experiments are performed multiple times using the same set of hyperparameters and the highest 1-score over 10 runs is recorded.
CalBERT outperforms the SAIL 2017 benchmark (Table 5) 1-score by 5.1 points, or 8.9% with the CalBERT-XLM-RoBERTa model (XLM-RoBERTa with CalBERT’s Siamese pre-training). We also improve upon the benchmark precision by 3.5% and the recall by 10.3%. Additionally, all other CalBERT architectures also outperform the benchmark 1-score, with the minimum improvement obtained by CalBERT-DistilBERT being 3%.
We observe that Transformer architectures also obtained improved performance metrics on the SAIL 2017 benchmark. For this reason, we evaluate CalBERT’s Siamese pre-training against a native Transformer that was used to Siamese pre-train CalBERT (Table 6). Our experiments show that CalBERT has improved the 1-score on all native Transformer architectures as well, thus showing that additional pre-training can improve performance on the given code-mixed task.
Pre-training a Transformer from scratch is computationally expensive as well as timeconsuming. Additionally, it requires a vast amount of data to efectively pre-train a Transformer to learn usable language representations. Since there are no code-mixed Transformer architectures that exist at the time of writing this paper, we experiment by pre-training some popular Transformers (Table 7) using Masked Language Modelling on a subset of our code-mixed data. The size of the subset taken is the same as that was used for CalBERT experiments to compare the two approaches. Our experiments show that the models pre-trained from scratch do not outperform the benchmark, and are significantly lower than all CalBERT architectures, thus proving that Transformers do need enormous amounts of data to provide good results. Additionally, it reinforces our hypothesis that it is far more optimal as well as efective to apply CalBERT’s Siamese pre-training to adapt a pre-trained Transformer’s representations for another code-mixed language. sab ka Bhai meri Jan Salman khan POSITIVE hahaha ! gazab imagination hai teri ! POSITIVE lagta hai aaj bhi bating nahi milega #ind- NEGATIVE vssa Or mastery kaisi chal ri hai apki NEUTRAL CalBERT-XLMRoBERTa POSITIVE POSITIVE NEGATIVE NEUTRAL
The IIT Patna product review dataset was released by the IndicNLP organization as part of the IndicGLUE collection of datasets that are meant to be used for the evaluation of models trained for NLU tasks on Indian languages. The dataset consists of product reviews in Hindi taken from a popular e-commerce website. It comprises 4182 train examples and 523 test and validation examples respectively. Like the SAIL dataset, there is a high degree of variability in this data too, with multiple forms existing for the same word and diferent styles of writing. The sentences are classified into 3 polarities – positive, neutral and negative.
We observe that CalBERT can beat the state-of-the-art accuracy on the dataset by 0.4 points,
or 0.5% with the CalBERT-XLM-RoBERTa model. We also improve on the score set by the
IndicBERT model, by achieving an improvement of 8.05 points, or 11.2%. However, we observe
that the other Transformer architectures do not perform well on this dataset, thus being
consistent with previous attempts made by other authors [
We also experiment with our custom pre-trained Transformers as used in the SAIL-2017 experiment on this dataset.(Table 11). As seen previously, the models again do not outperform the benchmark and perform very poorly. However, we observe that in certain cases, CalBERT outperforms the comparable from-scratch trained Transformer model, hence showing the efectiveness of Siamese pre-training as used in CalBERT, to learn language representations of code-mixed language.
We demonstrate the use of BERT and BERT-based architectures in learning code-mixed language representations for Hindi-English code-mixed language and evaluate the performance of the learned embeddings on a benchmark sentiment analysis task. We present a task and languageagnostic approach to generating cross-language representations for sentences, which can further be fine-tuned on any specific downstream task.
We show an 8.9% improvement in the 1 score achieved by the novel Siamese pre-training method over the existing benchmark score. We also show that CalBERT also outperforms the native Transformer architectures which were used to Siamese pre-train CalBERT, thus showing that Siamese pre-training can help existing Transformers adapt to a code-mixed language.
Owing to computational limitations at our end, we are yet to find out the extent of possible improvement that CalBERT can show on the benchmark dataset, but we postulate that training on more examples may result in a more significant increase in the performance of the model. [11] S. Kumar, S. Kumar, D. Kanojia, P. Bhattacharyya, A passage to india: Pre-trained word embeddings for indian languages, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), 2020, pp. 352–357. [12] A. Joshi, A. Prabhu, M. Shrivastava, V. Varma, Towards sub-word level compositions for sentiment analysis of hindi-english code mixed text, in: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, 2016, pp. 2482–2491. [13] N. Choudhary, R. Singh, I. Bindlish, M. Shrivastava, Sentiment analysis of code-mixed languages leveraging resource rich languages, CoRR abs/1804.00806 (2018). URL: http: //arxiv.org/abs/1804.00806. arXiv:1804.00806. [14] J. Bromley, J. W. Bentz, L. Bottou, I. Guyon, Y. LeCun, C. Moore, E. Säckinger, R. Shah, Signature verification using a “siamese” time delay neural network, International Journal of Pattern Recognition and Artificial Intelligence 7 (1993) 669–688. [15] Code-mixed indian social media text, 2016, https://amitavadas.com/Code-Mixing.html, 2016. Accessed: 2021-11-5. [16] Indic-transliteration, python package index, https://github.com/indic-transliteration/indic_ transliteration_py, 2020.