Code-mixing, the amalgamation of languages in speech, particularly common in India, generates informal, multilingual content on social media. Analyzing this content for linguistic tasks, notably Language Identification, is crucial. This study focuses on word-level Language Identification in Tulu-English codemixed words, using diverse embeddings and classifiers. Results show promising accuracy, afirming the viability of the proposed approach, with the best system achieving a weighted average F1 score of 0.799. The study enhances multilingual processing by providing insights into efective language identification in complex linguistic scenarios, with broader implications for communication understanding in multilingual societies. The proposed system ranked 3rd in the shared task.
CEUR ceur-ws.org
Language Identification (LID) in Natural Language Processing (NLP) refers to the process of determining the natural language in which a given piece of text is written. It involves analyzing various linguistic features and patterns within the text to accurately determine the language it belongs to.
Tulu, along with the state language Kannada is part of the cultural and linguistic landscape of
Karnataka, India. Those proficient in Tulu, known as Tuluvas, commonly exhibit fluency in both
Tulu and Kannada, encompassing reading, writing, and verbal communication. Moreover, the
Tulu language incorporates numerous lexical elements from Kannada. Additionally, the usage
of English characters holds prominence among many Tulu speakers, particularly those active
on social media platforms. Notably, the commentary contributed by Tulu users in response to
Tulu-focused content on social media platforms often manifests as a linguistic amalgamation,
involving Tulu, Kannada, and English. This intricate linguistic phenomenon has given rise
to a valuable collection of trilingual code-mixed data, an area that has remained relatively
unexplored within the realm of research. [
This paper delves into the realm of word-level LID within the context of code-mixed TuluEnglish (Tu-En) textual compositions. These textual instances have been sourced from comCEUR Workshop Proceedings
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
mentary sections of Tulu YouTube videos, consequently facilitating the construction of the
Code-mixed Tulu-English Language Identification (CoLI-Tunglish) dataset. This task was part
of the Word-level Language Identification in Code-mixed Tulu Texts (CoLI-Tunglish) shared
task[
In addressing the challenge of code-mixed language identification, several researchers have
contributed innovative approaches. Gundapu and Mamidi [
In the realm of code-mixed language identification, researchers have explored various
methodologies. Ojo et al. [
Researchers have developed versatile approaches to address code-mixed language
identification across various languages and contexts. Chittaranjan et al. [
Studies have also explored the efectiveness of BERT and Transformer models in code-mixed
language identification. Hidayatullah et al. [
The Code-mixed Tulu-English Language Identification (CoLI-Tunglish) dataset [
The undertaken methodologies within this study encompassed initial preprocessing steps, including the conversion of the provided text to lowercase, followed by its representation as strings for further analysis. The following embedding techniques were employed and tested to encapsulate the inherent linguistic characteristics of the text data:
Bag-of-Words (BoW) is a basic and widely used text representation technique in NLP. It treats each document (or piece of text) as a ”bag” of individual words, disregarding the order and structure of the words. The basic idea is to create a vocabulary of all unique words in the entire corpus (collection of documents). For each document, a vector is created where each dimension corresponds to a word from the vocabulary, and the value in each dimension represents the frequency of that word in the document. BoW is simple and eficient but does not capture word order or context.
Character n-grams are a more fine-grained technique that represents text by breaking it down into chunks of characters, rather than words. An n-gram is a contiguous sequence of n characters in a string. The character n-grams technique was trialled across varying n-gram intervals, specifically (1,2), (1,3), and (1,4), i.e., we are considering all possible combinations of character sequences with lengths ranging from 1 to 2 characters, 1 to 3 characters, and 1 to 4 characters. Character n-grams capture subword information.
A comprehensive array of models was applied in this study to address the task at hand. The Scikit-Learn library was employed for model implementation, and default parameters were utilized. The utilization of this diverse set of models aimed at exploring a wide spectrum of possibilities and capturing nuanced patterns within the code-mixed data. The descriptions of the models used is as follows:
RandomForest is an ensemble learning method that builds a forest of decision trees and combines their predictions to improve accuracy and reduce overfitting in classification and regression tasks. Multinomial Naive Bayes is a classification algorithm commonly used for text and document classification tasks. It’s based on the Bayes’ theorem and assumes that features are conditionally independent. Logistic Regression is a simple linear classification algorithm used for binary classification problems. It models the probability of a binary outcome. Linear Support Vector Classifier is a linear machine learning model used for binary classification. It aims to find a hyperplane that best separates the data into two classes. A Decision Tree is a tree-like model that makes decisions by recursively splitting the dataset based on the most significant feature at each node. KNN is a non-parametric and instance-based algorithm used for classification and regression. It classifies data points based on the majority class of their k-nearest neighbors.
AdaBoost is an ensemble learning technique that combines multiple weak learners (usually decision trees) to create a strong classifier. OneVsRest classifier was used, a multi-class classification strategy where a separate binary Logistic Regression classifier is trained for each class to handle multi-class classification problems. Gradient Boosting is an ensemble method that builds an additive model by training weak learners sequentially, where each new learner corrects the errors made by the previous one.
Stacking classifier was used, which combines multiple base models (LinearSVC, RandomForest, KNN) with a meta-learner (Logistic Regression) to improve overall model performance. The Voting Classifier combines the predictions of multiple classifiers (e.g., LR, RF, and SVC) using majority voting or weighted voting to make a final decision. Bagging (Bootstrap Aggregating) is an ensemble technique that trains multiple instances of the same base model (KNN) on bootstrapped samples of the data and combines their predictions.
After performing data preprocessing, each of the models mentioned in the previous section was trained separately using the various word embeddings discussed. Table 2 (refer to Table 2) displays the weighted average F1 scores of the classifiers when combined with diferent combinations of word embeddings, including Bag of Words (BoW) and Character n-grams with varying n-gram ranges. The evaluation of the models was based on their performance in terms of the weighted average F1 score. This particular metric is well-suited for assessing multiclass classification models because it accounts for class imbalances, provides a comprehensive measure of overall performance across all classes, and considers practical considerations such as the significance of individual classes. This metric delivers a balanced evaluation that combines both precision and recall, making it a valuable tool for selecting models and evaluating their performance in real-world applications.
Out of all the models, CountVectorizer with an n-gram range of (1,4), coupled with LinearSVC classifier was the most efective configuration observed for the language identification task. This combination adeptly captures linguistic nuances and establishes clear decision boundaries, showcasing superior accuracy and precision in distinguishing languages. With the development dataset, the model gives a weighted average F1 score of 0.87. The weighted average F1 score with this set-up for the test dataset was 0.799. This was the third best score in the CoLi-Tunglish shared task.
This study addressed the task of language identification within code-mixed Tulu-English words, prevalent in multilingual communication. Through the utilization of diverse word embeddings and classifiers, significant progress was made in efectively meeting this challenge. Notably, character n-grams in the range 1 to 4 with LinearSVC classifier demonstrated exceptional performance, yielding the highest weighted average F1 score compared to the other embeddingsmodel that were evaluated, highlighting the critical role of appropriate selection in achieving accurate language identification. Further exploration could involve refining embeddings and considering ensemble strategies to advance the accuracy and resilience of code-mixed language identification systems. puter Science 9 (2023). URL: https://www.readcube.com/articles/10.7717%2Fpeerj-cs.1312. doi:10.7717/peerj-cs.1312. [15] H. L. Shashirekha, F. Balouchzahi, M. D. Anusha, G. Sidorov, CoLI-Machine Learning Approaches for Code-mixed Language Identification at the Word Level in KannadaEnglish Texts, 2022. URL: http://arxiv.org/abs/2211.09847. doi:10.48550/arXiv.2211. 09847, arXiv:2211.09847 [cs]. [16] V. Vajrobol, CoLI-Kanglish: Word-Level Language Identification in Code-Mixed KannadaEnglish Texts Shared Task using the Distilka model, in: Proceedings of the 19th International Conference on Natural Language Processing (ICON): Shared Task on Word Level Language Identification in Code-mixed Kannada-English Texts, Association for Computational Linguistics, IIIT Delhi, New Delhi, India, 2022, pp. 7–11. URL: https: //aclanthology.org/2022.icon-wlli.2.