In this paper, a word-level language identification system is presented for the Dravidian languages Tamil, Telugu, Malayalam, Kannada, and Tulu using a combination of TF-IDF character n-grams, handcrafted features, and FastText embeddings. These languages have rich morphological structures and diverse scripts, making computational processing challenging. For each language, there is a training set in CSV format with many words and their labels (e.g., tam for Tamil, en for English, Location, Name, symbol, etc.). The test set contains only words, and the goal is to predict the labels. This approach combines multiple types of features, which are TF-IDF on character n-grams, handcrafted features, and FastText embeddings, which are given as input to a linear SVM LinearSVC, which gives the predictions as output. The validation set evaluates accuracy. Results show strong performance across all five languages, demonstrating that integrating statistical and embedding-based features is efective for Dravidian language identification.
Dravidian languages have beautiful and complex word structures that make automatic language
processing dificult [
The same machine is applied to all 5 main languages for evaluation, incorporating previous work in word-level language identification. The first step is pre-processing, where the training and validation sets are made clean. Rows with empty words are removed from both sets so that the model trains only on valid words and their labels. The next step is feature extraction. Three types of features are incorporated: TF-IDF on character n-grams, handcrafted features, and FastText embeddings. Character-level TF-IDF n-gram features detect patterns in sufix, prefix, or common root that frequently occur in Dravidian languages. Handcrafted features such as word length, capitalization, and the presence of digits or special characters provide hints at the characteristics of the word. FastText is an open-source library developed by Facebook AI Research for text classification. It works not only on each word but also on subwords to be more precise. The output from combining these features is fed into a Linear Support Vector Machine (LinearSVC), a fast and accurate classifier for high-dimensional data. The classifier assigns the ideal weight to each of the features by learning during training (with a maximum of 10,000 iterations). The machine uses these weights on the test words and gives their predicted labels as output. This model follows a machine learning approach rather than a rule-based or purely statistical model because it can learn useful language patterns directly from data instead of relying on manually crafted linguistic rules. Machine learning also allows easy extension of the same pipeline to multiple languages and updates it whenever new data becomes available.
Research on word-level language identification for Dravidian languages has grown steadily in recent years, driven by the need to handle code-mixed and morphologically rich text. Benchmark datasets and shared tasks, such as the CoLI-Dravidian 2025 challenge, have provided a unified framework for evaluating models across Tamil, Telugu, Malayalam, Kannada, and Tulu, revealing the dificulties of rare labels, multiple scripts, and complex word structures [13]. Prior eforts often focused on individual languages, including Tulu-English and Kannada-English code-mixed corpora, which enabled the development and testing of machine learning approaches combining n-gram statistics, handcrafted linguistic features, and embedding-based representations [14]. Beyond shared tasks, the creation of annotated corpora for under-resourced Dravidian languages has played a key role in advancing research. Such datasets not only support language identification but also facilitate related tasks like sentiment analysis and named entity recognition, providing a foundation for models to capture both frequent and rare patterns in the data [14]. Collectively, these eforts have established the resources, evaluation methodologies, and feature engineering strategies that inform the design of the current Dravidian language identification model, allowing it to efectively handle multiple languages within a single unified framework.
Language identification at the word level in multilingual and code-mixed contexts is an active area of
research in natural language processing. As mentioned in the previous section, for Dravidian languages,
which have complex structures and appear frequently in code-mixed data, shared tasks establish
benchmark datasets and evaluation frameworks that highlight the challenges of handling multiple
scripts and rare labels [
There are five subtasks under this task, each corresponding to a diferent Dravidian language. A training and validation set for each language contains words and their labels, and a test set contains words only. The goal is to predict the labels for the words in the test dataset, which is considered the run submission taken for evaluation. The shared task for this year includes all five Dravidian languages, each representing a separate subtask with its corresponding dataset. This difers from previous editions, which focused on individual languages separately rather than addressing all of them simultaneously. The labels are unique to each language, and some labels are challenging to predict precisely as they are rare in the training set. For example, there is no Location label for the words in the Malayalam training and validation datasets, but it is called Place instead. A maximum of five run submissions is allowed for each language.
As aforementioned, the same algorithm is used for all five languages. In this description of methodology, the Tulu language is used as a reference to explain the process. There are three main steps in this algorithm, which are pre-processing to clean the data, Feature engineering, including three types of features, and Prediction using LinearSVC. The design of the model is consistent across languages, which ensures that the same pipeline of preprocessing, feature extraction, and prediction can be applied without major modifications.
The first step of the pipeline is preparing the input data. The training set and validation set are carefully cleaned by removing rows that contain empty words. This prevents the model from training on incomplete entries, which could introduce errors or reduce classification accuracy. After cleaning, the training, validation, and test datasets are loaded into the algorithm. The training set is used to learn feature weights, the validation set is used to measure performance, and the test set is used to generate the final predictions. This structured handling of input data ensures that the model learns only from reliable examples and can generalize efectively to unseen words. The choice of preprocessing all the languages with the same steps is intentional. Each dataset follows a similar CSV format, and basic cleaning operations, such as removing empty words and normalizing text, are applied. The deeper language diferences are captured in the later stages, like feature extraction. As these representations automatically adapt to structural variations across languages, the universal preprocessing keeps the pipeline simple.
TF-IDF (Term Frequency-Inverse Document Frequency) on Character n-grams: Each word is broken
into small overlapping sequences of 2 to 5 characters (called n-grams). TF-IDF calculates how important
each n-gram is for a given word relative to the entire set of words [
IDF() = log ︂(
)︂ 1 + = Total number of words (documents) = Number of words containing n-gram where: where: N-grams that are common have low IDF, so they are down-weighted, and rare n-grams have high IDF and hence are up-weighted. To calculate TF:
TF(, ) = Number of times n-gram appears in word (document) Alternatively, if normalized frequency is more preferable:
TF(, ) =
The TF-IDF Score is obtained by multiplying TF and IDF: = n-gram = word (document)
TF-IDF(, ) = TF(, ) × IDF() Hence, if an n-gram occurs frequently in a given word but rarely in other words, it gets a high TF-IDF score, so it is important in classifying the word uniquely.
Custom Handcrafted Features: A few simple diferentiators are added, such as word length,
capitalization, alphanumeric check, and digit count. Capitalization is used as it helps identify proper nouns
(label = name). Word length is relevant as the number of characters is analyzed, since longer words
often have diferent linguistic roles [
FastText Embeddings:
FastText is a tool created by Facebook for representing words as vectors that capture their meaning
[
The model is trained using a linear Support Vector Machine. SVM is a powerful machine learning
algorithm used for classification when each data point (here, each word) needs to be assigned to one
of several categories (labels such as tam, en, sym, etc.) [
TF-IDF Character n-grams
Input Data Train, Validation, Test Set
Pre-processing Remove empty rows
Normalize text
Handcrafted Word length, digits, etc.
Feature Combination Concatenate all feature vectors
Model Training Linear SVM (LinearSVC) Prediction & Evaluation
Predicted Labels Accuracy, Precision, Recall, F1
The performance of this algorithm on these five Dravidian languages is shown in the table above. The rank column refers to the rank obtained using this algorithm in the CoLI-Dravidian task held on CodaBench. Wtd in the table refers to Weighted. Overall, the model achieves high accuracy and strong scores on most evaluation metrics across all five languages. Kannada obtains the best results among the languages with an accuracy of 0.96 and a macro F1 score of 0.92, showing that the classification is consistent and reliable. Telugu and Tamil perform well. Malayalam gives the lowest scores among the group. The diferences in the performance of the same algorithm across the five languages may be due to various reasons, such as training data size or particular language features. Overall, the combination of features such as FastText embeddings and TF-IDF on character n-grams processed by LinearSVC contributes to good performance, demonstrating that this model classifies Dravidian languages efectively despite their complexity.
The results show that while the model performs strongly overall, there are still clear areas for improvement. One important observation is that the same feature set does not behave equally across all languages, which suggests that some language-specific tailoring may be necessary. For example, Malayalam performs worse compared to the other languages. The first suspect for this was that the training set for the Malayalam language may have been significantly smaller than those of the other languages. However, this is not the case. The Malayalam training set has 25996 entries, whereas the Tulu training set has 29525 entries. Hence, this is likely an incorrect assumption. The poorer performance of the Malayalam language identification model may be due to diferent labels being inconsistent in frequency. For example, the label “PLACE” occurs only 123 times in the Malayalam training set, whereas the label “MALAYALAM” occurs 11794 times and “ENGLISH” occurs 5768 times, showing that the label “PLACE” is rare and underepresented. This indicates that adding more balanced datasets or applying data augmentation could improve performance. Another promising direction is the use of deep learning models, such as recurrent or transformer-based architectures, which may capture richer contextual information than handcrafted features alone. At the same time, a hybrid approach that combines TF-IDF and FastText with neural models could strike a balance between interpretability and performance. Finally, improving the handling of rare labels remains an open challenge, and exploring techniques like few-shot learning or external lexical resources may help address this gap.
In this task, the development of a machine learning approach for classifying Dravidian words was explored. The model’s performance was evaluated repeatedly during development using the validation dataset. By combining the three diferent types of features of TF-IDF on character n-grams, handcrafted features, and FastText word embeddings, the model captured the linguistic properties, including both the surface-level and deep information of each word by representing it in a numerical matrix form. LinearSVC was used as the classifier due to its speed and compatibility for diferentiating high-dimensional data. Overall, the results were satisfactory. However, the model could be improved by adding more or diferent layers or by having bigger training datasets with more instances of words with rare labels. Working on this project gave me a better understanding of Language Identification (LI) and the diferent tools one could use to build a model to tackle the task of LI and NLP in general.
In the preparation of this paper, a generative AI tool was used in a limited manner only for paraphrasing. The author(s) utilized QuillBot for rephrasing certain sentences to improve clarity and readability. All conceptual content, implementation details, analysis, and final editing were carried out by the author(s), who take full responsibility for the accuracy and integrity of the work. [13] Hegde, A., Balouchzahi, F., Butt, S., Coelho, S., Hosahalli Lakshmaiah, S., & Agrawal, A. (2025).
Overview of CoLI-Dravidian 2025: Word-level Code-Mixed Language Identification in Dravidian Languages. In Forum for Information Retrieval Evaluation (FIRE 2025), Varanasi, India. [14] Hegde, A., Anusha, M. D., Coelho, S., Shashirekha, H. L., & Chakravarthi, B. R. (2022). Corpus Creation for Sentiment Analysis in Code-Mixed Tulu Text. In Proceedings of the 1st Annual Meeting of the ELRA/ISCA Special Interest Group on Under-Resourced Languages (pp. 33–40), Marseille, France. European Language Resources Association.