The rapid global dissemination of false information on social media platforms poses a serious threat to public debate, particularly during significant events like the Russo-Ukrainian conflict. Typical machine learning techniques are inefective because real-world social media data might be complex, noisy, and severely uneven in terms of class. In order to identify false information in an unbalanced, multilingual dataset of tweets on the war, our group, Deepish with id 429337, employed a refined RoBERTa-based Transformer model to identify false information in a multilingual Twitter dataset.Our method uses enhancements like a dynamic optimal thresholding strategy to maximize the F1 score on the validation set and balanced class weighting in the loss function to minimize imbalance. In order to normalize noise and platform-specific information like URLs, mentions, and hashtags as unique tokens, it also uses proprietary pre-processing. Our optimized classifier placed fourth in the identification procedure with a strong weighted F1 score of 0.88 using a held-out test set. . In a dificult real-world case where the misinformation class makes up only 1% of the data, this result shows the robustness and efectiveness of our approach. Future research on automated, high-performance disinformation identification using complex language models will have a strong basis based on our work.
In this digital era, and with the advancement of GenAI, we are generated enormous amounts of information. Determining the authenticity of this information is very dificult. Misinformation can have significant social consequences and sometimes lead to violence. For example, during the 2016 US presidential election and the Russo-Ukrainian conflict, misleading information rapidly spread on Twitter .
In addition, researchers have discovered that false information spread faster. False news spreads
much more quickly and widely than accurate news, according to a groundbreaking study by Soroush
Vosoughi et. of MIT [
Public discourse is seriously threatened by the extensive transmission of false information via social
media and other channels. A relatively new advancement in deep learning, transformers are particularly
good at comprehending the contextual relationships seen in text [
Researchers in this field are working to identify and classify information as either misinformation or non-misinformation. Using artificial intelligence (AI), researchers have had success in this area. However, traditional machine learning (ML) algorithms struggle to perform when faced with complex real-world scenarios such as natural language processing tasks.
LLMs have significantly improved our capacity to distinguish between accurate and false information. Why are they efective for content analysis? They are particularly adept at seeing the obvious indicators of dishonesty, such as odd textual patterns, biased language, and blatant contradictions. We were aware of the obstacles that older approaches faced as well as the actual harm posed by false information. As a result, we took a calculated risk and used this cutting-edge technology to successfully navigate and fix those challenging problems. In our proposed method we overcome the limitation of previous studies in this area.
The contribution of our research is as follows: • Robust Preprocessing and Novel Feature Engineering • Advanced Training and Optimization Strategies • F1-Score Focused Prediction and Decision Making
There are multiple work done in this domain which examined ways to detect false information using
machine learning and deep learning approaches. For instance, Ahmed et al. achieved a precision rate of
92% using TF-IDF features and traditional classifiers; SVM performed best among these classifiers [
A recommendation was given to identify fresh rumor’s in the middle of breaking news. Word2Vec
was used with Long Short-Term Memory (LSTM) recurrent neural network based on word embedding.
Although it still need improvement, the model achieved an accuracy of 79.5% [
However, these studies generally rely on datasets that difer significantly from the one proposed in
this research. This research focuses on multilingual misinformation surrounding the Russo-Ukrainian
conflict using real-time retweet data [
The classification model was trained and validated using a special data set based on false information observed in the real world.
Context and source: The collection is made up of correctly annotated tweets collected over the ifrst year of the conflict between Russia and Ukraine. This source ofers high-stakes, contextually rich content that examines the model’s power to categorize quickly moving narratives.
Language and Multilingual Aspect: Given the global scope of the disagreement, it is anticipated that the tweets will be in several languages. Multilingual preparatory steps in our workflow are essential because LLMs can react to false content in diferent languages.
Severe Class Imbalance: The most obvious aspect is the severe class disparity, which poses a significant challenge to the model. This is the division as a whole:
Misinformation (Label 1, minority class): 156 rows for testing and 364 rows for training. Non-misinformation (label 0, majority class): 14,646 records for testing and 34,174 rows for training.
As covered in Section 3.3, this severe imbalance requires the application of particular mitigating strategies.
To maximize the signal extracted from the noisy social media text and address the model’s inherent limitations regarding platform-specific features, a custom preprocessing pipeline was developed (implemented in the preprocess_text function).
1. Noise Normalization: All text is cleaned, including the normalization of repeated punctuation (e.g., !!! to !) and the conversion of emojis to their text descriptions (de-tokenization) to retain semantic value. 2. Social Media Feature Injection: Key structural elements of the tweets are converted into dedicated special tokens before RoBERTa tokenization. This includes replacing URLs (http\S+) with [URL], user mentions (@\w+) with [USER], and formatting hashtags (#(\w+)) with [HASHTAG] tags. This process teaches the model that the presence of these elements is a relevant contextual feature, rather than treating them as noise. 3. Language Robustness: An internal mechanism attempts language detection. While full translation is simulated, the core function ensures that the model can handle diverse inputs, a necessary step given the global source of the dataset.
is an end-to-end architecture where the classification layer (a linear layer) is built directly atop the pooled output of the final Transformer layer, making the entire model diferentiable and trainable.
A. Model Initialization and Loss Function • Base Model: We used roberta-base, a 125M parameter model, as the underlying architecture. • Class Imbalance Mitigation: Due to the severe imbalance (Misinformation ≈ 1% of the data), balanced class weighting was computed using sklearn.utils.class_weight and integrated directly into the CrossEntropyLoss function. This ensures that misclassifying the minority Misinformation class incurs a significantly higher penalty than misclassifying the majority Non-Misinformation class.
The loss function for a binary classification problem with class weighting w is defined as: = −
1 ∑︁ w log(ˆ) =0 (1)
To handle the memory requirements of the transformer model and ensure eficient learning, several key optimizations were implemented: • Gradient Accumulation: Training utilizes a physical batch size of phys = 8 but applies Gradient Accumulation over = 8 steps. This simulates an efective batch size of ef = phys× = 64 , stabilizing gradient calculation and improving training convergence without requiring excessive GPU memory. • Learning Rate and Scheduler: The AdamW optimizer was initialized with a low learning rate (1 × 10 −5 ) and weight decay (0.01). A linear learning rate scheduler with zero warmup steps was used to gradually decrease the learning rate over the training process, promoting fine-tuning stability. • Early Stopping: The training process monitors the validation F1-score and employs an early stopping patience of 2 epochs to prevent overfitting on the training data.
The final classification prediction is not determined by the standard probability threshold a dynamic optimal threshold ( ) is calculated on the validation set: 1. The model generates probability scores (1) for all validation samples. 2. The P1 scores and true labels are used to construct the Precision-Recall Curve. 3. The threshold that maximizes the F 1-score on the validation data is selected. 0.5. Instead, The optimal threshold is determined by maximizing the F1-score ( 1) calculated as: Optimal = arg max ︂( 2
Precision( ) · Recall( ) · Precision( ) + Recall( ) ︂) This ensures that the final model is calibrated specifically for the task’s primary metric (F 1-score) and accounts for the high-imbalance context.
For the unlabeled test data, the predicted label (1 or 0) is determined by applying the optimized Predicted Label = {︃1 (Misinformation)
if 1 > 0 (Non-Misinformation) if 1 ≤ Texts filtered out during preprocessing are automatically assigned a label of 0 (non-misinformation). (2) (3) The performance metrics that the optimized RoBERTa classifier produced on the held-out test set are threshold :
shown in this section.
The checkpoint with the highest F1-score on the validation data, as identified by the ideal threshold, was used to evaluate the model on the held-out test set. The weighted F-score is used as the main indicator of overall system quality because of the extreme class imbalance (misinformation makes up about 1%).
The other teams taking part in the shared task [
(Section 3.2).
Achieving a high weighted F1-score validates the necessity and efectiveness of the core optimizations
The high F1 score indicates that the dynamic optimal threshold successfully located the optimal location on the precision-recall curve. Instead of depending on the poor default of 0.5, this guaranties that the model’s decision boundary is modified for balanced performance. This is necessary for practical implementation when reducing false positives and false negatives is crucial.
The performance achieved is due to the inherent contextual power of the RoBERTa architecture, which is enhanced by the special preprocessing pipeline (see Section 3.2). The model can leverage Social Media Feature Injection (tokenizing URLs, users, and hashtags) to analyze platform-specific cues as predictive language features. Misinformation propagators frequently use echo chambers (mentions) and viral dispersion techniques (URLs). Because of explicit tokenization, RoBERTa was able to encode these patterns more eficiently than ordinary clean-text tokenization.
This high-stakes, multilingual social media dataset is unusual, making a direct quantitative comparison with the literature mentioned in Section 2 dificult. Nevertheless, the performance (F_1 = 0.88) shows that the optimized RoBERTa classifier is very competitive. Previous research using simpler, more balanced, and cleaner benchmark datasets (like FakeNewsNet or LIAR) frequently reported higher accuracies (like in the mid-90% range). Our outcome, obtained on a clearly dificult, noisy, and very unbalanced dataset, demonstrates the resilience of the improved transfer learning method in an actual crisis situation.
We have classified the tweets as either misinformation or non-misinformation.In this study we took the dataset which is highly imbalanced. This dataset is a real world dataset which contain manually annotated tweets collected using the twitter API. In this our proposed model have achieved a F1 score of 0.88 . The F1-score was deliberately and inevitably selected as the primary criterion due to the inherent class disparity. This implies that the model performs well in terms of recall and precision for both groups. Furthermore, significant text pre-processing techniques were required to transform noisy, unstructured social media data into a format suitable for high-performance machine learning. This can act as a baseline for the new research in this domain. We have done the preprocessing of the text which is a foundation of this research.
During the preparation of this work, the author(s) used DeepL and Quillbot in order to: Grammar and spelling check. Further, the author(s) used Gemini-Banan for figures 1,2,3 and 4 in order to: Generate images. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.