The rapid growth of online misinformation poses serious risks to democratic societies and media trust, particularly in the context of Ukraine, where the ongoing Russian invasion has intensified the spread of disinformation. While large language models (LLMs) such as GPT-4 and Claude have shown impressive results in various natural language processing tasks, including misinformation detection in English, their effectiveness for low-resource languages like Ukrainian remains underexplored. In this study, we investigate the performance of modern LLMs from OpenAI, Google, Anthropic, and DeepSeek on the task of misinformation detection in Ukrainian news, using zero-shot and few-shot in-context learning (ICL) strategies. We also examine the impact of various prompting techniques, including direct classification and chain-of-thought reasoning, as well as the use of the Ukrainian language for prompting. To facilitate evaluation, we curated and cleaned a subset of Ukrainian misinformation data, building upon publicly available Ukrainian News dataset. Our experiments show that recent multilingual LLMs demonstrate strong comprehension and classification abilities for Ukrainian texts, achieving high F1scores even without fine-tuning. Importantly, we observe consistent performance when using either English or Ukrainian prompts, underscoring the cross-linguistic robustness of these models. However, the lack of large, high-quality Ukrainian misinformation benchmarks remains a major bottleneck for further progress. Our findings highlight both the potential and limitations of LLMs for misinformation detection in Ukrainian. While current ICL-based methods are promising, especially in low-resource settings, further gains are expected through supervised fine-tuning and improved data resources. This research lays the groundwork for future endeavors in developing more accurate and context-aware misinformation detection tools for the Ukrainian media.
The task of automated misinformation detection has undergone significant evolution over the
past two decades. Early approaches to text classification relied on classic machine learning (ML)
algorithms such as Naïve Bayes, Support Vector Machines (SVM), or decision trees, which operated
on manually engineered features, including TF-IDF or n-grams. These classic ML algorithms relied
heavily on the NLP-based preprocessing steps [
Building upon the transformer architecture, large generative language models (LLMs) such as
GPT-3 [
In the context of misinformation detection, LLMs have been explored under various in-context
learning (ICL) approaches and prompting paradigms. Zero-shot prompting involves asking the
model to classify a news item without any examples; few-shot prompting includes a small number
of labeled examples in the prompt to guide the model; fine-tuning involves updating the model's
weights using labeled misinformation datasets. Research in this area shows a lot of experimentation
and progress, and results grow with the increase in the LLMs’ performance. At the same time,
many studies demonstrated that fine-tuned BERT-based models often outperform LLMs in the
misinformation detection task, like Raza et al., 2025 [
While LLMs have shown promise in detecting misinformation in English, their effectiveness in
Ukrainian contexts is less clear. Applying these advances to Ukrainian news and social media posts
presents unique challenges. English-language datasets and benchmarks for fake news detection are
abundant (Mridha et al. provide a list of popular benchmark datasets [
This study aims to systematically evaluate the capabilities of major LLMs for detecting misinformation in Ukrainian news. Specifically, we investigate zero-shot, few-shot, and reasoningbased prompting strategies, analyzing how effectively these models generalize to Ukrainian content without fine-tuning. By presenting a rigorous comparison, we seek to identify strengths, limitations, and future directions for employing LLMs in Ukrainian-language misinformation detection, with a focus on practical applicability in low-resource settings.
The rapid rise of large language models (LLMs) has driven notable interest among researchers for
their application to text classification and misinformation detection. Unlike traditional deep
learning approaches and bidirectional transformer models, LLMs can analyze text in depth,
crossreference facts, and even perform basic fact-checking by detecting contradictions or checking
claims against their vast pre-trained knowledge. LLMs can be used without extensive task-specific
training, instead leveraging prompting (in zero-shot or few-shot modes) to adapt to the
misinformation detection task, which is attractive when labeled data is scarce. Wang et al. explored
the usage of LLMs in different modes (zero-shot, few-shot and fine-tuned) as text classifiers and
demonstrated their good performance on various datasets, which confirmed that LLMs can be
effective classifiers [
In the context of the misinformation detection task, Xu et al. [
In addition to text classification, LLMs can reason in natural language, which opens the door to
explainable outputs (e.g., generating a rationale for why a news piece is predicted fake) and
interactive fact-checking agents. Research by Pendyala et al. explored the capability of LLMs not
just to predict true/fake, but also to explain their decisions using chain-of-thought prompts or to
highlight evidence, and confirmed that LLM-based detectors could potentially provide richer
insights than a traditional black-box classifier [
However, there are also clear risks in relying on LLMs for misinformation detection. By design,
LLMs are trained to produce fluent and plausible text, which means they can hallucinate – i.e.,
generate information that sounds convincing but is false. An LLM might confidently assert a claim
is true or false based on incorrect “knowledge” it learned, leading to false positives or negatives.
Moreover, LLMs themselves can be double-edged: the same models can be misused by adversaries
to create more convincing fake news. For instance, Jiang et al. underscore that generative models
like ChatGPT have been used to produce highly persuasive but misleading content, raising the bar
for automated detectors [
Another important question is whether LLM-based detectors actually outperform the previous
generation of models (e.g. BERT-based classifiers) on misinformation tasks. Several recent studies
have conducted side-by-side comparisons, concluding that while BERT-like models often achieve
higher accuracy and efficiency on structured, pattern-driven tasks, LLMs consistently outperform
them on tasks requiring deep world knowledge, complex reasoning, or interpretability. Moreover,
prompt-engineering strategies – including zero-shot and few-shot prompting – enable LLMs to
perform well with minimal or even no task-specific training, a flexibility that classic models cannot
match. For example, Raza et al. reveals that BERT-like models generally outperform LLMs in
classification tasks, while LLMs demonstrate superior robustness against text perturbations [
LLMs can also be used to enhance smaller classifier models. An empirical study by Chen et al.
revealed that well-prompted LLMs achieve comparable performance in text-based misinformation
detection with SLMs, while LLM-enhanced detectors outperform plain SLMs in most cases when
generating richer textual features, producing analysis, and simulating user engagements [
Another noteworthy line of research is how to craft prompts or intermediate steps that enable
LLMs to make more accurate judgments about misinformation. One idea is to encourage explicit
reasoning, often via chain-of-thought (CoT) prompting. In a chain-of-thought approach, instead of
asking for a direct label, we prompt the LLM to first “think through” the content: for example, list
out the claims in the article, check each against known facts, and then conclude whether the article
is fake or real. This technique has yielded improvements in some domains of question answering
and reasoning. However, in the domain of misinformation detection, the evidence is mixed. Cao et
al. used chain-of-thought prompting for Financial Misinformation Detection and demonstrated that
it works better than a zero-shot approach for a number of different commercial LLMs [
Another interesting approach is the Hierarchical Step-by-Step (HiSS) prompting method by Zhang et al. [15] It directs LLMs to separate a claim into several subclaims and then verify each of them via multiple question-answering steps progressively. Experiment results on two public misinformation datasets show that HiSS prompting outperforms a state-of-the-art fully-supervised approach and strong few-shot ICL-enabled baselines. This technique addresses two main challenges in news claim verification: omission of necessary details and fact hallucination. 2.2. Misinformation Detection for Ukrainian language: Datasets and Approaches Most misinformation detection research is focused on English, but applying these approaches to Ukrainian presents some challenges. While English-language datasets and benchmarks for fake news detection are abundant (e.g., LIAR [16], FakeNewsNet [17]), resources for Ukrainian remain limited. A few notable datasets exist, such as the multilingual EUvsDisinfo corpus [18], the Mantis Analytics dataset [19], the UNLP 2025 Shared Task on Detecting Social Media Manipulation [20] and the Ukrainian News dataset [21], which targets the Russo-Ukrainian war discourse. These efforts are beginning to close the resource gap, but the scale and diversity of data in Ukrainian are still far behind those in English. Consequently, some researchers resort to cross-lingual transfer, adapting English models to Ukrainian tasks – an imperfect strategy that can introduce bias or produce misaligned representations, or opt for a more complex approach of crafting their own dataset.
For example, Dementieva et al. translated English datasets into Ukrainian using cross-lingual knowledge transfer methods, thereby avoiding manual data curation, including large multilingual encoders and translation systems, LLMs, and language adapters [22]. They then tested XLMRoBERTa on three text classification tasks: toxicity classification, formality classification, and natural language inference. Study from Vysotska et al. details the creation of a custom dataset for Ukrainian fake news detection based on a set of NLP techniques, providing a rare example of localized annotation [23].
Another notable effort to address the issue of the limited representation of low-resource languages like Ukrainian was done by Kiulian et al. [24], who tried to fine-tune the open-source Gemma and Mistral LLMs with Ukrainian datasets, aiming to improve their linguistic proficiency and benchmarking them against other existing models capable of processing the Ukrainian language. Additionally, they presented the Ukrainian Knowledge and Instruction Dataset (UKID) to aid future efforts in language model fine-tuning.
In terms of methods for Ukrainian misinformation detection, the progression is similar to the general evolution from traditional ML to LLMs, with additional constraints. Early attempts relied on translation or language-agnostic features. For instance, before Ukrainian training data was available, one might translate Ukrainian articles to English and then apply an English fake news detector – a cumbersome but sometimes effective workaround. With the advent of multilingual transformers, researchers moved to models like mBERT or XLM-RoBERTa [25], which are pretrained on dozens of languages, including Ukrainian. Fine-tuning such models on any available Ukrainian data has shown decent success. For example, Bazdyrev et al. (2025) employed a multilingual RoBERTa model adapted to the task and demonstrated reasonable accuracy in detecting certain propaganda messages, illustrating that transformer-based classifiers can be effective across languages when properly adapted [26].
In another study, Shupta et al. (2024) proposed an approach that utilizes different types of embeddings, including an LLM-based one, for misinformation classification and tested it on both English and Ukrainian fake news datasets [27]. They found that the classification results for the Ukrainian dataset were about 2-3% worse than for the English one.
Despite these challenges, the research work is pushing forward. The Ukrainian community and researchers are beginning to assemble more misinformation detection datasets in Ukrainian language. The good examples here are the Kaggle-based “Disinformation Detection Challenge” and the UNLP 2025 Shared Task on Detecting Social Media Manipulation mentioned above, which provided their own datasets and brought attention to the domain for many research teams to develop their custom models.
In summary, applying LLMs and other detectors to Ukrainian is a promising but still developing frontier – initial results show approaches similar to those in English can work, but they must be adapted to the linguistic and data constraints, and there is a need for more Ukrainian-specific resources to speed up this work.
Effective misinformation detection using LLMs critically depends on the availability of high-quality benchmark datasets. As it was mentioned in the previous section, while the English language benefits from a range of well-established datasets, which offer both diverse content and detailed annotations, the situation for Ukrainian-language resources remains more challenging. There are not many Ukrainian datasets on this topic, and the available ones are often small in size, exhibit thematic bias, and in some cases rely on automatic or source-based labeling strategies rather than human-verified fact-checking. As a result, no benchmark has been established yet, making it difficult to compare different research efforts with each other.
For example, EUvsDisinfo [18] mentioned above provides some support for Ukrainian through its annotations of pro-Kremlin disinformation. However, the Ukrainian subset of this dataset is notably small and unevenly distributed across topics, confirming a broader issue in sourcing reliable benchmarks for low-resource languages like Ukrainian. The Mantis Analytics dataset [19], introduced in the Kaggle-based “Disinformation Detection Challenge” is more focused on content relevant to the war, often collected from suspicious or fringe Telegram channels, thus reflecting realistic misinformation scenarios. The UNLP 2025 Shared Task on Detecting Social Media Manipulation provides a great humanly annotated dataset for manipulation technique classification and span identification [20], which is slightly different type of the problem than misclassification detection.
Another example of the Telegram-based datasets is Ukrainian News [21], which contains 10700 news from 2022. Though the news were collected from the start of the full-scale Russian-Ukrainian war, they cover not only war-related topics but also politics, culture and some other topics which makes it a valuable data source.
The lack of datasets in Ukrainian language motivate some authors to create their own datasets to get a well-annotated set of data for training and testing. We decided to use this approach for our study.
LLMs can be applied to the task of misinformation detection as text classifiers, typically framed as a binary classification problem: the model determines whether a given news article is “true” or “fake”. This can be achieved through various prompting techniques, even without fine-tuning. There are several choices of LLMs available currently from different companies, including OpenAI (GPT family), Google (Gemini), Anthropic (Claude), xAI (Grok), Meta (Llama), Mistral, Alibaba (Qwen), DeepSeek, and others.
Each of these companies produces models of varying sizes, suitable for different use cases. These models also vary in the following attributes: speed of inference, multimodality, context window, function calling, structured outputs, knowledge cutoff date, web search support, reasoning support, fine-tuning support, price for input and output tokens and many others. Therefore, when using them for our task, we need to try different models, as they can provide varying performance in misinformation detection.
Recently developed reasoning-optimized models, such as GPT-o3, Claude Sonnet 3.7, or Gemini 2.5 Pro, have shown promise in handling different tasks more effectively. These models are architecturally designed to support longer context, perform intermediate reasoning, and adhere more strictly to prompt constraints, making them well-suited for misinformation detection in complex or low-resource linguistic settings.
One compelling aspect of using LLMs is the variety of in-context learning (ICL) strategies available. Approaches to employing LLMs for misinformation detection fall broadly into zero-shot prompting, few-shot prompting, and fine-tuning.
In zero-shot learning, an LLM is prompted with only the task instructions (e.g., “Is this article credible?”) and the input text, relying entirely on its pretraining to make a decision. Zero-shot prompting enables immediate deployment across domains, though models typically lag behind in overall accuracy. This method is particularly attractive for low-resource settings such as Ukrainian, where labeled examples are scarce, but may struggle with nuanced or domain-specific content.
Few-shot learning improves upon zero-shot by providing the model with a handful of labeled examples directly in the prompt, offering implicit task guidance and contextual anchoring. While still non-parametric, few-shot prompts can improve performance on low-resource tasks through implicit adaptation.
Fine-tuning, though more resource-intensive, involves training the LLM on a labeled dataset, allowing it to adapt specifically to the task and language. This approach usually provides the best performance of the model. Fine-tuning requires having a special API provided by the company (e.g. GPT-3.5 or Mistral 7B) or open-sourced model weights (e.g., Llama or DeepSeek).
Prompt engineering plays a critical role across all approaches. Basic prompting strategies include direct task framing (e.g., “Classify this text as misinformation or not”) with definition support and instruction-based formulations. This approach can serve as a baseline for further improvements and complications.
More advanced techniques include Chain-of-Thought (CoT) prompting, where the model is encouraged to articulate reasoning steps before delivering a final verdict. In CoT, instead of asking for a direct label, we prompt the LLM to first “think through” the content: for example, list out the claims in the article, check each against known facts, then conclude if the article is fake or true. This technique has yielded improvements in some domains of question answering and reasoning. It can be especially valuable in misinformation tasks, as fake news often exploits subtle contextual cues that require multi-step reasoning.
Other interesting approaches include self-consistency decoding, which aggregates multiple generations to improve robustness, and retrieval-augmented generation (RAG), which supports evidence-grounded classification. In RAG, the relevant external evidence (e.g., fact-checked articles) is retrieved alongside the input, where possible, enabling the LLM to ground its assessments.
To explore the capabilities of large language models (LLMs) in detecting misinformation in the Ukrainian language, our experimental framework uses a carefully annotated benchmark dataset, a spectrum of LLMs, and several prompting techniques.
We based our evaluation on the Ukrainian News dataset [20] mentioned above – a collection of messages from Ukrainian Telegram channels during the Russian-Ukrainian War. This dataset is particularly valuable due to Telegram’s role as a major platform for both verified news and disinformation during wartime, making it a relevant source for real-world misinformation detection.
However, initial exploration of the dataset revealed several critical issues. The dataset contains duplicates, announcements, or posts lacking substantial factual content, making them unsuitable for evaluating a model's ability to distinguish truth from misinformation. Furthermore, label reliability was problematic: numerous "fake" and some “true” samples were mislabeled, most likely due to the source-based labeling approach rather than human-verified fact-checking.
To address these limitations, we randomly selected a significant subset of messages for manual curation. Each message was carefully reviewed to assess whether it contained a factual claim that could be evaluated as true or false. Messages that were non-factual, badly written, or too short for evaluation were removed. The remaining data was reannotated. After that, the final dataset of 400 samples was randomly selected in a balanced way: 200 “true” and 200 “false” samples.
For this experiment, we selected a set of LLM models of different sizes, novelty and capabilities, provided by OpenAI, Google, Anthropic and DeepSeek. The list of models with their main characteristics is provided in Table 1.
This selection of models allowed us to examine how factors such as model scale, novelty (knowledge cutoff), reasoning capacity, and architecture impact performance on misinformation detection in a low-resource language setting. In particular, we were interested in how reasoningaugmented models (e.g., GPT-o3, Gemini Pro 2.5, Claude Sonnet 4 and DeepSeek-R1) perform relative to lightweight or instruction-tuned variants (e.g., GPT-3.5, Gemini Flash 2.0, DeepSeek-V3). GPT-3.5 model is used as a baseline model which is expected to perform poorly compared to newer and bigger models with more recent knowledge cutoffs.
To test the models’ reasoning processes and understand how instruction phrasing and language affect performance, we implemented four prompting techniques:
Basic (no analysis), English: A simple, direct prompt in English asking whether the given Ukrainian-language text is true or fake, with no additional context or rationale required. Chain-of-thought (analysis), English: An English-language prompt instructing the model to provide a step-by-step explanation of its thinking before delivering its final judgment. Chain-of-thought (analysis), Ukrainian: The same analytical prompt translated into Ukrainian, intended to test whether linguistic alignment between input and instructions improves model reasoning. 4. Few-shot chain-of-thought (analysis), Ukrainian: A context-rich Ukrainian prompt that includes multiple labeled examples of true and fake samples, enabling in-context learning through few-shot prompting.
Each model was tested with all four prompting techniques to enable cross-comparison. Table 2 provides the texts of each prompt.
different perspectives and give your judgment whether it is true or fake at the end.
Verdict should be "true", "probably true", "probably false" or "false", don't add any other text on top of it.
If you are not sure about the verdict, do your best to suggest "probably true" or "probably false" verdicts.
Use the following structure for your answer: "Analysis: <put analysis here> Verdict: <put verdict here>" Text is provided below: Перевір факти, наведені в тексті новини нижче, проаналізуй їх з різних перспектив та надай свій вердикт, чи цей текст правдивий чи брехливий. Вердикт повинен бути "правда", "скоріше правда", "скоріше брехня" або "брехня", не додавай ніякого іншого тексту. Якщо ти невпевнений на рахунок вердикту, використовуй варіанти "скоріше правда" або "скоріше брехня". Використовуй наступну структуру для своєї відповіді: "Аналіз: <додай аналіз сюди> Вердикт: <додай вердикт сюди>" Текст для аналізу надано нижче: Перевір факти, наведені в тексті новини нижче, проаналізуй їх з різних перспектив та надай свій вердикт, чи цей текст правдивий чи брехливий. Вердикт повинен бути "правда", "скоріше правда", "скоріше брехня" або "брехня", не додавай ніякого іншого тексту. Якщо ти невпевнений на рахунок вердикту, використовуй варіанти "скоріше правда" або "скоріше брехня". Використовуй наступну структуру для своєї відповіді: "Аналіз: <додай аналіз сюди> Вердикт: <додай вердикт сюди>" Приклад 1: Текст: <текст1> Аналіз: <аналіз1> Вердикт: правда
The evaluation followed a consistent protocol across all models and prompting strategies. Each sample from the curated benchmark dataset was input to the model through its respective API using a standardized prompt template. To ensure comparability, the same prompt structure and parameters were used across models for each strategy.
For chain-of-thought prompts, both the reasoning and final answer were logged. Model outputs were post-processed to standardize label formats (e.g., "True", “Probably True”, “Probably False”, "False", etc.) for evaluation.
While conducting the experiments, evaluation metrics included: • • • •
This experimental protocol was designed to ensure fairness, repeatability, and depth in assessing how well LLMs can reason about misinformation in the Ukrainian language, especially when faced with real-world content from conflict-related media environments.
In this section, we present the results of the assessment of how existing reasoning and nonreasoning LLMs perform on the task of misinformation detection for the Ukrainian language. We provide the classification metrics for a selected set of models and hand-crafted prompts in tables and heatmaps.
Table 3 presents a comprehensive comparison of 13 LLMs with four prompts with Precision, Recall and F1-score. GPT 4o GPT o3-mini
Pr 0.72
To facilitate comparison, we also provide a heatmap of F1-scores by model and prompt type.
DeepSeek-R1 demonstrates the highest Precision (but poor Recall, very high bias), while GPT o3-mini demonstrates the highest Recall (and relatively good Precision).
Bigger models, such as Gemini Pro 2.5 or Claude Sonnet 3.7/4, provide better results for a basic prompt compared to their smaller versions (Gemini Flash 2.0/2.5 and Claude Haiku 3.5).
DeepSeek models perform the poorest with a basic prompt, they need analysis to compensate for their low Recall.
Example of the chain-of-thought output by GPT 3.5 and GPT o3 models for comparison (EN prompt):
“День міста у Харкові почався з обстрілів Київського района. Є влучання у приватний будинок, без постраждалих, - повідомив мер Харкова Ігор Терехов. Він привітав харків’ян зі святом свого незламного міста і нагадав, що Україна обов’язково переможе у війні з російським ворогом.”
“Analysis: The news report claims that the Day of the City in Kharkiv started with shelling in the Kyivsky district, with hits on a private house but no casualties reported. The mayor of Kharkiv, Igor Terekhov, is said to have greeted the residents and expressed confidence in Ukraine's victory in the war against Russia.
Verdict: Probably False. This news report contains several inconsistencies and inaccuracies. Firstly, Kharkiv is a separate city from Kyiv, so it is unlikely that shelling in Kharkiv would be described as occurring in the Kyivsky district. Additionally, there is no verifiable information or reports from credible sources to confirm the alleged shelling incident at the start of the City Day celebrations in Kharkiv. Therefore, the claim is likely false.”
“Analysis: 1. Date of the event. Kharkiv officially celebrates its City Day on 23 August every year. Therefore, a report that starts with “День міста у Харкові” (City Day in Kharkiv) logically refers to 23 August.
2. Claim of shelling. Russian forces have repeatedly shelled Kharkiv since February 2022, and the city’s Kyivskyi (Київський) district is often among the hit areas. On 23 August 2023, multiple Ukrainian news outlets (e.g., Suspilne, RBC-Ukraine, Ukrinform) quoted Kharkiv mayor Ihor Terekhov saying that the city was being shelled in the early morning, with at least one strike on a private house in the Kyivskyi district and no casualties reported. The wording in those reports is virtually identical to the wording in the text provided.
3. Source credibility. Ihor Terekhov maintains an official Telegram channel that Ukrainian media routinely cite for real-time information on shelling. His statements on 23 August 2023 match the claims in the text (strike on a private house, no casualties, morning of City Day).
4. No contradictory evidence. No reputable source has refuted Terekhov’s report or offered an alternate version of events for that date.
Considering the alignment between the text and multiple independent, reputable Ukrainian news reports of the same morning, the information appears accurate.
Verdict: True”
In this example, the GPT 3.5 model demonstrates several mistakes that prevent it from reaching the correct verdict. First, its knowledge cutoff (Sep 2021) doesn’t allow it to “know” the facts of shelling Kharkiv and even the Russian-Ukrainian war itself. Second, it connected the Kyivsky district to Kyiv city and didn’t understand that it is a district in Kharkiv. And last but not least, it ignored the instruction not to add any additional information into the Verdict section and actually added a part of the analysis there.
At the same time, the GPT o3 model not only demonstrated the capabilities to split the sample claim into several parts and verify each of them, but also used its own “memory” of the facts to verify and weigh the claims.
In Table 4, we present the aggregated average Precision, Recall and F1-score for each LLM across all prompts.
DeepSeek-R1
Based on Table 4 and Figure 2, we can conclude several trends about the overall model performance:
GPT o3 provides the best performance overall and across reasoning models. GPT 4o provides the best performance across non-reasoning models.
Models tend to have their own bias toward making True or False predictions. For example, OpenAI models have better Recall, while Gemini, Claude and DeepSeek models tend to have noticeable higher Precision, demonstrating higher bias. The only exceptions are GPT 4o and GPT o3, which provide both high Precision and Recall, making them the two bestperforming models.
Newer versions of models don’t mean better performance: for example, older version of Sonnet (Sonnet 3.7) performs better than a new one (Sonnet 4.0), and GPT o3-mini performs better than GPT o4-mini.
The last comparison, which is interesting for our analysis, is the comparison of prompting techniques. In Table 5, we present the aggregated average Precision, Recall and F1-score for each prompt type across all models.
Here are the main observations from Table 5 about the overall prompt type performance: 1. A basic prompt without analysis provides the worst average performance (average F1-score) across the models, showing higher Precision, but significantly lower Recall. 2. Reasoning prompts (with analysis) offer a better Precision/Recall balance, resulting in a significantly higher average F1-score. 3. Ukrainian vs. English prompts: The analysis prompt in both Ukrainian and English languages yields similar average F1-scores, suggesting that the Ukrainian language is wellsupported in the prompt. 4. The few-shot prompt outperforms the zero-shot one written in Ukrainian language by almost 2%.
The application of LLMs for misinformation detection in Ukrainian news texts presents several unique challenges and promising opportunities. One of the central issues identified in this research is the scarcity of well-annotated, publicly available, and sufficiently large datasets for the Ukrainian language. While English-language datasets such as LIAR and FakeNewsNet are frequently used for training and benchmarking misinformation detection models, equivalent resources in Ukrainian remain limited in scope, quality, and popularity. Existing Ukrainian datasets often suffer from inconsistencies in labeling, unbalanced distributions, and insufficient sample sizes, making their usage difficult for reliable benchmarking or training. This gap underlines the need for future work on the creation, validation, and dissemination of high-quality Ukrainian misinformation datasets. Such datasets would not only facilitate more accurate evaluations but also help establish reproducible benchmarks for Ukrainian NLP research.
Despite this limitation, our experiments show that current LLMs are capable of processing Ukrainian texts with high competence. When tested using both English and Ukrainian prompts, most reviewed LLMs exhibited consistent performance, demonstrating similar quality when the prompt language was switched to Ukrainian. This suggests that these multilingual models have acquired strong comprehension capabilities across languages and can accurately follow instructions and reasoning chains in Ukrainian. Moreover, the models typically respond in the same language as the prompt, ensuring consistency in multilingual workflows. This finding is encouraging for lowresource language communities, indicating that even without fine-tuning, state-of-the-art LLMs can be leveraged effectively for Ukrainian NLP tasks, including misinformation detection.
Nevertheless, there are limitations to our study. Not all recent LLMs were included in the evaluation, and only a subset of prompting techniques was tested. The focus was primarily on zeroshot and few-shot scenarios, with an emphasis on direct classification prompts and chain-ofthought prompting. The results reveal that the best evaluated models achieved F1-scores in the range of 0.8 to 0.9, highlighting strong zero-shot performance across a variety of LLMs. Still, these results likely represent a lower bound on the models’ true capabilities. Further performance gains are anticipated through more sophisticated prompting strategies, including multi-turn reasoning, contextual priming, or the use of external knowledge sources.
An important direction for future work is the use of fine-tuning. While our study did not include any fine-tuned models, prior research suggests that supervised fine-tuning on task-specific data can yield substantial improvements in classification accuracy, especially when combined with in-domain examples. Fine-tuning LLMs on Ukrainian misinformation data – once a reliable dataset is available – could help close the remaining performance gap and provide even more robust tools for misinformation mitigation.
Finally, beyond classification, our experiments highlight the potential of LLMs in supporting data annotation and fact-checking tasks. The explanation-oriented outputs generated by models such as GPT o3 demonstrate a level of reasoning that can be valuable for human annotators. These outputs often provide relevant contextual knowledge and reasoning paths that help justify predictions, making the annotation process faster and more informed. While these models are not yet perfectly reliable as autonomous misinformation detectors, their fact-checking capabilities can significantly assist journalists, researchers, and general users in evaluating the credibility of news claims. Another potential application of the reasoning output of the models could be in using them as additional input for the SLMs like BERT, which can improve the performance of such models further.
In this study, we explored the effectiveness of large language models (LLMs) for the task of misinformation detection in Ukrainian news. We examined the current research landscape with a focus on in-context learning (ICL) strategies, including zero-shot and few-shot prompting, as well as the impact of different prompting strategies, ranging from direct prediction requests to more elaborate chain-of-thought analyses. Our analysis highlights that modern reasoning LLMs, when equipped with appropriate instructions, can achieve significant performance on this task even without task-specific training.
We also assessed the current state of misinformation detection, specifically in the context of Ukrainian texts. Despite recent progress, there remains a significant lack of large-scale, highquality, publicly available datasets for Ukrainian-language misinformation detection. Existing datasets are often noisy, limited in size, or not annotated in a way that supports robust benchmarking. This gap hinders the development of accurate models and comparative evaluations, underlining a clear need for the creation and publication of better Ukrainian-language datasets in this domain.
To understand how current LLMs perform on this task, we evaluated a diverse set of modern models, including those developed by OpenAI, Google, Anthropic, and DeepSeek. These models were tested under various prompting strategies in both English and Ukrainian, revealing that language choice in prompts does not substantially affect model performance. The results also confirmed that many state-of-the-art models, particularly when using analytical prompts, achieve strong F1-scores (up to 0.895), demonstrating a high level of reasoning and instruction-following ability, even in a low-resource language context.
Nevertheless, our findings also suggest that zero- and few-shot settings may not fully exploit the potential of these models, especially when high precision is required for downstream applications such as fact-checking or content moderation. As a result, future work will explore the fine-tuning of LLMs and smaller language models (SLMs), such as BERT-based architectures, to further enhance accuracy and robustness. This next phase of research will aim to better align model outputs with the nuances of Ukrainian-language misinformation and contribute more reliable tools to the information verification ecosystem.
During the preparation of this work, the authors used Grammarly and ChatGPT in order to: grammar and spelling check, stylistic improvements to make text more formal, generate heatmaps for figures 1 and 2. After using these tools, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.