This study presents a comprehensive analysis of user interactions and assistant responses across 28 European languages using the WildChat and LMSYS datasets, addressing an existing gap in the understanding of multilingual conversational AI. We examine five specific dimensions: the topics discussed, the length of the conversations, the sentiment expressed, the toxicity of the interactions, and the quality of the responses. Our findings indicate significant cross-linguistic variations that have significant implications for the development and deployment of language models. Topic analysis shows a high degree of overlap across languages, indicating that users engage with similar subjects regardless of their linguistic background. We observe a positive correlation between the frequency of language use and conversation length, which suggests that different engagement patterns may be at play across language communities. Sentiment analysis indicates a high degree of consistency in neutral tones across languages, whereas toxicity levels vary considerably, with some languages exhibiting notably elevated scores. To assess response quality, we introduce a custom neural architecture based on the classification of user-assistant interaction triples. Our model achieved an accuracy of 0.82 and served to uncover variations in user satisfaction across language groups. Speakers of Romance languages exhibited higher levels of satisfaction, whereas those of Eastern European languages tended to show lower satisfaction with their interactions with the assistant. Our findings underscore the need for language-specific strategies in conversational AI development, particularly in content moderation, conversation design, and quality assessment. By highlighting the differences and commonalities in conversational interactions across languages, our work provides insights for researchers and developers seeking to better understand and address the needs of users across a diverse linguistic landscape.
The development of conversational assistants has seen significant advancements in recent years, following the advent of Transformer-based architectures [1], as exemplified by the GPT family of models. These assistants are capable of engaging in conversations with users on a wide range of topics, providing information, answering questions, and even engaging in small talk [2].
The popularity of conversational assistants has therefore surged across various applications, including customer service and language learning [3,4]. Notably, ChatGPT exceeded 100 million monthly active users within its first two months, a testament to the rising interest in conversational AI [5]. However, despite this booming interest, research and development in the field remain predominantly Englishcentric, with limited focus on other languages [6,7].
A branch of research that is key to the development of conversational assistants is the analysis of user interactions and assistant responses in a large scale. Two main works in this area due to their size and diversity are the WildChat and LMSYS datasets [8,9], which contain a large number of user interactions with conversational assistants and are publicly available. However, there is still work to be done in analyzing these datasets, due to their size and complexity.
This study aims to explore user interactions and assistant responses, focusing on linguistic differences across European languages. Previous research has only examined datasets as a whole [10], without delving into cross-language discrepancies-a gap our study seeks to fill. In contrast, we seek to provide insights into cross-language differences within large-scale conversational datasets, in order to inform the development of assistants that are better tailored to the needs of users of a wider range of languages. Our analysis focuses on language groups rather than the countries in which users reside. We acknowledge that languages like Spanish or Portuguese may have significant representation from Latin American users, contributing to the diversity of our findings. The rest of this paper is structured as follows. The subsequent sections examine whether notable differences exist across languages in terms of topics discussed (RQ1), length of interactions (RQ2), sentiment expressed (RQ3), toxicity (RQ4), and quality (user satisfaction) of responses (RQ5). We then proceed to an integrative discussion and conclusion.
In this section, we describe the datasets used for our analysis.
We combined the WildChat and LMSYS datasets [8,9], which contain 990,372 and 1,000,000 examples, respectively. Nevertheless, the count of examples we analyze is lower, as we applied several preprocessing steps to clean the data. First, we discarded examples where the user's initial message contained fewer than five words, as short inputs hinder the accuracy of language detection. For instance, messages like "Test" were often misclassified, such as being incorrectly identified as Estonian. Furthermore, we excluded interactions where the toxicity scores were not defined, as these annotations are essential for our toxicity analysis.
To ensure a comprehensive representation of European languages, we included 28 languages, shown in Figure 1, together with the number of examples in each. However, we excluded languages such as Basque and Breton because of the lack of a sufficient number of examples in the datasets, as well as the lack of language models trained specifically for these languages, which could introduce bias when using them in our analysis. After applying these filtering criteria, we obtained a total of 781,376 examples.
In this section, we analyze the distribution of topics across languages. We utilize language tags as a clustering feature for semantically-informed embeddings. We hypothesize that if speakers of different languages tend to discuss distinct topics, language tags should form clusters with clear boundaries. However, if topics are more uniformly distributed across languages, the boundaries will be blurrier.
We commence our analysis by selecting all initial user messages in the dataset, as these are the most representative, establishing the context for subsequent discourse. We then use two multilingual models from the Sentence Transformers library [11]-paraphrase-multilingual-mpnet-base-v2 and paraphrase-multilingual-MiniLM-L12-v2-to generate embeddings of these messages.
To evaluate how well messages cluster based on language, we conduct a silhouette score analysis (ranging from -1 to 1), which measures cluster cohesion and separation. A score of 1 indicates welldefined, separated clusters, while a score of 0 suggests overlap. Negative scores indicate poorly defined or incorrect clusters [12].
Due to the high computational cost of this analysis, which scales quadratically with the number of examples, we performed the silhouette score calculation on 20 randomly-selected subsets, each representing 5% of the dataset, and then averaged the results. The mean silhouette scores for the two models were β0.121(24) -mean(s.d.) -and β0.115(30). These negative scores indicate significant semantic overlap between clusters, confirming that languages are not effective clustering tags.
Per-language silhouette scores, which align with the overall results, are provided in the supplementary material. Based on this analysis, we conclude that users across linguistic groups engage with similar themes, rather than showing strong language-specific patterns. This finding highlights the universality of topics and suggests that conversational assistants must prioritize general topic coverage and flexibility. Moreover, the lack of clustering implies that cultural or regional nuances may play a smaller role in topic differentiation than previously expected.
In this section, we explore the correlation between the mean number of words written by users in a conversation and the number of examples per language. As both variables are not normally distributed, we study their interaction by calculating the Spearman Rank correlation coefficient, which ranges from -1 to 1. A value of -1 indicates a perfect negative correlation; 0, no correlation; and 1 signifies a perfect positive correlation. The coefficient evaluates to 0.51, indicating a moderate positive correlation between the number of user words and the number of examples in a language. With a p-value of 0.0055 < πΌ = 0.05, the correlation is statistically significant.
While not a perfect correlation, the strength of the relationship suggests that the number of user words can be considered a reasonably reliable predictor of the number of examples in a language. Figure 2 presents a scatter plot illustrating this relationship. Additionally, Table 1 provides the per-conversation mean number of messages, user words, and assistant words for each language.
In essence, we find that speakers of languages with more interactions in conversational datasets tend to engage in longer conversations, as measured by the number of user words. This trend provides moderate support for the notion that there exist behavioral differences across groups of language users, which may be indicative of cultural or linguistic factors influencing conversation length. However, further research is warranted to ascertain the underlying reasons for these differences.
In this section, we explore sentiment differences across languages, as it allows to understand the emotional tone of conversations and provides insights into both user experience and interaction quality.
We use the pre-trained multilingual classification model twitter-XLM-roBERTa-base [13] for our analysis. This model is particularly suited to our multilingual setting, as it has been trained on a large, diverse corpus of Twitter messages in various languages.
Our analysis focuses on both the user and the assistant first messages in each conversation, as these set the tone and are likely the most representative of the overall sentiment. We calculate sentiment scores for both messages, then aggregate these scores across languages. We show the results in Table 2.
One clear observation is that the assistant's messages tend to be more positive than the users', with an overall mean of 0.2205(15) compared to 0.16 (11). This difference likely reflects the Reinforcement
Mean and standard deviation for the sentiment scores of the first user (U) and assistant (A) messages. Learning from Human Feedback (RLHF) paradigm used for training [14], which encourages the assistant to maintain a more positive and helpful tone. Interestingly, the assistant also shows a slightly higher negative sentiment score (0.28 (20)) than the users (0.26 (20)). This could also be attributed to RLHF, which prompts it to refrain from engaging in potentially toxic conversations, thus increasing the frequency of negative sentiment classifications in those contexts [15]. Overall, however, both user and assistant messages tend to be neutral, with a mean neutral score of 0.50 (18) and 0.58 (19), respectively. These observations show that while there are differences in the sentiment expressed by users and assistants, the sentiment across languages tends to remain fairly neutral.
We perform a toxicity analysis to identify potential differences, for every message in the dataset (a total of 2,951,678). There exist 11 types of toxicity annotations generated by the dataset creators with the OpenAI Moderations API. To simplify our analysis, we aggregate these categories into two general toxicity scores using the mean and maximum values across the categories. We do this for each example. Then, we calculate the mean of the two across the examples, which we present in Table 3.
In the rest of this section, we focus on the perspective of the user messages, as they are generally causative of the toxicity in the assistant's side of the conversation. To identify significant differences in toxicity across languages, we employ the Kruskal-Wallis test. This is a non-parametric method well-suited to compare medians of toxicity scores across multiple independent samples, especially given the non-normal distribution of toxicity values, which cluster near the extremes. We test a null hypothesis π» 0 = "The medians of the toxicity scores are equal across languages" with πΌ = 0.05. Since toxicity scores are continuous and generally skewed towards values close to 0, directly applying the Kruskal-Wallis test might exaggerate differences between languages due to minor deviations in low-toxicity messages. To mitigate this, we round the scores to two decimal places, reducing the number of unique values and treating very close values as ties. The outcomes of the Kruskal-Wallis test, as well as independent analyses for each toxicity category and aggregated scores, are presented in Table 4.
With the exception of the "self-harm" category, the p-values for all other categories are less than πΌ, indicating significant differences in toxicity across languages. This necessitates pairwise comparisons to determine which languages exhibit meaningful disparities. We use Dunn's test for this purpose, with the null hypothesis π» 0 = "The probability that a randomly selected message from one language has a higher toxicity score than one from another language is 0.5. " To control for the increased risk of Type I errors due to multiple comparisons, we apply the Bonferroni correction to the p-values. The results of these pairwise comparisons for the averaged toxicity scores are shown in Figure 3, while results for other configurations are included in the supplementary materials.
Key findings from the pairwise comparisons include:
1. No Statistically Significant Differences: For most languages-such as Bulgarian, Croatian, Czech, Danish, Estonian, Finnish, Galician, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Norwegian, Polish, Romanian, Slovak, and Slovenian-the comparisons are not statistically significant. In these cases, we cannot reject π» 0 , indicating it is equally likely that a randomly selected message from one of these languages has a higher or lower toxicity score than another. 2. Significant Differences in Specific Languages: Pairwise comparisons involving Dutch, English, French, German, Spanish, Swedish, and Ukrainian reveal significant differences compared to several languages from the first group (Greek, Italian, Polish, Portuguese, Romanian, Spanish, Swedish and Ukrainian). These languages show distinct toxicity profiles, although notable exceptions exist within this subset. 3. Group of Similar Toxicity: While Dutch, English, French, and German exhibit significant differences compared to other languages, they are not significantly different from one another, suggesting a group of languages with similar, higher-than-average toxicity scores. 4. Highest Average Toxicity: Languages such as Catalan, Dutch, English, German, and Swedish display the highest average toxicity scores, with Dutch, English, German, and Swedish identified as significantly different in the majority of pairwise comparisons.
Overall, the results highlight considerable variability in the toxicity of conversations across languages. The distribution of toxicity scores varies significantly between some languages, suggesting that conversational toxicity may be influenced by a range of factors, including cultural backgrounds, the structure of the language itself, potential biases in toxicity tagging, or a combination of these influences.
In this section, we assess the quality of the assistant's responses, which we define as how well the assistant meets the user's needs and expectations. To the best of our knowledge, no large-scale multilingual dataset exists with labels for assistant response quality, such as those used in RLHF [16]. This motivated us to develop a custom architecture tailored for evaluating assistant responses in a multilingual setting.
Our approach is inspired by the siamese architecture proposed in Sentence-BERT [11]. While the original architecture encodes two inputs and trains a classification head that is later discarded to retain only the fine-tuned encoder, we focus on training the classification head with three inputs to evaluate the quality of assistant responses, the encoder remaining frozen. The classification task asks "Is the user satisfied with the assistant's response?", with classes "Yes, " "No, " and "N/A" (not applicable).
We structure the model to process β¨π’ 1 , π, π’ 2 β© triples, where π’ 1 is the user's initial message, π is the assistant's response, and π’ 2 is the user's second message. The goal is to capture the semantics of the user-assistant exchange and assess whether the assistant's response satisfies the user's original query.
For instance, if the user were to ask about the weather, the embedding of their first message may belong to a subspace "weather"; if the assistant were to respond with the weather forecast, the embedding of the assistant's response could also belong to the same subspace. The user, having received a satisfactory response, may then express their satisfaction in their second message, the embedding of which could encode a positive sentiment. We expect the model to be able to capture the distances between the embeddings of the messages (e.g., the assistant's response addressing the user's first message), as well as the semantics of the messages themselves (e.g., the user's satisfaction with the assistant's response). Formally, our model takes a tokenized input of shape 3 Γ πΏ, where πΏ is the maximum length of the tokenized input (shorter sequences are padded), the three messages being stacked along a new dimension. The model outputs a pooled representation π(π) for each message in the triple, where π β π’ 1 , π, π’ 2 , producing embeddings π(π’ 1 ), π(π), and π(π’ 2 ) of dimension π π = 768.
We concatenate these embeddings with their absolute differences into a single vector π πππ : We pass π πππ through a fully connected feedforward neural network with four layers, π π and π π being the weights and biases of layer π, each with a hidden size of 884 and Mish [17] activation functions to obtain the output logits π πππ :
Finally, we apply an argmax operation on π πππ to determine the class prediction. A diagram of this architecture is shown in Figure 4. We used Optuna [18] to optimize the hyperparameters of the classification head and training. Table 5 shows the search space and best values found.
For training, we built a dataset of β¨π’ 1 , π, π’ 2 β© triples by sampling up to 1000 conversations per language with at least three messages. We manually annotated 1000 examples using the Argilla platform [19]. We also generated translations for non-English conversations to assist the annotation process using the EuroLLM-1.7B model [20]. We did not assess the quality of the translations, as they were utilized solely for contextual purposes, thereby facilitating a general comprehension of the discourse. Furthermore, we established explicit guidelines to facilitate a uniform interpretation of the task. Our annotation process maintained reasonable balance across languages, with an average of 37(33) annotations/language. Specific counts and annotation guidelines are provided in the supplementary material.
We trained the model using cross-entropy loss, optimized with AdamW [21] for 10 epochs, with the default hyperparameters in the Hugging Face Transformers library [22], π½ 1 = 0.9, π½ 2 = 0.999, and π = 1 Γ 10 β8 . For the encoder, we utilized the paraphrase-multilingual-mpnet-base-v2 pretrained weights [11]. We reserved 10% of the data for validation and another 10% for testing, performing 10-fold cross-validation. The model achieved a mean test accuracy of 0.820(45), significantly outperforming both a random baseline (0.339(42)) and a majority class classifier (0.594(44)).
When applying the model to the full dataset (raw numbers are reported in the supplementary material), the results (Figure 5) reveal notable discrepancies in user satisfaction across linguistic groups. Disregarding languages with a limited number of examples, where the results may lack representativeness, we observe that the satisfaction of users with the assistant's responses is generally high and tends to be higher for users speaking languages belonging to the Romance language family [23], including Italian, Portuguese, Spanish, Catalan, French, and Romanian. In contrast, users speaking languages that originated in Eastern Europe, such as Bulgarian, Greek, and Hungarian, exhibit a lower level of satisfaction with the assistant's responses. English-speaking users demonstrate a satisfaction level that is comparable to the overall mean, similar to that observed in German and Polish, although the latter two languages exhibit a higher percentage of unsatisfied users.
Our analysis of the WildChat and LMSYS datasets reveals significant differences in user-assistant interaction across European languages. These findings contribute to a more nuanced understanding of multilingual conversational AI and highlight the importance of considering linguistic diversity in the development and evaluation of language models.
The lack of clear clustering by language in our topic analysis (RQ1) suggests that users across different languages engage with conversational assistants on a wide variety topics. This finding is encouraging, as it indicates that the assistants are capable of handling a wide range of subjects across multiple languages. However, it also emphasizes the need for language models to be equally proficient in diverse topics across all supported languages.
Our analysis of conversation length (RQ2) revealed a positive correlation between the number of user words and the number of examples in a language. While further investigation is needed to understand the underlying factors driving this correlation, it suggests that users in languages with more examples may be more likely to engage in longer conversations, possibly due to a higher level of comfort or familiarity with the conversational AI system or a better performance of the assistant in those languages.
The sentiment analysis (RQ3) showed that both user and assistant messages tend to be neutral, with assistant responses generally being either more positive or negative. This is a consistent result across languages, and suggests that the current training approaches are effective in maintaining a coherent tone across different linguistic contexts.
Perhaps our most striking finding relates to toxicity (RQ4). The significant differences in toxicity levels across languages highlight the need for language-specific approaches to content moderation and toxicity detection. This is particularly important for languages like Dutch, English, German, and Swedish, which also exhibited higher average scores.
Finally, our quality analysis (RQ5) revealed considerable variations in user satisfaction across different language groups. The higher satisfaction levels among Romance language speakers and lower levels among Eastern European language speakers underscore the importance of tailoring conversational AI to specific linguistic and cultural contexts.
These findings collectively emphasize the need for a more nuanced, language-specific approach to the development and evaluation of conversational AI. While current models show promise in their ability to engage across multiple languages, there is still significant room for improvement in addressing language-specific challenges and user expectations.
This study presents the first comprehensive analysis of differences in user interaction and assistant responses across a wide range of European-origin languages for the WildChat and LMSYS datasets. Our work addresses a critical gap in existing literature, mainly treating these datasets as homogeneous.
By examining topics, conversation length, sentiment, toxicity, and response quality, we have uncovered significant variations across languages that have important implications for the development and deployment of conversational AI systems. Our findings highlight the need for more nuanced, language-specific approaches in areas such as content moderation and quality assessment.
The insights gained from this study are crucial for ensuring that the perspectives and needs of non-English speakers are adequately represented in the development of conversational AI. As the use of these systems continues to grow globally, it is imperative that they are designed to provide equitable and high-quality experiences across all languages.
Future work should focus on developing language-specific strategies for improving conversational AI, particularly in areas where we observed significant differences, such as toxicity levels and user satisfaction. Additionally, expanding this analysis to include non-European languages would provide a more comprehensive global perspective on multilingual conversational AI.
In conclusion, our work seeks to contribute to a more inclusive and effective approach to conversational AI development, providing insight into the importance of linguistic diversity in creating truly global and user-centric AI systems. We hope that these findings will inform and inspire future research and development efforts in multilingual conversational AI, ultimately leading to more equitable and effective language technologies for users worldwide.
Supplementary materials are available at https://github.com/ACMCMC/eur-langs-convs-analysis.