This paper investigates transformations of “Essential Jobs” during the COVID-19 pandemic through sentiment analysis of social media data, specifically focusing on “X” posts. The study employs a comprehensive methodology consisting of traditional and modern sentiment analysis tools as well as advanced deep learning approaches to examine job-related sentiments across English and German languages. The research demonstrates that the “Twitter-XLM-RoBERTa” model outperforms other sentiment analysis tools in both base and enhanced implementations, challenging the assumption that deep learning enhancements necessarily improve sentiment analysis performance. The findings indicate significant variations between “Essential Job” designations. However, the high proportion of “No-Data” classifications and linguistic variability between English and German datasets suggest methodological limitations.
Social media has emerged as a potent instrument for capturing and shaping public opinion during significant events. Platforms such as “X” (formerly “Twitter”) can be seen as real-time indicators of societal sentiment. These platforms facilitate the aggregation of dynamic insights into public sentiment and reactions, thereby ofering researchers unprecedented opportunities to study human behavior and opinion at scale.
In order to comprehend the substantial and unstructured data generated on social media platforms such as “X”, researchers employ techniques such as sentiment analysis, which categorizes posts as positive, negative, or neutral. The eficacy of this process is augmented through the integration of machine learning and Natural Language Processing (NLP), a combination fostering the extraction of emotional meaning from textual data. NLP facilitates the nuanced interpretation of user-generated content, while machine learning models can learn from labeled datasets to classify new posts with a high degree of accuracy. Deep learning, a subset of machine learning, has further advanced this capability by more efectively handling the complexity of human language, especially with the introduction of Large Language Models (LLMs). These models have been demonstrated to possess the capacity to comprehend implicit sentiments and contextual meaning, rendering them particularly well-suited for the analysis of extensive social media datasets with enhanced precision and scalability.
These methods help using social media data to study public opinion on societal changes, such as the evolving definition of Essential Jobs during the course of the pandemic. Conventional data collection methodologies, such as surveys, have encountered significant limitations, as they are not only timeand ressource-intensive. Social media provides a viable alternative, ofering the capacity to capture real-time, spontaneous public reactions. This paper addresses the research gap on why more and more jobs were regarded as essential during the pandemic by conducting a comprehensive analysis of sentiments expressed on “X” with respect to Essential Jobs, building upon extant research on the German workforce. The objective of this study is to find hints on whether shifts in what constitutes an Essential Job were institutionally driven or arose “organically” from public discourse. However, we focus on two particular research questions: • Which sentiment analysis method is the most efective for assessing changes in sentiment towards
Essential Jobs? (RQ1) • How might contemporary deep learning algorithms improve the performance of the sentiment analysis process? (RQ2)
Tiemann et al. [
Miah et al. [
In order to explore the predictability of trends using recognized patterns over “X” social media
sentiment analysis, Di Tollo et al. [
3.1. Data
This paper is centered on the analysis of public opinion shared on X. Due to modifications in X policies
in 2023, it has become impracticable to retrieve the most recent X posts. It is fortunate that research
involving German occupations in the same year had already been conducted, as this thesis will build
upon this previous study [
The data collection process was executed in accordance with the protocol established in the preceding
study (see [
The initial step in this research will be the conceptualization and implementation of a variety of sentiment analysis tools, as well as their subsequent evaluation to determine the most efective tool. Subsequently, deep learning algorithms will be integrated with the previously mentioned sentiment analysis tools to assess the impact of deep learning models on the aforementioned tools. This analysis will identify the most efective tool for determining whether a job is essential or not.
of conceptualizing and implementing the tools is needed. This process involves a multi-step approach, incorporating various sentiment analysis methods and addressing the challenges posed by multilingual data. Ultimately, to address finding the most efective tool, a comprehensive performance evaluation will be developed and conducted.
tion of deep learning models with previously introduced sentiment analysis tools in order to respond to the second research question. Studies show that deep learning models are indeed practical for both sentiment analysis and emotion classification tasks, and they can outperform conventional machine learning in general for these tasks. Particularly, Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) variants have become increasingly popular for sentiment analysis tasks due to their state-of-the-art performance, as many studies have successfully applied these deep learning models for sentiment analysis (Kastrati et al., 2024). The endeavor focuses on harnessing the capabilities of both deep learning and machine learning techniques to enhance the accuracy and robustness of sentiment analysis.
Measuring the performance of the sentiment analysis tools is the first mandatory step in analyzing and evaluating their outcomes. This is accomplished through the utilization of continuous measurements, including MSE, MAE, R-squared, and correlation coeficients, as well as discrete metrics, such as accuracy, precision, recall, and F1-score. For brevity, this paper only reports detailed results on the discrete metrics.1
A cumulative score is introduced to give a fair perspective on the best sentiment analysis tool overall. However, to experiment with the impact level and magnitude of the metrics, they are fed to the score individually and then at once, meaning that first, only the first three error metrics, R-squared and 1Detailed results for the continuous metrics can be obtained via the authors. correlation metrics were taken into account, and then they were included stepwise to reach a cumulative score. Interestingly, the best tool for both languages didn’t change even after the inclusions.
For the English dataset, the initial evaluations showed that the Twitter-XLM-RoBERTa (‘twtr_xlm_rob’ model) consistently outperformed other tools across all continuous metrics where its superior performance can be attributed to its strong correlation with labels and lower error rates. The high correlation coeficients indicate a strong positive relationship between predicted and actual sentiment scores. The low error metrics suggest that Twitter-XLM- RoBERTa predictions are closer to the true labels compared to other tools. GPT4o showed the second-best performance in the continuous evaluation, though its performance was not as strong as Twitter-XLM-RoBERTa, it still demonstrated a good correlation with labels and relatively low error rates, outperforming the remaining tools.
For the German part, GPT4o emerged as the top performer in the continuous evaluation, achieving the highest scores in most metrics, which indicates its superior ability to predict sentiment scores that closely align with the true labels, showing both high correlation and low error rates. Despite trailing GPT4o by a small margin, the Twitter- XLM-RoBERTa displayed strong predictive capabilities, especially in minimizing absolute errors, and placed second in the continuous evaluation.
Similar to the continuous perspective, a cumulative score is calculated considering all key classification metrics, including accuracy, precision, recall, F1-score, macro-average, and micro-average, to have a thorough assessment of classification performance.
As with the continuous evaluation, the Twitter-XLM-RoBERTa again emerged as the top performer across all discrete metrics, indicating the tool’s strength in classifying “X” posts into the correct sentiment categories, while its performance was consistent across diferent sentiment classes (Table 1). Due to the outstanding classification performance of XLM-RoBERTa-German, it outperformed the other tools in the discrete evaluation, yet not as strong as Twitter-XLM-RoBERTa, and thus earned the second-best ranking.
For German “X” posts (Table 2), the Twitter-XLM-RoBERTa again surpassed all other tools, demonstrating its ability to properly classify sentiments across multiple groups while maintaining a balanced performance in both precision and recall. Even while GPT4o was powerful and ranked second in the discrete examination, it was not as dominant as it was in the continuous evaluation. This revealed that while it excels in predicting sentiment scores on a continuous scale, its performance in discrete classification is good but not superior to Twitter-XLM-RoBERTa.
Focusing on the ROC curves, for the tools employed for English “X” posts, the Twitter- XLM-RoBERTa consistently outperforms the others, with the highest AUC values across diferent classes, ranging from 0.69 to 0.82 (a good to excellent performance). The XLM-RoBERTa-German and GPT4o came next, as both tools demonstrated good to moderate performances, though their performances were not even comparable to Twitter-XLM-RoBERTa. Regarding German “X” posts Twitter- XLM-RoBERTa showed the best performance with an AUC of 0.89 and the ROC curve consistently above all others, indicating superior discrimination ability across all thresholds. The second-best performer with an AUC of 0.75 was GPT4o, also exhibiting good discriminative power.
In the next step the results of the most eficient model are evaluated to ensure they are accurate, besides realizing the opportunity to compare it to the results of RQ1. The relative performance of the enhanced sentiment analysis tools is therefore evaluated using the same metrics as in section 4.1, with the same continuous and discrete metrics.
The continuous evaluation results show varying performance across enhanced sentiment analysis tools and their variants. The gradient-boosting (twtr_xlm_rob_GB) enhancement of the Twitter-XLMRoBERTa and GPT4o models demonstrate superior performance in capturing the nuances of sentiments on a continuous scale across various metrics and evaluation approaches, indicating that both models are efective in predicting sentiment scores accurately, and consistently outperforming other tools across various metrics. Moreover, it demonstrates that the ensemble variant of each tool tends to perform better than the base-enhanced counterparts using the CNN-LSTM model.
Regarding the discrete viewpoint (Table 3), the gradient-boosting form of the enhanced Twitter-XLMRoBERTa along with the random forest (twtr_xlm_rob_RF) variant consistently outperforms other tools in discrete classification on almost all performance metrics, followed by the gradient-boosting version of the XLM-RoBERTa-German model which shows competitive performance in terms of accuracy and other metrics.
The results of the ROC curves and AUC values, as evident in Figure 2, show that the enhanced Twitter-XLM-RoBERTa with both random forest and gradient-boosting ensemble emerges as the overall best tool for sentiment analysis, demonstrating superior performance across all sentiment classes. Their ability to distinguish between diferent sentiment classes is significantly better than other models, as evidenced by the consistently showing the highest AUC values across diferent classes, often above 0.80, and curves furthest from the diagonal line. The GPT4o families exhibit competitive performance, particularly for certain sentiment classes.
The GPT4o model with both gradient-boosting and random forest enhancements performs best across all metrics in the continuous evaluation. Technically, it shows the highest correlation coeficients and the best error metrics – lowest MSE, MAE, and RMSE. While the Twitter-XLM-RoBERTa model with gradient boosting was not as strong as the GPT4o models, it still shows good performance, particularly in terms of Pearson correlation and error metrics.
In the discrete evaluation (Table 4), the Twitter-XLM-RoBERTa model with gradient boosting performs best overall, demonstrating particularly strong precision and a good balance between precision and recall, as reflected in its high F1-Score. The same model with base enhancement comes next, with slightly lower but still impressive scores. The GPT4o model with gradient boosting is the third-best performer in this category, indicating strong performance, particularly in terms of F1-Score and Macro-Average.
Based on the ROC curves shown in Figure 2 and related AUC values, the best tool appears to be Twitter-XLM-RoBERTa with gradient boosting with ROC curves consistently above others for most classes and AUC values close to 0.8, indicating strong discriminative power across all sentiment classes. The ROC curves for the gradient- boosting type of GPT4o are also among the top performers, with considerable AUC values, showcasing decent performance for some classes. The Twitter XLM-RoBERTa model with base enhancement also shows competitive performance, with ROC curves often close to the top two models.
In order to discover the best model overall, the results of the first research question regarding the evaluations of the sentiment analysis tools without enhancement from external deep learning and machine learning models will be called ERQ1. They are compared to the evaluation results of the enhanced sentiment analysis tools (here ERQ2) to reach a thorough examination of the second research question. This will be done by investigating the results of both Continuous and Discrete aspects of both English and German tools, which would strengthen the quality of the examination.
For English, the best overall tool comes from ERQ1, namely the Twitter-XLM-RoBERTa model. This tool achieved the highest evaluation score, outperforming its enhanced variants in ERQ2. Notably, it is the best performer for both continuous and discrete metrics in ERQ1, demonstrating its robustness across diferent evaluation criteria. The German language analysis revealed a more complex picture, as the best overall performance came from ERQ1, but with a mixed result. In fact, GPT4o performed best for continuous metrics, while Twitter-XLM-RoBERTa excelled in discrete metrics.
The analysis reveals that the best overall tool across both languages is Twitter-XLM- RoBERTa, which demonstrates no significant improvements over the baseline sentiment analysis tools (Table 5). The analysis of core tools further proves that it is indeed the best overall core tool due to appearing with the maximum number of 5 times among the top-performing models in both base and enhanced CNN-LSTM with machine learning ensemble models across both languages.
The Twitter-XLM-RoBERTa base model ultimately emerged as the best sentiment analysis tool due to demonstrating superior performance overall in both English and German datasets, highlighting the efectiveness of a robust pre-trained language model and transformer- based architecture and enabling efective cross-lingual sentiment detection. In a next step, a comprehensive assessment framework was implemented to determine whether or not jobs can be predicted as being Essential Jobs. The approach entailed utilizing the elected Twitter- XLM-RoBERTa model to conduct sentiment analysis on the “X” posts for all available datasets and aggregate the outcomes for each unique job title to use as a predictor of job essentiality. Two diferent methods were introduced to assess the essentiality of job titles: The Berlin Method solely focuses on the comparison of the test subject to the corresponding ground truth of the initial list of essential jobs. The Temporal Method compares the changes to the list of essential jobs over the pandemic’s duration to the initial list on the same dataset. And Both Methods which is the combination of the two methods, acting as a meta-classifier of the essentiality assessment mechanism, to make the final decisions based on the results of the other two methods.
The findings obtained from the final assessment reveal a significant degree of complexity and variability, underscoring the challenges inherent in sentiment-based assessments, specifically in lesspopular and intricate contexts such as job essentiality. The following content ofers an in-depth interpretation of the findings: Berlin Method: The Berlin Method exhibited a consistent trend across both languages, with the majority of unique job titles of both English and German versions of the Extended List being categorized as No-Data, suggesting a potential bias or overgeneralization in the classification process. The next highest proportion of jobs were allocated to the Essential category, with about half of the No-Data amount, highlighting the permissible performance of the method in discovering Essential Jobs. The overall outcome of the Berlin Method may reflect the inherent limitations of the ground truth dataset or the aggregation mechanism, which could have failed to capture delicate variations in sentiment for certain job titles.
Temporal Method The Temporal Method results revealed greater variability between the two languages. While the Extended List predominantly classified jobs as No- Data in both languages, the rest of the German dataset jobs revealed somewhat similar trends between the rest of the classes, while the English dataset classified nearly no job as essential. This disparity could be attributed to restricted classification criteria, variations in the temporal distribution of the datasets, difering patterns of sentiment evolution across languages, or the impact of language-specific nuances on sentiment scoring. The relatively balanced distribution across the remaining classes in both languages otherwise may suggest that the Temporal Method failed to capture a broader range of sentiment dynamics compared to the Berlin Method.
Both Methods The “Both Methods” approach demonstrated significant diversity in categorization between Essential and Non-Essential classes. The No-Data category appeared to be the dominant class as in most of the previous cases, for the English Added dataset, almost one-third of the remaining items were classified as Essential and a limited amount of the jobs to Non-Essential, while the German dataset exhibited equal job classification in the Essential and Non-Essential categories. This suggests that the integration of Berlin and Temporal Methods, while robust in combining complementary perspectives and resulting in depicting a better classifications, may still be influenced by the inherent biases or limitations of each approach. Notably, the excess of No-Data besides variation of Non-Essential classifications between languages underscores the dificulty of definitively identifying job essentiality through sentiment analysis alone.
This research has made significant contributions to understanding the dynamics of Essential Job classifications during the COVID-19 pandemic through advanced sentiment analysis of social media data and highlights both methodological advances and significant analytical challenges. While Twitter-XLMRoBERTa has emerged as the superior sentiment analysis tool within all competitors on both the English and German datasets, the results indicate that adding deep learning enhancements unexpectedly failed to surpass baseline performance. The new dual-stream comparative evaluation approach, which merges Berlin-based reference analysis with temporal sentiment evolution, sheds light on the intricate nature of Essential Job classifications. Significant methodological constraints, including concerns with data quality, language variation, and threshold setting, are highlighted through the considerable proportion of classifications becoming No-Data and substantial distinctions between the outcomes of English and German datasets. While the established framework seems promising in analyzing job essentiality through sentiment analysis, its practical application faces considerable constraints, particularly in predicting prospective Essential Jobs. Future developments ought to focus on enhanced job title grouping strategies, refined multilingual model adaptation, and more sophisticated comparative analysis design to minimize classification ambiguity. Despite these limitations, this research establishes a foundational framework for investigating job essentiality through sentiment analysis during global crises, providing valuable insights into the intersection of public sentiment and social policy determination.
In this study it was decided not to concentrate on job title pre-processing, separation, grouping, but instead execute just the minimal amount of data cleaning and pre-processing. This decision ended up with the study’s conclusion being based on raw job titles, leading to significant job title variation. Future research could investigate how prediction of job essentiality could be improved by more rigorous data pre-processing and grouping of job titles. Filtering jobs and focusing on those with a minimum amount of tweets related to them could also prove interesting, as well as job-specific sentiment analysis instead of aggregated sentiments for groups of jobs. Lastly, fine-tuning for multilingual contexts could also improve results. During the preparation of this work, the authors used DeepL in order to: Grammar and spelling check. After using these tool(s)/service(s), the authors reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.