The Forum for Information Retrieval Evaluation, a platform emphasizing on data evaluation and analysis, has conducted a novel challenge in the NLP domain titled Artificial Intelligence on Social Media. The main idea behind this task is to formulate an efective multi-label classifier that labels social media posts according to specific concerns towards vaccines expressed by authors of the posts. The critical point here is that each post has multiple unlikely characters and multiple labels, this would ideally require models with better attention mechanisms for accurate classifications. Upon appropriate pre-processing, our team performed three diferent transformer based approaches that provided optimal results compared to other models with lesser attention span. The models used were BERT, RoBERTa and XLNet. The BERT and RoBERTa models gave the same macro-f1 score of 0.57 but had a mild diference with the Jaccard score with RoBERTa giving 0.57 and BERT giving 0.56. The XLNet model gave out a 0.48 macro-f1 score and 0.49 Jaccard score. Our research primarily aims to throw light on why these state-of-the-art transformer approaches prove to be deal breakers in modern day NLP.
CEUR ceur-ws.org
In more recent times there is no greater illustration of how crucial vaccinations are as a vital weapon in fighting of diseases than how this has played out and the impact vaccines have had on controlling the COVID-19 pandemic. While there are numerous advantages to vaccination, there is still an extensive group of individuals who don’t trust in them. Such skepticism has been fed by a number of issues including doubts over the safety and efectiveness of vaccines, conspiracy theories about pharmaceutical companies and governments having hidden agendas. These issues need to be understood and addressed in order to encourage mass vaccination and secure the public health. In this paper, we propose an exploratory method to analyze social media sentiment regarding vaccines in general. They aim at building an eficient multi-label classification system for the task of identifying and labeling diferent types of issues raised in tweets about vaccines. Our typology includes many issues such as concerns over vaccine CEUR Workshop Proceedings (S. Karthika) safety and eficacy, concerns about government influence or manipulation, various forms of conspiracies, and more. To accommodate the possibility that one tweet could address more than one issue, we have built our classification framework in such a way that multiple labels can be assigned to each tweet. We utilize cutting edge NLP models namely BERT, RoBERTa, XLNET for the task, which can learn subtle linguistic features and context information with great efectiveness. These tools allow us to get a pretty accurate reading of what people are saying in tweet-form; it’s quite useful for understanding feelings and topics people are discussing on social media.The goal of these studies is to advance knowledge about public attitudes and receptivity towards immunizations to better develop public health information campaign and communication strategies. With the power of social media data analytics and sophisticated advanced NLP models at our disposal, we are attempting to illuminate what may contribute to anti-vaccine attitudes so as to increase awareness and immunization rates across the globe
This section provides an overview of research in developing architectures for comment
classification in the context of the COVID-19 pandemic’s impact on social media information flow.
The urgency of reliable classification methods for the surge in COVID-19-related tweets has
led to various approaches. One study [
The training dataset[
Upon receiving the training dataset for the AISoMe track as part of the FIRE 2023 conference we ifrst analyzed a few inconsistencies that the textual data had. To overcome such inconsistencies and to use a clean form of data for the multi-label classifier model, we first used certain methods to preprocess the data. In the provided dataset, each “post” started of with the usernames being tagged. In social media tagging a user usually involves using the “@” symbol. So the ifrst task involved removing the “@” symbol. Post this, we removed html tags, brackets and emojis and converted the characters to lower cases. This was followed by stop word removal and lemmatization. This preprocessing technique provided us with clean data that can be used for further processing. Since there are multiple labels for a single post, this task can be considered multi label classification. The processing of these labels involved splitting them up and encoding the labels, using multi-label binarizer technique to the appropriate numerical format. Upon understanding the attention mechanism of transformer based models, we proceeded to incorporate three such models namely BERT, RoBERTa and XLNET for the classification task. These models were chosen because of the self-attention mechanism that they possess which enables it to weigh the significance of diferent elements within a sequence simultaneously, this gives an edge over traditional models that relied on sequential data processing. 4.1. BERT The first model devised for this task is the Bidirectional Encoder Representation from Transformers, shortly known as BERT. It is a state of the art model used in a variety of NLP tasks but in this particular case, we are formulating one for multi-label classification. Any text classification model requires in depth context extraction/learning. BERT has this intricate feature called MLM that helps it learn this concept more eficiently. MLM refers to a masked language model which means, certain tokens in a text are intentionally masked or hidden, and the model’s task is to predict or fill in the missing tokens. BERT is bidirectional which means it takes into account both left and right contexts of the masked tokens in general. While using BERT for The input text is tokenized and fed into the BERT model, and the final hidden state corresponding to the [CLS] token (a special token added to the input) is used as the representation for the entire input sequence.classification, we append a classification layer upon the pre-trained BERT model.The output of the BERT model’s [CLS] token representation is passed through this classification layer. The number of output units in the classification layer corresponds to the number of classes in the classification task, for this task it is 12 output units representing 12 unique classes. 4.2. RoBERTa RoBERTa, an extension of the BERT model, excels in various NLP tasks, including multi-label classification. It utilizes a Masked Language Model (MLM) during pre-training, similar to BERT, where certain tokens in the input text are concealed, and the model’s objective is to predict these hidden tokens, taking both left and right contexts into account. The key diference between BERT and RoBERTa here is in BERT architecture, the masking is performed once during data preprocessing, resulting in a single static mask. To avoid using the single static mask, training data is duplicated and masked 10 times, each time with a diferent mask strategy over 40 epochs thus having 4 epochs with the same mask. This strategy is compared with dynamic masking in which diferent masking is generated every time we pass data into the model.. In multi-label classification, RoBERTa adapts by adding a classification layer atop its pre-trained layers. Input text is tokenized, and the final hidden state associated with the [CLS] token is used as the sequence representation. The number of output units in the classification layer matches the number of classes in the classification task. During inference a sigmoid activation was used and a threshold is applied to determine the assigned labels. 4.3. XLNET Much similar to BERT and RoBERTa, XLNET also has a bidirectional approach for concept learning but what sets XLNET apart is the autoregressive method that it employs. This helps XLNET to capture contextual dependencies .Much like RoBERTa, XLNet employs a Masked Language Model (MLM) during pre-training, akin to BERT, where specific tokens within the input text are concealed, and the model’s primary objective is to predict these concealed tokens, factoring in both left and right contexts. The pivotal distinction between BERT and XLNet lies in the methodology of masking. While BERT employs a static mask, applied once during data preprocessing, XLNet opts for a more dynamic approach. During training, the data is duplicated and masked tenfold, with each instance employing a distinct masking strategy. This dynamic masking strategy evolves with each pass through the model, enriching context comprehension. In the realm of multi-label classification, XLNet adapts seamlessly by introducing a classification layer atop its pre-trained layers. Text inputs are tokenized, and the ultimate hidden state corresponding to the [CLS] token functions as the sequence representation. The number of output units in our case is 12 mirroring the number of classes in the classification task. During inference, sigmoid activation function was used, with a threshold applied to establish the assigned labels similar to the other 2 models.
The training phase of our models was a meticulous process that involved fine-tuning several critical hyperparameters, each with its unique role in shaping model performance. Weight decay, represented by λ and set at 0.001, acted as a vital regularizer. It controlled the influence of model weights on the loss function, efectively curbing overfitting by penalizing large weight values. The learning rate, denoted as η and thoughtfully configured at 0.0001 or 0.00001, played a pivotal role in determining the step size for weight updates during training. This parameter significantly impacted convergence, with an appropriate learning rate ensuring that the model efectively learned from the data without overshooting optimal parameter values. Dropout rates, which ranged between 0.4 and 0.5, were judiciously chosen to regulate neuron activations. Dropout is a regularization technique that prevents over-reliance on specific neurons by randomly deactivating a fraction during each training iteration, promoting generalization. Furthermore, the ”epochs” parameter governed the number of complete iterations through the training data, allowing models to progressively learn and adapt over time. Finally, the ”maximum sequence length” parameter was set at 300, ensuring that input sequences were appropriately processed during training, thereby enhancing the model’s overall ability to achieve peak accuracy. These hyperparameters collectively contributed to the fine-tuning process, facilitating the models’ optimization and ensuring their robust performance.Table 1 specifies the various hyperparameters and their corresponding values that gave maximum training accuracy
In the evaluation phase, a dedicated test dataset was provided to rigorously assess the performance of the trained models. Each model underwent a single run, generating predictions that were then submitted for evaluation. The evaluation process hinged on two pivotal metrics: the Macro F1 Score and Jaccard Score.Together, these two reliable assessment criteria provided reliable benchmarks for carefully assessing and contrasting the performance of the models 1 = 2 ∗ ( ∗ )/( + )
The Macro F1-Score is a metric used to evaluate the performance of a classification model. It is calculated by taking the harmonic mean of precision and recall. The Macro F1-Score is useful in multiclass classification problems where the classes are imbalanced. It gives equal weight to each class, making it a good metric to use when the data is imbalanced.
∑ F1 scores Macro F1 Score = (2)
Number of classes
The Jaccard Index, also known as the Jaccard similarity coeficient, is a statistic used in understanding the similarities between sample sets. The Jaccard Score is particularly valuable for evaluating models dealing with binary or multiclass classification, as it doesn’t take into account true negatives, focusing solely on the overlap between predicted and actual positive instances.
Jaccard( , ) = | ∩ | | ∪ | (1) (3)
This study harnessed advanced transformer-based models, BERT, RoBERTa, and XLNET, to address the complex task of multi-label classification in the context of tweets expressing concerns about vaccines, primarily comprising anti-vaccine sentiments. Notably, RoBERTa demonstrated superior performance, followed closely by BERT and XLNET. In this predominantly anti-vaccine dataset, these models showcased their prowess in discerning and categorizing various concerns. To further enhance our model’s performance, future investigations could explore data augmentation techniques, given the data-hungry nature of transformer-based models. Additionally, fortifying model robustness through methods like adversarial training presents a promising avenue. These eforts align with our mission of accurately categorizing concerns within the domain of vaccine-related discussions.