Vaccination is one of the most important prerequisites to fight against all the downsides of various diseases like COVID-19. We propose a modified BERT model, VaxTweetClassifier, to classify tweets into 12 diferent categories such as Mandatory, Conspiracy, Side-Efect, etc. VaxTweetClassifier is our submitted work for the task Artificial Intelligence on Social Media (AISoMe) in FIRE 2023 [ 1]. We have built a robust multi-label classifier that categorizes tweets according to the specific concern(s) they are exhibiting. These types of classification tools can help in analyzing the sentiment of the public in general and also Governments of diferent countries with respect to the acceptance of vaccination. They can get a rough idea about what people are thinking, whether fake news is spreading or not, if the people have a good or bad point of view about the whole process, etc. For this task, we have chosen threshold-based criteria to select the final categories for the aforementioned tweets. The evaluation score of our submitted run is reported in terms of macro-F1 score and Jaccard metric. We achieved a macro-F1 score of 0.54 and a Jaccard score of 0.58.
CEUR ceur-ws.org
Vaccination is emerging as a critical tactic in the continuous fight to protect public health to reduce the dangers and spread of many diseases. Recent events highlight the crucial role that immunization plays in limiting the COVID-19 pandemic’s widespread efects. However, the need for mass vaccination goes beyond the current emergency and includes a wider range of disorders like childhood illnesses, recurring flu outbreaks, and more. Nevertheless, the skepticism that clouds the vaccination adoption landscape is a complicated scenario made of political machinations, worries about potential adverse efects, and other nuanced issues. Understanding the various issues that surround vaccines is essential to efectively addressing this complex topic. In this age of digital communication, the breadth of social media appears as
https://www.dasdipankar.com/ (D. Das) CEUR Workshop Proceedings a formidable library of viewpoints, enabling the quick gathering of insights on vaccine-related conversations.
People throughout the world have diferent perspectives about vaccination, which they communicate on social media channels like Twitter. A single tweet can convey a variety of viewpoints and gently impact the perceptions of others. Some of these tweets may not be in line with the good purpose of vaccination and should not be distributed widely. As a result, there is a pressing need to develop machine-learning algorithms capable of categorizing COVID-19 vaccine-related tweets.
In this research article, section 2 describes the details of this shared task whereas the modelbased related works are discussed in section 3. In sections 4 and 5, we have mentioned the dataset and pre-processing steps respectively. In sections 6 and 7, we have provided our proposed methodology and evaluation of the same. Finally, section 8 concludes the paper.
The primary goal at hand is to create a strong multi-label categorization system capable of assigning meaningful labels to tweets. These labels are intended to encompass the authors’ wide range of vaccine-related concerns. It’s worth noting that a single tweet may have many categories, indicating the author’s plethora of concerns.
The classification task’s taxonomy of concerns covers the following: 1. Unnecessary: The tweet implies that immunizations are unneeded or that other treatments are preferable. 2. Mandatory: Against compulsory vaccination — According to the tweet, immunizations should not be made mandatory. 3. Pharma: Against Big Pharma — The tweet implies that the Big Pharmaceutical firms are only interested in making money, or that the tweet is against such corporations in general due to their past. 4. Conspiracy: Deeper Conspiracy — The tweet implies a deeper conspiracy, other than Big Pharma’s desire to profit (e.g., vaccinations are being used to track individuals, COVID is a fraud). 5. Political: The political aspect of vaccinations — The tweet expresses worry that governments/politicians are using vaccines to further their agenda. 6. Country: Country of origin — The tweet is critical of vaccination because of where it was created or manufactured. 7. Rushed: Untested / Hasty Process – The tweet raises the worry that the vaccinations have not been adequately tested or that the reported data is incorrect. 8. Ingredients: Vaccine ingredients/technology — The tweet shows worry about the vaccine components (for example, fetal cells, chemicals) or the technology utilized (for example, mRNA vaccines can afect your DNA). 9. Side-efect: Negative Efects/fatalities — The tweet indicates worry about the vaccinations’ negative efects, including fatalities. 10. Inefective: Vaccine is Inefective — The tweet conveys worry that vaccinations are inefective and worthless. 11. Religious: Religious grounds – The tweet is anti-vaccines for religious grounds. 12. None: No specific reason stated in the tweet, or some reason other than the given ones.
People frequently utilize social media platforms such as Twitter for several objectives, including
sharing their ideas and feelings about COVID-19, vaccination, and related initiatives. Extraction
of useful information from textual tweets is a key aspect of social computing. Basic machine
learning approaches such as the Naive-Bayes classifier, Linear classifier, and Support Vector
Machine are insuficient for such dificult problems. Advanced deep learning approaches, such
as Long Short Term Memory (LSTMs) and Bidirectional Recurrent Neural Networks (RNNs)
have shown great success in text categorization tasks. Recent advances in natural language
processing have resulted in the introduction of models such as BERT (Bidirectional Encoder
Representations from Transformers) [
BERTweet is a game-changing advancement in natural language processing designed exclusively for English Tweets. BERTweet, which is distinguished by its architectural closeness to BERTbase as proposed by Devlin et al. in 2019, is trained using the RoBERTa pre-training approach described by Liu et al. in 2019. Extensive testing confirms BERTweet’s superiority against tough competitors, including RoBERTa-base and XLM-R-base, as clarified by Conneau et al. in 2020. Notably, BERTweet outperforms prior state-of-the-art models in three critical Tweet NLP tasks: part-of-speech tagging, named-entity identification, and text categorization. This achievement highlights BERTweet’s pioneering role in improving the eficacy of NLP in the Twitter conversation space.
RoBERTa, a BERT refinement, improves critical hyperparameters by removing the next-sentence pretraining target and increasing mini-batch size and learning rates. Language model pretraining has resulted in significant performance increases, but comparing diferent techniques is dificult due to diferent datasets and computing costs. This work replicates the pretraining of BERT, indicating that BERT was under-trained and capable of outperforming succeeding models. The top model obtains state-of-the-art performance in GLUE, RACE, and SQUAD, highlighting the significance of previously missed design decisions and raising concerns about recent advancements. The researchers have made their models and code public for review.
We were given a training dataset from a previous paper titled ”CAVES: A Dataset to facilitate
Explainable Classification and Summarization of Concerns towards COVID Vaccines” (the
current version is available on Arxiv [
As a test set, we were given 500 tweets without labels. It goes beyond conversations regarding COVID-19 vaccinations to include debates about other types of vaccines such as the MMR vaccine and the Flu vaccine. 4.1. Trends in the dataset • The dataset is divided into 12 categories. • With 3,805 occurrences, ’side-efect’ is the most common category. • With 64 instances, ’Religious’ is the least frequent category. • ’Pharma,’ ’Inefective,’ ’Rushed,’ and ’Mandatory’ are moderately common. • ’None’ is related to 629 items, suggesting that the data is neutral or unlabeled. • The categories ’Conspiracy’ and ’Political’ imply talks on potentially contentious issues. • ’Pharma’ and ’Side-efect’ are major health-related categories. • The ’Mandatory’ and ’Country’ sections hint at social efect considerations. • ’Inefective,’ ’Rushed,’ ’Unnecessary,’ and ’Conspiracy’ are examples of unfavorable attitudes.
The tweets in the dataset have been processed by removing irrelevant expressions in order to improve the eficiency of word embeddings produced by the BERT model. Our tweets contain links, emojis, punctuation marks, and Twitter embeds(images or GIFs) that may afect the performance of the BERT model.
We have used the following processing pipeline for cleaning the tweets:
1. Conversion of words to lowercase: All the words in the tweet are converted to
lowercase to enhance the classification process
2. Conversion of emojis to words: Emojis and emoticons in tweets significantly contribute
to the context of the tweet and can be valuable for sentiment analysis. We converted
them into the :emojiname: format by using the ‘emoji’ library [
Input: #VAERS 16y♀#Pfizer died pulmonary embolism suspected #Cardiacarrest #vaccinedeaths #adolescent Symptoms: Cardiac arrest, Circulatory collapse, Computerised tomogram thorax abnormal, Death, Lung assist device therapy, Pulmonary embolism https://t.co/dTAdjik8gO
Output: vaers 16y female sign pfizer died pulmonary embolism suspected cardiacarrest vaccinedeaths adolescent symptoms cardiac arrest circulatory collapse computerized tomogram thorax abnormal death lung assist device therapy pulmonary embolism
6.1. Model
BERT-BASE-UNCASED [
The tweets are first tokenized and the labels are converted into a binary matrix which can be fed into the model for classification. We used sklearn’s MultiLabelBinarizer for this task.
For instance, in our current task, we have 12 labels for tweet classification. The labels in the training will be converted into a 1x12 matrix.
The tokenized tweets have three parts - input_ids, attention_mask, and segment_ids. The input_ids include the raw tokenized tweets, and attention_masks are used to distinguish between the relevant input tokens and the tokens that need to be ignored. Finally, the segment_ids help the model to distinguish between diferent parts of the input token such as diferent sentences.
The aforementioned input nodes are fed into a keras layer which embeds and encodes the input tokens into a dense vector. Our next layers consist of two dense layers one having 128 units and the next having 64 units. These dense layers process the embeddings produced by the Keras layers and produce a new set of features.
The output layer, also a dense layer, has 12 units as we have 12 labels for classification. It takes the features produced by the two dense layers as input and produces a set of probabilities, one for each label.
In order to prevent overfitting in the training process, dropout layers are used after every dense layer while fine-tuning the model.
Figure 1 shows the graphical representation of our model.
We used the fine-tuned BERT model to generate the embeddings for the tweets and to generate the probability scores of each tweet against the twelve classes. For a tweet, the classes having probabilities greater than the threshold i.e., 0.25 were treated as the predicted classes for that tweet. The final prediction file containing the tweet IDs and predicted classes was submitted in the requested format.
Run file run1.csv run3.csv run2.csv
Summary of Methodology Overall model is
the same as discussed above and for this run,
We set the learning rate at
1e-4 Overall model is
the same as discussed above and for this run,
We set the learning rate at
1e-6 Overall model is
the same as discussed above and for this run,
We set the learning rate at 1e-8
Macro-f1
The AISoMe task involves categorizing data into multi-classes and multi-labels. There are two measures to evaluate the performance: the macro-F1 score and the Jaccard Coeficient, which are applied to 12 unique categories. Table 1 shows the results of our submission for this challenge. The VaxTweetClassifier placed 20th out of all entries, with a macro-F1 score of 0.54 and a Jaccard Coeficient score of 0.58.
In this work, we used VaxTweetClassifier, a transformer-based model that was pre-trained on a large dataset of anti-vaccination tweets related to COVID-19. Our major goal is to categorize tweets into 12 unique groups. Surprisingly, our results show that the transformer-based model outperforms classic natural language processing classifiers like Naive Bayes, Logistic Regression, and Support Vector Machine. This advantage is due to the transformer model’s ability to produce more expressive word embeddings, which results in improved performance on the task at hand.
Given that transformer-based models have a strong thirst for data, we recommend exploring data augmentation approaches to improve our model’s performance. In addition, we propose the use of adversarial training to improve the model’s resilience and robustness. These advancements may improve our categorization accuracy even further.
• GitHub, • Overleaf template.