Determining sentiments of the public with regard to COVID-19 vaccines is crucial for nations to efifciently carry out vaccination drives and spread awareness. Hence, it is a field requiring accurate analysis and captures the interest of many researchers. Microblogs from social media websites such as Twitter sometimes contain colloquial expressions or terminology dificult to interpret making the task a challenging one. In this paper, we propose a method for multi-label text classification for the track of ”Information Retrieval from Microblogs during Disasters (IRMiDis)” presented by the ”Forum of Information Retrieval Evaluation” in 2022, related to vaccine sentiment among the public and reporting of someone experiencing COVID-19 symptoms. The following methodologies have been utilised: (i) Word2Vec and (ii) BERT, which uses contextual embedding rather than the fixed embedding used by conventional natural language models. For Task 1, the overall F1 score and Accuracy are 0.503 and 0.529, respectively, placing us fourth among all the teams, while for Task 2, they are 0.740 and 0.790, placing us second among all the teams who submitted their work. Our code is openly accessible through GitHub. 1.
We all have just faced a life-threatening world pandemic, and the only long-term remedy seems to be through society-scale vaccination drives. Researchers from all over the world have created a number of vaccines to combat the COVID-19 virus. Covaxin, Covishield, and Spikevax are some of the well-known vaccines. Many people, however, are dubious about the use of vaccines for a number of reasons, including the politicisation of the issue and the hasty development of vaccines. There is reluctance among some people to get vaccinated since they believe that vaccines have not been tested and there have not been enough trials to make sure they are safe. To make the vaccination drive a success, it is imperative to understand the sentiments of the public towards vaccines. This task can be accomplished using data from social networking websites like Twitter because most of us use social media to communicate our views and opinions in today’s world. In Task 1, data from social media and microblogging websites like Twitter was obtained to understand people’s sentiments on vaccinations. The tweets are classified into three categories: “ProVax”, “AntiVax”, “Neutral” based on their willingness to get vaccinated as 1https://github.com/RohitJain001/FIRE-Conference-BERT-Natural-Language-Preprocessing analysed from their tweets.
Using tweets gathered between February 2 and March 15, 2020, Abd-Alrazaq et. al. [
In task 2 of the “Information Retrieval from Microblogs during Disasters” track, we specifically explore if tweets that report someone experiencing COVID-19 symptoms can be automatically identified. It focuses on classifying these tweets into four categories: “Primary Reporting”,” Secondary Reporting”, “Third-party Reporting” and “Non-Reporting”. Our team has contributed to the problem as part of the shared task efort by cleaning the tweets and trying various NLP models including BERT and Word2Vec to classify the tweets.
We have made the following contributions in context of the problem of the shared tasks: • A transformer model i.e. BERTl which uses bidirectional word embeddings as opposed to the fixed embeddings used by traditional Natural Language Processing models is implemented to accurately identify the target tweets class. • Word2Vec was also implemented as a part of the efort. The results showed that models using contextual embeddings give relatively better results when compared to models using fixed embeddings.
On the basis of tweets and facebook postings, several analyses of the public’s opinions on
the COVID-19 vaccination have been made. Samuel, et al. [
About 4.5 million tweets from the COVID-19 pandemic were the subject of a research in
Europe. Multilingual language embeddings were used to do sentiment analysis using neural
networks [
3.1. Task 1 The dataset provided contained 4392 tweets for training and 717 tweets for testing. Every tweet in the dataset was labelled as one of the 3 stances related to vaccine sentiment expressed in the tweet, as explained below: 1. AntiVax - The tweet expresses reluctance on the part of the user who tweeted it to get vaccinated against COVID-19. 2. ProVax - The tweet endorsing or promoting the use of vaccinations. 3. Neutral - The tweet does not clearly reflect any opinion about vaccines or has nothing to do with vaccines. 3.2. Task 2 The dataset provided contained 1574 tweets for training and 400 tweets for testing. Every tweet in the dataset was labelled as one of four classes, which identifies reporting of COVID19, as explained below: 1. Primary Reporting - The tweet’s submitter is outlining personal symptoms. 2. Secondary Reporting - The user is outlining the symptoms of a friend, family member, neighbour, or recent acquaintance. 3. Third-party Reporting - A third party or famous person’s symptoms are being referenced by the user. 4. Non-Reporting - The user is not claiming that anyone is displaying COVID-19 symptoms and is instead utilising symptom-words in a diferent context. This collection of tweets only ofers general information on COVID-19 symptoms; no specific people who may be experiencing these symptoms are mentioned.
The tweets contain some colloquial expressions and terminology which might be dificult to interpret; because of which, it could be challenging to determine the stance of a tweet. Hence, an eficient method is to be established so these kinds of tasks can be accurately performed.
We use a method which focuses on learning the semantic relationships between the sentence tokens of tweets and map them with the given label. The work was seen as an extension of a multi label phrase classification problem. Task 1 required the prediction of three labels linked to public opinion on vaccines, and Task 2 required the prediction of four labels pertaining to the reporting of someone who had COVID-19 symptoms. After analysis of data during text Exploratory Data Analysis, the model’s accuracy was improved with efective stop-word removal including removal of all special characters and website URLs.
We used these preprocessed tweets to train specialised algorithms such as Word2vec and BERT (as shown in Figure 1) to eficiently handle sentence classification. The process of a deep learning-based transformers technique was applied. It helped in getting the correct interpretation of the tweets and eficiently classifying them as one of the labels. For getting optimised results from the BERT model, we tuned the hyperparameters including the max length limit of the tokens, the training batch size parameter to determine how many tweets were processed at once before the model was updated, the validation batch size and the learning rate of the model. We chose the max length parameter to be 256, train and valid batch size to be 32 and the learning rate is 10-5. We chose BCE with Logits Loss in Pytorch, a combination of a Sigmoid layer and BCE Loss into a single class which is numerically stable, as our loss function and kept saving the checkpoint at each epoch if the accuracy improved. Using the best saved model that we obtained after running the epochs, the tweets were classified into respective labels.
In Word2Vec methodology (as shown in Figure 1), initially a unique word dictionary was created using the Keras Tokenizer() and the text_to_sequence() function was used to get a list of lists where each value represent the index of that word in a unique word dictionary. Now as every row has a diferent dimension, so the pad_sequences() method is used to add paddings and generate an embedding matrix. The embedding matrix was fed to embedding layers in Keras as weights. The input dimensions of the embedding layer is the total number of unique words, and the output dimensions are the embedding vector dimensions. We received the word embeddings as the output from the Keras embedding layer. Furthermore, the Long Short Term Memory, Recurrent Neural Network Model was trained with the word embeddings and output
Input data Pre Processing [CLS] Don't trust vaccines [SEP] Not enough trials [SEP] Token Embeddings Segment Embeddings Position Embeddings
E[CLS]
EDon't
Etrust Evaccines E[SEP] ENot Eenough Etrials
E[SEP] EA E0
EA E1
EA E2
EA E3
EA E4
EB E5
EB E6
EB E7
EB E8
Transformer Blocks Pretrained BERT base-uncased
Output Label
Input Tweets Keras Tokenizer()
Preprocessing
Unique Word Dictionary text_to_sequence() & pad_sequences() Keras Embedding Layer Embedding Matrix Word embeddings Recurrent Neural Network
LSTM Output Label
Before training the model, the data was thoroughly cleaned for the right text analysis and to remove noise. For preparation, we used stop-word removal and tokenization. The words were initially divided up into smaller parts called “tokens.” The text was changed to lowercase, the urls were taken out, and words like “the”, “a”, “and”, “is”, and “will”—common stopwords in English—were removed. This was done since none of these help to the goal of text classification.
BERT [
Word2Vec: Mikolov [
The test data was tested using the proposed methodology and multiple runs were submitted. We experimented with various models including BERT and Word2Vec; and also tested for varying number of epochs and learning rates for the BERT model. After a thorough analysis of the test dataset processed by our proposed methods, the best results were obtained on using BERT; with the optimal hyperparameters coming out to be 15 training epochs and a learning rate of 10−5. As mentioned earlier, multiple runs were done using diferent learning rates and number of epochs, but BERT outperformed Word2Vec every single time and generated optimal results. BERT performs better as compared to Word2Vec since it uses a bidirectional transformer. As compared to traditional language models like OpenAI GPT and Word2Vec which take the previous ‘n’ tokens and predict the next token, BERT trains in such a manner which takes into account both the previous and the next tokens making it deeply bidirectional. Word2Vec uses ifxed word embeddings, not taking into account its neighbouring tokens, which hinders its ability to capture the contextual understanding of the sentence.
As mentioned in Table 3, training and validation accuracies for Task-1 turned out to be 0.986 and 0.942 respectively with BERT and in case of Word2Vec, training and validation accuracies were 0.582 and 0.524 respectively. Similarly, as mentioned in Table 4, we achieved a training accuracy of 0.991 and validation accuracy of 0.975 with BERT for task 2. With Word2Vec, the training accuracy was 0.534 and validation accuracy was 0.480.
For Task-1, Word2Vec gave an accuracy of 0.402 and a macro F1 score of 0.401 on the test dataset (shown in Table 5). Observing the results of Task-1, when we compare the best run of BERT (with 15 epochs) which gave a F1 score of 0.503, with the run with 20 epochs which gave a macro F1 score of 0.482, we can observe a direct improvement of 4.35% has been achieved. The result for the model with 20 epochs is low in comparison since the model is a victim of over-fitting. As mentioned in table 4, for task 2, we attained an accuracy of 0.790 and a macro F1 score 0.740 on a test dataset using BERT with 18 epochs which again turned out to be better than Word2Vec due to its ability to better understand the contextual meaning of the sentence.
Task 1 of the challenge aimed to classify tweets based on sentiments for vaccination against COVID-19 into three categories, namely “ProVax”, “AntiVax” and “Neutral”. For Task 2, our aim was to classify whether a tweet was reporting symptoms of COVID-19 into four categories - “Primary Reporting”, “Secondary Reporting”, “Third-Party Reporting” and “Non-Reporting”. After preprocessing the tweets and implementing stop word removal, we applied the BERT model which makes use of contextual embeddings to classify the tweets. In addition, we also implemented the Word2Vec model which makes use of fixed embeddings for labelling the tweets. We conclude from our experiments, which were carried out on corpora of around 4300 tweets for Task1 and around 1600 tweets for Task2, that BERT is the superior algorithm for analysing and classifying tweets based on COVID-19 and is significantly more capable of capturing the underlying meaning in the best way possible. Multiple labels in the corpus make it challenging for diferent models to perform well. However, the task was made easier to tackle by the use of deep learning techniques and the bidirectional embeddings of BERT. In further work, we aim to augment our current work with better hyperparameter modelling (RoBERTa, ERNIE) and optimization techniques such as Bayesian Optimization and Population based Training.