In this paper, team University of Botswana Computer Science (UBCS) investigate the opinions of Twitter users towards vaccine uptake. In particular, we build three diferent text classifiers to detect people's opinions and classify them as provax-for opinions that are for vaccination, antivax for opinions against vaccination and neutral-for opinions that are neither for or against vaccination. Two diferent datasets obtained from Twitter, 1 by Cotfas and the other by Fire2022 Organizing team were merged to and used for this study. The dataset contained 4392 tweets. Our first classifier was based on the basic BERT model and the other 2 were machine learning models, Random Forest and Multinomial Naive Bayes models. Naive Bayes classifier outperformed other classifiers with a macro-F1 score of 0.319.
In recent years, there has been rapid increase in internet access and usage all over the world.
This increase in internet access has resulted in large volumes of structured and unstructured data
being deposited on online repositories [
For the world to attain herd immunity or break the spread of the COVID-19 pandemic, Trapman
[
Sentiments Analysis is becoming a widely used natural language processing technique to mine
people’s opinions from diferent media sharing platforms. It has been used in many social issues
in the past to cope with mining people’s opinions from text [
A study by Machucha et al. [11] used sentiment analysis to get the people’s general opinions on corona virus using binary logistic regression. They classified the sentiments as positive and negative only. After extracting the data, they preprocessed it by removing all punctuation marks, transforming each tweet to lowercase and tokenizing tweets to facilitate the removal of non-English words and stopwords. The tweets were stemmed and rejoined. TF-IDF vectorizer was used for vectorization in this study. When the evaluated the performance of their classifier, they obtained a classification accuracy of 78.5%.
Delizo et al. [12] conducted a study that uses sentiment analysis to examine the polarity of COVID-19 related opinions on Twitter using Multinomial Naïve Bayes algorithm, targeting tweets from Philippines. Their preprocessing stage included removing special characters, hyperlinks, hashtags and mentions, and removing tweets containing less than 2 words and less than 3 characters from each tweet. 10% of the cleaned tweets was manually labelled as either positive or negative Words were then converted to their lowercase equivalents. TF-IDF vectorizer was used for feature extraction from the training data. Multinomial Naïve Bayes algorithm was employed to test the efectiveness of the proposed it achieved an overall accuracy score of 72%.
The study of [
In another study that aims to use sentiment analysis to analyses people’s opinions on
COVID19 vaccine, Trapman [
The study by Kitchat et al. [10] also used sentiment analysis to investigate people’s opinions from tweets in New York City. Keywords relating to COVID-19 vaccines were used to filter out relevant tweets. To further filter out relevant tweets, the location from which the tweets were from was also specified and in this study, New York City was specified as the location of tweets. After collecting the data, they cleaned the data by removing hashtags, punctuation marks, URLs, emojis stopwords (through tokenization) and converting all letters to lowercase. The tweets were stemmed and rejoined after tokenization. The tweets were classified as positive, negative and neutral. To test the performance of the proposed system, they used 6 machine learning models, namely, Random Forest, Decision Trees, Support Vector Machine (SVM), Naïve Bayes, Logistic Regression and Multi-Layer Perceptron (MLP). According to their results, all machine learning models used performed exceptionally well, giving an accuracy of over 90% except for Naïve Bayes which registered an accuracy of 82.13%. Multi-Layer Perceptron model performed better than all other models, with an accuracy of 93.63
In this Section, we present our BERT and machine learning approaches for classifying tweets into three classes, namely: ProVax, AntiVax and Neutral. AntiVax indicates that there is hesitancy in the tweet (of the user who posted the tweet) towards the use of vaccines. Neutral indicates that the tweet does not have any discernible sentiment expressed towards vaccines or is not related to vaccines. We propose using the basic BERT classifier and 2 machine learning classifiers provided by scikit-learn library 1, namely, Multinomial Naive Bayes classifier, which is a Nave Bayes variant used in text classification in python as well as the Random forest classifier.
The training dataset was pre-processed to make it compatible with machine learning models and BERT model by converting words to numbers which are easily understood by the models. In addition, the following pre-processing steps were applied to the main dataset: • For the BERT experiment, BERTokenizer provided by the Transformer library was used to convert the tweets into tokens and tokenIDs. • For the machine learning algorithms, the Tfidf Vectorizer from scikit-learn library was employed to perform feature extraction from the training as well as stopwords removal. • The training data was stemmed and rejoined.
• Hashtags and mentions not removed, as well as punctuations not removed.
The training dataset contains 4392 tweets. The dataset was obtained from Twitter by diferent teams as follows: • Cotfas dataset: Crawled from Twitter between November-December 2020. 2792 tweettexts were crawled along with tweetIDs and labels (proVax, neutral, antiVax). This crawled data formed the first part of the training data. • IRMiDis Fire2022 organizing team dataset: Tweets crawled from March to December 2020. Tweets were annotated with three labels by crowdworkers. A total of 1600 tweets were crawled, and this formed the second part of the training data.
Of this 4392 tweets, 1676 are provax, 1081 are antivax and 1635 are neutral. During training, the training dataset was subdivided such that 3952 tweets train our classification models and 440 tweets are used for validation.
The same pre-processing done in the training dataset was performed on the test data set, except for pre-processing that deals with labelling the tweets as provax, neutral and antivax.
We submit 3 runs for: Task 1: COVID-19 vaccine stance classification from tweets. Below is a brief description of each run: 4.1. Run 1 - UBCS This is our baseline run. We used BERT model to build a BERT classifier for the identification of sentiments. Since BERT works with fixed-length sequences, we set a maximum length for the sequences, guided by the number of tokens in the tweets. Most tweets seemed to contain less than 140 tokens and we then set our maximum length to 150 to allow for tweets that may be longer than the ones we used for training during testing. We also set the batch size to be 16 because 16 and 32 are recommended by BERT authors as the best sizes for fine tuning the model. We built our sentiment classifier on top of the basic BertModel. In our classifier, we used a dropout layer for some regularization and a fully connected layer for our output. . Both the training and test dataset underwent the same pre-processing steps as described in Section 3.1. 4.2. Run 2 - UBCS and Run 3 - UBCS In these runs, we used the Random Forest and Multinomial Naïve Bayes models respectively. We vectorized our tweets into numbers using TFIDFVectorizer. We defined variable X, to hold all the features to be used in predictions while variable y holds the labels. We then imported the Machine Learning models, Random Forest and Multinomial Naives Bayes (an implementation of Naïve Bayes algorithm in scikit-learn), from scikit-learn library using their default parameters. For all the three runs, we divided the data into training data(90% of the data), validation data(10%). Data loaders for the 3 sets of data were created for each set of data.
In this paper, we employ diferent text classifiers and compare their performance to determine the best classifier among them. Table 1 presents the results of our investigation. Run 3 - UBCS is our best run as it performed better than all the other runs in all the evaluation metrics, with a Macro F1 score and Accuracy of 0.319 and 0.337 respetively.
The results of our investigation suggests no significant diference in the performance of the diferent classifiers used in this study. The results might have been afected by not preprocessing data further and not converting the emojis to text. A study by [14], suggest that BERT based models produce better performance as BERT is empirical and a powerful tool. This is also evidenced by the overall performance of teams that used BERT models in the previous studies. Further work needs to be carried out in order to determine whether emojis can significantly improve the classification accuracy when building a classifier to detect the opinions of Twitter users towards vaccine uptake. A further study with more focus on the efects of stopword removal and semantic enrichment of tweets when build classifiers to detect people’s opinions is also recommended. [10] K. Kitchat, K. Limjumroonrat, T. Surasak, Twitter sentiment analysis towards covid-19 vaccines: A case study in new york city, in: 2021 IEEE 2nd International Conference on Technology, Engineering, Management for Societal impact using Marketing, Entrepreneurship and Talent (TEMSMET), IEEE, 2021, pp. 1–6. [11] C. R. Machuca, C. Gallardo, R. M. Toasa, Twitter sentiment analysis on coronavirus: Machine learning approach, in: Journal of Physics: Conference Series, volume 1828, IOP Publishing, 2021, p. 012104. [12] J. P. D. Delizo, M. B. Abisado, M. I. P. De Los Trinos, Philippine twitter sentiments during covid-19 pandemic using multinomial naïve-bayes, International Journal 9 (2020). [13] A. Thomson, K. Moon, G. Vallee-Tourangeau, The 5as: A practical taxonomy for the determinants of vaccine uptake, Vaccine 34 (2015). doi:10.1016/j.vaccine.2015.11. 065. [14] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics, 2019, pp. 4171–4186. URL: https://doi.org/10.18653/v1/n19-1423. doi:10.18653/v1/n19-1423.