The coronavirus outbreak has resulted in unprecedented measures, forcing authorities to make decisions related to establishing lockdowns in areas most affected by the pandemic. Social Media have supported people during this difficult time. On November 9, 2020, when the first vaccine with an efficacy rate over 90% was announced, social media reacted and people around the world began to express their feelings about this vaccination. This paper aims to analyze the dynamics of opinion on COVID-19 vaccination, in which the civil society is highly manifested in the vaccination process. We compared classical machine learning algorithms to select the best performing classifier. 4,392 tweets were collected and analyzed. The proposed approach can help governments create and evaluate appropriate communication tools to provide clear and relevant information to the general public, increasing public confidence in vaccination campaigns.
The coronavirus outbreak caused by the novel coronavirus SARS-CoV-2 has brought a series
of changes in many aspects of people's economic and social life. Since its occurrence, the
coronavirus pandemic has continued to monopolize the different parts of the world, reaching 220
countries and territories by December 9, 2020 [
Among some of the popular social media platforms, Twitter receives special attention. This is
because users can easily disseminate information about their opinions on a particular topic
through public messages called tweets [5]. In addition to the information voluntarily provided by
the user, Tweets may also contain information about the user's location and may include links,
emoticons, and hashtags that allow the user better express the emotions, making it a source of
valuable information [5], [
In this context, this paper analyses public opinion regarding the vaccination process in case of COVID 19, considering news posted on Twitter. Clean dataset was extracted, including 4392 tweets. The performance of multiple machine learning algorithms (both traditional and deep learning algorithms) was compared using annotated data sets. Best performing algorithms were selected and used to analyse the dataset. We collected and annotated the COVID-19 vaccination dataset, determined the best classifier for stance detection of COVID-19 vaccinations, and associated the number of tweets with stance (e.g., ProVax, Neutral and AntiVax). Selected approaches can be easily integrated into systems that allow interested organizations to adequately monitor public opinion regarding the vaccination process in case of the novel coronavirus.
The steps taken to analyze public opinion on COVID-19 vaccination from social media messages are stated below. The first step is to collect the COVID-19 vaccination stance dataset, which contains tweets in English. A randomly selected subset of this dataset was manually annotated as Neutral, ProVax or AntiVax to be used in the training phase of the pose classification algorithm. Due to its unstructured nature and informal writing style, tweets from the collected dataset were preprocessed in the next step to improve the performance of the pose classification algorithm.
In the current work, the performance of several classic machine learning algorithms was evaluated based on the following widely used metrics: accuracy, precision, recall and f-score. Accuracy is the ratio of correctly predicted observations to all observations and is defined as shown in (1). where TP, TN, FP, and FN refer to true positives, true negatives, false positives, and false negatives.
Thus, TP represents the number of real positive tweets classified as positive, FP is the number of real negative tweets classified incorrectly classified as positives, TN represents the number of negative tweets correctly classified as negative and FN is the number of real positive tweets incorrectly classified as negative. (1) (2) Precision, which represents the ratio of correctly predicted positive observations to the total predicted positive observations, is computed as shown in (2).
Recall, representing the ratio of correctly predicted positive observations to all the observations in the actual class, is computed as shown in (3).
=
+ + + + = + (3) (4) = + = ∗ ∗ + Starting from Precision and Recall, the F-Score can be computed as a weighted average, as shown in (4).
Finally, the best performing algorithm has been used to analyze the evolution of the public stance towards vaccination in the considered period.
The main components of the stance detection process are the pre-processing, the feature extraction and the machine learning classification. The preprocessing step cleanses the text and the feature extraction transforms the raw text data into feature vectors.
We have performed various pre-processing steps on the dataset that mainly dealt with removing stop words. The text document is then converted into the lowercase for better generalization. Subsequently, the punctuations were cleaned and removed thereby reducing the unnecessary noise from the dataset. After that, we have also removed the stop words from the words along with removing the URLs as they do not have any significant importance.
At last, Lemmatization (reducing the derived words to their root form known as lemma) was performed for better results. Stop words are the words in any language which does not add much meaning to a sentence. They can safely be ignored without sacrificing the meaning of the sentence.
Countvectorizer: Machines cannot understand characters and words. So, when dealing with text data we need to represent it in numbers to be understood by the machine. Countvectorizer is a method to convert text to numerical data. The text is transformed to a sparse matrix. Countvectorizer makes it easy for text data to be used directly in machine learning and deep learning models such as text classification. We have used CountVectorizer for tuning the learning process and set its hyperparameters as the following: • max_features:5000, which implies that top 5000 frequent words from the data is selected • stop_words: an array of redundant words has been passed.
We used inbuilt functions mentioned below to train our model: • train_test_split(): This function splits the dataset into a train and test set with a specified criteria of split, we started with a fraction setting of 0.2. This means we used 80% of our dataset for training our model and 20% for testing and evaluating our model. • •
TfidfVectorizer() : Tf-idf is used to handle text data for machine learning purposes, it stands for term frequency — inverse document frequency and is represented by the formula below, using this function we convert all words into tf-idf scores.
TF — IDF = TF (Term frequency) * IDF (Inverse document frequency) Term frequency — The number of times the term occurs in a given document. IDF — The number of documents in which the given term is found. make_pipeline() : This function is used for defining our data pipeline. In this we can apply a list of transforms, followed by a final estimator. We used Bernoulli for our case.
A machine learning approach has been used in order to accurately determine the stance towards vaccination in the collected tweets. Starting from the annotated dataset, the performance of several popular classification algorithms has been investigated: Bernoulli Naïve Bayes, Support vector machine (SVM), Multinomial logistic regression, Logistic Regression Machine Learning, The KNN classifier, Gradient Boosting.
Multinomial Naive Bayes algorithm is a probabilistic learning method that is mostly used in Natural Language Processing (NLP). The algorithm is based on the Bayes theorem and predicts the tag of a text such as a piece of email or newspaper article. It calculates the probability of each tag for a given sample and then gives the tag with the highest probability as output.
Support Vector Machines (SVM) [99] are a family of supervised learning algorithms used for classification, regression and other tasks such as outlier detection. While other classification algorithms suffer from overfitting, one of the advantages of SVM is that they are less prone to this situation [100]. Another advantage resides in the fact that besides binary classification, multiclass classification can be performed by combining several binary classification functions. For this, each class is considered individually at a time, and for each class a classifier is searched that separates it from the other classes [101].
This is used for discrete data and it works on Bernoulli distribution. The main feature of Bernoulli Naive Bayes is that it accepts features only as binary values like true or false, yes or no, success or failure, 0 or 1 and so on. So, when the feature values are binary, we know that we have to use Bernoulli Naive Bayes classifier.
The K-Nearest Neighbor or the KNN algorithm is a machine learning algorithm based on the supervised learning model. The K-NN algorithm works by assuming that similar things exist close to each other. Hence, the K-NN algorithm utilizes feature similarity between the new data points and the points in the training set (available cases) to predict the values of the new data points. In essence, the K-NN algorithm assigns a value to the latest data point based on how closely it resembles the points in the training set. K-NN algorithm finds application in both classification and regression problems but is mainly used for classification problems.
Logistic Regression Machine Learning is basically a classification algorithm that comes under the Supervised category (a type of machine learning in which machines are trained using "labelled" data, and on the basis of that trained data, the output is predicted) of Machine Learning algorithms. This simply means it fetches its roots to the field of Statistics. The main role of Logistic Regression in Machine Learning is predicting the output of a categorical dependent variable from a set of independent variables. In simple words, categorical dependent variable means a variable that is dichotomous or binary in nature having its data coded in the form of either 1 (stands for success/yes) or 0 (stands for failure/no). It is a popular boosting algorithm. In gradient boosting, each predictor corrects its predecessor’s error. The weights of the training instances are not tweaked, instead, each predictor is trained using the residual errors of predecessor as labels. Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees.
mn= [KNN( ), SVM(), LogisticRegression(), GradientBoostClassifier(), BernoulliNB(), MultinomialNB()] ; for (i=0; i<6; i++) Model= mn[i]; Model.fit(); Model.predict(); print(Accuracy(i), confusion_matrix, classification_report); end loop
Let’s view the performance of each of the machine learning algorithms/classifications by representing evaluation metrics such as recall, precision and f1-score. Confusion matrix for the KNN classifier is as shown in the above figure. Confusion matrix for the Gradient Boosting Classifier is as shown in the above figure. Confusion matrix for the Support Vector Machine Classifier is as shown in the above figure. Confusion matrix for the Multinomial Naïve Bayes is as shown in the above figure.
After applying various Machine Learning Algorithms on the Training data-set we got accuracies as mentioned below in table-7.
K-Nearest Neighbor Gradient Boosting Classifier Logistic Regression Support Vector Machine Multinomial Naïve Bayes Bernoulli Naïve Bayes 53.4% 61.4% 64.2% 62.9% 63.3% 60.5% Algorithm
Accuracy
F1 - Score Bernoulli Naïve Bayes Support Vector Machine Logistic Regression Multinomial Naïve Bayes K-Nearest Neighbor Gradient Boosting Classifier Below is the pictorial representation of the accuracies obtained by each machine learning classifier:
Accuracy 70 60 50 40 30 20 10 0
K Nearest neighbor
Logistic Regression
Multinomial Bernoulli Naïve Naïve Bayes Bayes
Gradient Boosting
Support Vector
Machine
Accuracy 4. Conclusion In the current study, the initial announcement of a coronavirus vaccine and the first real vaccination process initiated outside of limited clinical trials were analyzed using machine learning-based stance detection. Several classical machine learning and deep learning algorithms were compared, and the best performing classifier was selected based on the performance metrics. The proposed approach used Bernoulli Naïve Bayes with an accuracy of 49% to classify tweets into three main classes: ProVax, AntiVax, and Neutral regarding the COVID-19 vaccination. The purpose of this paper was to monitor changes in the stance towards COVID-19 vaccination through tweets.
With many countries around the world planning to initiate vaccination processes for COVID19, early detection of changes in opinion can be very useful and help government decision makers to take steps to curb infections. This can be very helpful as it allows us to drive targeted actions. Possible future research directions include the development of better performing stance classification algorithms, as well as extending the analyzed period, especially given the fact that the vaccination process is expected to take a relatively long period of time. 5. References