The rapid development of the Covid-19 vaccines, concerns about its safety contributed to vaccine hesitancy globally. Social media platforms transfer knowledge on such global concerns and are a good source for investigating public opinions. This study proposes a machine learning based analysis of Covid-19 vaccine public opinions using Twitter data where a tweet post is classified into multiple labels which describes various concerns. We experimented with supervised learning algorithms wrapped along with multiple label classifier algorithms. We have achieved an average F1 micro score of 62% which suggested improvement.
CEUR ceur-ws.org
Vaccination is a highly efective public health strategy, saving lives and reducing disease burden. However, vaccine hesitancy persists, especially in the digital age with easy access to both credible and misleading information. Crowdsourced data platforms now allow individuals to share their vaccine-related experiences and concerns, ofering valuable insights into this issue.
Machine learning has become a vital tool in public health and epidemiology. It can analyse large datasets, uncover patterns, and provide insights that traditional methods struggle to achieve. Machine learning algorithms can sift through vast amounts of unstructured text data from social media and other platforms to reveal patterns and concerns related to vaccines.
This research paper aims to contribute to the growing body of knowledge in the field of vaccine hesitancy and public health by presenting a comprehensive analysis of crowdsourced vaccine data using machine learning techniques. This study is driven by the fact that a deeper understanding of the concerns expressed by individuals through crowdsourcing regarding vaccines can inform targeted public health interventions and communication strategies. By CEUR Workshop Proceedings leveraging machine learning algorithms, we aim to shed light on the intricate dynamics of vaccine hesitancy, ultimately contributing to more efective vaccination campaigns and improved public health outcomes. Additionally, comprehending these perceptions within communities, states, and the nation across diferent time frames can furnish us with precise data for crafting specialised strategies to enhance immunisation education programs and public health campaigns.
In the subsequent sections of this paper, we will discuss the methods employed, present our ifndings, and discuss the implications of our analysis.
The utilisation of Machine Learning for analysing vaccine-related concerns is of paramount
significance in the current era of pandemics, and numerous studies have delved into this field.
This paper specifically concentrates on developing a predictive model for assessing public
sentiment-related concerns, primarily sourced from social media platforms. One study has
highlighted the utilisation of social media bots to intentionally sow discord and confusion
regarding vaccination, potentially dissuading people from getting vaccinated [
Additionally, another research emphasises the pivotal need to confront and counteract
rumours and conspiracy theories in public health campaigns, particularly in the context of
mitigating vaccine hesitancy and ensuring the success of vaccination initiatives [
A study conducted from April to August 2019 aimed to develop and validate deep learning
models to understand public perceptions of the HPV vaccine using data from social media [
A study [
Moreover, a study has been conducted to demonstrate the eficient collection and
preprocessing of Twitter data, encompassing information related to vaccines as well as other disaster-related
data [
We aim to develop a multi label classification on public opinion tweets of the Covid-19 vaccines
which is a methodology proposed as part of the AISoMe (Artificial Intelligence on Social Media)
track [
The proposed methodology aims to perform multi label classification on the given dataset. In depth study of the literature [9][10][11] describes various methods of machine learning based multi label classification methods. We experimented with a problem transformation method, namely classifier chains, which transforms a multi label classification problem into multiple binary classification problems.
To comprehend and interpret the given data in depth, we begin with an EDA. The given data initially had 3 columns, ‘ID’, ‘tweet’ and ‘labels’. For a multi label classification problem, one hot encoded data is appropriate and hence the data was modified where the labels are one hot encoded. The given data contained no null or NaN values. The one hot encoded dataset contains 9921 rows and 14 columns. Figure 1 shows a sample of the one hot encoded dataset.
The tweets in the data are labelled about concerns of covid vaccines (see section 3) and have categorised tweets under 12 labels. Figure 2 shows the number of tweets that are categorised under a particular label. From the figure we can see that the label ‘side-efect’ is the highest in number and ‘religious’ is the lowest. A tweet could be categorised by 1, 2 or 3 labels. Figure 2 shows the number of tweets that got categorised under a single label, two labels or three labels. From the figure we can see that the majority of the tweets were categorised by a single label.
To understand the use of vocabulary in the tweets, word clouds were generated of the most frequent and less frequent labels in the dataset, which are ‘side-efect’ and ‘religious’ labels respectively. Figure 3 shows the generated word clouds. From the word clouds we can observe that the terms ‘vaccine’, ‘covid’ and ‘pfizer’ have frequent occurrences. Keywords similar to ‘side-efect’, such as ‘death’, ‘adverse reaction’, ‘blood clot’ etc are found to occur frequently in the ‘side-efect’ word cloud. Keywords similar to ‘religious’ label, such as ‘religion’, ‘faith’, ‘psalm’ etc were found, but were less frequent in the ‘religious’ word cloud.
Basic data cleaning methods have been applied onto the tweet data. Before removing the unwanted text, we word tokenize the tweet and each character to lowercase. The cleaned tweet is further fed into the model allowing to reduce unnecessary processing. We use Python libraries such as NLTK and Regular expressions to eliminate the following: • URLs often found along with a tweet (image, video urls) • Stopwords (the, a, is etc) are removed except for ‘not’ and ‘no’ to maintain the context • Special characters, smileys
We have experimented with a multi label classification where each data point is associated with multiple labels. Among various multi label classification approaches, we have experimented with classifier chains and multi output classifier methods. A classifier chain initially starts with a set of binary classifiers, one for each label in the multi label classification problem. When making predictions for a new instance, you start by predicting the first label using its binary classifier. Then, you use this prediction, along with the instance’s features, to predict the second label. This process continues until all labels have been predicted. To perform the classification we can wrap any classification algorithm which is capable of a binary classification in the classifier chain. We have also experimented using the multi output classifier algorithm, a wrapper that takes a single-output classifier and extends it to work with multiple output labels.
Initially, we stratified the dataset with a train-test split of 70% to 30% respectively. Both the training and validation data was preprocessed as described in section 4.2. The resultant data was used as the input data for both classifier chain and multi output classifier models. We experimented two model creations with the classifier chains by wrapping a Logistic Regression and Support Vector Machines model. Both these models are widely used for classification problems. We experimented one model creation with the multi output classifier where it was wrapping a Support Vector Machine model. All 3 experiments showed a moderate result in the initial phase which led to fine tuning.
The created models are evaluated using accuracy, precision, recall and F1 score. Table 1 shows the evaluation results of the 3 models in the initial run. Table 3 shows the performance evaluation of the run files submitted to the AISoMe track.
To evaluate the model better, we plotted the learning curve of the classifier chain models which led to fine tuning it further. Figure 4 shows the learning curves of the Logistic Regression model wrapped in a classifier chain, before and after fine tuning. Figure 5 shows the learning curves of the Support Vector Machines model wrapped in a classifier chain, before and after tuning. Fine tuning certainly improved the performance, but also suggests the need for more data for training.
We have achieved slight improvement in the performance of the classifier chain models after ifne tuning. Table 2 shows the performance results of the 2 fine tuned models.
This research work uses the Covid vaccine social media data which showed the concerns of the public related to the usage of vaccinations. We experimented multiple models with the given data and chose the top 3 performing models to showcase in this report. We used a classifier chain model which wraps Support Vector Machines and Logistic Regression and a multi output classifier which wraps Support Vector Machines. We achieved the highest score for Multi Output Classifier with a F1 score of 64%. The performance can be improved either by improving the dataset or by other preprocessing methods or data augmentation strategies.
The input data, test data and implemented Python code are made available on ”https://github.com/lakshmiSGopal/AISOME-FIRE-2023”. [9] C. Prathibhamol, G. Amala, M. Kapadia, Anomaly detection based multi label classification using association rule mining (admlcar), in: 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), IEEE, 2016, pp. 2703–2707. [10] C. Prathibhamol, K. Jyothy, B. Noora, Multi label classification based on logistic regression (mlc-lr), in: 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), IEEE, 2016, pp. 2708–2712. [11] R. Ramanathan, K. Soman, P. Rohini, G. Dharshana, Investigation and development of methods to solve multi-class classification problems, in: 2009 International Conference on Advances in Recent Technologies in Communication and Computing, IEEE, 2009, pp. 805–807.