This paper implements a binary sentiment classi cation task on datasets of online reviews. The datasets include the Amazon Fine Food Reviews Dataset and the Yelp Challenge Dataset. The paper performs sentiment classi cation via two approaches: rstly, a non-neural bag-of-words approach using Multinomial Naive Bayes and Support Vector Machine classi ers. Secondly, a Long Short-Term Memory (LSTM) Recurrent Neural Network is used. The experiment is designed to test the role of word order in sentiment classi cation by comparing bag-ofwords approaches where word order is absent with an LSTM approach which can handle sequential data as inputs. For the LSTM approaches, we test the role of various features such as pre-trained Word2vec and Glove embeddings as well as Word2vec embeddings learned on domain speci c corpora. We also test the e ect of initialising our own weights from scratch. The tests are carried out on balanced datasets as well as on datasets which follow their original distribution. This measure enables us to evaluate the e ect of ratings distribution on model performance. Our results show that the LSTM approaches using GloVe embeddings and self-learned Word2vec embeddings perform best, whilst the distribution of ratings in the data has a meaningful impact on model performance.
Sentiment Analysis is a fundamental task in Natural Language Processing (NLP).
Its uses are many: from analysing political sentiment on social media [
Pioneering approaches for sentiment classi cation include Pang et al. [
Another key downside to using bag-of-words approaches is that they are
unable to deal e ectively with negation. For example, if the model sees words like
\great" or \inspiring" in a review, it will likely prompt a positive classi cation.
However, if the actual sentence was, \The cast was not great, nor was the movie
inspiring.", it has a completely di erent meaning which the model will fail to
pick up. Additionally, bag-of-words features have very little understanding of
the semantics of the words, which can be measured as the distances between
words in an embedding space [
The inclusion of word embeddings in NLP tasks enables us to overcome such
problems. Recently, Mikolov et al. [
Furthermore, the addition of word embeddings to the eld of NLP has
enabled practitioners to use more advanced learning algorithms which can handle
sequential data as inputs such as Recurrent Neural Networks (RNNs). An
important development in the eld of RNNs was the introduction of the Long
Short-Term Memory (henceforth LSTM) RNN by Hochreiter and Schmidhuber
[
Concerning sentiment classi cation, Pang et al. [
Studies which use neural network architectures include Socher et al. [
Our datasets include the Amazon Fine Food Reviews dataset1 and the Yelp Challenge dataset 2, both of which contain a series of reviews and labeled ratings. For this project, as it is a sentiment classi cation task, only the data containing the raw text reviews and their equivalent rating were parsed. An example of a positive and a negative review from the Amazon dataset is given below:
Positive Review: These bars are great! Great tasting and with quality wholesome ingredients. The company is great and has outstanding customer service and stand by their product 110% I highly recommend these bars in any avor.
Negative Review: These get worse with every bite. I even tried putting peanut butter on top to cover the taste. That didn't work. My ve-year-old likes them. That is the only reason I didn't rate it lower. 3.1
The Amazon Fine Food Reviews Dataset contains 568,454 reviews. The dataset contains almost 46 million words and comprises 2.8 million sentences, with an average of 5 sentences per review. 3.2
We use the Yelp Academic Reviews dataset from the Yelp Dataset Challenge, which contains written reviews of listed businesses. We parse data from two 1 https://snap.stanford.edu/data/web-FineFoods.html 2 https://www.yelp.com/dataset/challenge elds: \stars" and \text", where \stars" is the customer's rating from 1 to 5 and \text" is the customer's written review. There are 4,153,150 reviews in the dataset. Out of a sample of 100,000 reviews, the number of sentences is 829,165, while the number of words is 11.57 million. The average number of sentences in the reviews is 8.
(a) Distribution of Ratings Amazon (b) Distribution of Ratings Yelp 3.3
For the pre-trained Word2vec word embeddings, we use the GoogleNews embeddings 3 which were trained on 3 billion words from a Google News scrape. The data contains 3 million 300-dimensional word vectors for the English language. We also use GloVe embeddings as a comparison. We use the GloVe embeddings which were trained on a crawl of 42 billion tokens, with a vocabulary of 1.9 million words. 4 Similarly, these vectors also have a dimension of 300. 3.4
The ratings in the Amazon and Yelp datasets were turned into binary positive and negative reviews where negative labels were assigned to ratings of 2 stars and below. Positive labels were assigned to ratings of 4 stars and above. Neutral, 3-star reviews were excluded so that our data would be highly polarised. From Figures (a) and (b) we can see that there is a large number of 5-star reviews in both datasets. In order to test the e ect of using a balanced dataset, with an even number of positive and negative reviews and a dataset which follows the original distribution, we carry out two di erent sampling techniques: First, we separate the datasets into an evenly split distribution of 82,000 positive and 3 The GoogleNews embeddings are available at: https://github.com/mmihaltz/ word2vec-GoogleNews-vectors. 4 The GloVe 42B embeddings are available at: https://nlp.stanford.edu/projects/ glove/. 82,000 negative reviews (as there are only around 82,000 negative reviews in the Amazon dataset). For the second test, to analyse the e ect of the original distribution, we randomly sample 164,000 reviews from the datasets, which should have a proportionally higher number of positive reviews re ecting the original distribution. We use 164,000 reviews in both datasets and distributions to ensure di ering results are not attributed to varying dataset sizes. For our experiments, we partition each of our datasets by an 80:20 training/test split. As the split is made after the various dataset sampling measures, the distribution of ratings in the training/test sets should be representative of the speci ed sampling approach. By doing so, we avoid the situation whereby models trained on balanced data are used to predict on the original distribution and vice versa. 4 4.1
Support Vector Machines are a type of machine learning model introduced by
Cortes and Vapnik [
!w := X j cj !dj ; j
0; j
Looking at the above solution, the idea behind the training method of the
SVM for this task is to nd a maximum separating hyperplane !w, that
separates the di erent document classes. The corresponding search is a constrained
optimisation problem, letting cj 2 f-1:1g (where 1 refers to positive and -1 is
negative) be the correct class of document dj . The j s are obtained by solving
a dual optimisation problem. The documents !dj where j is greater than zero
are called support vectors since only those document vectors are contributing to
the hyperplane !w. We are able to classify test instances by determining which
side of the of the hyperplane !w they lie [
SVM Implementation For the bag-of-words approach, the reviews were cleaned
via a text-processing algorithm to remove any unwanted characters, HTML links
or numbers and retrieve only raw text. The next stage involves converting the
words in the reviews from text to integers so that they have a numeric
representation which can be used in machine learning models.5. The bag-of-words model
builds a vocabulary from all of the words in the documents. It then models each
document by counting the number of times a word in the vocabulary appears
in the document. Considering the datasets contain a large number of reviews,
resulting in a large vocabulary, we limit the size of the feature vectors by
choosing a maximum vocabulary size of the 5000 top-occurring words. Bag-of-words
features were created for both training and test sets. We used an 80:20 train/test
split for our experiment. GridSearch cross-validation with 3 folds was used to
nd the optimal Cost parameter for our Linear Support Vector Classi er (SVC)
on the training sets. A form of feature scaling used in text classi cation tasks
includes converting the words to tf-idf features, which stands for term frequency
inverse document frequency, which is a value that corresponds to how distinctive
a word is in a corpus [
As a second baseline classi er, we use the Multinomial Naive Bayes (MNB) model. It is worth noting that Naive Bayes operates under the conditional independence assumption, that given the class, each of our words are conditionally independent of one another. This is not true in reality, as the order of words in a sentence plays an important role in the overall sentiment of a sentence. That said, Naive Bayes models using bag-of-words features can still achieve impressive results, making it a valid baseline classi er.
For our context, we can state Bayes theorem as follows:
P (C(j) w(j),...,wn(j)) = j 1
P (C(j))P (w1(j),...,wn(j)jC(j))
P (w1(j),...,wn(j))
To carry out this task we want to know P (C(j)jw1(j),...,wn(j)), that is the probability of the class of the document C(j) given its words w1(j),...,wn(j), where j is the document.
Multinomial Naive Bayes Implementation The MNB model was used as a baseline model in order to compare the results with the linear SVM. GridSearch cross-validation was used to nd the optimal value for each MNB model. As with the SVM approach, both tf-idf and regular features were evaluated. 4.3
For our neural network approach, we use LSTM RNNs because they generally
have a superior performance than traditional RNNs for learning relationships in
5 Both the Multinomial Naive Bayes and Support Vector Machine Classi ers were
implemented in Python using the Scikit-learn library
sequential data. A problem arises when using traditional RNNs for NLP tasks
because the gradients from the objective function can vanish or explode after a
few iterations of multiplying the weights of the network [
ht = ot tanh(ct) 1. New Memory Generation: The input word xt and the past hidden state ht 1 are used to generate a new memory c~t which includes aspects of the new word xt. 2. Input Gate: The input gate's function is to ensure that new memory is generated only if the new word is important. The input gate achieves this by using the input word and the past hidden state to determine whether or not the input is worth preserving and thus controls the creation of new memory.
It produces it as an indicator of this information.
3. Forget Gate: The forget gate is similar to the input gate but instead of
determining the usefulness of the input word, it assesses whether the past
memory cell is useful for the computation of the current memory cell. Here,
the forget gate looks at the input word and the past hidden state and
produces ft.
4. Final Memory Generation: For this stage, the model takes the advice
of the forget gate ft and accordingly forgets the past memory ct 1. It also
takes the advice of the input gate it and gates the new memory c~t. The
model sums these two results to produce the nal memory ct.
5. Output/Exposure Gate: This gate's purpose is to separate the nal
memory from the hidden state. Hidden states are present in every gate of an
LSTM and consequently, this gate assesses what part of the memory ct needs
to be exposed/present in the hidden state ht. The signal ot is produced by it
to indicate this and is used to gate the point-wise tanh of the memory [
LSTM Design As with the baseline approach, we use a text processing algorithm to remove any unwanted characters. We then convert all text samples in the dataset into vectors of word indices which involves converting each word to its integer ID. For this study, we permit the 200,000 most commonly occurring words in our vocabulary. We truncate the sequences to be a maximum length of 1000 words. Once the words in the reviews are converted into their corresponding integers, for the word embedding approaches, we can prepare an embedding matrix which contains at index i, the embedding vector (e.g. Word2vec or GloVe embedding) for the word at index i. The embedding matrix is then loaded into a Keras embedding layer and fed through the LSTM.6
Param
Value
Param Input length 1000 LSTM size 200 Dropout 0.25 Activation ReLU Batch size 64
Embedding size Hidden layer size Recurrent Dropout
Optimizer
Output
Value 300 128 0.25 Nadam
Sigmoid
As with the baseline approaches, we partition the training and test data by an
80:20 split. We perform GridSearch cross-validation with 3 folds to nd the
optimal model hyperparameters on the training data. We tested for several
parameters including LSTM and hidden layer sizes, batch size and the dropout value.
After conducting GridSearch cross-validation, the following hyper-parameters
were chosen: The optimal LSTM layer size was found to be 200, while the
hidden layer size was found to be 128 units. A Dropout value of 25% was selected,
which helps our models to prevent over- tting by randomly turning o nodes in
the network. The activation function we use on the inner layer is ReLU (Recti ed
Linear Unit), which is a function that maps negative values to 0 and positive
values linearly which helps transmit errors during back-propagation. The
optimizer we use is Nadam, which is a variation of the popular Adam optimizer
6 Keras was used as the deep learning library to build the LSTM network. The GPU
version of Tensor ow was used to speed up training times signi cantly.
which incorporates Nesterov momentum [
The results of our various models and dataset distributions are shown in this section. The metrics we use are accuracy and AUC (referring to area under ROC curve), which gives a measure of the relative share of true positive and false positive rates depending on a threshold. The inclusion of this metric will help shed light on the role of ratings distributions and how they a ect the model's classi cation ability. For example, a model trained on a dataset containing a higher proportion of positive reviews may perform better on a super cial level due to it being more likely to predict the majority class prevalent in the data.
From Table 2, with respect to the balanced dataset, from the bag-of-words approaches, the SVM using TF-IDF features performed the best achieving an accuracy 88.95% on the Amazon dataset. Similarly, the SVM TF-IDF model performed best on the Yelp dataset with an accuracy of 92.91%. In both cases, the AUC score is very similar to the accuracy score. The MNB classi ers performed worse across the range of tests but still achieved satisfactory accuracy and AUC scores, rendering the use of MNB models as a baseline.
For the LSTM models, we can see that they generally perform better than the baseline bag-of-words methods with the exception of the LSTM using Word2vec embeddings learned on the Amazon dataset, which lags behind the other LSTM variations and even the SVM models and some MNB models on the Yelp dataset. The LSTM with GloVe embeddings performs the best across the Amazon dataset with an accuracy of 95.03% and and AUC score of .9889. Meanwhile, the LSTM using dataset speci c Word2vec embeddings performs best with 95.75% accuracy and an AUC score of .9924 on the Yelp dataset.
The promising results from the LSTM models indicate that the LSTMs can handle sequential information well and the addition of word embeddings help improve model performance, where the models using embeddings narrowly outperform the models using self-initialised weights. An interesting result is the di erence in performance between the LSTM models using domain-speci c Word2vec embeddings on the Amazon and Yelp datasets, where these features resulted in a worst performing and best performing LSTM model respectively. A reason for this could be due to attributes of the corpora involved. The Amazon dataset had an average of ve words per sentence, whilst the Yelp dataset had an average of eight words per sentence. The longer sentences in the Yelp dataset could give rise to a scenario where more semantic information can be captured by the Word2vec model, resulting in better embeddings.
With respect to the original datasets: As with the balanced datasets, the SVMs perform better than the MNB models, with SVMs using TF-IDF features Distribution Model Balanced
SVM SVM TF-IDF MNB
MNB TF-IDF Original
LSTM Word2vec LSTM Self Initialised LSTM GloVe LSTM Word2vec-domain SVM SVM TF-IDF MNB MNB TF-IDF LSTM Word2vec LSTM Self Initialised LSTM GloVe LSTM Word2vec-domain Accuracy AUC
Accuracy AUC performing better on the Amazon and Yelp datasets. Whilst the baseline models look good at an initial glance in terms of accuracy, the AUC scores on the datasets following the original distribution paint a rather di erent picture. We are able to observe that there is a much wider gap between the accuracy and AUC scores in the original distribution than in the balanced distribution. This vindicates a prior assumption, that the greater success of a number models on the original distribution in terms of accuracy can somewhat be attributed to the increased likelihood of classifying the majority class. This is not as relevant to the LSTM models on the original datasets as they still manage to achieve very successful AUC scores.
With regard to the LSTMs, the LSTMs which use GloVe embeddings perform best in terms of accuracy on the Amazon dataset and Word2vec embeddings learned on domain-speci c corpora perform best on the Yelp dataset and on the Amazon dataset in terms of AUC. The models using GloVe embeddings narrowly outperform the models using Word2vec embeddings across all tests, which echoes the results on the balanced datasets where GloVe scores, in the majority of cases, outperformed the scores of models using pre-trained Word2vec embeddings. Similarly, models using self-initialised weights slightly lag behind models using pre-trained weights in most cases. Despite this, the features which consist solely of word indices and have no prior knowledge of word meaning, still act as good features for the LSTM. The performance of LSTM models using self-initialised weights is very stable across both datasets and distributions, indicating that the LSTMs can learn meaningful information from the words in the corpora without having a semantic understanding of the words. The fact that these models do not use pre-trained weights and still outperform the baseline bag-of-words methods, which also have a limited understanding of word meaning, gives credence to the role of LSTMs in tasks which involve modeling sequential data such as text. 6
In this study we have compared bag-of-words and neural network based approaches for sentiment classi cation. Firstly, we used unordered bag-of-words models, and secondly, we used an LSTM model which can handle sequential data as well as leverage the use of pre-trained word embeddings. From our analysis, for the baseline approaches, the SVM models outperform the Multinomial Naive Bayes classi ers. The LSTM models outperform the bag-of-words models across both metrics for the majority of tests. Nevertheless, bag-of-words models can still perform very well, particularly with respect to their shorter training period. LSTM models using pre-trained GloVe embeddings and Word2vec embeddings learned on domain-speci c corpora performed best. In most cases, pre-trained GloVe embeddings served as better features than pre-trained Word2vec embeddings. The strong performance of the models using domain-speci c Word2vec embeddings could justify using such an approach provided there is an adequate amount of text to train on.
We also compared our results across two di erent dataset distributions, a balanced distribution and one which follows the original distribution. While the greatest accuracy was achieved on the original distribution using Word2vec domain-speci c embeddings, there was less disparity among AUC scores on the balanced datasets, particularly with respect to the baseline models. The inclusion of sampling measures which balance the distribution of ratings can help ensure the models are less likely to over t on positive reviews given their higher respective share in the data. The fact that the LSTM models achieved greater AUC scores than the baseline models highlights their ability in NLP tasks. The LSTM models are able to learn more subtle relationships which the baseline models fail to pick up on as evident in their comparative AUC scores.