Selection of Intelligent Algorithms for Sentiment Classification Method Creation

Abstract

-The main goal of this paper is to select two single intelligent algorithms for sentiment classification method creation. We perform set of experiments to recognize positive or negative sentiment, using single intelligent methods and combination of them. It was observed that the better results were obtained by the single methods: Logistic regression, SVM and Na¨ıve Bayes, also the combination of Logistic regression with SVM. Index Terms-Sentiment analysis, Logistic Regression, Na¨ıve Bayes classification, Support Vector Machines, Random Forest.

-

I. INTRODUCTION

Nowadays sentiment analysis is a very popular research area. A lot of works are done, but still there are no good enough method for sentiment classification. Many authors declare results of average slightly above 80%, but it is not enough if we need more accurate results.

Pang et al. in [ 17 ] evaluated the performance of Na¨ıve Bayes, Maximum Entropy, and Support Vector Machines in the specific domain of movie reviews, obtaining that Na¨ıve Bayes shown the worst and SVM the best results, although the differences aren’t very large. Later Go et al. in [ 12 ] obtained similar results with unigrams by introducing a more novel approach to automatically classify the sentiment of Twitter messages as either positive or negative with respect to a query term. The same techniques were also used by Kharde and Sonawane in [ 14 ] to perform sentiment analysis on Twitter data, yet resulting in lower accuracy; again, SVM proved to perform best. Davidov et al. in [ 11 ] also stated that SVM and Na¨ıve Bayes are the best techniques to classify the data and can be regarded as the baseline learning methods, by applying them for analysis based on the Twitter user defined hashtag in tweets. Tian et al. in [ 25 ] applied seven classification algorithms: J48, Random Forest, ADTree, AdaBoostM1, Bagging, Multilayer Perceptron and Na¨ıve Bayes for imbalanced sentiment classification of Chinese product reviews. They found that their proposed method helps a Support Vector Machines (SVM) to outperform other classification methods. Singh et al. in [ 21 ] used a novel technique to predict the outcome of US presidential elections using sentiment analysis. To accomplish this task they used SVM. Jayalekshmi and Mathew in [ 13 ] proposed a system that automatically recognize the facial Copyright held by the author(s). expression from the image and classify emotions for final decision. For classification they used SVM, Random Forest and KNN classifier. Ahmed et al. in [ 2 ] did investigation on a new approach of finding sentence level sentiment analysis using different machine learning algorithms. SVM (Support Vector Machines), Na¨ıve Bayes and MLP (Multilayer Perceptron) were used for movie reviews sentiment analysis. Moreover they used two different classifiers of Na¨ıve Bayes and two different types of SVM kernels to identify and analyze the difference in accuracy as well as to find the best outcome among all the experiments. Tayade et al. in [ 23 ] used sentiment analysis through machine learning to identify the targets of trolls, so as to prevent trolling before it happens. Na¨ıve Bayes, Support Vector Machines (SVM) and Maximum Entropy (MaxEnt) classifiers have shown very promising results. Maheshwari et al. in [ 16 ] applied Support Vector Machines, Logistic Regression and Random Forests machine learners to identify the best linguistic and non-linguistic features for automatic classification of values and ethics. Pranckevic˘ius and Marcinkevic˘ius in [ 19 ] did experiments on short text for product-review data from Amazon in case to compare Na¨ıve Bayes, Random Forest, Decision Tree, Support Vector Machines, and Logistic Regression classifiers implemented in Apache Spark by evaluating the classification accuracy, based on the size of training data sets, and the number of n-grams. Ahmad et al. in [ 1 ] did a review of different machine learning techniques and algorithms (Maximum Entrophy, Random Forest, SailAil Sentiment Analyzer, Multilayer Perceptron, Na¨ıve Bayes and Support Vector Machines) which were applied by the researches on movie reviews and product reviews for the evaluation. Brito et al. in [ 7 ] presented how different hyperparameter combinations impact the resulting German word vectors and how these word representations can be part of more complex models. For prediction whether a user liked an app given a review with three different algorithms: Logistic Regression, Decision Trees and Random Forests. Ashok et al. in [ 4 ] proposed a social framework, which extracts user’s reviews, comments of restaurants and points of interest such as events and locations, to personalize and rank suggestions based on user preferences. Na¨ıve Bayes, Support Vector Machines with two different kernels (Gausian and Linear), Maximum Entropy and Random Forest have been used in this work.

Such results led to the conclusion that Logistic Regression, SVM, Na¨ıve Bayes and Random Forest are still prominent for future research. Therefore, in this paper we perform experiments with each of them also with various combinations of two of them (depending on results of previous) to recognize positive or negative sentiment and to compare accuracy between them. The rest of the paper is organized as follows. In section II, a description of techniques used in research. In section III, presented method for combining results. In section IV, described preparation of dataset, experiments, experimental settings, effectiveness measure and results. In section V, we conclude and give tasks of our future works.

II. DESCRIPTION OF TECHNIQUES USED IN RESEARCH

A. Logistic Regression

The logistic regression model arises from the desire to model the posterior probabilities of the K classes via linear functions in x, while at the same time ensuring that they sum to one and remain in [ 0, 1 ]. The model has the form log

P r(G = 1jX = x) log

P r(G = KjX = x)

P r(G = 2jX = x) log

P r(G = KjX = x) . .

.

P r(G = K 1jX = x)

P r(G = KjX = x) = 10 + = 20 + 1T x 2T x

(1) = (K 1)0 +

KT 1x The model is specified in terms of K 1 log-odds or logit transformations (reflecting the constraint that the probabilities sum to one). Although the model uses the last class as the denominator in the odds-ratios, the choice of denominator is arbitrary in that the estimates are equivariant under this choice. A simple calculation shows that

P r(G = kjX = x) =

exp( k0 + kT x) 1 + PK l=11 exp( l0 + lT x) ; k = 1; : : : ; K 1; 1 P r(G = KjX = x) = 1 + PlK=11 exp( l0 + lT x) ; (2) and they clearly sum to one. To emphasize the dependence on the entire parameter set = f 10; 1T ; : : : ; (K 1)0; KT 1g, we denote the probabilities P r(G = kjX = x) = pk(x; ) (Hastie et al. in [ 26 ]).

B. Na¨ıve Bayes Classification

A Na¨ıve Bayes classifier is a simple probabilistic classifier based on Bayes’ theorem and is particularly suited when the dimensionality of the inputs are high. In text classification, the given document is assigned a class

C = arg max p(cjd) c (3) (4) (5) Its underlying probability model can be described as an “independent feature model”. The Na¨ıve Bayes (NB) classifier uses the Bayes’ rule Eq. (3), p(cjd) = p(c)p(djc) p(d) Where, p(d) plays no role in selecting C . To estimate the term p(djc), Na¨ıve Bayes decomposes it by assuming the fi’s are conditionally independent given d’s class as in Eq.(4), pNB (cjd) := p(c) m Y p(fijc)ni(d) i=1

! p(d) Where, m is the no of features and fi is the feature vector. Consider a training method consisting of a relative-frequency estimation p(c) and p (fijc) (Pang et al. in [ 17 ]).

In our experiments are used Multinomial Na¨ıve Bayes, presented in [ 18 ]. It implements the Na¨ıve Bayes algorithm for multinomially distributed data, and is one of the two classic Na¨ıve Bayes variants used in text classification (where the data are typically represented as word vector counts, although tf-idf vectors are also known to work well in practice). The distribution is parametrized by vectors y = ( y1; : : : ; yn) for each class y, where n is the number of features (in text classification, the size of the vocabulary) and yi is the probability P (xi j y) of feature i appearing in a sample belonging to class y.

The parameters y is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting: ^yi =

Nyi + Ny + n where Nyi = Px2T xi is the number of times feature i appears in a sample of class y in the training set T , and Ny = PjiT=j1 Nyi is the total count of all features for class y.

The smoothing priors 0 accounts for features not present in the learning samples and prevents zero probabilities in further computations. Setting = 1 is called Laplace smoothing, while < 1 is called Lidstone smoothing [ 18 ].

C. Support Vector Machines

Support vector machines were introduced by Boser et al. in [ 5 ] and basically attempt to find the best possible surface to separate positive and negative training samples. Support Vector Machines (SVMs) are supervised learning methods used for classification.

Given training vectors xi 2 Rn, i = 1; : : : ; l, in two classes, and an indicator vector y 2 Rl such that yi 2 f1,-1g, C SV C (Boser et al. in [ 5 ]; Cortes and Vapnik in [ 9 ]) solves the following primal optimization problem [ 8 ].

l min 1 wT w + C X w;b; 2

i=1 subject to yi(wT (xi) + b) i; i 1 where (xi) maps xi into a higher-dimensional space and C > 0 is the regularization parameter. Due to the possible high dimensionality of the vector variable w, usually we solve the following dual problem. where e = [1; :::; 1]T is the vector of all ones, Q is an l by l positive semidefinite matrix, Qij yiyj K(xi; xj ), and K(xi; xj ) (xi)T (xj ) is the kernel function.

After problem (6) is solved, using the primal-dual relationship, the optimal w satisfies.

w =

l X yi i (xi) i=1 and the decision function is

sgn(wT (x) + b) = sgn (Chang and Lin in [ 8 ])

D. Random Forests

l X yi iK(xi; x) + b i=1 ! (6) (7) (8) (9) (10)

Random Forests were introduced by Leo Breiman in [ 6 ] who was inspired by earlier work by Amit and Geman [ 3 ]. Random Forest is a tree-based ensemble with each tree depending on a collection of random variables. More formally, for a p-dimensional random vector X = (X1; : : : ; Xp)T representing the real-valued input or predictor variables and a random variable Y representing the real-valued response, we assume an unknown joint distribution PXY (X; Y ). The goal is to find a prediction function f(X) for predicting Y. The prediction function is determined by a loss function L(Y, f(X)) and defined to minimize the expected value of the loss

EXY (L(Y; f (X))) where the subscripts denote expectation with respect to the joint distribution of X and Y [ 10 ].

Intuitively, L(Y, f(X)) is a measure of how close f(X) is to Y; it penalizes values of f(X) that are a long way from Y. Typical choices of L are squared error loss L(Y; f (X)) = (Y f (X))2 for regression and zero-one loss for classification: L(Y; f (X)) = I(Y 6= f (X)) = (0 if Y = f (X) 1 otherwise: It turns out that minimizing EXY (L(Y; f (X))) for squared error loss gives the conditional expectation f (x) = E(Y jX = x) (11) (12) (13) otherwise known as the regression function. In the classification situation, if the set of possible values of Y is denoted by Y, minimizing EXY (L(Y; f (X))) for zero-one loss gives f (x) = arg max P (Y = yjX = x)

y2Y otherwise known as the Bayes rule [ 10 ]. Ensembles construct f in terms of a collection of so-called “base learners” h1(x); : : : ; hJ (x) and these base learners are combined to give the “ensemble predictor” f(x). In regression, the base learners are averaged

J f (x) = 1 X hj (x)

J j=1 while in classification, f(x) is the most frequently predicted class (“voting”)

J f (x) = arg max X I(y = hj (x))

y2Y j=1 In Random Forests the jth base learner is a tree denoted hj (X; j ), where j is a collection of random variables and the j ’s are independent for j = 1; : : : ; J (Cutler et al. in [ 10 ]).

III. THE METHOD FOR COMBINING RESULTS

The method for combining results is presented in this section. Proposed method is based on our introduced method (Algorithm for sentences) in paper [ 15 ]. We modified this algorithm for using it with different machine learning algorithms. This algorithm is presented below.

Algorithm for combining results

Input: Let us denote ML1 as the strongest classifier and ML2 as the weakest classifier.

RML1 = fML1 sent; pg – set of the first algorithm results, obtained after performing machine learning algorithm ML1 classification; ML1 sent – sentiment; p – the probability of classification;

RML2 = fML2 sent; vg – set of the second machine learning ML2 classification results obtained after performing ML2; ML2 sent – sentiment;

v – ML2 results value, contains “positive” or “negative” sentiment;

th2 = 0:8. The threshold value was selected by manually investigating the results;

th3 = min(RML1fpg) + ( RML1fpg n 2) 0:01 (used our proposed formula), where RML1fpg is the standard deviation of RML1fpg.

Algorithm for results combining:

1) Find results which are the same in both ML1 and ML2.

Results = RML1 \ RML2 = fx : x 2

RML1fML1 sentg and x 2 RML2fML2 sentgg

2) Find results which are different between ML1 and ML2.

RML1fML1 sentg RML2fML2 sentg and

RML1fpg < th2 (

Results [ RML1; if jRML1fpgj < th3

3) Results =

Results [ RML2; if jRML1fpgj th3

Output: set of classification results Results = fSentence; Sentimentg and Accuracy (Korovkinas et al. in [ 15 ]).

IV. EXPERIMENTS AND RESULTS

A. Dataset

In this paper are used two existing datasets: The Stanford Twitter sentiment corpus (sentiment1401 ) dataset and Amazon customer reviews dataset2 . The Stanford Twitter sentiment corpus dataset is introduced by Go et al. in [ 12 ] and contains 1.6 million tweets automatically labeled as positive or negative based on emotions. The dataset is splitted into training dataset 70% (1.12M tweets) and testing dataset 30% (480K tweets). Amazon customer reviews dataset contains 4 million reviews and star ratings. The dataset is splitted into training dataset 70% (2.8M reviews) and testing dataset 30% (1.2M reviews).

Training and testing data has been preprocessed and has been cleaned before it was passed as the input of intelligent algorithm. It included removing redundant tokens such as hashtag symbols @, numbers, http for links, punctuation symbols, etc. After cleaning was performed all datasets were checked and empty strings were removed.

B. Experiments

In this paper are performed four experiments: two experiments with The Stanford Twitter sentiment corpus (sentiment140) dataset and two experiments with Amazon customer reviews dataset.

In the first and second experiments are used above described datasets, using split into 70% for training and 30% for testing, and apply them to four machine learning algorithms: Logistic Regression, Na¨ıve Bayes classification, Support Vector Machines and Random Forest.

In the third and fourth experiments the best three machine learning algorithms are selected, depending on results of the previous experiments, for the creating various combinations of two different single methods and apply them on above described datasets.

C. Experimental settings

Data cleaning and preparing are performed with R [ 24 ]. The experiments are implemented with Python programming language and scikit-learn [ 18 ]: library for machine learning.

Machine learning algorithms are used with their default parameters. They are described below.

Logistic Regression default parameters [18]:

C (Inverse of regularization strength): float, default: 1.0. dual (Dual or primal formulation): bool, default: False fit intercept (Specifies if a constant should be added to the decision function): bool, default: True intercept scaling: float, default 1 1http://help.sentiment140.com/ 2https://www.kaggle.com/bittlingmayer/amazonreviews/ max iter(Maximum number of iterations taken for the solvers to converge): int, default: 100 multi class: str, default: ‘ovr’. With ‘ovr’ a binary problem is fit for each label. n jobs (Number of CPU cores used when parallelizing over classes if multi class=‘ovr’): int, default: 1 penalty (Used to specify the norm used in the penalization): str, ‘l1’ or ‘l2’, default: ‘l2’ solver (Algorithm to use in the optimization problem): ‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’, default: ‘liblinear’.

tol (Tolerance for stopping criteria): float, default: 0.0001 Na¨ıve Bayes default parameters [ 18 ]: alpha (Additive (Laplace/Lidstone) smoothing parameter (0 for no smoothing)): float, optional (default=1.0) fit prior (Whether to learn class prior probabilities or not): boolean, optional (default=True) class prior (Prior probabilities of the classes): array-like, size (n classes), optional (default=None)

Support Vector Machines default parameters [18]:

C (Penalty parameter C of the error term): float, optional (default=1.0) kernel (Specifies the kernel type to be used in the algorithm): string, optional (default=‘rbf’). We used ‘linear’ kernel instead. loss (Specifies the loss function): string, ‘hinge’ or ‘squared hinge’ (default=‘squared hinge’). ‘squared hinge’ is the square of the hinge loss. max iter (The maximum number of iterations to be run): int, (default=1000) multi class (Determines the multi-class strategy if y contains more than two classes): string, ‘ovr’ or ‘crammer singer’ (default=‘ovr’). ‘ovr’ trains n classes onevs-rest classifiers. penalty (Specifies the norm used in the penalization): string, ‘l1’ or ‘l2’ (default=‘l2’) tol (Tolerance for stopping criteria): float, optional (default=0,0001) Random Forest default parameters [ 18 ]: n estimators (The number of trees in the forest): integer, optional (default=10) max features (The number of features to consider when looking for the best split): int, float, string or None, optional (default=“auto”) max depth (The maximum depth of the tree): integer or None, optional (default=None) min samples split (The minimum number of samples required to split an internal node): int, float, optional (default=2) min samples leaf (The minimum number of samples required to be at a leaf node): int, float, optional (default=1) min weight fraction leaf (The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node): float, optional (default=0.0) max leaf nodes (Grow trees with max leaf nodes in best-first fashion. Best nodes are defined as relative reduction in impurity): int or None, optional (default=None) min impurity decrease (A node will be split if this split induces a decrease of the impurity greater than or equal to this value): float, optional (default=0.0) bootstrap (Whether bootstrap samples are used when building trees): boolean, optional (default=True) oob score (Whether to use out-of-bag samples to estimate the generalization accuracy): bool (default=False) n jobs (The number of jobs to run in parallel for both fit and predict): integer, optional (default=1) verbose (Controls the verbosity of the tree building process): int, optional (default=0) warm start : bool, optional (default=False) criterion (The function to measure the quality of a split): string, optional (default=“gini”)

D. Effectiveness

Effectiveness is measured using statistical measures: accuracy (ACC), precision (PPV – positive predictive value and NPV – negative predictive value), recall (TPR – true positive rate and TNR – true negative rate) and F1 (Harmonic mean of PPV and TPR). Formulas are presented below (Sammut and Webb in [ 20 ]): Accuracy (ACC):

ACC =

T P + T N

T P + T N + F P + F N Positive predictive value (PPV): Negative predictive value (NPV): True positive rate (TPR): True negative rate (TNR): Harmonic mean of PPV and TPR (F1):

P P V = N P V = T P R = T N R =

T P T P + F P

T N T N + F N

T P T P + F N

T N

T N + F P F1 =

2

TABLE I contains the results of standard single machine learning algorithms with their default parameters. Results show that Logistic Regression (LR) obtained the best accuracy (ACC) in both experiments 79,67% and 90,21%. Other methods are arranged in the following order: SVM (ACC) – 79,16% and 90,00%, Na¨ıve Bayes classification (ACC) – 76,72% and 84,18%, Random Forest (ACC) – 75,81% and 80,15%.

The better accuracy obtained when was used Amazon reviews dataset, while it significantly bigger than sentiment140 dataset. This happened because tweets are very short, contain noises, slangs, acronyms and etc.

Logistic Regression and SVM provided more uniform recognition of both classes; PPV, NPV, TPR, TNR, F1, are almost even, compared to other methods.

Depending on results presented in TABLE I, for the further experiments were selected Logistic Regression, SVM and Na¨ıve Bayes. Various combinations of two different single algorithms were performed in these experiments.

Table II shows that using proposed method (see Section III) for combination of two single methods let us to obtain the better accuracy to compare with a single method. where TP – count of correctly classified “positive” sentiments, TN – count of correctly classified “negative” sentiments. FP – count of incorrectly classified “positive” sentiments. FN – count of incorrectly classified “negative” sentiments.

LR-SVM (Logistic Regression and SVM combination) shows the better accuracy (ACC) 81,83% and 90,22%, while (ACC) of other combinations are smaller: LR-NB (Logistic Regression and Na¨ıve Bayes combination) – 79,81% and 90,22%, SVM-NB (SVM and Na¨ıve Bayes combination) – 79,26% and 89,98%. Our introduced method also outperformed single LR algorithm in all experiments, except the fourth experiment where SVM-NB obtained accuracy (ACC) 89,98% to compare with Logistic Regression 90,21%.

Our method also provided more uniform recognition of both classes PPV, NPV, TPR, TNR, F1.

V. CONCLUSIONS AND FUTURE WORK

The main idea of this paper was to select two single intelligent algorithms to create a combined method for sentiment classification.

Results show that combination of two almost equal intelligent methods, which shown the best results (in our case Logistic Regression and SVM) can obtain the bigger accuracy (ACC) 81,83% and 90,22% to compare with the best results obtained single method like Logistic Regression 79,67% and 90,21%.

Combination between the strongest and the weakest method (in our case Na¨ıve Bayes classification with accuracy (ACC) 79,81% and 90,22%) also outperform the best results obtained single method Logistic Regression.

The main advantage of methods combination is that combined method provided more uniform recognition of both classes PPV, NPV, TPR, TNR, F1 to compare with Na¨ıve Bayes and Random Forest.

Such results let to conclude that Logistic Regression and SVM, and combination of these methods fit the best for our further work. Our method presented in [ 15 ] can be applied with different algorithms and obtain the better classification accuracy. The goal of this approach was to test proposed method with existing datasets to be able in the future continue work with real-world data.

References

[1] M. Ahmad , S. Aftab , S. S. Muhammad , and S. Ahmad . Machine learning techniques for sentiment analysis: A review . Int. J. Multidiscip. Sci. Eng , 8 ( 3 ): 27 - 32 , 2017 .

[2] E. Ahmed , M. A. U. Sazzad , M. T. Islam , M. Azad , S. Islam , and M. H. Ali . Challenges, comparative analysis and a proposed methodology to predict sentiment from movie reviews using machine learning . In Big Data Analytics and Computational Intelligence (ICBDAC) , 2017 International Conference on, pages 86 - 91 . IEEE, 2017 .

[3] Y. Amit and D. Geman . Shape quantization and recognition with randomized trees . Neural computation , 9 ( 7 ): 1545 - 1588 , 1997 .

[4] M. Ashok , S. Rajanna , P. V. Joshi , and S. Kamath . A personalized recommender system using machine learning based sentiment analysis over social data . In Electrical, Electronics and Computer Science (SCEECS) , 2016 IEEE Students' Conference on , pages 1 - 6 . IEEE, 2016 .

[5] B. E. Boser , I. M. Guyon , and V. N. Vapnik . A training algorithm for optimal margin classifiers . In Proceedings of the fifth annual workshop on Computational learning theory , pages 144 - 152 . ACM, 1992 .

[6] L. Breiman . Random forests . Machine learning , 45 ( 1 ): 5 - 32 , 2001 .

[7] E. Brito , R. Sifa , K. Cvejoski , C. Ojeda , and C. Bauckhage . Towards german word embeddings: A use case with predictive sentiment analysis . In Data Science-Analytics and Applications , pages 59 - 62 . Springer, 2017 .

[8] C.-C. Chang and C.-J. Lin . Libsvm: a library for support vector machines . ACM transactions on intelligent systems and technology (TIST) , 2 ( 3 ): 27 , 2011 .

[9] C. Cortes and V. Vapnik . Support-vector networks . Machine learning , 20 ( 3 ): 273 - 297 , 1995 .

[10] A. Cutler , D. R. Cutler , and J. R. Stevens . Random forests . In Ensemble machine learning , pages 157 - 175 . Springer, 2012 .

[11] D. Davidov , O. Tsur , and A. Rappoport . Enhanced sentiment learning using twitter hashtags and smileys . In Proceedings of the 23rd international conference on computational linguistics: posters , pages 241 - 249 . Association for Computational Linguistics, 2010 .

[12] A. Go , R. Bhayani , and L. Huang . Twitter sentiment classification using distant supervision . CS224N Project Report , Stanford, 1 ( 12 ), 2009 .

[13] J. Jayalekshmi and T. Mathew . Facial expression recognition and emotion classification system for sentiment analysis . In Networks & Advances in Computational Technologies (NetACT) , 2017 International Conference on, pages 1 - 8 . IEEE, 2017 .

[14] V. Kharde , P. Sonawane , et al. Sentiment analysis of twitter data: a survey of techniques . arXiv preprint arXiv:1601.06971 , 2016 .

[15] K. Korovkinas , P. Dane˙nas, and G. Garsˇva. Svm and na¨ıve bayes classification ensemble method for sentiment analysis . Baltic Journal of Modern Computing , 5 ( 4 ): 398 - 409 , 2017 .

[16] T. Maheshwari , A. N. Reganti , S. Gupta , A. Jamatia , U. Kumar , B. Gamba¨ck, and A. Das . A societal sentiment analysis: Predicting the values and ethics of individuals by analysing social media content . In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1 , Long Papers , volume 1 , pages 731 - 741 , 2017 .

[17] B. Pang , L. Lee , and S. Vaithyanathan . Thumbs up?: sentiment classification using machine learning techniques . In Proceedings of the ACL02 conference on Empirical methods in natural language processingVolume 10 , pages 79 - 86 . Association for Computational Linguistics, 2002 .

[18] F. Pedregosa , G. Varoquaux , A. Gramfort , V. Michel , B. Thirion , O. Grisel , M. Blondel , P. Prettenhofer , R. Weiss , V. Dubourg , et al. Scikit-learn: Machine learning in python . Journal of machine learning research , 12 (Oct): 2825 - 2830 , 2011 .

[19] T. Pranckevicˇius and V. Marcinkevicˇius. Comparison of naive bayes, random forest, decision tree, support vector machines, and logistic regression classifiers for text reviews classification . Baltic Journal of Modern Computing , 5 ( 2 ): 221 , 2017 .

[20] C. Sammut and G. I. Webb . Encyclopedia of machine learning. Springer Science & Business Media , 2011 .

[21] P. Singh , R. S. Sawhney , and K. S. Kahlon . Forecasting the 2016 us presidential elections using sentiment analysis . In Conference on eBusiness, e-Services and e-Society , pages 412 - 423 . Springer, 2017 .

[22] J. T. Starczewski , S. Pabiasz , N. Vladymyrska , A. Marvuglia , C. Napoli , and M. Woz´niak. Self organizing maps for 3d face understanding . In International Conference on Artificial Intelligence and Soft Computing , pages 210 - 217 . Springer, 2016 .

[23] P. M. Tayade , S. S. Shaikh , and S. Deshmukh . To discover trolling patterns in social media: Troll filter . 2017 .

[24] R. C. Team et al. R: A language and environment for statistical computing . 2015 .

[25] F. Tian , F. Wu , K.-M. Chao , Q. Zheng , N. Shah , T. Lan , and J. Yue . A topic sentence-based instance transfer method for imbalanced sentiment classification of chinese product reviews . Electronic Commerce Research and Applications , 16 : 66 - 76 , 2016 .

[26] H. Trevor , T. Robert , and F. JH. The elements of statistical learning: data mining, inference, and prediction , 2009 .

[27] A. Venckauskas , A. Karpavicius , R. Damasˇevicˇius , R. Marcinkevicˇius, J. Kapocˇiu¯te-Dzikiene´, and C. Napoli . Open class authorship attribution of lithuanian internet comments using one-class classifier . In Federated Conference on Computer Science and Information Systems (FedCSIS) , pages 373 - 382 . IEEE, 2017 .