The use of sarcasm dates as far back as communication goes. Sarcasm is expressed in words, facial expression and can be noticed in the intonation of voice. In this digital age, sarcastic comments are passed everyday, via tweets, comment sections of different social media outlets, and even in news headlines. Some of these headlines can be misunderstood and taken to mean something different than the original intentions. This leads to a need to detect sarcasm especially in the news and on social media. Detection of sarcasm from text has its challenges because text lacks intonation and deflection of voice that occurs when a sarcastic statement is made vocally by a human. This paper concentrates on the effect of different feature encoding techniques applied to text for feature extraction for machine learning models. A deep learning model is also applied and the results are compared. Prior to feature extraction, data pre-processing techniques like tokenization, removal of stop-words and punctuation are applied by researchers in any work involving text analysis. These pre-processing techniques are widely used and accepted and are also applied in this project. Different feature extraction methods like Count Vectorizer, Term Frequency-Inverse Document Frequency and word embedding were implemented in this experiment. The Support Vector Machine, Naive Bayes and Logistic Regression were the traditional machine learning algorithms used in this research. Convolutional Neural Network was also used. Results of these algorithms are recorded in this paper and these include the F1-score, precision, recall and the accuracy scores. The results of these different algorithms for the methods of feature extraction examined in this paper show that a combination of more than one technique is better suited for the purpose of classifications. The results of this paper show
In the last decade, there has been an increase in the use of online resources for dissemination of information. These texts have different characteristics that are explored by the use of Natural Language Processing techniques of machine learning for knowledge and insight. One of such important characteristics is the presence of sarcasm in text. Sarcasm is a complex act of communication that 2
Chudi-Iwueze and Afli
allows speakers the opportunity to express sentiment-rich opinions in an implicit
way [
Although sarcasm is well known and widely used, it is challenging not only for
computers but also for humans to detect promptly. Some humans find it difficult
to identify and understand the use of sarcasm [
The rest of this paper is arranged as follows: Section 2 goes over some related work both in sarcasm detection and feature extraction methods. Section 3 details the methodology and which models were applied for the analysis. Section 4 outlines the results of the application of the different feature extraction techniques to the chosen models and describes which performed best. 2 2.1
Text extraction describes the process of taking words from text data and
transforming them into a numerical feature set that is useful for a machine
learning classifier [
The detection of sarcasm is very crucial in the domain of sentiment analysis and
opinion mining. Different machine learning algorithms have been applied to the
problem. Some researchers used the Naive Bayes Classifier and Support Vector
Machines for analyses of social media data in Indonesia [
The data used for this analysis was taken from Kaggle. The news headlines were curated into two different files for competitions on Kaggle. Each file was a JSON file, containing news headlines and the label named ’is sarcastic’ that indicates the presence or absence of sarcasm in that headline. The two files were joined together to create a bigger data set for the analysis. The complete data used in the analysis contains over 50000 headlines. The data set is not significantly imbalanced but there were a few more non-sarcastic news headlines than the sarcastic headlines. Table 1 below shows a few rows in the data and the labels for those data points.
Headline Is sarcastic
1 former versace store clerk sues over secret ’black code’ for minority shoppers 0
2 why writers must plan to be surprised 0
3 boehner just wants wife to listen, not come up with alternative debt-reduction ideas 1
4 remembrance is the beginning of the task 0
5 4 lessons prison taught me about power and control 0
6 top snake handler leaves sinking huckabee campaign 1
7 courtroom sketch artist has clear manga influences 1
8 stock analysts confused, frightened by boar market 1
9 gillian jacobs on what it’s like to kiss adam brody 0
10 diy: sports equipment closet 0
The first step in the pre-processing pipeline was the conversion of all the text to
lower case. This is done to achieve uniformity of words. Unwanted symbols like
digits and newlines were removed also. All punctuation were also removed from
the data set. After these, the texts were tokenized. Tokenization is the breaking
down of sentences into words [
Count Vectorizer The Count Vectorizer is a simple means used to tokenize and build vocabulary of known words. It is also used to encode new documents using the vocabulary that has been created. The product of the count vectorizer is an encoded vector which has the length of the entire vocabulary of the document and a count for the number of times each word appeared in the document. Count Vectorizer was implemented using scikit-learn for this project.
a popular and well recognized method for the evaluation of the importance of 6
Chudi-Iwueze and Afli a word in a document. Term Frequency measures the frequency of a term in a document. For a variety of documents with different lengths, the probability of a word occurring more times in a longer document is present. Due to this fact, normalization is required.
N
T F (t) = T where N = Number of times term t appears in a document and TN = Total number of terms in the document
The Inverse Document Frequency is the measure of importance of a term. This is done to reduce the influence of words e.g ‘of’, ‘the’ that could be used multiple times in a document but have no real meaning and importance in context.
T D
IDF (t) = loge N D where TD = Total number of documents and ND = Number of documents with term t in it
For most traditional applications the bag-of-words model was used. However,
words that occurred frequently in the document were given significant weights
in the Bag-OF-Words model and this affected the accuracy of these results. The
use of TF-IDF was introduced to solve these problems of the typical
Bag-OfWords Model. The IDF of an infrequent term due to the log applied is high and
the IDF for common words is low [
A couple of classification algorithms were chosen for this experiment. These
algorithms are briefly described below:
Support Vector Machines The Support Vector Machine is efficient for both
classification and regression. It is known to give good results for classification.
The classes are separated by a hyper plane found by the algorithm. The
LinearSVC from scikit-learn was implemented for this analysis.
Naive Bayes The Naive Bayes is a powerful algorithm used for classifying
data based on probabilities [
Logistic Regression Logistic Regression is a popular classification algorithm. It belongs to the class of Generalized Linear Models. Its loss function is the sigmoid function which minimizes the results to be a value between 0 and 1. The Logistic Regression Model was implemented for this analysis using the function available in the scikit-learn package.
Convolutional Neural Networks (CNN) CNNs are very commonly used in text classification problems due to their success and great results. This has contributed to the decision to use CNN for this aspect of the experiment. In the first experiment with CNN, one CNN model is used on the features extracted using the Count Vectorizer. Another CNN model is also trained on the features extracted using Word2Vec. The results of these two models are plotted and compared. 4 4.1
8 4.2
For the supervised learning techniques, the results to be examined are the results of the SVM, Naive Bayes and Logistic Regression when text features are extracted using the Count Vectorizer technique and different levels of the TF-IDF technique. These results are detailed in Table 3 above and can be seen graphically in 3 below.
The bar plot that follows in Figure 3 is pictorial representation of the results of the analysis performed using the supervised learning methodologies. It is clear that the SVM outperforms both the Logistic Regression and the Naive Bayes in all the cases, except the N-Gram level TF-IDF where they all perform similarly poor. The figures that follow concentrate mainly on the results of the SVM algorithm due to this fact.
The results of the work done in our research shows that the use of Count
Vectorizer and TF-IDF gives an accuracy of 93% when this feature set is fed
to SVM. This is an improvement on the work of [
The results as seen above corroborate the known fact that Support Vector Machine usually outperforms other algorithms for classification purposes, depending on the measure of data available. This is clear in the measure of the accuracy as seen in the bar plots in Figure 3. The accuracy of the SVM with the Count Vectorizer is about 90%. The results for the n-gram level TF-IDF are the worst of all. This is similar to the Bag-Of-Words model and as such the bad results are expected.
Figure 4 below shows the training and validation accuracy scores for the model trained over 5 epochs. Unsurprisingly, the accuracy of training for the CNN is very close to 100%. The validation accuracy however is about 92.5%.
The loss also reduces for both the training and validation data. 10
Chudi-Iwueze and Afli The training and validation accuracy score is higher when word embedding is applied to the CNN model. However, the Figure in 5 shows an interesting pattern for the validation loss when word2vec is applied with the CNN. The loss reduces and then begins to increase again for the data used in validation. This suggests that the model does not actually perform as expected when word embedding is applied. This is due to the fact that word2vec is a neural network based solution and these work optimally when the volume of data is substantial.
Sarcasm can be found in a wide variety of areas. To be able to accurately detect sarcasm from different aspects and topics, a large enough data set will be needed. The accuracy of prediction as seen in this paper greatly depends on the mode of feature extraction. The use of N-grams and characters produced less desirable results than the use of words. The combination of the Count Vectorizer and the Term Frequency-Inverse Document Frequency for the supervised learning techniques gave the most satisfactory performance metrics.
This results of this research favor the supervised machine methods over the deep learning methods. Although this could be due to the limited amount of data available. This goes to show that the use of simple methods can produce great results, especially in the situation that the data available is limited and not suitable for more complicated deep learning techniques. The CNN performed satisfactorily also. The use of word embedding for this task showed great performance in terms of accuracy, both in the training and validation sets so does the use of the features extracted using the count vectorizer. The overall performance of both the CNN and supervised learning methods suggests that the use of Count Vectorizer is most appropriate for this task. As stated above, the volume of data available limits the strengths of the Word2Vec method, this the Count Vectorizer outperforms in this situation. This is due to the fact that simpler methods are always a better choice than more complicated methods provided that the results are similar or better with the simpler methods.
Sarcasm can also be multi-modal, contained in images and GIFs. This is especially available on social media like Twitter where tweets can be replied with GIFs and videos. Future research would involve an extension of this system to include multi-modal methods for sarcasm detection. Social media data for sarcasm detection will also contain emoticons (emojis) and analysis of those emojis could add more meaning to the text. A bigger data set will allow for the use of feature extraction techniques that require large data sets. This paper has not explored the use of word embedding for both supervised and deep learning techniques. More exploration into these methods of feature extraction available is warranted and would be a focus of future research in this area.