In this paper, we present our team participation at Author Profiling and Deception Detection in Arabic (APDA) - Task 2: Deception Detection in Arabic Texts. Our analysis shows that the accuracy of a unigram method outperforms both bigrams and hybrid modeling based on unigram and bigrams. We show that the accuracy of the unigram modeling can achieve performance above 76%.
Social networks such as Facebook, Twitter or Instagram are excellent
communication channels between customers and diferent brands. According to statistics,
approximately 92% of business-to-business marketers in North America use
social networks as part of their marketing tactics [
Being aware of the importance of social networks in the commercialization
of the companies, it is key to detect when a comment in social networks is real
or when it is suspected of deception and tries to damage the reputation of the
brand. This problem is constant throughout the world, regardless of the
country. However, for instance, the Arabic language sufers from a lack of Natural
Language Processing (NLP) techniques that does not happen in the English
language. In particular, one of the first contributions can be found in [
In this paper, we present our participation in Author Profiling and
Deception Detection in Arabic (APDA) - Task 2 carried out together with the FIRE
2019 Forum for Information Retrieval Evaluation 12-15 December 2019, Kolkata,
India [
Given a set of Arabic messages on Twitter or new headlines, the goal of the task is to predict when a given message is deceptive when it was written trying to sound authentic to mislead the reader. Therefore, the target labels are Truth or Lie. The organizers provided 1 1443 new headlines (APDA@FIRE.Qatar-Newscorpus.training.csv) and 532 tweets (APDA@FIRE.Qatar-Twitter-corpus.traini ng.csv) to train our model. Both files have identically the same fields, that is, an ID (identifier of each text), a label (to describe if a text is true or false), and a text (words to analyze). It is important to say that some tweets or news were repeated throughout the file and we decided to delete them so as not to overtrain the model with those messages. Similarly, organizers also provided 370 tweets (APDA@FIRE.Qatar-Twitter-corpus.test.csv) and 241 new headlines (APDA@FIRE.Qatar-News-corpus.test.csv). In this case, the files contain only the ID and the text but not the label, which has to be predicted with a statistical model. 3
In this section, we analyze the n-grams methods that can be found in the literature in their standalone version (unigrams or bigrams) or hybrid (unigrams and bigrams) for text processing. 3.1
n-Grams An n-gram model consists of a probabilistic method that attempts to predict the xi item based on the items xi−(n−1), ..., xi−1, that is:
Pxi = P (xi|xi−(n−1), ..., xi−1). (1) 1 https://www.autoritas.net/APDA/corpus/ In our case, an item is a word but it could be a syllable, a set of letters, phonemes, etc. For instance, for n = 1, the probability of the word example is written along with the word for in English is very high. However, the probability of the word can is written together with the word would is zero, Pcan = P (can|would) = 0.
The probabilistic method proposed in this paper is based on the next steps: 1. We divide L sentences of Twitter or headlines with the following delimiters: spaces ( ), dots (.), question marks (?), exclamation marks (!), parentheses (()), semicolons (;) or commas (,). 2. We eliminate a set of words (stop words) that are common in the Arabic language. 3. We remove punctuation marks, numbers, hashtags, and extra whitespaces. 4. We get all combinations of n words from all sentences. For example, in English, the sentence In summer the temperatures are high in middle east is divided in the following combinations: in summer, summer the, the temperatures, temperatures are, are high, high in, in middle, middle east. 5. We calculate the Pxi probability of all combinations. 6. Given the large number of combinations, we select the T combinations with the highest probability. 7. We build the final dataset with L rows and T columns. 8. Finally, with machine learning, we classify tweets or headlines.
Note that this approach has been carried out for the training and test data. 4
To carry out the Task 2, we use a unigram modeling to predict whether a tweet or headline is misleading or not, and we compare its performance with the following methods: 1. Bigrams. 2. A hybrid method that uses unigram and bigrams.
We use 751 common words to remove them from the texts to analyze 2. Besides, instead of using the probability as a metric of deception, we use the number of times each combination appears. Finally, we have tested several machine learning algorithms with the qdap, tm, XML, splitstackshape, caret, and RWeka libraries of the R programming language. In particular, we have tested k-Nearest Neighbour, Neural Networks, Random Forest, and Super Vector Machine (SVM). We verified that SVM is the learning algorithm that ofers the best performance with a 10‐fold cross‐validation experiment and three repetitions.
Figure 1 shows the performance evolution of the proposed methods based
on n-grams to detect the deception in Arabic. This performance evolution is
calculated for Twitter (right figure) and new headlines (left figure). We show
the accuracy with the number of combinations T explained in Section 3.1. From
2 https://github.com/mohataher/arabic-stop-words/blob/master/list.txt
the figures, we can conclude that there is a significant performance gap between
Twitter and the headline texts. In general, as the number of combinations
increases, the accuracy increases for both texts. However, 500 combinations could
be good enough to get a good performance at a reasonable cost. We also
highlight an erratic performance behavior in the figures with few combinations (see
the hybrid method unigram and bigrams). Therefore, it is key to select correctly
which combinations can be removed. From both figures, we can conclude that
the unigram method is the best algorithm to detect deception in Arabic texts.
Table 1 shows the best accuracy values that can be achieved with the proposed
methods. Note that the Arabic language is a relatively complex language from
the NLP point of view. For instance, some prepositions are joined with some
words: ملاعلا سٔاك (the world cup) and ملاعلا سٔاكل (for the world cup). This fact causes
a performance loss in the n-grams that should be considered. This problem
can be solved by using handmade rules that separate prefixes and suffixes from
words [
(a) Twitter (b) New headlines
76.19 71.45
72.12 60.05
In this paper, we presented our participation in Author Profiling and Deception Detection in Arabic (APDA). We worked on a unigram approach and compared its performance with bigrams and a hybrid method that used unigrams and bigrams. Our proposed approach achieved good performance compared to the other two methods. In particular, we obtained an accuracy of 76.19% for Twitter texts and 71.45% for new headlines. As future work, we could include letters as items instead of words to deal with the complexity of the Arabic morphology where some prepositions are joined with words.