This article describes the system submitted by the RGCL team to the IDAT 2019 Shared Task: Irony Detection in Arabic Tweets. The system detects irony in Arabic tweets using deep learning. The paper evaluates the performance of several deep learning models, as well as how text cleaning and text pre-processing in uence the accuracy of the system. Several runs were submitted. The highest F1 score achieved for one of the submissions was 0.818 making the team RGCL rank 4th out of 10 teams in nal results. Overall, we present a system that uses minimal pre-processing but capable of achieving competitive results.
According to philosophers such as Grice, irony could be de ned as an utterance
which violates a conversational maxim [
On one level unexpectedness is achieved by what can be dubbed as \sentiment imbalance". It refers to the use of a negative or positive word where the opposite polarity is expected from the context. The ironic tweet, \... YÓA éJºË ¬Q K øQåÖÏ@ XAJ¯B@" (The Egyptian economy is crumbling yet it is unfaltering) is a good example of this. The use of the word ` YÓA' (`unfaltering or rm') with its positive polarity where a negative item is expected creates an ironic e ect.
Moreover, the out-of-context ironic element can be pointed out by a typographical element such as quotes, bold typing, ellipsis and emoticons. Other observed features of the ironic data are the frequent use of parallel syntactic structures, repetition of one or more lexical items and the use of polysemy or multi-sense words.
Recently, a lot of research has been carried out in the eld of natural
language processing in order to detect irony. This is evidenced by an increase in
irony detection shared tasks. SemEval-2018 Task 3 focused on irony detection in
English tweets [
Another such task, which is at the FIRE 2019 conference, is Irony Detection in Arabic Tweets (IDAT). Given a tweet, systems have to classify it as either ironic or not ironic. This paper describes our submission to the IDAT shared task in irony detection. We propose a simple, low-e ort approach, with minimal data processing. We employ six di erent neural network architectures in order to detect irony in tweets, evaluate each network and select the three best performing architectures for our nal submissions.
The paper is structured as follows. Section 2 describes the system developed for this shared task and the dataset used to train and test it. Section 3 presents an analysis of the results of our evaluation of the ve di erent architectures (Section 3.1), as well as of the nal submission (Section 3.2). Section 4 describes the error analysis we performed for the test set. The paper is concluded by Section 5 o ering some nal remarks. 2
This section describes the shared task data, as well as the system that was used to classify the data. We use minimal preprocessing in order to use the data. For classi cation, we used and compared six di erent neural network architectures suited to this task. Our implementation has been made available on Github.1 2.1
The dataset contains tweets related to di erent political issues and events related
to the Middle East that hold during the years 2011 to 2018 [
The main objective of the classi cation task is to capture the irony-de ning features in the Arabic tweets, and hence the data has been cleaned accordingly. The team has chosen to delete a number of textual features that do not contribute to the classi cation and maintain others which may be signi cant in spotting an ironic tone in the tweet.
Twitter users writing in Arabic can either write in Modern Standard Arabic (MSA) or in their particular dialects. If the tweet is written in MSA, it may or may not include diacritics whereas dialectical Arabic does not include any. Ironic tweets can be formulated in any of these di erent versions of the Arabic transcript. Thus, in order to avoid false classi cations due to a non-de ning feature, regular expressions were used to delete diacritics.
Moreover, native Arabic tweeters can either include punctuation or ignore it completely. Again, regular expressions were used to delete a number of punctuation markers, and multiple spaces were reduced to a single space. Some punctuation markers were not deleted if it was assumed to be signi cant for capturing irony. For example, it has been observed that the ellipsis is often used in ironic tweets to indicate a pause, a hesitation or a trailing-o in thought that marks a shift to an ironic tone.
Similarly, emojis were not omitted as they were deemed important for the classi cation. Finally, English characters which are either used to refer to the user's account or to a word in Arabizi (the Arabic Chat Alphabet) were also deleted. Arabizi characters were removed as they are not written in an Arabic script, but rather a mixture of English characters and symbols which would not be captured by the Arabic word-embedding model employed for classi cation. 2.3
After data processing each text is encoded using Arabic fasttext [
The parameters of each architecture were optimised using 10-fold
crossvalidation considering a binary cross entropy loss function and using adam
optimiser [
embedding vectors is fed into a bi-directional LSTM-layer [
This section presents the results of the evaluation of the six architectures, as well as the evaluation of the nal submissions. The competitors were allowed to submit three runs of their system to the evaluations. Therefore, we compare the performance of six di erent neural network architectures in order to select three submissions for the task. 3.1
This section describes how we selected the architectures for the nal submission. To evaluate the architectures, we used 20% of the available training data, and used the rest of the data for training. Table 1 shows the evaluation results of each architecture. We used three evaluation metrics: Precision (P), Recall (R) and F1 score as denoted in Table 1. As shown, 2D Convolution with Pooling, GRU with Capsule and Pooled GRU architectures had the best F1 scores of the six experimented architectures. Therefore, we submitted outputs from those three architectures as our nal submissions.
Architecture
P
R
F1 This section presents the results of the evaluation of our submission. The evaluation was carried out by the task organisers, and at the time of writing the paper the GOLD standard for the test set is not available. Therefore, we report only the evaluation provided to us by the task organisers which is solely based on F1 score. Our selected architectures 2D Convolution with Pooling, Pooled GRU and GRU with Capsule had F1 scores of 0.818, 0.816 and 0.804, respectively. Our best result 2D Convolution with Pooling ranked seventh in the nal results. 4
The deep learning architecture used in our models is fundamentally based on word-embeddings and their sequential relations. Accordingly, the rst step for error analysis was to explore the most informative features on the word-level which were crucial in classi cation. In order to achieve this, the top 30 vectors for ironic and non-ironic tweets were extracted (see Figure (1) for the most ironic vectors and Figure (2) for the most non-ironic).
A number of observations were made on the above lists. First, the most informative tokens for ironic tweets revolve around Middle East political gures who were usually the target of jokes in social platform discussions during the time frame of the compiled data (e.g `Mursi', former president of Egypt, `Bashar', president of Syria, `Mubarak', ousted president of Egypt, `Okasha', an Egyptian TV anchor famous for propagating hoax news). These are essentially divisive political gures who typically provoke ironic comments on their actions or political stance. Second, elongated words such as ` éêêêêêë' (hahaha) referring to an exaggerated sentiment is irony de ning. Third, the most non-ironic features similarly include words referring to political gures or entities such as `Trump', `Isis' and `Putin'. However, it is noticeable that a less formal reference to the political gure is included in the ironic tweet. For example, in the most informative ironic features, reference to `Vladimir Putin' is informally given by his rst name `Vladimir' whereas his surname `Putin' becomes the most informative feature in non-ironic tweets. Fourth, the most informative features for non-ironic tweets seem to include particular hashtags which may have been retweeted in the corpus. This is concluded from the use of the underscore, ` ' , in more than one non-ironic vector (e.g. `àAJ« ú×A' (Sami Anaan), seems to be a retweeted hashtag concerned with the imprisonment of the Egyptian chief of sta who was imprisoned after announcing his run for the Egyptian presidential elections). These hashtags refer to current news which may be more factual than sarcastic. Finally, 50% of the most ironic tokens belong to di erent Arabic dialects whereas only 10% of the non-ironic features was dialectical.
The second step in the error analysis was the investigation of the misclassied instances in the test set. Given that access to the o cial annotation for the test set was not available, one of the authors of the paper, who is an Egyptian Arabic native speaker analysed the automatic classi cation. The total misclassied tweets was 188 tweets: 133 belong to the false positives, where the tweet is wrongly classi ed as ironic, and 55 false negatives, where the tweet is mistakenly considered non-ironic. Generally speaking, the false positive instances typically include reference to one of the political gures contained in the most informative ironic features shown in Figure (1) above. For example, the following tweet: (Asmaa, wife of the Syrian president Bashar, is pregnant in her fth month) is misclassi ed as ironic because of the presence of the word `PA . ' (Bashar) which is one of the most informative features for the positive class. Thus, tweets reporting news about one of the gures typically mentioned in ironic tweets are misclassi ed. It was also noticed that a number of the false positives were wrongly annotated tweets. For example: (Oh you who see through the future .. oh you who see through pants #bassem yussif style) is an ironic tweet mistakenly annotated as non-ironic. Interestingly, our model correctly classi ed it as ironic. Moreover, a number of tweets that were misclassi ed included slang or dialectical words. For example, is part of a tweet written in dialectical Egyptian Arabic that was misclassi ed as ironic.
As for false negatives, it was observed that the ironic tweet typically included an emoticon indicating a positive sentiment (such as laughter or a smile). The preprocessing stage for our model deleted any non-characters and hence emoticons were not captured by the classi er. In the future, we plan to investigate whether the use of other features such as emoticons can improve the classi cation accuracy. 5
In this paper, we have presented our system for identifying irony in Arabic tweets. The system uses minimal preprocessing and relies on word embeddings. We experimented with di erent neural network architectures in order to determine the most suitable for this task. Going by our evaluation, and the results provided by the task organisers, it is clear that 2D Convolution with Pooling scores highest overall.
While the word embeddings performed well in the task there were a lot of out
of vocabulary words. In the future, we would like to experiment with the
performance of contextualised word embeddings, such as ELMo [