Spam filtering is a non-standard derivative data science problem aiming to catch unsolicited and undesirable messages and prevent those messages from reaching a user's inbox. To solve the abovementioned problem, we propose a text augmentation approach using the most similar synonyms called TAMS. We used Random forest and Bidirectional LSTM classification models for the experimental part to assess the proposed approach. The results indicate that training the classifiers with synthesized spam messages generated by TAMS reduces the influence of the imbalance problem present by nature in the dataset and improves the overall performance of the classification models. Hence, this study shows the potential of using TAMS to enhance the classification performance on textual data where the imbalance scenario is present.
minority class. The most common solution to the imbalanced datasets problem is generating new samples Spam filtering is a non-standard derivative data science belonging to the minority class to make the dataset balproblem [1]. Derivative because it is an extension of anced. core problems, i.e., classification problems. Non-standard, The paper is organized as follows: Section 2 describes since the data has an unusual distribution on the target the imbalance problem, data augmentation, used mavariable, such problems belong to the imbalance problems chine learning models, some related terminologies and family. Spam filtering is also one of the most common related works. Section 3 introduces the proposed text problems in the Natural Language Processing domain. augmentation approach TAMS with a detailed explanaThe main target of this problem is to identify the spam tion and practical examples. The experimental results, messages and filter them out from the legitimate mes- including the dataset, evaluation metrics, and results with sages. Solving this problem will be very beneficial for discussion, are presented in Section 4. Lastly, Section 5 telecommunication companies, where text messaging is outlines the conclusion about the conducted research and the most common non-voice use of a mobile phone. In the potential research direction to improve learning and fact, according to security firm Cloudmark, about 30 mil- classification of similar problems. lion spam messages are sent to cell phone users across North America, Europe, and the U.K.
This study aims to improve the SMS spam filtering 2. Background by solving the most common problem in the available datasets, which is the imbalance problem, where most of the samples belong to the legitimate class, i.e., legitimate messages, the so-called majority class − , and a small proportion of the samples belongs to the spam class, i.e., spam messages, so-called minority class +. Dealing with an imbalanced dataset is one of the main challenges in machine learning, especially in classification problems, where most well-known classification models tend to be biased toward the majority class and fail to identify the Data science techniques are used to solve many problems. These methods can learn and likely extract hidden patterns from the data used as input. Regardless of data modality (e.g., textual, visual, tabular), we classify data science problems into standard and non-standard problems. Standards problems mainly concern supervised learning (predictive problems) and unsupervised learning (descriptive problems). However, there exist more complex (non-standard) problems than the cited ones.
These complex problems are derived or hybridized from the standard, i.e., core problems.
Technically, we call a dataset imbalanced regardless of and monitors the classification performance of RF. its data modality when there is a disproportion among the number of instances of each class, making classes 2.3. Data augmentation for textual data under-represented. Therefore, traditional machine learning (ML) algorithms have complications defining the tar- Obtaining accurate results while training a classifier beget class’s decision boundaries. comes dificult due to the lack of available, varied, and
As various real-world applications and diverse do- meaningful data, especially when the imbalance
probmains fall under the imbalance problem, researches on lem is present. Therefore, additional samples should be
imbalanced data classification have expanded and gained added to train the classifier more eficiently. However,
more interest [2]. Mainly, we face the imbalance problem gathering such data is time-consuming and needs domain
in credit card fraud detection [3], anomaly detection [4], experts that examine and annotate the data.
e-mail foldering [5], medical diagnosis [6] [7] [8], parti- As assembling such data is costly, synthesizing new
cles identification [ 9], face recognition [10], fault diag- data from the existing ones seems to be a promising
apnosis [
Despite the numerous proposals, most anti-spam such as imbalance learning, as both data and its desired strategies have some inconsistency between false nega- label are present. In this paper, we are considering only tives (missed spam) and false positives (rejecting good two classifiers, random forest and bidirectional LSTM. emails) due to imbalance problems that act as an obstacle for most systems to make anti-spam systems suc- 2.4.1. Random forests cessful. Therefore, an adequate spam-filtering system that addresses imbalance issues is the prime demand for according to [29], random forests are a combination of web users. Recently, the authors in [19] presented an tree predictors such that each tree depends on the valimproved random forest for text classification that in- ues of a random vector sampled independently and with corporates bootstrapping and random subspace methods the same distribution for all trees in the forest. Random simultaneously and tested its performance on the SMS forests have similar hyperparameters to the Decision binary class dataset. The method removes inessential Tree. In addition, they have a very important hyperfeatures, adds some trees in the forest on each iteration, parameter, which is the number of estimators, i.e., the number of trees in the forest.
Hochreiter and Schmidhuber firstly proposed LSTM back in 1997 to overcome the gradient vanishing problem of RNN [30]. Its main idea is to introduce an adaptive gating mechanism, which decides the degree to keep the previous state and memorize the extracted features of the current data input. LSTM models can recognize the relationship between values at the beginning and end of a sequence. For the sequence modeling tasks, it is beneficial to have access to the past and future contexts. By the end of 1997, Schussed and Palatal proposed BiLSTM to extend the unidirectional LSTM by introducing a second hidden layer, where the hidden to hidden connections flow in the opposite temporal order. Therefore, the model can exploit information from both the past and the future, which can improve model performance on sequence classification problems. BiLSTM models are primarily used in natural language processing applications like text classification because BiLSTM is a powerful tool for modeling the sequential dependencies between the words and phrases in both directions of the sequence.
The diagram displayed in Fig. 1 summarizes the integration between the proposed text augmentation approach TAMS and the standard supervised learning training and evaluation process. It starts with data cleaning and common NLP preprocessing steps. After that, the training set is augmented by using TAMS. TAMS generates synonyms for each word, then filters them, and keeps the most similar synonyms to generate new messages. Then the augmented data will be used to train the chosen classifiers defined in Section 2.4 and evaluate their performance on the test set.
To prepare the textual data for the model building we performed the following text preprocessing steps: • Duplicates removal: duplicate samples are problematic as when the same sample appears more than once; it receives a disproportionate weight during the training phase. Thus models that succeed in recurring instances will look like they perform well, while in reality, this is not the case. Additionally, duplicate samples can ruin the split between train, validation, and test sets in cases where identical entries are not all in the same set, leading to biased performance estimates leading to disappointing models in the prediction phase. • Tokenization: this step aims to split each message into a list of words, and this is necessary for two reasons, it is required to recognize the stop word and remove them in the next step, and it is also a requirement to use the Word2Vec to compute a continuous vector representation for each word in the message. • Removal of stop words: stop words are the most frequent words in any language, such as articles, prepositions, pronouns, and conjunctions. They do not add much information to the text. Examples of stop words in English are words like the, a, an, so, what, and many more. Stop words are available in abundance in any human language. By removing these words, we remove the low-level information from the text to focus on critical ones. • Message representation: this step aims to compute a numerical representation for each message by computing a vector that represents the message simply by taking the average of vectors representing each word in that message, where these vectors are computed using the Word2Vec model. The resulting vector will be the feature vector of the message. Word2Vec [31] is a model that computes continuous vector representations of words from large data sets. These word representations help establish the relationship between a word and the other similar meaning words through the created vector representations. Word2Vec models produce real-valued vectors, which allow the machine learning algorithm to deal with the textual data, and at the same time, these vectors keep the semantic meaning of the represented words, where the similar meaning words are closer in space, which indicates their semantic similarity.
The proposed text augmentation method aims to generate new spam messages based on the original ones by replacing some words in the message with their most similar synonyms. Fig. 2 summarizes the proposed TAMS approach, starting from the preprocessed message text tokens and it ends by generating a set of semantically similar messages. In the following subsections, we will explain each step in detail.
• Synonyms extraction: extracting all possible synonyms for each word in the sentence is done with the help of the WordNet database. WordNet is a lexical database of semantic relations between words introduced by [32]. It links words into semantic relations, including synonyms, antonyms, hyponyms, and other morphological relations. Fig. 3 shows synonyms of the word Car in a tree-like structure where the tree’s root is the main word, and each node of the first level of the tree represents synonyms. Each synonym is connected to the root via an edge with a weight representing its similarity. • Finding most similar synonyms: In order to choose the most similar synonyms, every synonym is represented using the Word2Vec representation, and the cosine distance is used to measure the similarity between the word and its synonyms.
The most similar synonyms are the synonyms that have a similitude greater than or equal to a predefined similarity threshold . Fig. 3 shows the most similar synonyms for the word car in case = 0.5.
Optimizing is essential as it explicitly impacts the classification performances. A grid search-like process is done to discover the optimal by training multiple models using diverse augmented data according to candidate similarity thresholds. Candidate similarity thresholds are = [0.625, 0.65, 0.675, 0.7, 0.725, 0.75]. These candidates are selected according to the augmented data’s spam percentage . For example, by using = 0.625, TAMS will generate data with an = 55.98%, in this case, the augmented data is approximately balanced. However, lower values ( < 0.625) will yield an imbalanced data situation. For the highest candidate value i.e., = 0.75, the corresponding spam percentage is 20.10%, and for higher values( > 0.75), the augmented data will be the same as the original data with a high imbalance. To determine the best threshold , We calculate a rank for each candidate as shown in Eq 1: = () + () + ( ) + (1) 4 (1) is the mean average of Spam Caught (), Blocked Ham (), Matthews Correlation Coefifcient ( ), and F1 Score (1) ranks divided
1https://colab.research.google.com/drive/ 12LGx9j6OtEIadERJXJtmqaBaUaqVjz9S?usp=sharing
is a technique for summarizing the prediction results of a classification model, Table. 4 well defines the Confusion Matrix (CM) for binary classification problems. 4.3.2. F1 score is Precision-Recall trade-Of i.e. it combines the precision and recall metrics into a single metric, and it is calculated using Eq. 3, F1 score has been designed to work well on imbalanced data.
1 = 2 × × +
Where is computed by Eq 4: While the is computed by Eq 5 =
+ =
+ 4.3.3. Matthews Correlation Coeficient is used to measure to the quality of the binary classifications, and this measure takes values in the range [-1,+1], where +1 means a perfect predication, 0 indicates the random prediction and -1 means an inverse prediction, (a) confusion matrix of RF and it is given by Eq. 6 model tested on the test data.
( × ) − ( × ) = √ Where: = ( + ) × ( + ) × ( + ) × ( + ). (3) (4) (5) (6) is equivalent to the True Positive Rate (TPR) or Recall, and it means the number of the spam messages which are detected by the spam filter over the number of all the spam messages,i.e. it is the measure of correctly identifying True Positives by the model, and it is defined by the Eq. 7 =
+ 4.3.5. Blocked Hams is equivalent to the False Positive Rate (FPR). A low score close to 0 is preferred as it reflects that we have few false predictions. Furthermore, FPR is the number of legitimate messages which are classified as spam by the spam filter over the number of all the legitimate messages, and it is defined in Eq. 8 =
+ (7) (8)
In this experiment, a random forest classifier with its hyperparameters: _ = 200, __ = 20, _ = 25, and criterion is used to implement the spam filter.
The first part of the experiment depends only on the original data i.e. the imbalanced data, Fig. 4a shows the confusion matrix which summaries the trained model’s predictions on the test set, and Fig. 4b shows the results of the 10-folds cross-validation (10-CV) applied on the original training set, while Table 5 shows the resulting evaluation measures based on the test set. As previously discussed, the optimal threshold is determined based on Table 1. Therefore, we choose = 0.625 as it has the highest rank . Results of TAMS-RF are displayed in Fig. 5a, which shows the confusion matrix obtained using the test set solely, and Fig. 5b shows the results of the 10-folds cross-validation applied on the augmented training set.
As Fig 4, Fig 5, and Table 5 shows, the model trained on the augmented data using the TAMS approach (TAMSRF) performs better than the model trained on the original data solely in most used evaluation metrics. TAMS improved the MCC by 12.36%, SC by 30.96%, and 1-score by 13.67%, which means that the proposed text augmentation improved spam detection meaning that we reduced the bias of RF toward the Ham class and improved the generalization ability of the models. We can conclude that TAMS generated data that has new linguistic patterns that are pertinent to the task and have not yet been seen in pre-training. However, the trained RF with the original data has a better BH score than TAMS-RF, but that does not mean it is better than the second one. On the contrary, it means that the first model is biased toward the ham class and has fewer spam predictions, i.e.; it could not detect the spam messages properly.
In this experiment, a Bidirectional LSTM model with optimizer and activation function at the output layer is used to implement the spam filter. It was trained with a Batch size of 10 for ten epochs. f
Similarly to RF, the first part of the experiment depends only on the original data, Fig. 6a shows the confusion matrix with = 6 and = 13, and Table 5 shows the resulting values of evaluation metrics. While in the second part, a BiLSTM model with the same structure is trained using the augmented data generated by TAMS. For TAMS-BiLSTM, we used = 0.65 as it has the highest rank (see Table 1).
The confusion matrix in Fig. 6b shows a decrease in false predictions as = 4, and = 11. While Table 5 shows that the TAMS-BiLSTM model overcomes BiLSTM according to all the suggested evaluation metrics, and the improvements are as follows: MCC by 2%, Sc by 1.7 %, BH by 33%, F1-score by 1.76%.
Both experiments proved that augmenting the training set by using the proposed TAMS approach helped the classification models to increase their ability to detect spam messages and not get biased toward the majority class and that synthesizing new data from the existing ones is indeed an excellent alternative to data collection and annotation.
Nowadays, the spam filtering task is still a real challenge because most of the available datasets are imbalanced. Dealing with such non-standard derivative datasets is a common problem in classification tasks, especially in the spam filtering case. We proposed TAMS, a text augmentation based on the most similar synonyms replacement to enhance the quality of supervised learning models and solve the spam filtering problem. Experimental results showed that generating additional samples from the existing ones using TAMS added new linguistic patterns pertinent to the task and helped in improving the classification performance of traditional classifiers like the random forest and deep learning models like the Bidirectional LSTM. We can deduce that TAMS increased the ability of both used models to identify spam messages, reduce the bias toward the majority class, and improve the trained models’ generalization ability.
In future work, we aim to improve TAMS further and enhance the quality of its generated data. Furthermore, we have to test our method on other textual datasets and compare it with other text augmentation methods to ensure that the proposed model is generic and not specific to spam filtering exclusively.
This research is supported by the ÚNKP-21-3 New National Excellence Program of the Ministry for Innovation and Technology from the source of the National Research, Development and Innovation Fund.