In this paper, we introduce DeepQuarantine (DQ), a cloud technology to detect and quarantine potential spam messages. Spam attacks are becoming more diverse and can potentially be harmful to email users. Despite the high quality and performance of spam ltering systems, detection of a spam campaign can take some time. Unfortunately, in this case some unwanted messages get delivered to users. To solve this problem, we created DQ, which detects potential spam and keeps it in a special Quarantine folder for a while. The time gained allows us to double-check the messages to improve the reliability of the anti-spam solution. Due to high precision of the technology, most of the quarantined mail is spam, which allows clients to use email without delay. Our solution is based on applying Convolutional Neural Networks on MIME headers to extract deep features from large-scale historical data. We evaluated the proposed method on real-world data and showed that DQ enhances the quality of spam detection.
deep
Nowadays it is hard to imagine a life without e-mail communication, particularly
in business area. The growth of e-mail's popularity is accounted for low cost and
high e ectiveness of exchanging messages. The same factors contribute to the
increasing amount of spam. According to a report by Kaspersky [
Anti-spam software companies aim to protect users against malicious mail,
and, crucially, ensure the delivery of all legitimate messages to them. Otherwise,
even one misclassi ed message, for example, from a business conversation, can
lead to signi cant reputation risks. To reach a low false positive rate, anti-spam
decisions must be very reliable, which obviously reduces detection rate. To solve
this problem, commercial anti-spam products delay potential spam messages to
recheck them after a certain time to improve the reliability of the anti-spam
solution. The Axway Inc. in [
In this paper, we describe a novel approach to quarantine messages. Our
work focuses on applying Deep Learning [
We evaluated our approach on a large-scale dataset. In the experiments, we showed that combination of our proposed model and traditional spam lters improves in classi cation rates. 2
Cybercriminals continue to look for new ways to spread spam and improve previous techniques. Traditional signature approaches are becoming less e ective compared to previous years. The reasons are poor generalization ability and the need to use human resources to nd new attacks and develop signatures to block them.
Machine learning techniques have recently become very e ective to ght
spam. Most research papers propose di erent methods to handle body content
of a message. [
There are also related works that use non-content features for spam
detection. [
Publicly available benchmark datasets on e-mail spam highlighted in [
In this section, we introduce the design of DQ. We describe three main parts of the new technology. First, we focus on backend logic, which is responsible for message transactions and the system-customer relationship. Then we illustrate preprocessing of message headers. Finally, we show design of model for spam classi cation. 3.1
According to Figure 1a messages in origin-based scheme are processed by complicated system of spam lters before delivered to user. Moreover, spam lters are regularly updated because statistical properties of spam campaigns change over time. Indeed, this approach can be used and potentially provides high detection rate. In real life, most missed spam is detected shortly after updating lters. Unfortunately, the considered scheme delivers these messages to users because spam decisions are made once when a message is received.
To solve this problem we implemented DQ, illustrated on Figure 1b. DQ is a cloud technology, which provides request-response logic with an installed anti-spam service on user's machines. The main objective of DQ is classi cation of potential spam. After a messages passed through lters, the service sends a request to DQ with message headers and waits for a response. Meanwhile, DQ handles input data and returns true if message should be delayed or false otherwise. Of course, in real life organization of this communication to process the big data that accumulates from di erent user nodes is not a trivial task. We do not go deep into implementation details and focus on logic of the technology. As shown on Figure 1b, suspicious mail is put in the Quarantine folder for a while, others are delivered to user. When the time is over, quarantined messages pass through lters again. It should be noted that DQ only receives required headers and returns the quarantine decision, all delayed mail in Quarantine folder is located on the user PC.
The proposed scheme allows to gain the time to update lters and doublecheck suspicious messages to improve the reliability of the anti-spam solution. Moreover, this implementation provides a low-cost way to update the model that is extremely important to adapt to new spam tactics or changing mail transfer protocols.
(a) Origin-based
(b) DQ implementation It is well known that the feature selection plays a big role in model performance. The e-mail provides a large amount of information about a sender and content of the message. Some of this data can be absolutely useless and add unwanted noise that can be a reason of lower model performance. Our solution is based on non-content classi cation. Due to this fact, we are able to transfer data to cloud service and collect this type of information using simpler way than in contentbased case. Another important aspect is the ability to quickly extract features from message. As far as DQ is a real-time service, performance is very important to ensure email communication without delay.
At the moment, the model takes: Message-ID, a sequence of message headers (HeaderSeq) and X-mailer. To bypass protection systems and spread malicious mail, spammers often use their own Mail User Agent (MUA). MUAs are responsible for preparing email messages for transferring to a Mail Transfer Agent (MTA). One of the MUA tasks is to create and ll correct MIME headers. Some of attackers ignore it and can use random content for headers. Others try to fake headers to make them look like real ones. We focus on Message-ID and HeaderSeq for several reasons. Firstly, these features have non-trivial structure. Secondly, the form of Message-Id and the order of headers in HeaderSeq can vary depending on the type of MUA, which creates a tight connection between features. These facts make compromising more di cult, which helps the model to detect spam. We also added X-mailer to de ne MUA. Below we describe features and their representation for the classi er.
The Message-ID provides an identi er for messages and looks like a sequence of US-ASCII characters between an angle bracket pair. For example: Message-ID consists of two parts splitted by @. The left part of the MessageID is a hash that has a speci c structure for di erent MUAs. The right part is a domain. The Message-ID is transformed to a tensor size l, where each row vector is a char embedding. For encoding, we build a vocabulary that maps USASCII chars (without special characters) to trainable embeddings. In addition, we added two symbols < EOS > for the end of a string and < U N K > for unknown characters. In case the length of Message-ID is greater than l, the rst l-characters are taken. In case length of Message-ID is less than l, the sequence is lled with < EOS > to the length l.
The HeaderSeq is a sequence of MIME headers in the message. The order of headers can vary depending on the type of MUA. The encoding of HeaderSeq has the same scheme as the Message-ID. The only di erence is that we operate with header names, not characters. For example:
subject:from:to:date:message-id:content-type: is a possible HeaderSeq. The nal representation is a tensor with xed shape where each row is an encoded header. The number of rows was estimated from statistics as a 95-percentile of HeaderSeq length.
The X-Mailer is the name of a MUA. Before encoding, we preprocess the X-Mailer to get information only about the type of MUA. For an actual e-mail program, we drop information about version and release. For example:
Microsoft Windows Live Mail 14.0.8117.416 is transformed to Microsoft. This helps signi cantly reduce the size of the feature space. We also conducted experiments that used the name and version of MUA, but this did not increase performance. For an unknown e-mail program, we created a special category. The encoding is done by using one-hot encoding.
In this section, we describe the architecture of the spam classi er. Despite the fact that DQ does not block messages, we cannot delay all of them for re-checking, because this signi cantly increase the delivery of an e-mail. Moreover, to ensure that e-mail work without delays, DQ has a time limit for the response. If the time is over, the message is delivered to the user without applying DQ. For these reasons, we have a trade-o between model complexity and computation time.
Figure 2 demonstrates the model architecture. Following [
For the Message-ID we designed a subnet with four temporal convolution
layers with a xed number of lters for each of them. We applied relu as activation
function. Initially we use a layer with biggest lter size to extract information
from longer subsequences. After the rst and last layers, we inserted a temporal
max-pooling layer to ensure stability of training. In the HeaderSeq branch, we
used two layers: a temporal convolution layer and a temporal max-pooling layer.
The shallow architecture is the result of a small length of HeaderSeq. The
outputs from the convolutional nets are concatenated with the encoded X-Mailer to
a one-dimensional tensor as illustrated in Figure 2. Finally, we added two fully
connected layers and inserted a dropout [
In many research papers it is stated that a CNN usually requires large-scale datasets to work and achieve competitve performance in di erence areas. Unfortunately, public datasets for spam classi cation are fairly small and do not show actual threats because they are not regularly updated.
In this work, we used a collection that consists of metadata from tens of millions of real-time e-mail scans. We split the data into training and test datasets by timestamp to avoid leaking information from the future into the past. We sampled 120 million objects for training and 40 million objects for testing. In both datasets, the proportion of spam is about 40 percent. We optimized weights of the model using SGD with momentum of 0:9 to minimize the cross-entropy loss. We initialized the model weights using a Gaussian distribution and trained all layers together throughout nine epochs and halve the learning rate every three epochs.
We show the PR-curve in Figure 3 to demonstrate the model performance on the test data. As mentioned earlier, a classi er should have high precision to deliver legitimate e-mail messages without delay. We de ned a probability threshold for which the precision is equal 0:998 and the recall is 0:823. We tested the DQ with this classi er in the course of 4 weeks in the real world. Our internal tests showed that the proposed technology detects up to 30% of previously missed spam.
This article proposes a non-content-based classi cation approach to delay potential spam messages in real time. On the one hand, we demonstrated a novel feature set and way to handle it for a spam classi cation task. On the other hand, we show that this method is well-suited for enterprise solutions because it has a simple update scheme, high performance and a low false positive rate. Furthermore, combining this technology with resource-intensive checks that require additional time for veri cation/response (such as a Whois requests, in-depth content veri cation, etc), we can get a fast and cost-e ective system for detecting spam messages.