Over the past decade, the darknet has created unprecedented opportunities for traficking in illicit goods, such as weapons and drugs, and it has provided new ways to ofer crime as a service. Natural language processing techniques can be applied to find the types of goods that are traded in these markets. In this paper we present the results of evaluating state-of-the-art machine learning methods for the classification of darknet market ofers. Several embeddings, such as GloVe embeddings [20], Fasttext [15], Tensor Flow Universal Sentence Encoder [7], Flair's contextual string embedding [2] and term-frequency inverse-document-frequency (TF-IDF), as well as our domain-specific darknet embedding have been evaluated with a series of machine learning models, such as Random Forest, SVM, Naïve Bayes and Multilayer Perceptron. To find the best combination of feature set and machine learning model for this task, the performance was evaluated on a publicly available collection covering 13 darknet markets with more than 10 million product ofers [6]. After extracting unique advertisements from the corpus, the classifier was trained on a subset with those advertisements that contain strings related to weapons. The purpose was to determine how well the classifier can distinguish between diferent types of advertisements which seem all to be related to weapons according to the keywords they contain. The best performance for this classification task was achieved using the
Linear Support Vector Machine model with the Tensor Flow Universal Sentence Encoder for feature extraction, resulting in a micro-f1-score of 96%.
Darknet markets (DNMs) provide a largely anonymous platform for the trade in
illegal goods and services, and drugs represent a large part of the product range
[
Classification is one of the common tasks of text mining and natural language processing (NLP). It is used to divide a set of texts into groups according to predefined labels, and this technique can also be applied to automatically classify product listings of DNMs. Since it is a supervised machine learning technique, ground truth training data is needed to build the classifier. Many darknet market websites provide menus which allow their customers to navigate between product categories. In principle, these categories could serve as product labels. However, ifrst, this categorization was created by the vendor and is therefore not necessarily trustworthy. Second, most markets do not indicate a product category and therefore would be excluded from the analysis. Third, some markets and vendors use false categories to obscure what is being ofered. Finally, the categories vary between diferent DNMs which makes it dificult to create a consistent cross-DNM overview about the product ofers available. To overcome these dificulties, we have created a manually annotated dataset that was obtained from thirteen DNMs and which therefore contains consistent labels over a high range of examples.
In principle, classification by keywords is a simple and eficient way to
characterize texts. However, the selection of the right keywords requires a profound
understanding of the domain, which is labor intensive and hard to obtain in a
hidden society. Furthermore, words are often used diferently in the darknet and
homonyms are frequent. For example, fruits, weapons, popular brands or celebrity
names are sometimes used as brands for drugs and as code words. A string search
for “ak47” results in many listings related to drugs, some literature on the weapon
and a few ofers for a weapon in the Gwern dataset [
In the following, we present state of the art word and document embeddings. Then, in section 3, we briefly discuss the literature on classification of product ofers on illicit online markets, before describing the dataset we used for our experiments in section 4. Finally, we cover the experimental setup in section 5 and our results in section 6.
Recent advances in NLP have shown the benefit of pre-trained word embeddings
for various tasks such as named entity recognition [
For our experiments, that will be described in section 5, we calculate vector representations for documents using the following techniques:
Term frequency-inverse document frequency (TF-IDF) serves as a simple
benchmark to advanced document embeddings. TF-IDF relies on the assumption
that the occurrence of a frequent word adds little information about a document
compared to the occurrence of an infrequent word. The frequency of the word “is”
carries little information about a document, whereas the existence of a word like
‘perceptron’ usually indicates that the topic of the document is related to machine
learning as ‘perceptron’ is a technical term only used in machine learning. TF-IDF
is the normalized number of occurrences of a word in a document, weighted by the
number of its occurrences in the whole corpus. The TF-IDF representation of a
document consist of the TF-IDF values for all words in the document [
The GloVe [
The contextual string embedding Flair [
Fasttext [
Pre-trained embeddings are usually trained on texts taken from a non-criminal
context, e.g., Wikipedia or news articles. The language used in the darknet can
vary significantly to those sources mentioned above. This is due to diferences in
contents itself, genre (article versus product ofer), style (formal versus colloquial)
and the use of domain-specific expressions. Therefore, the transferability of those
embeddings to the darknet domain can be limited. The benefit in generalisation
of a pre-trained model might be counteracted by diferent usage of words in the
darknet. To answer the question whether a domain specific embedding trained
on the darknet outperforms a general embedding that was trained on clear net
data, we trained a fasttext darknet embedding [
to calculate feature vectors of dimension 512. It was trained on NLP tasks in eight
languages [
We selected our document embeddings to represent three categories: State-ofthe-art methods that use a bag of words approach and therefore don’t take the order of words into account (GloVe and FastText), embeddings that depend on the order of words (Flair, Universal Sentence Encoder) and a simple statistical benchmark (TF-IDF)
Since Christin [
Automated keyword extraction for product categories was discussed by Ghosh
et al. [
More recently, LSTMs [
Our contribution is the comparison of multiple state of the art document embeddings using a range of established classifiers on a darknet dataset. We evaluate whether pre-trained embeddings improve the classification performance over simple vector space models, such as TF-IDF, when being transferred to the darknet domain. Further, we test our models on a dataset which contains similar product descriptions aggregated by a keyword based search.
Our dataset is based on a subset of the Darknet Market Archives [
Hence we are interested in a dataset that is related to firearms, the final dataset
for our experiment contains only product descriptions with at least one occurrence
of a keyword related to firearm names. The list of firearm keywords was extracted
from a publicly available dataset [
Drug Weapon Book Crime as a Service (CaaS) Excluded
977 284 153 116 60
In our experiments we examine combinations of machine learning classifiers and
document embeddings for a text classification task in the darknet domain. We have
selected a range of commonly used models for classification tasks. We use the
scikitlearn implementations for: RandomForestClassifier, LinearSVC, GaussianNB,
LogisticRegression (LR), DecisionTreeClassifier, AdaBoostClassifier, KNeighbors
Classifier, MLPClassifier using default parameters [
We use our dataset to train and evaluate classifiers on the task of assigning a label to new product ofers on DNMs. To find the best combination of feature set and machine learning model for this task, several embeddings, GloVe , Fasttext, Tensor Flow Universal Sentence Encoder, Flair’s contextual string embedding and our darknet embedding have been used.
After extracting unique advertisements from the corpus, the classifier was trained using the subset with those advertisements that contain strings related to weapons. The purpose was to determine how well the classifier can distinguish between diferent types of advertisements which seem all to be related to weapons according to the keywords they contain. For our experiments we train all models with all embeddings. We use a fourfold cross-validation that preserves the percentage of samples for each class which is consistent for all experiments. The model’s score is the average of the scores for each fold in the cross-validation.
In binary classification, the precision is the number of true positives over all
predicted positives and recall is the number of true positives over all actual
positives. The f1-score is the harmonic mean between precision and recall. For
multiclass classification, the micro f1-score is calculated globally by counting the total
true positive, false negatives and false positives, while for the macro-f1 score, the
standards f-1 score is calculated for each class and the scores for all classes are
averaged without weighing them. To evaluate the performance of our models, we
report micro-f1-score as well as macro-f1-score, to measure the overall performance,
but also take the performance for classes with fewer samples into account. [
We present the macro f1-score and the mirco f1-score for each combination of embedding and classifier in Figure 1. The overall best performance was achieved with Tensorflow Universal Sentence Encoder and a linear SVM resulting in a score of macro f1-score of 0.93 and a micro f1-score of 0.96. Analysis of the best results per classifier shows that Tensorflow Universal Sentence Encoder performs best for ifve out of eight embeddings. The simple TF-IDF performs better than others for Decision Tree and Gaussian Naive Bayes. Only AdaBoostClassifier works best with our darknet embedding trained with fasttext. Further, the comparison of classifiers shows that Linear SVC performs best for five out of six embeddings. The good performance of SVMs is expected, as it is the prevalent model in previous works. Overall, Tensorflow Universal Sentence Encoder appears to generate the best features for our task. However, TF-IDF ranks second place with a simple and lightweight implementation that doesn’t require pre-training on huge datasets. The comparison of micro and macro scores indicates that the performance for all classes is balanced for Tensorflow Universal Sentence Encoder.
The comparison of the pre-trained fasttext embedding and our darknet embedding shows similar performance as each of the embeddings outperforms the other one in four cases. The best score for the darknet embedding is 0.03 lower than the best score for the pre-trained embedding and therefore no benefit over the pre-trained embedding could been shown for the darknet embedding.
Further, we present a detailed analysis of the best combination, which is linear SVM with Tensorflow Universal Sentence Encoder. To achieve more significant results, we reduce the proportion of training data to 25%. The training dataset contains a total of 382 samples, with 242, 69, 36, 35 texts for Drug, Weapon, Book and CaaS respectively. The now larger test set contains a total of 1148 samples, with 735, 215, 117, 81 texts for Drug, Weapon, Book and CaaS respectively. The predictions for this experiment are shown in a confusion matrix (Figure 2).
We show the metrics for each class in Table 2. It can be seen that precision as well as recall achieve almost perfect scores over all four classes. The class "Drug" performed best with a f1-score of 0.99, whereas "Book" achieved the lowest score with 0.88. Further, micro and macros averages and the counts per group (Support) are listed.
In this paper, we evaluated state of the art text embeddings for classification tasks in the darknet domain using multiple classifiers. To show the benefits of a text classification compared to a keyword-based search, we have trained the classifier on the results of a keyword-based search. We used a subset of the Grams crawl in Gwern’s achieve, that contains strings related to weapons and we showed that a text classifier is able to correctly determine labels with an overall accuracy of 97 %. Best results are achieved with features generated by the Tensorflow Universal Sentence encoder using SVMs. However, other state of the art embedding do not beat the established TF-IDF vectorization in this task.
Acknowledgements. The research described in this paper was carried out as part of the COPKIT project which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 786687