We discuss the archiving procedure of a corpus comprising posts submitted to Torilauta, a Finnish dark web marketplace website. The site was active from 2017 to 2021 and during this time one of the most prominent online illegal narcotics markets in Finland. As a result of the presented work, a reduced version of the corpus, Finnish Dark Web Marketplace Corpus (FINDarC), has been archived in the Language Bank of Finland. Researchers can apply for access rights to the corpus under the CLARIN RES licence. The discussion presented in this paper addresses the archiving process, including assessment of the risk and impact of data subject re-identification, assessment and implementation of viable data anonymization/reduction approaches, assessment of privacy and security measures implemented by the Language Bank of Finland, and future corpus management plan coordinated by the Language Bank of ∗Corresponding author. †These authors contributed equally.
dark web, illegal narcotics, online marketplace, data sharing
We discuss the archiving procedure of a corpus comprising posts submitted to Torilauta, a dark web marketplace website, for research purposes. The site was active from 2017 to 2021 and during this time one of the most prominent online illegal narcotics markets in Finland. Functionally, the site consisted of discussion imageboards where vendors and customers were able to set up instances of face-to-face trading, typically with the assistance of instant messaging software such as Wickr or Telegram. The original, unmodified data set comprising 3,104,976 CEUR CEUR Workshop Proceedings
ceur-ws.org ISSN1613-0073 posts was collected and handed over to the ENNCODE consortium1 by the site administration to be archived and shared for research purposes, as permitted by the site’s Terms of Service. As a result of the presented work, a reduced version of the corpus comprising 3,104,515 posts, referred to as the Finnish Dark Web Marketplace Corpus (FINDarC), has been deposited in the Language Bank of Finland, a language resource service coordinated by the national FIN-CLARIN consortium formed by Finnish universities and other research organizations. Researchers can contact the Language Bank and apply for permission to access the corpus under the CLARIN RES license.2
While the dark web online market places, including Torilauta, emphasize user anonymity,
the posts submitted to such sites can nevertheless contain personal information, such as unique
usernames and personal names, enabling data subject re-identification. Therefore, as described
in this paper, we have made our best efort to assess and identify the type and amount of
personal information in the original unmodified data set, to assess and implement viable data
anonymization/reduction approaches, to assess privacy and security measures implemented by
the Language Bank of Finland, and to put in place a future corpus management plan coordinated
by the Language Bank of Finland. Moreover, any future research based on the corpus is
encouraged to implement appropriate ethical proofreading measures (see e.g. [
The rest of the paper is organized as follows. We first discuss previous studies on Torilauta and existing related corpora in Section 2. We then provide an overview of the data set in Section 3. In Section 4, we present our experimental work, the purpose of which was to assess the amount of personal and sensitive data in the corpus. Subsequently, given the experimental results, we discuss in detail our data release plan in Section 5. Finally, we provide conclusions on the work in Section 6.
In this section, we discuss previously published studies using the Torilauta site as a data source and related corpora in Sections 2.1 and 2.2, respectively.
Prior to the work presented here, Torilauta was utilized as a data source in multiple
linguistic and social science studies [
It is notable that none of the previous studies attempted to share their data sets with the research community in a systematic manner. This practice negatively impacts the replication and verification of the published studies and potentially discourages further research on the topic. On the other hand, given the sensitive and potentially incriminating nature of the data, not releasing the data is an understandable approach since preparing and managing such a resource gives rise to multiple technical, ethical, and legal challenges. The purpose of this paper is to describe and discuss these challenges and how we approached them.
To the best of our knowledge, there exist relatively few published dark web corpora or text
data sets. Three notable exceptions include the Dark Net Market archives (2013–2015) [
Existing Finnish web forum corpora include texts collected from the Ylilauta imageboard [35]
and Suomi24 social networking site [
The Finnish Internet Parsebank [
Finally, in a recent study, Leedham et al. [
In this section, we describe the original, unmodified corpus received by the ENNCODE consortium. We provide an overview of the data as a whole and discuss in more detail the contained data fields, post lifespans, and thread and message lengths in Sections 3.1, 3.2, 3.3, and 3.4, respectively.
The original data set received by the consortium included all posts submitted to Torilauta between 2019-09-11 and 2020-05-20 (1,863,639 posts in 251 days) and 2020-06-17 and 2020-10-31 (1,099,710 posts in 136 days). In addition to the posts collected during these active collection periods, the data contained “residue” posts submitted between 2017-11-02 and 2019-09-11 (141,627 posts in 678 days). Meanwhile, posts submitted between 2020-05-20 and 2020-06-17 were missing completely. Therefore, the original unmodified corpus consisted of 3,104,976 posts in total.
Table 1 presents an overview of the post and thread frequencies in the data grouped by boards along with brief topic descriptions. Of the 32 boards, the board with the highest activity measured by the total number of submitted posts and threads was the market board dedicated to narcotics transactions within the city of Helsinki (/hki). The total number of posts submitted to this board was 787,459 corresponding to 25.4% of all posts in the data. Meanwhile, in total 96.5% (2,997,624) of all posts were submitted to the 16 boards dedicated to transactions (denoted with (market) in the topic column of Table 1).
Each post in the corpus is represented as a data structure with 8 fields as shown in Table 2. Each field belongs to one of the following three types: a string, an integer, or a date. All dates are in timezone UTC+0 (GMT+0). Note that throughout this paper, we refer to a set of these 8 data fields as post, whereas the content of the text data field within a single post is referred to as message or text body. Missing values have diferent meanings depending on the field. For deletions, a missing value means the post was never deleted. The first post of a thread, referred to as the original post (OP), always has a missing postId value and is instead identified by its (boardUri, threadId) pair. The subject field is missing for 46.2% of all posts since it was common practice to omit the subject. Similarly, the poster name field is missing for 54.1% of all posts which is in line with the anonymous nature of the site and since any optional contact information, such as an instant messenger username, was often included in the message text body instead.
Finally, the posts submitted to Torilauta optionally contained an attached image. However, no images were included in the original data set. Moreover, the data fields comprising a post did not include information on whether the post contained an image or not.
Submitted posts were deleted from the site for three main reasons. First, the site hosted a fixed number of threads on each board at a given time and so inactive threads were regularly removed by an automatic pruning mechanism to make room for new, active threads. Second, posts which violated the site rules (e.g. spam) were removed by the site administration. Third, the site interface did not provide users with means to edit messages and, therefore, the only way to correct erroneous message content (e.g. typos, updates) was to delete the post and resubmit. Lastly, a small portion of posts were ”pinned” by the site administration, that is, they were meant to stay available on the site indefinitely. hki tre vnt oulu muut tku esp kpo jkl lti bulk vsa t roi seka hm b h y pm a hox hax kkk meta test spam rotta tt k fap pgp total
City of Helsinki (market) City of Tampere (market) City of Vantaa (market) City of Oulu (market) Other areas (market) City of Turku (market) City of Espoo (market) City of Kuopio (market) City of Jyväskylä (market) City of Lahti (market) Bulk transactions (market) City of Vaasa (market) Dates City of Rovaniemi (market) Miscellaneous (market) Narcotics markets Random Narcotics Jobs Mail orders (market) Everyday/mundane Hormones (market) Hacking Cultivation Meta discussion Testing Spamming Vendor feedback Health Getting and staying sober Porn PGP public keys
# posts posts (%) # threads threads (%) 3,104,976
There are two known caveats related to the deletion timestamps. First, while the data set included the creation and deletion times of posts, it unfortunately did not include information about the reason for the deletion. Second, all posts deleted during the pause in collection 2020-05-20 - 2020-06-17 had their deletion value marked as missing and, therefore, appeared as if they were not deleted. The amount of these potentially erroneous missing values was, however, relatively small and 97.22% of all posts (3,104,976) in the data had reliable deletion time information.
Finally, we estimated the median lifespans of submissions to the market and non-market boards to be 23 and 238 hours, respectively. The diference was mainly due to the lower posting frequency and consequently lower thread pruning frequency of the non-market boards. For noise filtering purposes, we are mostly interested in the messages with short lifespans. To this end, we note that 5% of all messages had a lifespan of less than 32 minutes. Since posts with such short lifespans were likely removed by the user and resubmitted after minor modifications, they may be discarded as noise.4
By and large, the threads in the market boards were rather short as the majority (57.9%) consisted of a single post (the thread has no reply and/or follow-up messages to the original post) and 99% of the threads have 23 posts or less. Similarly, on the non-market boards, 64.18% of the threads consisted of a single post and 99% of the threads have 35 posts or less.
In order to examine individual message lengths, we tokenized the message bodies using the tokenizer included in the Finnish Tagtools (v1.5) software developed at the University of Helsinki.5 The tokenizer splits running text into sentences and extracts punctuation and special characters from word bodies into separate tokens. The median length of original posts and reply/follow-up posts to the market boards were 36 and 6 word tokens, respectively. In other words, the original posts were typically short (a few sentences) and reply/follow-up posts even shorter (a few words). As for the reply/follow-up posts, this was because of the prevalence of short update posts, the purpose of which was to keep the thread alive and close to the front page. These types of posts comprised roughly one fith of all market board reply/follow-up posts. Therefore, one can reduce the amount of noise in the FINDarC considerably by ignoring reply/follow-up posts with, e.g. less than 5 word tokens (or less than 20 characters) in the message body.6
In this section, we present our experimental work, the purpose of which was to assess the amount and types of personal and sensitive data in the FINDarC corpus. As the size of the corpus exceeded over 3,000,000 messages, it was deemed infeasible to curate all the posts manually given the consortium resources. In Section 4.2, we discuss experiments utilizing full-text search using hand-crafted regular expressions. Finally, the implications of the results on data reduction and data release are discussed in Section 5.
In this part of the experiments, we examined the types of personal data contained in the message bodies of posts using a manual annotation approach. In particular, we were interested in types of personal information found in the market board submissions which contained the majority of posts in the data set (97%) and represent the primary function of the site as a narcotics marketplace. 4.1.1. Definition of Personal Data We adopted the Article 4(1) of the GDPR7 which defines personal data as any information relating to an identified or identifiable natural person (data subject); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person; and divided potential identifiers into the following 5 classes 1. person_name (e.g. first name, family name) 2. id_number (e.g. the Finnish social security number) 3. location (e.g. street address, city, city district) 4. online_id (e.g. instant messaging username, email address, IP address) 5. other (e.g. phone number, identifying physical appearance) Moreover, GDPR Article 9 (1)8 further defines the special categories of personal data as Processing of personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person’s sex life or sexual orientation shall be prohibited.
For cases belonging to this group, we added a sixth potential identifier class
7 https://gdpr-info.eu/art-4-gdpr/
8 https://gdpr-info.eu/art-9-gdpr/
6. special_category
Given the personal data class specifications, the annotation task then comprised two subtasks:
entity span detection and class assignment on a word token level to continuous, non-overlapping
sequences of tokens.9 For example, consider
priimaa kukkaa [Helsingin keskustassa]location ! yhteydenotot Wickerillä W //
[example-name]online_id
excellent bud [in the centre of Helsinki]location ! contact Wicker W //
[examplename]online_id
4.1.2. Personal Data versus Named Entities
The definition of personal data and the personal data classes described in the previous section
are closely related to named entities [
Sauli Niinistö is the President of Finland.
where Sauli Niinistö and Suomen (= Finland) are defined as person and location named entities,
respectively, according to the Finnish NER guidelines [
In this section, we assess the amount and types of personal data in the data set using a full-text search approach. In particular, we are interested in finding common personal identifiers with relatively rigid formats, such as social security numbers and phone numbers. We describe the experiment setup in Section 4.2.1, the applied textual patterns (regular expressions) in Section 4.2.2, and the obtained results in Section 4.2.3. 4.2.1. Setup We define a target set of textual patterns (regular expressions), search for matches in message bodies. Specifically, we are interested in finding expressions matching 1. (Finnish) social security numbers 2. (Finnish) phone numbers 3. Email addresses 4. IBAN bank accounts 5. IP addresses all of which have relatively rigid formats. The employed regular expressions are presented in Section 4.2.2. We apply the search to all posts in the data and assign the matches manually to personal data and non-personal data according to post context. The pattern matching is performed using MongoDB (v.5.0) full text search.10 In contrast to the manual annotation experiment discussed in Section 4.1, we do not filter out noise from the data and instead apply the search to all 3,104,976 posts in the original corpus. 4.2.2. Regular Expressions In what follows, we provide brief descriptions of the applied regular expressions. Social security numbers The Finnish social security number (SSN) is a sequence of 11 characters assigned to individuals by the Finnish government based on their date of birth and gender. The first 10 characters of the sequence are 6 numbers (date of birth) followed by a hyphen or A, followed by 3 numbers. The last character is alphanumeric, i.e., a number or a letter. Valid sequences likely have, therefore, format ”121212-1234” and ”121212-123A”. We detect the sequences using the regular expression \ d \ d \ d \ d \ d \ d \ - \ d \ d \ d [ a - z A - Z 0 - 9 ] . Persons born in the 2000s, who would have an ”A” instead of hyphen, were not found in the sample. Phone numbers According to the specification of the Finnish telephone network numbering, Finnish mobile phone numbers begin with a routing number (04-, 050, or 059) and are followed by a subscriber number, such as, ”040 1234567”, ”059 1234567”, and so forth.11 The first zero (”0”) of the number can optionally be replaced by the country code of Finland +358 (e.g. ”+358 10https://www.mongodb.com/ 11Specification of numbers in the Finnish phone network is available at: https://www.finlex.fi/fi/viranomaiset/normi/480001/47180 40 1234123”, ”+358 59 4321432”, etc.). Based on a preliminary examination of the data set, we detect common phone number formats using two regular expressions: [ \ + ] ? 3 5 8 [ \ - \ s ] ? 0 [ 4 5 ] [ \ \ s ] ? \ d \ d \ d [ \ - \ s ] ? \ d [ \ - \ s ] ? \ d \ d \ d which matches numbers starting with the country code and 0 [ 4 5 ] \ d [ \ s \ - ] ? \ d \ d \ d [ \ - \ s ] ? \ d [ \ - \ s ] ? \ d \ d \ d which detects numbers with the country code omitted. Moreover, the expressions detect most commonly used grouping patterns using hyphens (e.g. ”+358-40-12345-567”) and whitespaces (e.g. ”059 123 4567”). While the subscriber part of the number can, in principle, vary in length, the patterns match the most common length of 7 digits. Landline numbers would be shorter but follow the same principles; none were however found in the data.
Email addresses According to the RFC 5322 standard12, an email address as an identifier which contains a locally interpreted string followed by the at-character (”@”) followed by an internet domain, such as ”name@domain.com”, ”firstname.surname@subdomain.domain.com”, and ”underscore-hyphen-plus+sign@domain.com”. We detect the addresses using a regular expression \ S + \ @ \ S + \ . \ S + which successfully detects all the above examples from a running text.
IBAN bank accounts We search for bank account numbers matching the International Bank Account Number (IBAN) structure specified by the ISO 13616-1:2020 standard 13. The IBAN formatted numbers consist of the Finnish bank account number (14 digits) preceded by a two letter country code (”FI” for Finland) and two check digits (e.g. ”FI72 1234 5678 1234 12”). We detect the pattern using the regular expression [ F f ] [ I i ] \ d \ d [ \ s \ - ] ? \ d \ d \ d \ d [ \ s \ ] ? \ d \ d \ d \ d [ \ s \ - ] ? \ d \ d \ d \ d [ \ s \ - ] ? \ d \ d which takes into consideration the letter case of the country code and the commonly used grouping whitespaces.
IP addresses IP (internet protocol) addresses are unique addresses which identity devices
on the internet and local networks. We search for IP addresses using the following regular
expression ( 2 5 [
Finally, we created a second regular expression list using words and prefixes related to the
personal information contained in the identified 295 targeting posts. This list consisted of 77
keywords and parts of person names and addresses.14 After performing a second search with
these patterns and a subsequent manual inspection, we identified an additional set of 166 posts
submitted as a means of targeting.
4.2.4. Discussion
Similarly to automatic NER, rule-based search for text patterns is a commonly used part of text
anonymization pipelines to varying extents [
In this section, we draw on the discussion and experimental results presented in the previous sections and present an outline of the FINDarC data release.
Data Reduction Conventionally, the most direct approach to protect data subjects from
re-identification has been to anonymize the data by removing/obscuring the parts containing
personal information. [
Due to the problematic privacy-utility trade-of, we posit here that reducing the FINDarC extensively would not be appropriate even if suficient resources could be allocated for domainspecific tool development and manual labour. Furthermore, we note that Torilauta and other drug trading sites have also been under observation by other parties, including both criminals and law enforcement agencies. Therefore, it is our assessment that leaving the sell/buy posts, which form the majority of the FINDarC, largely intact poses few additional risks to the studied populations. However, as discussed in 4.2, in addition to the sell/buy posts, the data also contains posts with the intention of doxxing/targeting individuals. Here, our position is that removing these submissions is warranted from an ethical point of view while not decreasing the value of the corpus as a data source significantly. This is because these posts are not directly related to the main functionality of the site as an online marketplace. Accordingly, we removed from the corpus all 461 posts containing identified doxxing/targeting information described in Section 4. The reduced corpus, therefore, comprises 3,104,515 posts. A summary of the removed posts by board is presented in Table 5.
Finally, as per the Terms of Service of Torilauta, the site users gave consent to data collection for academic use by using the site. Consequently, site users could opt out of the data collection by not submitting new posts and/or contacting the site administration about previously submitted posts. However, it could be argued that by removing a previously submitted post, a user has withdrawn the permission to use the data. Unfortunately, as discussed in Section 3.2, the original data set received from the site administration did not include information about the reasons behind post deletions. Therefore, we were not able to exclude any posts from the corpus based on the deletion status.
board
# posts
Access Restrictions Due to the limited applicability of data reduction as a means of protecting
data subjects from reidentification, we next discuss restricting the access to the corpus. In
general, the Language Bank recommends sharing resources using standard CC-BY or other
open source licenses, in which case only the metadata of the resource needs to be registered
with the Language Bank, although the Language Bank also hosts openly available and publicly
accessible resources (PUB) in its Language Bank Download service. However, since the FINDarC
resource in its current form contains personal data, both copyright and personal data legislations
apply and the corpus cannot be published with open access. Instead, FINDarC is provided via
protected access under the CLARIN RES licence which means that permission to download and
use the corpus is only granted to researchers based on written applications reviewed by the
data controller (principal investigator of the ENNCODE consortium) including a data protection
impact assessment. The purpose of this limitation is to ensure that the material is accessed
only by verified researchers for legimitate research purposes. It also lessens sharing-related
risks to both the researchers and the subjects of study, as mandated by the consortium’s data
management policy. Restricting access to the corpus as described here is in line with the current
literature on data sharing [
We discussed the archiving procedure of FINDarC, a Finnish dark web marketplace corpus, in the Language Bank of Finland. The discussion included an overview of the data, assessment of the risk and impact of data subject re-identification, assessment and implementation of viable data reduction approaches using manual and automatic text processing, assessment of privacy and security measures implemented by the Language Bank of Finland, and a future corpus management plan implemented and coordinated by the Language Bank of Finland. As a result of the presented work, a reduced version of the corpus has been archived in the Language Bank of Finland. Researchers can apply for access to the corpus under the CLARIN RES licence.
As this article shows, the data set was cleared using best practices for ethical proofreading, which consistently sought to prioritize the protection of the posters on the forum. Given how even indirect identifiers could be utilized against the site’s users by either law enforcement or other members of the drug-using community, it was necessary to opt for maximal removal eficiency whenever possible. Nevertheless, the amount of data is so high that no clearing can be considered ethically suficient for the purpose of releasing the data openly, so we opted for a gatekeeping approach in addition to clearing everything that could be found.
LVI. [33] Virtanen, A., J. Kanerva, R. Ilo, J. Luoma, J. Luotolahti, T. Salakoski, F. Ginter, and S. Pyysalo.
2019. Multilingual is not enough: Bert for Finnish. arXiv preprint arXiv:1912.07076. [34] Weischedel, R., M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, N. Xue, A. Taylor, J. Kaufman, M. Franchini, and Others. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA 23. [35] Ylilauta. 2016. Ylilauta-korpuksen ladattava versio [tekstikorpus]. Kielipankki. Available at http://urn.fi/urn:nbn:fi:lb-2016101210.