We present the BETH cybersecurity dataset for anomaly detection and out-of-distribution analysis. With real “anomalies” collected using a novel low-level tracking system, our dataset contains over eight million data points tracking 23 hosts. Each host has captured benign activity and, at most, a single attack, enabling cleaner behavioural analysis. In addition to being one of the most modern and extensive cybersecurity datasets available, BETH enables the development of anomaly detection algorithms on heterogeneously-structured real-world data, with clear downstream applications. We give details on the data collection, suggestions on pre-processing, and analysis with initial anomaly detection benchmarks on a subset of the data.
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).
Proceedings of the Conference on Applied Machine Learning for Information Security, 2021 using a novel honeypot tracking system, our dataset has the following properties that make it attractive for the development of robust ML methods: 1) at over eight million data points, this is one of the largest cyber security datasets available; 2) it contains modern host activity and attacks; 3) it is fully labelled (for verification); 4) it contains highly structured but heterogeneous features; and 5) each host contains benign activity and at most a single malicious user, which is ideal for behavioural analysis and other research tasks. In addition to the described dataset, further data is currently being collected and analysed to add alternative attack vectors to the dataset.
There are several existing cybersecurity datasets commonly used in ML research: the KDD
Cup 1999 Data [
This paper begins with a description of the data collection process and the relevance of
the available features. We then perform an analysis of the first set of kernel-level process logs
collected, including anomaly detection benchmarks2. Our benchmarks include both traditional
baselines [
The BETH dataset currently represents 8,004,918 events collected over 23 stand-alone honeypots, running for about five non-contiguous hours on a major cloud provider. For benchmarking and discussion, we selected the initial subset of the process logs. This subset was further divided into training, validation, and testing sets with a rough 60/20/20 split based on host, quantity of logs generated, and the activity logged. Only the test set includes malicious activity, as expected when training unsupervised anomaly detection models. Table 2 provides a summary of the dataset, while Table 3 and Table 4 provide a description of the kernel-process and DNS log features, respectively.
In this section, we first detail the log collection methodology, followed by a description of the overall dataset. The final subsection discusses potential research questions that could be investigated using our dataset. 2.1
The challenge of crafting a honeypot is two-fold: make it tempting enough to infiltrate, and track activity without being detected. The former is typically done by providing “free” resources to an attacker, i.e., easily accessible computer power. Our implementation currently runs hosts 2https://github.com/jinxmirror13/BETH_Dataset_Analysis d c u u o r 6 6 6 6 3 l t
s C a
m N o 6 6 6 6 3 h
r c e i n l v e p n b e r u t r f n
I k r t o n w e t m e n im E S e o K r
P l a i t s i t in s r o se y r
t u h 8 4 1 31 23 U i o f
v o e i 6 6 6 6 3 g l t n g c i #
d in A u
l S c t
in se d , b % % %
s u 8 6 6 % et ic go s .8 .5 .5 00 l f 6 6 6 1
o 6 1 1 a f e f r a 3 3 3 3 6 s
s -c r ec % e T o R r
p 8 8 le ssse l-reen ethng ,44316 ,76898 ,76898 ,7014 ,1940 n e k L 7 1 1 ,1 ,0 r c 3 6 6 6 3 1 8 ,ed sseed secrdo secrdo slfow sseperdm secrdo iirttssecca settaaD iiranngT iiltaadoV isetngT sebtuoT ltaoT t r r r r r S z t m n n no co n h e a p a iS tS co ilo io il B io lc o B il ill ilm G:1 ill rea N G m m 2 18 m8 n
e 4 2
G + 7
&
] ] 9 e 11 ] ]0 [2 m [
[8 [3 2 a 8 1 n m R N 99 99 D 02 H irsao ecnh ithw seta 1PA 19D -DK ISD EBT p b d ta R D SL X m le e A K
D
D
N C
S
I : 2 e l b a T
l n a t timestamp Seconds since system boot processId* Integer label for the process spawning this log threadId Integer label for the thread spawning this log parentProcessId* Parent’s integer label for the process spawning this log userId* Login integer ID of user spawning this log mountNamespace* Set mounting restrictions this process log works within processName String command executed hostName Name of host server eventId* ID for the event generating this log eventName Name of the event generating this log argsNum* Length of args returnValue* Value returned from this event log (usually 0) stackAddresses memory values relevant to the process args List of arguments passed to this process sus Binary label as a suspicious event (1 is suspicious, 0 is not) evil Binary label as a known malicious event (0 is benign, 1 is not) with a single ssh vulnerability: any password will be accepted to login. This is enough to entrap automated, naive SSH scanning and brute force methods. In the future we plan to deploy hosts with other vulnerabilities, with which we hope to observe other attack vectors.
To log activity in real time, each host runs Ubuntu 18.04 with a Docker container [
The first sensor is set to listen and exfiltrate relevant data packets resulting from internal activity. In particular, this sensor tracks all Linux system calls to create, clone, and kill processes. The second sensor logs network traffic, specifically DNS queries and responses from all processes on the host machine, including those processes running within the hosted Docker containers. When the desired packet appears, it is parsed out to pre-defined fields and then transmitted to a collection server.
These process and network data packets are gathered on a separate node hosting a message queue server. Sensor clients connect to this server and transmit their individual data packets as events occur, without batching. This allows the collection and re-transmission of data at an atomic level. The data is then stored locally in two correlated files with respective logs, allowing analysts to understand both local and network effects that occur jointly. 2.2
The dataset is composed of two sensor logs: kernel-level process calls and network traffic. As the initial benchmark subset only includes process logs, this section only covers these; a description of the network logs can be found in Table 4.
Each process call consists of 14 raw features and 2 labels, described in Table 3. These largely contain categorical features with some containing large integers, necessitating further processing. Thus, for our benchmarking, we converted several fields to binary variables based on field expertise, as described in Appendix A. We note that this conversion process creates more duplicates due to the removal of critical fields, such as processName and args, which have no widely-accepted encoding scheme for any machine learning model to utilise. There are some inherent duplicates which we keep within the dataset prior to conversion for the accuracy of real logs and to not limit users of the data.
Each record in the process logs and DNS logs was manually labelled suspicious (sus) or evil to assist post-training analysis3. Logs marked suspicious indicate unusual activity or outliers in the data distribution, such as an external userId with a systemd process4, infrequent daemon process calls (e.g. acpid or accounts-daemon), or calls to close processes that we did not observe as being started. Evil5 indicates a malicious external presence not inherent to the system, such as a bash execution call to list the computer’s memory information, remove other users’ ssh access, or un-tar an added file. Events marked evil are considered “out of distribution,” as they are generated from a data distribution not seen during training.
A subset of the kernel process logs were divided into a typical 60/20/20 split for training, validation, and testing, based on the amount of activity recorded, and evil labels from each host. As is typical in unsupervised anomaly detection, our training and validation sets are each composed of logs generated from multiple hosts which only recorded activity from the OS and cloud infrastructure management. Because there are no signs of infiltration in these process logs, activity resulting from these hosts is considered benign and the events generated by the OS and cloud infrastructure management to be “in-distribution”.
Our initial testing dataset contains all activity on a single exploited host, including its 3These labels are provided from a single reviewer and should not be solely relied upon for industry systems. 4In the scope of our honeypot any external user traffic is suspicious, but some of these events were initiated by the cloud provider.
5We note that presence in this dataset does not constitute a “conviction”, as no real damage was done. (a) Fitted on Training and transformed Testing (b) Same as (a), coloured with evil labels
OS and cloud infrastructure management. The first attack we logged is an attempt to setup a botnet; more details are available in Appendix B. The full dataset contains other malicious activity performed within our honeypots, including cryptomining and lateral movement (between servers). These various attacks may also be compared to answer alternative research questions with our data, as discussed in Subsection 2.3. As each exploited host only contains a single staged attack, with no artificial noise in the benign activity, BETH is one of the cleanest cyber security datasets available to distinguish between malicious and benign activity.
As an initial investigation of the data, we visualised the (pre-processed) training and testing
datasets with uniform manifold approximation and projection (UMAP) [
The BETH dataset could answer other cyber security questions than just OoD analysis. Unlike
logs within real deployed systems that contain no labels for malicious events, our BETH dataset
contains (recently recorded) real data with labels. One use for this dataset would be to profile the
attacker or malware’s behaviour [
In this section, we provide anomaly detection benchmarks on our initial subset of logs. We chose
both standard anomaly detection baselines [
Robust covariance [
Given the scale of the dataset, we also considered DL-based OoD detection methods. In
particular, DoSE uses summary statistics (such as the log-likelihood or posterior entropy),
calculated over the training set by a trained generative model, in order to characterise the “typical
set”. In our work we train a variational autoencoder (VAE) [
As seen in Table 5, the iForest performs best at differentiating sus events from the benign in our testing dataset. We attribute this to the small set of discrete features available and the conspicuous nature of the attack. DL-based models are less competitive on these sets of features, but have the potential to deal with more raw categorical and even text-based features, which we hope to explore in future work. Finally, we note that imbalanced labelling, summarised in Table 6, necessitates further investigation of what each model predicts is benign or not. 4
In this paper, we present our BETH cybersecurity dataset for anomaly detection and OoD analysis. The data was sourced from our novel honeypot tracking system recording both kernellevel process events and DNS network traffic. It contains real-world attacks in the presence of benign modern OS and cloud provider traffic, without the added complexity of noisy artificial user activity. This cleanliness is ideal for OoD analysis, such that each host in the dataset only contains one or two data-generating distributions. We also include baselines for anomaly detection trained on a subset of the BETH dataset: robust covariance, one-class SVM, iForest, and DoSE-SVM (with a VAE).
For future work, we plan to collect and publish more attacks for alternative testing datasets. This will also allow investigations in comparing attacks or perhaps testing in a continual learning setting.
We are thankful for the many incredible academics who supported this work. Dr. Arinbjörn Kolbeinsson inspired the included UMAP visualisations. Professor Sergio Maffeis provided incredible advise on the presentation and edits to the paper. Also, thank you to the reviewers for their positive feedback.
This work was also supported by The Alan Turing Institute, during an author’s participation in the Turing Enrichment Program (October 2021 - June 2022), under the EPSRC grant EP/N510129/1.
arXiv preprint [10] A. Krizhevsky, V. Nair, and G. Hinton. Cifar-10 (canadian institute for advanced research).
URL http://www.cs.toronto.edu/~kriz/cifar.html.
A
In this section, we provide more details on the raw features in the dataset, as well as preprocessing suggestions, which we used in our baselines:
timestamp: We left this field out to consider the dataset as a sample from a distribution rather than time series. We recommend using the values as they are or also leave them out, depending on the method chosen.
processId: Process IDs 0, 1, and 2 are meaningful since these are always values used by the OS, but otherwise a random number is assigned to the process upon creation. We recommend replacing processId with a binary variable indicating whether or not processID is 0, 1, or 2.
threadId: While this value did not appear useful in our analysis, it might suggest how to link process calls if obfuscated in the system. No conversion is recommended at this time. parentProcessId: Same as processId, the same mapping to a binary variable should suffice. userId: The default in Linux systems is to assign OS activity to some number below 1000 (typically 0). As users login, they are assigned IDs starting at 1000, incrementally. This can be altered by a user, but none of the current logs gave evidence an attacker did this. We used a binary variable to indicate userId < 1000 or userId 1000. Alternatively, one could use an ordinal mapping that buckets all userId < 1000 at zero and then increment upwards for each new user. Also, no more than four logins were viewed per host in our current datasets.
mountNamespace: This field is somewhat consistent across our hosts and determines the access a certain process has to various mount points. The most common value for this feature is 4026531840 or 0xF0000000, which is for the mnt/ directory where all manually mounted points are linked. It is noted that all logs with userId 1000 had a mountNamespace of 4026531840, while some OS userId traffic used different mountNamespace values. We converted this feature into a binary mapping for whether or not mountNamespace = 4026531840.
processName: This is a string field of variable length (ranging from one to fifteen characters). When manually analysing the data, this was a critical field in conjunction with the eventName.
For our baselines, we refrained from utilising this, although the model should be given an encoding of this using a hash or ability to learn a useful encoding on its own. It is noted that attackers can easily change the processName to override a benign one so their traffic looks regular. This was not observed within the current dataset.
hostName: This field is useful for grouping the dataset into related subsets of data generated from the same honeypot. The name of the host name does not transfer between the model development subsets described in this paper.
eventId: Linux systems assign an integer corresponding to the eventName. We include this field as-is for our benchmarks.
eventName: Event names uniquely map to eventId, so we drop it from training. argsNum: This raw feature is included as-is, since, at this time, adequately parsing args requires either more sophisticated pre-processing or a more complex ML model.
returnValue: This is also called the exit status and can be used to determine whether a call completed successfully or not. Mappings for this can vary, as this value is decided between the parent and child process. We mapped returnValue into three values based on the common usage of the field: -1 when negative (error), 0 when zero (success), and 1 when positive (success and signalling something to the parent process).
stackAddresses: It is difficult to clearly relate this feature during manual analysis and the large values within a variable size list make processing automatically difficult without encoding or learning an extra embedding. Thus this field was dropped from training our baselines.
args: There are many options in this variable list of dictionaries. For simplicity, we refrain from utilising any of these values. However, more features can and should be created for future work.
Finally, BETH contains two binary, manually-labelled flags: sus and evil. Examples and the explanation of how these labels were created are detailed in Section 2.2. A breakdown of these labels within the subsets for model development is given in Table 6.
This testing dataset was extracted from a single honeypot. The overall attack appears to be instantiating a botnet node. The timeline of the events recorded is provided in Figure 2; this is the typical attack pattern. The server is initially accessed, it may run a few setup operations in the environment to send some details to its Command and Control (C2) for a customised attack, it sleeps for a while, intermittently checks in with the C2 or a clock, and then launches its attack until complete.
In this case, the honeypot is first accessed at around 411 seconds from booting. Several thousand lines are then recorded in the process logs denoting the setup of the new user profile. This happens within milliseconds; these are detailed logs of everything the OS does during the short pause before the terminal opens for user entry when ssh-ing into a server. This user then sleeps, pausing all user activity for some number of seconds. This appears to happen at random intervals—a more sophisticated technique than using consistent intervals—of which the latter would give a clear signature of automated activity.
After a few minutes, it sets up an SFTP server to download a file called dota3.tar.gz (known botnet malware) and scopes out the system using common commands such as whoami, ls, and cat /proc/cpuinfo. After about 7.5 minutes, it unpacks the dota3.tar.gz and runs over a hundred threads, all attempting to connect with different servers.