Existing research on malware classification focuses almost exclusively on two tasks: distinguishing between malicious and benign files and classifying malware by family. However, malware can be categorized according to many other types of attributes, and the ability to identify these attributes in newly-emerging malware using machine learning could provide significant value to analysts. In particular, we have identified four tasks which are under-represented in prior work: classification by behaviors that malware exhibit, platforms that malware run on, vulnerabilities that malware exploit, and packers that malware are packed with. To obtain labels for training and evaluating ML classifiers on these tasks, we created an antivirus (AV) tagging tool called ClarAVy. ClarAVy's sophisticated AV label parser distinguishes itself from prior AV-based taggers, with the ability to accurately parse 882 diferent AV label formats used by 90 diferent AV products. We are releasing benchmark datasets for each of these four classification tasks, tagged using ClarAVy and comprising nearly 5.5 million malicious ifles in total. Our malware behavior dataset includes 75 distinct tags - nearly 7 × more than the only prior benchmark dataset with behavioral tags. To our knowledge, we are the first to release datasets with malware platform, exploitation, and packer tags.
The malware ecosystem is both massive and diverse. Novel malware emerges regularly and
existing malware is continually being updated to add functionality or improve evasion [
We have assembled a collection of four benchmark datasets named MalDICT (Malware Datasets for Infrequent Classification Tasks), with each dataset supporting a diferent, underrepresented malware classification task. MalDICT is being published in the hope that it will encourage increased awareness and study of these tasks. We are also publishing benchmark results after training two standard malware classifiers on each of these datasets. This will enable researchers to compare the performance of their own models against a baseline and against each other. The four benchmark datasets within MalDICT are: 1. MalDICT-Behavior: 4,317,241 files tagged according 75 common malware categories and malicious behaviors. 2. MalDICT-Platform: 963,492 files tagged according to 43 common file formats, operating systems, and programming languages. 3. MalDICT-Vulnerability: 173,886 files tagged according to 128 common vulnerabilities exploited by malware.
4. MalDICT-Packer: 252,148 files tagged according to 79 common malware packers.
1. TR/Andromeda.B MalDICT was tagged by combining out- 2. Trojan.Win32.Andromeda.xyz puts from multiple diferent antivirus 3. Backdoor.Androm.99 (AV) products. We developed a custom tool named ClarAVy for this, which we 4. Win32/Gamarue.1234 describe in Section 2. For the remainder 5. Trj.Gamarue!1.23W of this section, we introduce terminology 6. W32.TrojanDownloader.Wauchos.A about AV products and survey related AV- 7. Trojan.TR/Backdoor.Gen based taggers. 8. Malware (ai Score=99)
When detecting a file as malware, an 9. BehavesLike:W32/Zbot-abc
AV product will produce an output called
an AV label. An example AV label Figure 1: Fictitious AV scan report for a file. Labels 1-6
is Trojan:Win32.Androm.abc. Each correctly classify this file as belonging to the Andromeda
portion of the label describes a charac- family, which has the aliases “Androm", “Gamarue", and
teristic of the file that the AV detected as “bWutauincdhiocsa"t.eLsatbhealt7thdeoeits bneoltoansgssigtno tthhee mTraolwjaanreanadfaBmacilky-,
malicious, such as its behavior, file for- door categories. Labels 8-9 are heuristics, and Label 9
mat, or family. In some cases AV labels incorrectly classifies the malware into the Zeus family.
may also include a threat group the
malware is attributed to, a vulnerability the malware exploits, or the packer the file was packed with
[
AVClass is the seminal work on labeling malware using AV scan reports [
To our knowledge, the following AV-based taggers can assign non-family tags to malware.
EUPHONY creates a graph with weighted edges between related reports, forms clusters from
communities in the graph, and assigns labels based on the majority family, category, or file
type in the cluster [12]. SMART distills AV scan reports into a multi-label representing the
ifle’s behaviors [ 13]. However, it supports only 11 malicious behaviors. It is the tagging method
used by the SOREL dataset. AVClass2 is an update to AVClass by its original creators [
1
5 Raw Scan Reports
AV Label Parsers The purpose of ClarAVy is to clarify the 2 noisy outputs from a collection of AV 4 products into simple, easy-to-interpret tmaaglsw.aItreisinthMeatloDoIlCuTs.eFdigfuorrel2absehloinwgs tthhee ATloiakseens 3 TaTxookneonmy major stages of the ClarAVy architecture.
First, ClarAVy ingests a corpus of AV scan Figure 2: ClarAVy Architecture. reports and tokenizes each label. A lexical category is assigned to most tokens (i.e., whether the token indicates a malware behavior, packer, etc.). Once all scan reports are processed, ClarAVy reviews any tokens with incomplete or ambiguous parsing and attempts to assign a global lexical category to them. Next, ClarAVy identifies tokens which are aliases of each other. Finally, ClarAVy re-processes all of the scan reports, this time using its newly-obtained information about lexical assignments and token aliases. For each scan report, ClarAVy outputs a token ranking and the lexical category each token was assigned to. In the remainder of this section, we provide technical details for each of these stages.
Ranking Token Transformer
A taxonomy of the diferent lexical categories which ClarAVy can assign to tokens is provided in Table 1. The FAM, PLAT, and BEH lexical categories are analogous to the "Family", "Platform", and "Type" fields in the CARO malware naming scheme. Some AV labels in- Table 1: Taxonomy of Tokens in AV Labels clude additional information, such as BEH The malware category or behavior the packer used to pack the file (PACK) PLAT The OS, file format, or programming language or a vulnerability the malware exploits VULNA vulnerability exploited by the malware (VULN). Some AV labels may not be able PACK The packer used to pack the file to be fully parsed, necessitating the PRE FAM The malware family that the file belongs to and UNK lexical categories. The sufix of SUF A sufix token at the end of the AV label the AV label is assigned the SUF lexical PRE Ambiguous, but not a FAM or SUF token category.
UNK A token whose lexical category cannot be determined
While parsing AV scan reports, ClarAVy
identifies tokens within AV labels and at- Exploit:Win32/MS08067.xyz
tempts to assign each token to a lexical Dfoerlimmaitter nToorkmenailzizee,
category in its taxonomy. ClarAVy tok- TOK:TOK/TOK.TOK [exploit, win32, ms08067, xyz]
enizes AV labels by splitting them on "de- Determine
limiter" tokens, which are any tokens in parser
the label that are non-alphanumeric. We def parse_fmt(tokens):
call the sequence of delimiter tokens in if rer.emtautrcnh([rB"E^HM,S[P0L-A9T],+$V"U,LNt,okSeUnFs][
After identifying the delimiter format ploit:Win32/MS08067.xyz". ClarAVy identifies the for an AV label, ClarAVy selects an ap- delimiter format of the label and selects a parsing propriate parsing function. The pars- fbuentwcteioenn fAoVritl.abTehliss isnimthpilse dpealrisminigtefrufnocrtmioantdtihsatitncgounisthaiens ing function attempts to assign a lexi- VULN or FAM tokens. The assigned lexical categories cal category to each token in the label. indicate that this AV label detects exploitation of the Most of ClarAVy’s parsing functions do MS08-067 vulnerability in Windows. this by applying regular expressions and boolean logic to the tokens. Figure 6 shows how theTokAen VTransformer label Exploit:Win32/MS08067.xyz is parsed. This label is applied to malware which exploits the MS08-067 vulnerability. The delimiter format for this label is TOK:TOK/TOK.TOK, where TOK represents the locations that tokens may appear in the label. AV labels that have this delimiter format always have a CAT token in the first position, a TGT token in the second position, and a SUF token in the fourth position. A token in the third position of the label may either be a VULN or a FAM token, and the parsing function uses a regular expression to determine which lexical category should be assigned to it.
ClarAVy includes parsing functions for 882 diferent delimiter formats across 90 AV products - over 8,000 lines of Python code in total. A few AV products use only a single delimiter format, while others have dozens. Parsing functions range from trivial to complex, depending on how standardized the labels of an AV product are. We identified the most common delimiter formats used by each AV product to ensure maximal coverage. Then, we manually implemented and verified each parsing function to ensure that the lexical categories ClarAVy assigns to tokens are accurate.
In some cases, ClarAVy’s parsing functions cannot assign lexical categories to some tokens in an AV label. This is most often due to there being no programmatic way to distinguish tokens indicating behavior, platform, vulnerability, and/or packer from each other or from other generic tokens. The parsing function assigns these tokens the PRE lexical category to indicate that there is some ambiguity, but it is not a FAM or SUF token. More rarely, there are edge cases where tokens are truly ambiguous. The parsing function assigns the UNK lexical category to these tokens.
After all scan reports are parsed for the first time, ClarAVy attempts to determine the lexical category of each token that had some parsing ambiguity. Even if a token is assigned PRE or UNK by one parsing function, it may appear in other AV labels where it can be parsed correctly. If a token is unanimously assigned to a lexical category (not counting PRE and UNK), it is permanently assigned to that category when ClarAVy is used in the future.
ClarAVy is provided with a default wordlist that maps tokens to their lexical categories. This
wordlist was generated by running ClarAVy on ≈ 40 million AV scan reports from VirusTotal
[15]. We describe how we collected these scan reports in Section 3. Users can add to or alter
this wordlist if they have diferent preferences. For example, we manually removed the “trojan"
and “win32" tokens from the BEH and PLAT categories, respectively, since they are nearly
ubiquitous in AV scan reports. In particular, AV products tend to use the “trojan" tag generically
rather than for actual trojan malware [
During its next stage, ClarAVy attempts to identify tokens that have identical meaning. In FAM tokens, aliases may have very distinct spellings (e.g. Andromeda, Gamaue, and Wauchos in Figure 1). However, we observe that aliases for tokens in most other lexical categories generally have similar spellings. Our approach to token alias resolution uses a metric based on edit distance in addition to token co-occurrence percentage. We identify two diferent classes of token aliases, which we call trivial aliases and parent-child aliases.
We say that a pair of tokens are trivial aliases if they share a lexical category and are nearly identical in spelling, where a single minor edit can transform one token into the other. For example, if one token can be transformed into a second token adding extra digit or character to the end (e.g. “backdoor" and “backdoor0"), ClarAVy considers the pair to be trivial aliases. Additionally, ClarAVy uses a small list of common substrings that are frequently observed at the beginning and end of tokens. If two tokens are identical except for the substring, it assigns them as aliases. Trivial aliases are very frequent in AV scan data, and this procedure is simple but highly efective.
ClarAVy also recognizes aliases from token pairs which have a “parent-child" relationship. Two
conditions must apply to satisfy this relationship. First, the less common token (the child)
must co-occur with the more common token (the parent) in a suficient percentage of scan
reports. Additionally, a score based on edit distance must be suficiently high. We adapt metrics
from Sebastián and Caballero [
A high co-occurrence percentage indicates that the child token may be related to the parent token, but other factors (such as spurious correlations between the outputs of diferent AV products) may cause dissimilar tokens to co-occur frequently. To reduce false positives, we also require that pair of tokens is similar in spelling. Let len() be the number of characters in token . We define a custom edit score based on edit distance:
edit_score(, ) = 1 − edit_dist(, ) / min(len(), len( ))
Afterwards, we apply two heuristics to the edit score which we frequently observe in token aliases. If the shorter token is a substring in the longer token, or if the two tokens are anagrams, the edit score is capped at a minimum of 0.75. ClarAVy uses threshold parameters (0.6 by default) and (0.5 by default) to control parent-child aliasing. If edit_score(, ) >= and co_occur(, ) × edit_score(, ) >= , then has a parent-child relationship with .
Pairs of tokens with parent-child relationships are not immediately considered to be aliases. This is because a token may share a parent or child relationship with multiple other tokens. Algorithm 1 shows how ClarAVy identifies aliases from the set of tokens with parent-child relationships. Let be a list of all known tokens within the same lexical category (e.g. all of the BEH tokens) sorted by token frequency, descending. At each iteration of the algorithm, the curAlgorithm 1 Parent-Child Alias Resolution Require: Sorted list of tokens 1: function Alias_Resolve( ) 2: ← ∅ 3: for ∈ do 4: ← ∅ 5: ← Queue 6: . enqueue() 7: while not . is_empty() do 8: ← . dequeue() 9: if ∈/ and ∈/ then 10: ← ∪ 11: for ∈ . children do 12: . enqueue() for ∈ do ← ∪ if ̸= then
resolve_alias_pair(, ) rent token is treated as the canonical name for all of its aliases (i.e., all of its aliases will be renamed to the current token). A set of all “descendants" of the current token is created by recursively visiting child tokens.
Then, each descendent token is assigned as an alias of the current token, provided that it has not been assigned a diferent alias already.
By default, the canonical name for a group of aliases is the most frequently-appearing token in the provided AV scan report dataset.
ClarAVy comes with a text file which maps tokens to their canonical alias names. It was generated by using the previously-described alias resolution process on a dataset of ≈ 40 million AV scan reports from VirusTotal [15]. Clar- 13: AVy users can easily customize this mapping 14: with their own alias pairs by editing the file. 15: Furthermore, the canonical names in the alias 16: mapping have priority over automaticallyidentified canonical names, allowing users to set their naming preferences.
After assigning lexical categories to tokens and after resolving aliases, ClarAVy parses all scan reports a second time. This time, tokens with known aliases are replaced with their canonical names. Additionally, tokens which were previously assigned PRE or UNK may receive a more informative lexical category. For each AV scan report, ClarAVy outputs a ranking the BEH, PLAT, PACK, and VULN tokens in the report. Each token is given a score based on the number of times it appears in the scan report, adjusted for known correlations between AV products. PRE, SUF, and UNK tokens in the scan report are considered generic and discarded. Accurately ranking FAM tokens is much more challenging and is a target of our future work.
The existence of correlations between AV products is well-known in the malware analysis
industry. Leading causes include AV products sub-licensing their engines to others, AV products
owned by the same company, and AV products “copying" another product’s detection results
[
Invincea
nProtect Agnitum
Sophos
TACHYON
AVware AntiVir
Avira Baidu-International AvastMobile
ALYac GData BitDefenderTheta
ViRobot Emsisoft Qihoo 360 BitDefender F-Secure
Arcabit FireEye
MicroWorld eScan Ad-Aware K7AntiVirus
eTrust K7GW
TotalDefense
Zonealarm
Kaspersky Cyren F-Prot
TrendMicro
TrendMicro-Housecall McAfee
McAfee-GW-Edition available information, shown in Figure 5. Like prior work, we observed that the main sources of correlation were due to AV products owned by the same company (e.g. McAfee and McAfee-GWEdition) and AV products licensing their technology to others (e.g. ZoneAlarm previously used Kaspersky’s engine). In our dataset, there are 11 diferent AV products which use the BitDefender engine to varying degrees, often in combination with their own detection technologies. ALYac and Qihoo 360 use multiple other engines. We also noticed multiple instances of AV products being renamed or acquired by other companies (e.g. Commtouch was renamed to Cyren and aquired F-Prot).
VirusBuster
NOD32 ESET-NOD32 Baidu Avast AVG
Commtouch BEH ransom 12 worm 5 ClarAVy assigns a score to each token based on the number of times it appears in the scan report. This approach as- PLAT sumes that if multiple independent AV products output the same token, then the token is likely to be an accurate tag for VULN CVE-2017-0144 2 the file. For this assumption to be valid, correlations between AV products must be accounted for. If two or more AV prod- PACK UPX 3 ucts with known correlations output the same token, ClarAVy combines them into Figure 6: Example ClarAVy output for a malicious file. a single “vote". It is tagged as having ransomware and worm behavior.
A threshold parameter is used to The file exploits the CVE-2017-0144 vulnerability and is control the minimum token score al- packed with UPX. No platform tags are identified. lowed in the ranking that ClarAVy outputs. Tokens with fewer than votes are excluded from the ranking. Higher values of decrease the amount of noise in the outputs, but may also cause correct tokens to be omitted from the ranking. can be set separately for each lexical category. By default = 5 for BEH and PLAT tokens and = 1 for VULN and PACK tokens. As we later show in Section 3.2, agreement of at least 5 independent AV products has a very low false positive rate. VULN and PACK tokens are much less frequent than BEH and PLAT tokens, but also much less noisy. Any threshold above = 1 for these lexical categories would cause a high false negative rate.
ClarAVy was developed with AV scan reports for 40,307,433 malicious files from vdiorwusnloader jsscript ccvvee__22001104__26536382 unpsixs chunks 0 through 465 of the VirusShare riskware html cve_2017_17215 nsanti corpus [18]. We queried the VirusTotal adware pe cve_2017_11882 upack API for these files between Feb. and Apr. dropper vbs cve_2010_0188 aspack 2023 to get these reports [15]. When de- pua hllo cve_2017_0199 themida veloping each parsing function in Clar- packed msil cve_2010_2586 nspack AVy, we randomly selected 10,000 AV la- worm pdf cve_2010_2586 pecompact bels with the corresponding delimiter for- rbeadcikrdeoctoorr amnudlrtoiid ccvvee__22001122__00550077 fvsmgprotect mat from this dataset. After creating a parsing function, we performed a brief visual inspection of the resulting tokens and lexical assignments to ensure they were correct. After finishing the entire ClarAVy implementation, we ran it on these ≈ 40 million AV scan reports with default settings ( = 5 for BEH and PLAT tokens and = 1 for VULN and PACK tokens). Then, we inspected lexical categories that ClarAVy assigned to each token and the alias mapping which it created. We manually verified both of these, correcting any errors if necessary. We identified 1,307 aliases for 92 malware behaviors, 194 aliases for 47 file-related tokens, and 53 aliases for 24 packers. The ClarAVy output included 134 distinct BEH tokens, 91 distinct PLAT tokens, 440 distinct VULN tokens, and 90 distinct PACK tokens. The ten most common tokens of each type are listed in Table 2.
PACK
ClarAVy’s comprehensive collection of parsing functions distinguishes it from other AV-based
taggers. Most prior work uses hard-coded lists and/or heuristic methods for assigning tokens to
lexical categories [
We experimentally test ClarAVy’s ability to tag malware according to behavioral attributes. We
do this using the SOREL dataset, which has 9,919,065 malicious PE files labeled according to 11
separate behavioral tags [
Labeled malware data which can be used to evaluate ClarAVy is extremely limited. With the
exception of SOREL, nearly all malware reference datasets either use benign/malicious labels
or family labels [
Suppose a dataset of malicious files = {1, 2, ...}, where is the number of files in the dataset. Let ∈ be the set of files that a malware tagging tool assigns tag . For example, could be the set of files that ClarAVy assigns the “ransomware" tag to. A malicious ifle may be assigned multiple tags. Then, let = {}1≤ ≤ partition , where is the set of malicious files belonging to family . Each file is assigned to exactly one family. Then, for each predicted label , let = ⋃︀ , if |∩|
|| ≥ 0.5. This constructs a set of malicious =1 ifles from malware families where at least 50% of files have tag predicted. Using this, we can define metrics which are analogous to per-tag Precision and Recall: (, , ) = (, , ) =
Under these definitions, Precision measures the “noisiness" of a tag. It penalizes instances where a tag is assigned to a file, but where most files in its family are not associated with that tag. Conversely, Recall measures coverage of a tag within malware families. It penalizes instances where a family is likely to be associated with that tag, but there are files within that family where the tag is not assigned.
We acknowledge that there are flaws in this Table 5: MOTIF Evaluation (Micro Avg.) evaluation strategy. It is possible that fami- ClarAVy AVClass2 lies which are truly associated with a tag may be “missed" due to incorrect predictions. Fur- Precision .828 .694 thermore, it does not necessarily confirm that Recall .912 .796 predicted tags are correct (although we be- F1-Measure .868 .741 lieve this is likely in most instances due to the high precision observed in our previous experiment using the SOREL dataset). However, Table 6: MOTIF Evaluation (Weighted Avg.) in the absence of better-labeled data, we be- ClarAVy AVClass2 lieve that this is a reasonable approach for Precision .880 .723 measuring ClarAVy’s tagging consistency. Recall .912 .796
We then used ClarAVy and AVClass2 to tag F1-Measure .896 .758 VirusTotal reports for the MOTIF dataset. MOTIF contains 3,095 malware samples from 454 families, labeled with ground-truth confidence. ClarAVy and AVClass2 were run on default settings, and any tags in AVClass2’s family (FAM) or unknown (UNK) taxonomy were discarded because they are not output by ClarAVy. Additionally, the AVClass2 “windows" tag was discarded, since it appears in nearly all scan reports in MOTIF and ClarAVy treats it as generic. We computed Precision, Recall, and F1-Measure for each label using the method described above. Results are show in Tables 5 and 6. ClarAVy clearly outperforms AVClass2 in this experiment. ClarAVy tags malware within the same family more consistently and there is less tagging noise.
To build the MalDICT datasets, we ran ClarAVy on 40,307,433 VirusTotal reports for the malware in VirusShare chunks 0-465. We reviewed the tags that ClarAVy had assigned to these files and observed significant class imbalances. To account for this, we discarded tags which were too rare and down-sampled tags which were very common. BEH tags with less than 1,000 instances, PLAT tags with less than 500 instances, VULN tags with less than 100 instances, and PACK tags with less than 50 instances were not included. Tags which were too frequent were randomly down-sampled, so that they were no more than 100× more common than the minimum threhsold in the training set and no more than 25× more common than the minimum threshold in the test set.
Depending on the lexical category, we selected between two diferent methods for dividing ifles into a training and test set. For MalDICT-Behavior and MalDICT-Platform, we selected from ifles in VirusShare chunks 0-315 for the training set and from VirusShare chunks 316-465 for the test set. Chunks 0-148 contain 131,072 files each, and the remaining chunks contain 65,536 ifles each. This supports an approximately 80% - 20% train-test split. More recent VirusShare chunks contain newer forms of malware that do not appear earlier in the dataset [18]. MalDICTBehavior and MalDICT-Platform test sets contain malware added to the VirusShare corpus between July 2018 and Apr. 2023, while all of the malware in the training sets were added prior to July 2018. The first chunks were added to VirusShare in 2012, but we are aware of malware in VirusShare which was uploaded to VirusTotal in 2006 [18, 19]. Since new types of malware are continually being observed, This enables a temporal train-test split which simulates model performance on novel types of malware that do not appear in the training set. With up to nearly a five-year gap between the chunks in the training and test sets, MalDICT-Behavior and MalDICT-Platform can unveil whether a malware classifier is robust against out-of-distribution (OOD) data from a "future" time period.
The training and test sets for MalDICT-Vulnerability and MalDICT-Packer do not use a temporal split. This is because VULN and PACK tags are much less frequent, and we observed that multiple VULN and PACK tags only appear in the dataset over a short time interval. If we had used a temporal split, this would have resulted in a number of tags appearing in only the training set but not the test set or vice-versa. Instead, we used a stratified 80% - 20% train-test split to ensure even proportions of tags in the training and test sets.
Table 7 lists the number of files and number
oDfuuenitqouesotamges infilestheocfocuurrrMinaglDiInCTmdualttiapsleets. PBVleuahltnafoevrrimaobrility
datasets, MalDICT includes 5,457,778 unique Packer
malicious files in total. We are releasing the
ifle hashes and ClarAVy token rankings for each of these files. Since they are a subset of the
VirusShare corpus, the corresponding malicious files can be downloaded by any malware analyst
who has been granted a VirusShare login [18]. Furthermore, we are releasing the disarmed
executable and EMBER raw metadata for each PE file in MalDICT. Files were disarmed by
zeroing out the OPTIONAL_HEADER.Subsystem and FILE_HEADER.Machine fields in their PE
headers, which is the same method used by SOREL and MOTIF [
We now survey potential sources of bias in the MalDICT datasets. To counteract the questionable accuracy of individual AV labels, we chose to only include BEH and PLAT tags for which there is consensus between at least five uncorrelated AV products [ 21, 22]. We judge this to be necessary for tag accuracy, but we are aware that it may cause a selection bias [23]. Omissions or errors in AV labeling is in of itself another source of bias in our dataset [21, 22, 24]. However, there is no other source which can be reasonably used as a source of malware labels at this scale [25]. Finally, the methods we used for selecting files to include in MalDICT changed the tags and their distributions from what would be observed in the wild. We have already justified these design choices earlier in this section.
We are releasing models trained on four MalDICT datasets. These models serve as measurements
of baseline ML performance in this problem space. We selected MalConv2 and LightGBM as
baseline models, since they are similar to those used by other notable datasets [
Our first baseline model is MalConv2, a convolutional neural network that ac- PRreeccailslion ..645912 ..771580 ..898286 ..880917 cepts raw file bytes as input [ 26]. Mal- F1-Measure .560 .733 .906 .846 Conv2 was also used as a baseline model ROC-AUC .929 .965 .995 .987 by the MOTIF dataset, and the original MalConv was used by the EMBER dataset Table 9: MalConv2 Evaluation (Weighted Avg.) [20, 28, 29, 26]. Our baseline MalConv2 Behavior Platform Vulnerability Packer model truncates any files greater than 1MB to 1MB in order to lessen GPU mem- PRreeccailslion ..469127 ..771782 ..898286 ..880912 ory usage. The remaining hyperparame- F1-Measure .512 .718 .903 .842 ters were kept as the MalConv2 defaults. ROC-AUC .896 .960 .995 .980 Then, we trained MalConv2 classifiers on the four MalDICT training sets using eight NVIDIA RTX 6000 GPUs in parallel. The MalConv2 model for MalDICT-Behavior was trained for 33 epochs (approximately 24 hours), and the other three MalConv2 models were trained for 100 epochs each. When a file is provided as input to the baseline MalConv2 model, it outputs the probability of each tag being associated with that file. For the purposes of computing Precision, Recall, and F1-Measure, we consider an output greater than or equal to 0.5 as the threshold for predicting a tag. We used standard definitions of Precision, Recall, and F1-Measure for these results rather than our own definitions in Section 3.3. MalConv2 results on MalDICT are shown in Tables 8 and 9. MalConv2 displays good performance when classifying malware by vulnerability and packer. Performance is lower when classifying by behavior and by platform, and this is almost certainly due to the temporal train-test split present in MalDICT-Behavior and MalDICT-Platform, but not in MalDICT-Vulnerability or MalDICT-Packer.
The EMBER feature vector format has be- Behavior Platform Packer come a de-facto standard for represent- Precision .177 .682 .783 ing malware in the Windows Portable Recall .555 .953 .948 Executable (PE) file format [ 28]. Like the F1-Measure .268 .795 .857 EMBER, SOREL, and MOTIF datasets, we ROC-AUC .897 .958 .992 use a LightGBM classifier trained on EMBER feature vectors as a baseline model.
MalDICT includes malware that is not Table 11: LightGBM OvR Evaluation (Weighted Avg.) in the PE format, as well as files with Behavior Platform Packer corrupt or invalid PE header fields. EMBER vectors for these files could not be Precision .363 .889 .844 computed, so they were excluded from Recall .555 .953 .948 this experiment. Nearly all of the mal- F1-Measure .385 .911 .884 ware in MalDICT-Vulnerability are mali- ROC-AUC .805 .955 .991 cious scripts rather than PE files, so we did not train a LightGBM model on this data. In the remaining three MalDICT datasets, there were a small number of tags which contained little to no PE files, and they were also excluded. Since this is a multiclass, multilabel problem, we trained one-versus-Rest (OvR) LightGBM classifiers on each tag for 100 iterations each. Results are displayed in Tables 10 and 11.
The LightGBM classifier performed well on MalDICT-Platform and MalDICT-Packer, but
was extremely poor at classifying malware in MalDICT-Behavior. MalConv2 performance on
MalDICT-Behavior was substandard as well, but not to such an extent. MalConv2 and LightGBM
both performed worse on MalDICT-Behavior than the SOREL dataset’s feed-forward neural
network (FFNN) baseline classifier, which achieved ROC-AUC scores above 0.97 for all 11 of
its behavioral tags [
To our knowledge, MalDICT includes the first public malware datasets labeled according to platform, vulnerability, and packer. It also includes the most diverse public dataset of malware labeled by behavior, containing over 4.3 million malicious files and 75 distinct behavioral tags. We are releasing the file hashes and tags for the nearly 5.5 million malicious files in MalDICT. We are also releasing the EMBER raw features and disarmed executable files for all of the malware in MalDICT with the PE format. All of the malware in MalDICT can be obtained by researchers who have been granted to the VirusShare corpus.
Additionally, we are publishing ClarAVy, the tool that was used to accurately tag the malware in MalDICT. With support for 90 diferent AV products and 882 diferent AV label formats, ClarAVy ofers more comprehensive parsing than any other AV-based malware tagging tool. ClarAVy can extract tags from tens of millions of AV scan reports, which can then be used to train production malware classifiers.
Our baseline classifier results indicate that there is significant room for improvement on all four tasks that MalDICT supports, especially malware behavior classification. The development of a classifier with strong performance on MalDICT-Behavior would represent a major success towards resisting concept drift over years of malware evolution. It is our hope that these contributions will facilitate and encourage further study of atypical malware classification tasks, fostering improved understanding and defense. [12] M. Hurier, G. Suarez-Tangil, S. K. Dash, T. F. Bissyandé, Y. Le Traon, J. Klein, L. Cavallaro, Euphony: Harmonious unification of cacophonous anti-virus vendor labels for android malware, in: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), 2017, pp. 425–435. doi:10.1109/MSR.2017.57. [13] F. N. Ducau, E. M. Rudd, T. M. Heppner, A. Long, K. Berlin, Automatic malware description via attribute tagging and similarity embedding, arXiv preprint arXiv:1905.06262 (2019). [14] P. García-Teodoro, J. A. Gómez-Hernández, A. Abellán-Galera, Multi-labeling of complex, multi-behavioral malware samples, Computers & Security 121 (2022) 102845. [15] VirusTotal, ???? https://www.virustotal.com/, Last accessed on 2023-06-09. [16] A. Mohaisen, O. Alrawi, Av-meter: An evaluation of antivirus scans and labels, in: S. Dietrich (Ed.), Detection of Intrusions and Malware, and Vulnerability Assessment 11th International Conference, DIMVA 2014, Egham, UK, July 10-11, 2014. Proceedings, volume 8550 of Lecture Notes in Computer Science, Springer, 2014, pp. 112–131. URL: https://doi.org/10.1007/978-3-319-08509-8_7. doi:10.1007/978-3-319-08509-8\_7. [17] R. J. Joyce, E. Raf, C. Nicholas, Rank-1 similarity matrix decomposition for modeling changes in antivirus consensus through time, 2021. arXiv:2201.00757. [18] Virusshare.com - because sharing is caring, ???? https://virusshare.com/, Last accessed on 2023-06-09. [19] R. J. Joyce, T. Patel, C. Nicholas, E. Raf, Avscan2vec: Feature learning on antivirus scan data for production-scale malware corpora, arXiv preprint arXiv:2306.06228 (2023). [20] R. J. Joyce, D. Amlani, C. Nicholas, E. Raf, Motif: A large malware reference dataset with ground truth family labels, 2021. arXiv:2111.15031. [21] M. Botacin, F. Ceschin, P. de Geus, A. Grégio, We need to talk about antiviruses: challenges & pitfalls of av evaluations, Computers & Security 95 (2020) 101859. URL: http://www.sciencedirect.com/science/article/pii/S0167404820301310. doi:https://doi. org/10.1016/j.cose.2020.101859. [22] A. Mohaisen, O. Alrawi, M. Larson, D. McPherson, Towards a methodical evaluation of antivirus scans and labels, in: Y. Kim, H. Lee, A. Perrig (Eds.), Information Security Applications, Springer International Publishing, Cham, 2014, pp. 231–241. [23] P. Li, L. Liu, D. Gao, M. K. Reiter, On challenges in evaluating malware clustering, in: S. Jha, R. Sommer, C. Kreibich (Eds.), Recent Advances in Intrusion Detection, Springer Berlin Heidelberg, Berlin, Heidelberg, 2010, pp. 238–255. [24] A. Mohaisen, O. Alrawi, M. Mohaisen, Amal: High-fidelity, behavior-based automated malware analysis and classification, Computers & Security 52 (2015) 251 – 266. URL: http://www.sciencedirect.com/science/article/pii/S0167404815000425. doi:https://doi. org/10.1016/j.cose.2015.04.001. [25] A. Kantchelian, M. C. Tschantz, S. Afroz, B. Miller, V. Shankar, R. Bachwani, A. D. Joseph, J. D. Tygar, Better malware ground truth: Techniques for weighting anti-virus vendor labels, in: ACM Workshop on Artificial Intelligence and Security, 2015. [26] E. Raf, W. Fleshman, R. Zak, H. S. Anderson, B. Filar, M. McLean, Classifying sequences of extreme length with constant memory applied to malware detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, pp. 9386–9394. [27] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, T.-Y. Liu, LightGBM: A Highly Eficient Gradient mohaisen2013 Decision Tree, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., 2017, pp. 3146–3154. URL: http://papers. nips.cc/paper/6907-lightgbm-a-highly-eficient-gradient-boosting-decision-tree.pdf. [28] H. S. Anderson, P. Roth, Ember: An open dataset for training static pe malware machine learning models, 2018. URL: https://arxiv.org/abs/1804.04637. doi:10.48550/ARXIV. 1804.04637. [29] E. Raf, W. Fleshman, R. Zak, H. S. Anderson, B. Filar, M. McLean, Classifying sequences of extreme length with constant memory applied to malware detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, pp. 9386–9394.