We have released a malware collection in TREC style. It contains scripts, html documents and text files extracted from binary files of about 650K malwares. The objective of the project is to index, extract significant features and classify them into malware families. At this aim we will also release a TREC style set of queries for classification tasks. In this abstract, we briefly describe the test collection, the project aims and the problems underlying the use of text mining and information retrieval techniques to malware classification.
Malware analysis is a growing research area but with still many open
problems [
655,361 #Tokens
153,587,253 4,222,109,462
mensions 21GB
The malware collection was obtained from the VirusShare.com project. VirusShare was born in 2011 with the aim of collecting, indexing and freely sharing malware samples for analysts, researchers and the computer security community. At the moment the site provides about 30 million malwares. We have downloaded a portion containing 655,361 of the most recent malware files (i.e. collected by VirusShare in the last 6 months). Initially the collection was about 286 compressed GB (11 zip archives). We extracted the text part and formed a collection of about 66GB of uncompressed data, or equivalently of about 30GB of compressed data, and obtaining 21 GB of indexes. The text part of the whole collection should therefore contain approximately 14 TeraBytes of compressed data for 9TB of Terrier’s indexes.
The malware collection has been subjected to the following operations: Text extraction The text part was first extracted through the unix script strings. From 286GB of compressed data, 30 GB of compressed data were obtained.
Tagging The collection was then labeled by introducing the following new tags: DOCNO, DOC_TYPE, SCRIPT, TYPE_SCRIPT, CDATA, DOMAIN, SOURCE, RUN_MODE, RUN_MODE_NOT. The labeling module was obtained through a set of syntactic rules of the regex type. We get all domains and URLs, to index them separately, trasforming strings such as http://xxx.yy.201.53/guodanpi/dhnchia.exe into: < DOMAIN > xxx.yy.201.53</DOMAIN > and <SOURCE> xxx.yy.201.53/guodanpi/dhnchia.exe</SOURCE> The new tags contain the following information: – DOC: Initial malware tag, and DOCNO, the malware file identifier that contains the MD5 hash value of the file; DOC_TYPE, a tag for a html document or not. – SCRIPT, the tag that encloses a script, and CDATA, the tag that contains data in a document of markup type. – SOURCE: a complete URL address, and DOMAIN: an internet domain https://VirusShare.com These are Terrier indexes, with both inverted and direct files http://terrier.org. – TYPE_SCRIPT, a tag with the type of script VBscript, javascript etc., and RUN_MODE, RUN_MODE_NOT, tag for Win32 or DOS etc.
Pre-classification Documents are pre-classified according to their type and script (javascript for example). In Figure 1 there are two examples of processed malware.
Tokenization Finally, the collection was indexed by considering the text within tags, e.g. Html, script etc., and any sequence of characters as meaningful tokens. The separator characters between indexed tokens were any blank type character according to the UTF8 encoding. Therefore, the typical punctuation characters (comma, points, etc.) were not considered separators. This extremely loose and permissive indexing makes possible the simultaneous presence in the dictionary as indexed terms (that is, in the lexicon) both words belonging to the natural language of text documents (such as those with the DOC_TYPE tag equal to html) and commands or tokens belonging to scripts (within the SCRIPT tags). 4.2 billion tokens were obtained with 154 million unique terms across the whole collection. The average frequency of terms is 3.58 occurrences per malware, much higher than the average word frequency in natural language texts.
Indexing Thanks to the tagging operations, one can activate or disable any possible tag during the indexing to obtain either dedicated indexes to only scripts (SCRIPT), to only textual parts, to only URL addresses (SOURCE), to only the domain (DOMAIN) or to a combination of these tags. Thanks to the DOC_TYPE and TYPE_SCRIPT fields you can also obtain statistics on the distribution of malware in the different types of documents.
Malware classification is a hard task because very few labeled training sets exist for detection. Clustering can be an alternative because it can automatically aggregate malware into different groups. However, the very first step forward for a classification task would be to separate malware files from non-malware ones in a very large document collection. We have released a TREC like collection of malware text that was still missing. This collection allows for researchers from different areas to cooperate and apply IR, Data Mining and Big Data technologies to the problem of malware classification. At this aim a set of classification queries will be soon released.