Many open sources of binaries, including malware, have emerged in the landscape in recent years. Their quality compares very favourably with commercial sources, as emphasised by Thibaud Binetruy (Twitter influencer under a pseudonym, Société Générale CERT, 2020): “Integrating operational threat intel in your defense mechanisms doesn't mean buying Threat Intel. You can start by using the [mass] of open source indicators available for free.” Some are provided by oficial sources (Abuse.ch, with data supplied by the Swiss national CERT, among others), while others are made available in more obscure ways, sometimes anonymously (VirusShare, VX-Underground, etc.). Our examination of these sources underlines the wide disparity in quality and quantity between them. We have had to take this diversity into account in our research, designing a dedicated platform that enables us to supply information to our binary analysis products and to conduct daily analyses of correlations between and within malware families on a large scale. This work can then be applied to concrete cases such as Babuk, Ryuk and Conti. We have been able to highlight links for these families by immediately identifying correlations, with additional manual analysis then confirming the genealogy of the samples precisely.
When an attack takes place, there is an urgent need to find out who the attacker is, and how they operate. But the heat of the moment is not the time to examine the details. This is when you need the services of a threat intelligence analyst, whose job it is to answer these questions. Analysts work over long time scales, and their constant monitoring gives them an up-to-date understanding of how families of attackers are evolving. But they themselves need to be wellequipped in order to pick up the weak signals providing news about these groups. How can we use public data (files, virus databases, etc.) to derive a better understanding of attackers’ TTPs (tactics, techniques and procedures)? How can a threat intelligence analyst’s tools provide useful information about how to respond to an incident?
To our knowledge, the academic sector has not so far covered the subject of the large-scale
automation of end-to-end malware analysis. But the problems we have encountered in our
work have led us to the conclusion that the whole chain (from collection to analysis) must
be taken into account if we want to achieve a high-quality end result. Here we report on the
infrastructure and techniques we have put in place to respond to this problem – how can we fuel
cyber threat intelligence with precise knowledge about malware families and their relationships,
reliably, quickly and fully automatically? We will use a correlation method based on concept
code, as this is well suited to correlating large numbers of binaries. This is not an exclusive
property of this method, though – other methods can also deliver similar results [
This section provides a detailed inventory of the open sources of binaries we have identified. The list is not intended to be exhaustive, and focuses primarily on data accessible publicly with no subscription or registration, combined with a validation or peering process (except VirusTotal, which is listed because it is a standard reference).
Our research has led us to develop capacity for the mass processing of open sources, from automatic collection to generating the daily Docker images that constitute our products. In this section, we will present our methods and the collection infrastructure, which are built around three major contraints: • respect for sources: it is important to prioritise local cache capacity as much as possible, avoid sending multiple requests to servers and respect a reasonable rate of requests. Failing to do this leads to multiple risks, including being blacklisted from a server, but above all it has a negative impact on the ecosystem and endangers the model of open-source data sharing that contributes so efectively to the cyber ecosystem. • control data volumes: our goal is to collect and store relevant data without exceeding our storage capacity. This makes it important to take account of future needs and availability of sources: what is it essential to conserve? What needs to be duplicated to guard against data loss? What can we allow ourselves to collect again in the future if necessary? • manage computing time and deadlines: the goal is to be able to ingest data continuously.
But we still need to to have the possibility (availability of data) and the capacity (volume/computing time) to submit all the files for reanalysis if we update the computation method we use. Otherwise, we run two risks: having to reduce the update frequency, or having to deliberately exclude data that is too old, which could ultimately impair our detection and analysis capacity.
To collect malware, we use a pipeline (see Figure 1) consisting of three major stages: ingestion, tagging and conversion into concept code. These stages take place in the form of a CI flow on a GitLab CI orchestrator. The orchestrator starts jobs regularly and automates the whole process.
False
Download the archive
Deocfotmheprfeilsession Forfieleach Ingestion
SHA256 computation ...
True Reject the archive colUeRcLtion 1E0vemriyn
URL exists in
Elasticsearch? ...
SHA256 exists?
The first phase, which manages ingestion, is itself divided into the following steps: download, extraction, filtering and finally storage of the data.
Download During this first step, we use web collection (Beautiful Soup 1) to find download links in the pages of various data sources. We implement a number of mechanisms to avoid overloading the source servers: minimal parallelisation, resuming interrupted downloads from the point where they stopped, using mirrors (or torrent links) whenever possible, etc. 99% of the time, the links point to archives – we maintain a cache of downloaded URLs to avoid ever having to download the same data twice. These measures enable us to limit our own load, but also and above all the load on the source services.
Extraction The extraction phase is specific to each data source. It allows us to extract additional information depending on the source. For example, for files downloaded from VXUnderground, we can retrieve information on the malware type when it is included in the file name. The way the links are organised on the collection page also allows us to extract the attacker family (APTxx, etc.), which we can use in the tagging phase. For VirusTotal (if we have a subscription): the archives contain the malware and a JSON file summarising the analysis, which can be used to deduce metadata for the family.
Filtering For data filtering, we rely on two types of hash: SHA256 and SSDEEP. The first is a very quick way of making sure we are not downloading an archive or malware that has already been ingested. This obviously avoids duplicates, particularly among diferent data sources.
SSDEEP, meanwhile, allows us to compute the level of similarity with other malware in the database: as our correlation algorithms are fairly flexible, we prefer to avoid downloading malware that is too similar to what we already have, whereas other chains may need maximum numbers of samples to be collected. The SSDEEP similarity computation is relatively complex, especially as the database grows larger. We thus carry out a preliminary partial search in an Elasticsearch database using an N-gram tokeniser of size 7. This tokeniser selects a very limited subset of the full database, on which a full comparison computation is carried out. With the results of this query, if the score exceeds a set threshold, the malware is rejected on the basis that it is too similar to a strain that has already been ingested.
Storage The storage infrastructure needs to allow us to store data of varying types and to retrieve it on demand. We also need to be able to upgrade it easily as the project progresses. We chose to store metadata in an Elasticsearch cluster and data on a Nexus server (using only a tiny proportion of its features).
1https://en.wikipedia.org/wiki/Beautiful_Soup_(HTML_parser) 3.2.2. Tagging We used two types of taggers for the tagging phase: metadata and YARA rules. The first type uses the source metadata (typically, MalwareBazaar tags or VX-Underground families, for example) while the second uses a series of YARA rules to identify specific families. Each piece of malware stored passes through both taggers to complete the malware information in Elasticsearch.
Once the binaries have been collected and identified, the final transformation is carried out to convert them into what we call concept code. This process is only relevant for the malware correlation method based on concept code, and is not necessary if other methods are to be used instead. It could be replaced by a disassembly process, for example, which extracts the basic blocks from each binary to enable a similarity calculation based on the basic block hashes. Another option is to extract character strings in order to detect binaries sharing the same strings and thus produce a correlation score based on this metric. We will not examine these methods in detail here, because they are highly dependent on capacity within the organisation wishing to implement a binary analysis automation process (in terms of both computation time and access to the diferent technologies required).
To automate this sequence, we use the following infrastructure: • A GitLab server for orchestration, triggering, managing and supervising jobs. • Elasticsearch to store the metadata on the collected files.
• Nexus to store all the binaries.
The collection sequence is asynchronous, making it easier to resume operation if it needs to be interrupted and restarted later. Link searches for each source are carried out every 10 minutes, and tags are updated hourly. So far we have 8,262,249 items of processed malware, of which 4,430,562 are non-rejected and 3,282,091 are tagged. This represents around 3 TB of malware, taking one to two months to ingest. Updates currently take about an hour for a 5 GB archive.
The archives can vary widely in size, from a few megabytes to several hundred gigabytes – the largest archive we have had to ingest was 800 GB. At this size, it becomes dificult to fully decompress the archive, which would require around twice the disk space available. This makes it essential to plan the process from the beginning for “flow”-based operation, partially decompressing and analysing the archive as we progress. Our collection process has provided us with archives in a variety of formats: zip, rar, 7z and tar. Partial decompression works very well for the zip, tar and rar formats. For 7z, partial decompression still works, but very slowly. We have thus chosen to decompress these archives in full before analysis, as we have not come across 7z archives that were too large for this mode of operation. 3.4.2. Tagging With the two taggers we use, we only manage to tag a fairly small proportion of the malware (a few percent), mostly using metadata and in a few cases using YARA rules. Additional work (which we have not yet carried out) will be needed to find better YARA rules, and probably more of them. This proves the significant limitations of this detection method, which ultimately is based on searching for signatures. For our purposes, this is not a major stumbling block (our algorithms are very tolerant of changes, so only a small number of samples are needed for efective detection), but it is important to be aware of it if large-scale use is to be made that requires a higher percentage of correctly identified malware.
For tagging, we are planning to use the correlation developed at GLIMPS (based on concept code) to compare malware with specific families. This is possible as long as the pipeline is completely reliable and fairly stable, because the volumes involved are not compatible with the use of this sequence for every update. This will thus be done later and will resolve the tagging problem.
Once the data has been collected from the public sources and processed in our infrastructure, we can compute the links between pairs of malware threats, family by family and between families. Computations of similarity between two binaries are carried out using the GLIMPS correlation technology (see Figure 2), based on code conceptualisation. The benefit of code conceptualisation lies in its robust handling of modifications, including changes in compilation chains or compiler versions. However, as described in section 3.2.3, other methods exist for computing similarity, and for this reason we will focus more closely on the use of the results obtained.
In addition, the computation is very fast – a matrix of 1,000 binaries (a million comparisons) can be produced in less than a minute, and the computation time is linear relative to size rather than quadratic, which most other methods are, enabling computation on families of several tens of thousands of binaries. Finally, this method is logarithmic when adding a binary to a collection (adding a row/column depends logarithmically on the size of the matrix, which is very beneficial in terms of computation time for large families). By carrying out the computation on a large scale, we can automatically generate correlation matrices for each family and study the evolution of the samples by reordering these matrices by value (i.e. performing a clustering). Binary
Tagging
Transformation and code analysis
Large scale comparisons
Similar binaries Clustering
Matrix
Clustering
Similarity scores
Machine Learning scoring model
To highlight the links for and between malware families, we automatically define family clusters that group similar binaries together (see Algorithms 1 and 3). These are constructed from a correlation matrix generated by analysing pairs of binaries to calculate their similarity scores.
The rows and columns of the matrix represent the binaries, each identified by the first five characters of its hash. In Figures 3.b and 4, each identifier is also prefixed by the cluster to which it belongs. The value scale ranges from 0 (blue) to 1 (red), and represents the similarity score between each pair of binaries, where 1 is the maximum value, representing perfect similarity between two binaries (typically the same file or a copy). The absence of any colour indicates that no similarity was detected between two binaries. An example of a correlation matrix (before clustering) is shown in Figure 3.a.
Once the matrix has been constructed, we use it as the starting point for two clustering
algorithms, which we use in combination: DBSCAN[
Algorithm 1: best_clustering
input : Correlation matrix M 1 c1 ← optimal dbscan clustering on M ; 2 c2 ← optimal agglomerative clustering on M ; 3 if silhouette_score(c1) ≥ silhouette_score(c2) then 4 return c1; 5 else 6 return c2; 7 end The optimum clusterings are identified by testing a set of values for both methods.
• Agglomerative clustering: number of clusters to create ∈ [2, |M |/2]. • DBSCAN: parameter epsilon (maximum distance between two samples for one to be considered a neighbour of the other) ∈ [0.02, 0.55] (pas de 0.02).
Algorithm 2: improved_majority_vote input : binary x, correlation Matrix M , number of neighboors K output : Associated cluster 1 xnn, lnn ← find binaries and clusters of the K-NN of x in M ; 2 mnn ← mean correlation value on xnn by cluster label from lnn; 3 vmax ← max(mnn); 4 if ∃! v ∈ mnn | v = vmax then 5 return cluster with mean correlation value = vmax; 6 else 7 8 end
return cluster with mean correlation value = vmax that is the most represented in lnn;
2The computation time for these 69 samples was around 2.5 seconds. method does not identify any other links between the Babuk binaries. For the Ryuk and Conti families, this method highlights only very few similarities, and none between the families. However, there is a strong link between the f0278, 3cd91, c5032 and fa13f binaries in the Conti family, which belong to cluster c5 in Figure 4. We can thus find shared similarities using either of the methods, but we can identify more links between the binaries, including between families, with our method for computing similarity.
The correlation matrices used to generate these clusterings (and other examples) are available from our GitHub repository3. In each case, we provide the input data (i.e. the matrix of correlations between samples) and images of the clusterings obtained with our method.
The clusterings generated in this way make the threat intelligence analyst’s work easier by specifying which grouping a piece of malware belongs to and any links to other groups or binaries. This cuts down on analysis time, because the analyst can target the relevant binaries for comparison. The output from this section will be used as the basis for a threat intelligence analysis in section 5 to illustrate this and validate the proposed approach.
This kind of analysis is not the only potential application. The clusterings can also be used, for example, to automatically retag binaries (based on the clusters found), providing better ground truth for learning (malware detection, malware type classification, etc.). Ultimately, this will make it possible to improve the results of trained algorithms.
It is important to note that a clustering can easily be evolved by automatically injecting a new binary. We just calculate its similarity scores with the binaries already clustered and then allocate it a cluster using majority voting (for example) among the clusters of its N closest neighbours.
To validate the approach presented above, it is important to compare it with a manual analysis
of the samples. We thus cross-referenced these results against a reverse engineering analysis
conducted with standard tools such as IDA4, which enables users to browse through a
disassembled binary and gain a detailed understanding, sometimes at the cost of many hours of
work.
5.1. Babuk
Babuk is a computer virus in the ransomware family. It neutralises the victim’s computer by
encrypting all the user’s files, or even the whole system if the ransomware is run with the
right permissions. The ransomware appeared on VirusTotal for the first time on 22 December
2020 under the name of Vasa Locker, and the group operating it calls itself Babuk. The group
appeared on RaidForums on 2 January 2021[
These core ransomware functions are altered as the malware evolves, corresponding very closely to the correlation matrix presented in the previous section – see Figure 3.b.
In May 2021, the group changed its name to PayloadBin and stopped its ransomware
activity[
Manual analysis through reverse engineering tells us that the first cluster (named C1) uses: • a queueing system that traverses the file system recursively, • the exchange of a shared private key using the public key algorithm curve255196, • mostly the hc-128 algorithm for data encryption, except for one sample that uses Salsa20.
The second cluster, C2, is older in terms of compilation date (the matrix is ordered based on cluster size, with no reference to date of appearance) and shows the use of less refined algorithms (particularly for traversing the file system and encryption): • a parallelised mechanism for traversing the file system with threads based solely on the disks present. This results in a very slow encryption process. The process for traversing the file system stops at the 16th level of the folder structure. If files are located below this level, they will not be afected. • The key exchange algorithm is ECDH, using an open-source library7 directly, or RSA. • The private key algorithm is Salsa20.
However, the samples 704a0 and 58ccb provide a vital information – they make a link between the two clusters, enabling us to establish connections between the two series of ransomware. This link is even more visible when we compare it with the compilation timeline (see Figure 6). Finally, we note the presence of an outlier, the sample 3dda3: it has functions in common with cluster 1, but they have been altered during compilation. It thus appears as separate from the cluster, but with weak links to two samples.
7https://github.com/kokke/tiny-ECDH-c/blob/master/ecdh.c
This initial analysis of a group with a limited number of samples validated the approach based on creating cross-correlation matrices and exploiting the information this provides. We then successfully applied this method to a larger number of samples from two malware groups (Ryuk and Conti), for which links seem to exist according to the literature.
Like Babuk, Ryuk belongs to the malware family of ransomware. Unlike Babuk, however,
Ryuk’s developers are not the direct operators of the malware. Ryuk is operated by afiliates,
as demonstrated by several articles including an ANSSI publication in March 2019[
Conti is another member of the same family. It emerged in 2020. In the same way as Babuk, Conti’s operators publish a website listing the victims and the data encrypted by the attackers. Again, as with Babuk, this technique aims to put pressure on the victim and guarantee that as high a ransom as possible will be paid. Unlike Babuk, however, these two families use techniques to obstruct reverse engineering analysis. In the case of Conti, for example, no calls are made via the import table – instead, strings of characters are decoded on the fly to reconstruct the table and the addresses of the stored functions so that they can be called later.
The correlation matrix reveals the evolution of these techniques and how they have been
implemented in the two families. The techniques are not specific to these malware families,
and other viruses have already used them. However, here it is the implementations of these
techniques that were compared, revealing the link between the Conti and Ryuk families. A
literature review also confirmed that links have been demonstrated showing that the same
coders are responsible for the two pieces of ransomware[
A traditional threat intelligence approach can provide global information about the threat associated with a specific group. But as we have seen, a single name (Ryuk, Conti, Babuk, etc.) can hide many variants of the same threat. When an incident occurs and a company or organisation needs to respond efectively, it is vital to know which sample the threat is associated with. The continuous collection of malware and binaries from open sources, together with clustering using a high-performance correlation technology, enables us to keep our exhaustive knowledge of attacker groups and their various malware releases constantly up to date. This means that when threat intelligence analysts are called in to support the teams responding to an incident, they can guide the response diferently depending on whether the binary they are dealing with focuses on encryption, lateral movement or leaking data.
In addition, an efective incident response usually requires decision-makers to be involved in the process, enabling them to be kept up to date with the progress of the response and to manage the crisis. As well as the standard CTI indicators, which are purely technical markers (YARA rules, hashes, etc.), the approach presented here ofers the prospect of automatically proposing a concrete, explicit view of where a particular sample found in the IT system is positioned within the “galaxy” of malware used by the attacker. This communication tool helps make the threat more tangible and more easily understood by business executives, which means it can be taken into account more thoroughly in decision-making processes.
We are very grateful to everyone who contributes to the sharing of data and information about cyber threats. The efort to guard against these threats is currently faced with two challenges: the availability of relevant data, and the capacity to process this data on a time scale compatible with an efective response. Our activity is automation, but it relies on data. We would also like to thank everyone who has supported GLIMPS since its creation and enabled this work, including the French armed forces ministry, the DGA, the Cyberdéfense Factory, Poool, Rennes Métropole, the BDI and all our partners.