Software vulnerability represents a flaw within a software product that can be exploited to cause the system to violate its security. In the context of large and evolving software systems, developers find it challenging to identify vulnerable functions efectively when a new vulnerability is reported. Existing studies have underutilized vulnerability reports which can be a good source of contextual information in identifying vulnerable functions in source code. This study proposes an information retrieval based method called Vulnerable Functions Detector (VFDetector) for identifying vulnerable functions from source code and vulnerability reports. VFDetector ranks vulnerable functions based on the textual similarity between the vulnerability report corpora and the source code corpora. This ranking is achieved modifying conventional Vector Space Model to incorporate the size of a function which is known as the tweaked Vector Space Model (tVSM). As an initial study, the approach has been evaluated by analysing 10 vulnerability reports from six popular open-source projects. The result shows that VFDetector ranks the actual vulnerable function at first position in 40% cases. Moreover, it ranks the actual vulnerable function within rank 5 in 90% cases and within rank 7 for all analysed reports. Therefore, developers can use these results to implement successful patches on vulnerable functions more quickly .
report corpora. In addition, programming language spe- out manual expert intervention. Recently, DL based
techcific keywords is removed for generating code corpora. niques [
In experiments, as an initial study ten Common Vul- vulnerability detection treats a program’s source code as nerabilities and Exposures (CVE) reports are chosen ran- text and employs natural language processing techniques domly from six open source GitHub repositories. Based to identify vulnerabilities. Russell et al. [3] introduced on the commit link available in reports we crawled the TokenCNN model, which utilizes lexical analysis to the corresponding projects before the vulnerability was acquire source code tokens and employs a Convolutional patched. The result analysis shows that VFDetector ranks Neural Network (CNN) to detect vulnerabilities. the vulnerable functions at the first position in 40% cases, Li et al. [4] proposed Vuldeepecker, a method that whereas it ranks the actual vulnerable function within collects code gadgets by slicing programs and transforms top 5 in 90% cases and within top 7 in 100% cases. them into vector representations, training a Bidirectional
It is evident from the results that VFDetector performs Long Short Term Memory (BLSTM) model for vulnerapromisingly in detecting vulnerable functions against a bility recognition. vulnerability report in a large scale software systems. It Zhou et al. [5] introduced µVulDeePecker, which enis also observed that in Top 5 and Top 7 ranking, the hances Vuldeepecker by incorporating code attention functions which ranks above the actual vulnerable func- with control dependence to detect multi-class vulneration are the related functions of that vulnerability which bilities. However, the performance of these text-based acquires higher similarity. It guides a developer to patch approaches is limited because they rely solely on static those related functions too in order to mitigate that vul- source code analysis and do not account for the program nerability from the system. semantics.
The remainder of this paper is structured as follows: Graph based methods: To address the limitations Section 2 gives an overview of previous studies on vul- of text-based methods, researchers have turned to dynerability detection at file level or function level. Section namic program analysis to convert a program’s source 3 describes our methodology for detecting vulnerable code semantics into a graph representation facilitating functions in a project. Section 4 reports our experimental vulnerability detection through graph analysis. Zhou et ifndings and the analysis thereof. Section 5 demonstrates al. [6] introduced Devign which employs a graph neural the threats to validity of our work. Section 6 motivates network for vulnerability identification. This approach future research directions and concludes this paper. includes a convolutional module that eficiently extracts critical features for graph-level classification from the learned node representations. By pooling the nodes, a 2. Related Work comprehensive representation for graph-level classification is achieved.
In recent years, the research community has directed Cheng et al. [7] introduced a diferent approach named significant attention toward the issue of vulnerability Deepwukong which divides the program dependency detection, primarily due to the complex challenges it graph into various subgraphs after distilling the program presents. The existing body of literature has introduced semantics based on program points of interest. These numerous methodologies in response to these challenges. subgraphs are then utilized to train a vulnerability deThese methods can be classified into three distinct cate- tector through a graph neural network. While these gories based on the degree of automation: manual, semi- graph-based techniques prove more efective in identifyautomatic, and fully automatic techniques. ing vulnerabilities but it is important to note that their
Manual techniques rely on human experts to create scalability is worse than text based methods due to large vulnerability patterns. However, all patterns can not be number of graph nodes in complex program. generated manually, which leads to reduced detection efi- Exploring the existing literature, it is evident that textciency in practical scenarios. In contrast, semi-automatic based methods lacks in incorporating program semantics techniques involve human experts in the extraction of while graph-based methods achieve high accuracy considspecific features like API symbols [ 9] and function calls ering source code semantics but have scalability issues in [10], which are then fed into traditional machine learn- complex scenarios. Moreover, due to the underutilization ing models for vulnerability detection. Full-automatic of contextual information like vulnerability reports with techniques utilize Deep Learning (DL) to automatically source code existing methods fails to detect complicated extract features and construct vulnerability patterns with- vulnerabilities in real-world projects. Because whenever a new vulnerability is reported in a system it is hard to detect in which function the vulnerability exist as the system consist of huge volume of functions. Before using vulnerable reports as a source of contextual information in existing methods, it is important to verify whether vulnerable functions can be identified efectively using these reports. Moreover, identifying vulnerable functions using vulnerability reports can play an efective role to minimize the search space in existing methods.
to assess similarity with vulnerability report corpora.
Therefore, the precision of code corpora generation directly impacts the precision of matching, consequently enhancing the accuracy of vulnerability localization. In this step all the folders are extracted from a system with their corresponding C/C++ files. From each of these files all functions are extracted automatically in individual C ifles which ensures function level analysis. For Example: 3. Methodology CVE-2014-2038 of Linux version 3.13.5 consist of 15,675 ifles which has total 229,682 functions.
This study proposes an approach which detects vulner- This stage generates a vector of lexical tokens by
doable functions from huge volume of files of a large soft- ing lexical analysis on every source code file. There are
ware system using vulnerability reports. The overall unnecessary tokens in source code which do not contain
process of this approach consist of three distinct steps any vulnerability related information. These tokens are
and those are Source Code Corpora Generation, Vulnera- discarded from source code such as programming
lanbility Report Corpora Generation, Ranking Vulnerable guage specific keywords (e.g., int, if, float, switch, case,
Functions. Each of these steps encompasses a series of struct), stop words (e.g., all, and, an, the). Many words in
tasks as illustrated in Figure 1. At first, all files and their the source code may include multiple words. For
examcorresponding functions are extracted from a particular ple, the term "addRequest" consists of the keywords "add"
version of a software system. Then these source code is and "Request". Mutiwords are separated using multi word
processed to create code corpora. Similarly vulnerability identifier. Furthermore, statements are divided according
report is processed to produce report corpora. Finally, to certain syntax-specific separators like ‘ , ’, ‘=’, ‘(’, ‘)’, ‘{’,
similarity between the report and code corpora is mea- ‘}’, ‘/’, and so on. WordNet2 is used to derive each word’s
sured using tweaked Vector Space Model (tVSM) to rank semantic meanings because a term might have more than
the vulnerable source code functions. one synonym. In specific cases, developers and Quality
Assurance (QA) personnel may employ diferent
termi3.1. Dataset nology, even though they are referring to the same
scenario with equivalent meanings. For example, the term
We used the benchmark dataset Big-Vul1 developed by ‘finalize’ may have multiple synonyms such as ‘conclude’
Fan et al. [
CSV_Dataset
A software vulnerability report contains information like description about the vulnerability, severity rating, vulnerability identifier (CVE-ID), reference to additional sources which gives valuable insights about a software vulnerability issue. However, these reports can also include irrelevant terms such as stop words and words in various tenses (present, past, or future). To refine vulnerability reports, pre-processing is necessary. In the initial stage of vulnerability report corpora creation, stop words are eliminated. We apply WordNet Lemmatizer, similar to what’s used for source code corpora generation, to generate refined report corpora containing only relevant terms.
based on the textual similarity between the query (report corpus) and each of the function in the code corpus. Vulnerable functions are ranked by applying tVSM. We employ tVSM, which modifies the Vector Space Model (VSM) by emphasising large-scale functions. In traditional VSM, the cosine similarity is used to measure the ranking score between the associated vector representations of a report corpus (r) and function (f), according to Equation 1.
(, ) = (, ) =
Here, ⃗ and ⃗ are the term vectors for the
vulnerability report (r) corpus and function (f) corpus
respectively. Throughout the years, numerous adaptations of
the tf(t,d) function have been introduced with the aim of
enhancing the VSM model’s efectiveness. These
encompass logarithmic, augmented, and Boolean modifications
of the traditional VSM [
(, ) = 1 + = ( # ) Here, represents the frequency of a term appearing in a function , # denotes the total count of functions within the search space, signifies the overall number of functions that include the term . Thus in equation 4 each term weight is calculated as follows: ℎ∈ = ( ) × ( ) = (1 + ) × ( # ) The VSM score is calculated using equation 5. (2) (3) (4) calculate the length value for each source code function based on the number of terms contained within the function. Here we apply the normalized value of ’#terms’ as the argument for the exponential function − . The normalization process is defined in Equation 7.
Let z denote a set of data, with and representing the maximum and minimum values of z term, respectively. The normalized value for z term is determined as: () = − − Considering the above analysis, tVSM score is calculated by multiplying the weight of each function, denoted as x(terms), with the cosine similarity score represented by cos(r, f), as described in Equation 8: (, ) = () × (, ) (7) (8) (, ) = √︀∑︀(1 + log ) × 2 × ∑︁ (1 + log ) × (1 + log ) × 2× ∈∩ 1
puted, a list of vulnerable functions is arranged in de1 scending order of scores. The function with the highest √︀∑︀(1 + log ) × 2 similarity score is positioned at the top of the ranked list.
(5)
Traditional VSM tends to give preference to smaller
functions when ranking them, which can be problem- This section provides information on the practical
impleatic for large functions because they may receive lower mentation, the criteria used for evaluation and
experisimilarity scores. Past research [
dataset which contains large scale vulnerability reports of C/C++ code from open source GitHub projects. Other C/C++ datasets can also be used. Based on the highest number of vulnerabilities reported, we choosed top six well known projects from this dataset which are Chrome, Linux, Radare2, ImageMagick, Tcpdump and FFmpeg as shown in Table 1. As the selected projects are opensource in nature and are hosted on GitHub, serving as the primary platform for storing code and managing version control. It allows us to extract all essential commits for our analysis. Additional information about the repositories can be found in Table 2. As an initial study, VFDetector was evaluated using ten vulnerability reports from these six open-source projects which are chosen randomly from the dataset. Table 1 lists the analysed project name, CVE ID of report, and the source code link.
To measure the efectiveness of the proposed vulnerability detection method, we use the Top N Rank metric. This metric signifies the count of vulnerable functions ranked in the top N (where N can be 1, 5, or 7) in the obtained results. When assessing a reported vulnerability, if the top N query results include at least one function that corresponds to the location where the vulnerability needs to be addressed, we determine that the vulnerable function is detected successfully. Table 2 includes ten vulnerability reports from six open source projects with their number of commits, total files, total functions, actual vulnerable functions name and finally VFDetector ranking in Top N ranked functions in output. The results of Table 2 shows that among the ten CVE reports VFDetector ranks the actual vulnerable function at the 1st position for four (40%) reports which are CVE ID #13000, #14470, #15033, #16359. For five reports (50%) with CVE ID #3916, #2094, #2038, #10190 and #11339 it ranks the vulnerable function in Top 5 rank. It indicates that nine (90%) reports are ranked in Top 5. For one report CVE-2013-6763 of Linux Kernel version 3.12.1 it ranks the vulnerable function in Top 7 rank i.e., in 7th position out of total 273,898 functions. Upon manual inspection, we observed that the six functions preceding the vulnerable function exhibit a higher similarity score compared to the actual vulnerable function. The reason behind this can be the inter-connectedness of these six functions with the vulnerable function through function calls. It is also noticeable that projects with less number of functions ranks the vulnerable function in 1st position and with large number of functions the ranking decreases slightly. The reason behind this is larger projects might contain more associated functions which are needed to be fixed in order to address a particular vulnerability.
In summary, the experimental results show that VFDetector can detect vulnerable functions from a huge volume of functions and can also suggest developers with the related functions having highest similarity scores which might need to be patched to address the reported vulnerability. Moreover, to the best of our knowledge we are the first to incorporate vulnerability reports in software vulnerability detection from the concept that vulnerability report’s description contain conceptual information about a reported vulnerability. Based on the promising results in this initial evaluation, the future work can be analyzing more vulnerable reports from diverse projects to make the approach comparable and generalizable.
Threats to external validity: The generalizability of the acquired results poses a threat to external validity. The dataset that we used in our research was gathered from open-source. Open-source projects may contain data that difers from those created by software companies with sound management practices. Seven Apache projects are examined in this study. More projects from other systems are needed to be evaluated for the generalisation. However, to overcome this threat large-scale diversified projects with long change history is to be chosen.
Threats to internal validity: One limitation of our approach is its reliance on sound programming practices when naming variables, methods, and classes. If a developer uses non-meaningful names, it could have an adverse impact on the efectiveness of vulnerability detection.ay not fully represent the characteristics of the whole program. Additionally, our model is evaluated with C/C++ functions and it may encounter challenges in detecting vulnerabilities in other programming languages.
Threats to construct validity: We used the WordNet database and lemmatizer of NLTK library as essential components in text pre-processing to extract word semantics and reduce words to their base forms. Since these resources are well known for their usefulness in NLP, we relied on their accuracy. Moreover, vulnerability reports ofer essential information that developers rely on to address and patch vulnerable functions. A bad vulnerability report delays the fixing process. It’s worth noting that our approach is dependent on the quality of these reports. If a vulnerability report lacks suficient information or contains misleading details, it can have a detrimental impact on the performance of VFDetector.