-We have realised a Java source code analyser that firstly extracts code from a repository and then analyses the code to gain some knowledge. In particular, for each method of each class the internal variables used are automatically determined. For Java classes, we find method bodies and variable declarations by using regular expressions. The tool has been developed by means of Java and MrJob. Python with the MapReduce model have been used to perform data analysis in a distributed manner.
We have developed a system that analyses different versions
of a software system, which is automatically extracted from
a git hub repository [
The analysis is performed by two different tools. The first tool, developed in Java, takes as input the source code and get classes, methods and variables. The second tool, developed in Python, analyses the output of the first tool and gets the variables within each method. To mine the data from a set of Java classes, we use regular expressions, a sequence of characters that define a search pattern, mainly used in pattern matching with strings, or string matching, i.e. “find and replace”-like operations.
Since the said computation can become expensive with the
increasing quantity of code to analyse, the second tool can be
executed in a distributed manner using Hadoop, an Apache
framework providing the MapReduce model, for providing
support to distributed systems accessing big data [
Finally, the data obtained from the computation are saved to a .csv file into the output directory, and can be read using tools like CSV Viewer. Both the first and second tool will be subject to evaluation tests, to calculate execution time and obtain some results from a chosen Java repository.
Copyright c 2017 held by the authors.
The proposed analysis can be valuable when examining
existing code in view of refactoring opportunities [
The following of this paper firstly describes the problem at hand, then describes the proposed solution in detail, then results are shown, and finally conclusions are drawn.
Requirements analysis in systems engineering and software engineering encompasses tasks that determine the needs or conditions to meet for the product. As said previously, the entire software product is divided in two tools, for each the following requirements have been identified.
Analysis of Java source code: the first tool will be able to analyse Java source codes, obtaining the desired data, i.e. the names of classes, methods within the classes, the number of start and end line of each method, and the variables used by such methods. The results are stored in appropriate data structures.
Reading and writing information into files: the second tool will be able to read .txt files placed in a specific directory. These files contain the bodies of methods previously extracted. The extracted data will be placed in .csv and .txt files. In particular, the overall structure of the examined Java files will be placed in a .csv file, while the bodies of the identified methods will be included in a .txt file.
Extract variables: the second tool will be able to extract the names and types of the variables used by each method, by using the MapReduce paradigm. The variables will be associated with the method that uses them.
Comparison of files: the second tool should analyze the variables that are used within methods, with the purpose of tracking changes between two different versions of the same method. The difference will be performed in both directions, i.e. by identifying variables that can be within the first version of the method and not in the other version and vice versa.
The development of the proposed tools involved different programming languages combined together in a pipeline with a bash script. For the Tool 1, used for the pre-processing, we used Java. For the Tool 2, that computes the differences, we used Python with MrJob toolkit. Figure 1 shows a general overview of the developed components.
All the algorithms of the said tools described here are valid for an arbitrary number of Java classes. However, to make it easier to understand the algorithms used in this project, all the examples and the related images refer to the analysis of the Java files in Figure 2 and Figure 3.
The analysis of the class code has been performed by a ParseText class taking as parameters: (i) an input string holding the code of the class under analysis, and (ii) a string that represents the output folder.
The source code of the class under analysis is read line by line, and regular expressions (regex) are used on each line to identify the signature of the methods, and the respective row in which they occur. Since the lines of code holding method signatures can have different forms (e.g. the constructor does not have a return type), then there will be more regex, contained in a list (classRegexList). By knowing all the starting lines of code for methods, it is possible to extract the ending lines of the methods by simply calculating the differences between the start line of the i-th method and the beginning of the next method.
Once these two pieces of information have been obtained, we can analyse the method body. Obviously, the code will check that the structure of the method is compliant with the Java specifications. The next step uses regular expressions again, and by manipulating strings (by means of replace, replaceAll, trim) extracts the class name, the signature and the method body.
The output generated by the first tool consists of a number of files in two categories.
A group of txt files, one for each method found on the analysed code, containing the name of the reference class, the start and the end line of the method within the class, and the method body. A sample output is shown in Figure 7.
A cmList.csv file containing the list of the identified methods, variables, starting and ending lines as shown in Figure 8.
The initial step of the second Tool is to scan the directory in which the first Tool created the .txt files containing the methods bodies. When Tool 2 identifies the files, it stores the filenames in a list called “names”. Furthermore, the number of files taken as input is stored in the variable “n methods”.
Afterwards, the previously listed files are opened. The ClassFind job is launched on each of them using the run() command.
The MapReduce job consists of two main steps. The first step consists of a map function and a reduce function, with the goal to detect variables and insert them into pairs “methodName, [variablesList]”. The second step consists in a further reduce function that has as the goal to calculate variables differences between the examined methods.
The mapper detects the presence of variables by means of regular expressions. Each regular expression identifies a particular sequence of characters to search for in the text.
The first reducer receives pairs “methodName, variableName” detected by the mapper and groups them according to the “methodName” key, forming “methodName, [variablesList]” pairs.
The second reducer is more complex than the previous. It takes as input the identification key of the current method and the list of variables associated with it. Variables are read by using the “next” command and stored in the “variables” string, then used for printing the “variables.txt” file, which will contain pairs “methodName, [variablesList]”. The dict type (key-value dictionary) structure “variables list” is used to store “methodName, [variablesList]” pairs. The variable “count” is a counter incremented whenever the MapReduce process has been executed on a method. When it reaches the value “n metodi”, then all methods have been processed, and the used variables have been identified and stored in “variables list”. Now differences can be easily computed.
Leveraging the “itertools” library, it is possible to calculate all combinations of pairs of methods, so as to compare each method with all the others, which is the task of the “combinations” function.
The calculation of the differences is managed by Counter type structures, which allow to take into account the presence of variables with the same name. Let us suppose, for example, that a method has 2 i named variables, and the second method has only one, the difference between the first and the second method will result in one i variable.
The difference is performed in the same way also in the opposite direction. Then, for each pair of methods, two differences are generated. Different prints for various output files are managed along the entire process. Finally, the various strings that contain the results are written on ad-hoc files.
The variables-finding function of the Tool 2 has been implemented as a variant of Tool 1, hence both tools detect methods variables and associate them to the respective methods. Nevertheless, the two tools achieve this goal in different ways. Tool 1 uses a classical sequential approach without employing the MapReduce paradigm, whereas Tool 2 uses Python and MrJob as described in this section.
The Tool 2 generates four output files.
output.txt: created by redirecting the standard output, it is a file containing, for each pair of analysed methods, the name of the method, the set of variables of each method and the two differences between the sets of variables. The output is shown in Figure 16. variables.txt: contains the list of the identified methods and their variables, as shown in Figure 17. differences.txt: shows the differences between the sets of variables only, as shown in Figure 18. time.txt: contains a string giving the time in seconds spent to search all the methods variables, as shown in Figure 19.
The configuration used for the test is the following.
LAPTOP: ASUS A56C CPU: Intel (R) Core (TM) i7-3537U @ 2.00GHz RAM: 4.00 GB OS type: 64-bit OS: Linux Ubuntu 16.04 LTS
The two tools show similar results from a reliability standpoint. Both seek the variables by means of regular expressions using the same approach. All internal variables of methods of the code under analysis are detected successfully.
Extraction times for finding the internal variables of the methods (and inclusion in key-value lists, in the case of Tool 2) were compared. Execution times are listed in Table I.
Number of files 2 3 4 5 6 6 8 9 10
Tool 1 (Java) 0.004s 0.006s 0.007s 0.009s 0.011s 0.012s 0.013s 0.013s 0.014s
TABLE I ANALYSED EXECUTION TIME STATS.
Tool 2 (Python) 0.001s 0.002s 0.002s 0.003s 0.004s 0.005s 0.006s 0.007s 0.009s
Both tools show linear progression for the measured
timings, increasing with the number of methods for which
variables are extracted. In general, the Tool 2 is quicker than the
Tool 1. The performance results are summarised in the graph
of Figure 20 (by using gnuplot [
This paper described a pair of tools that analyses Java source code and extract from them some relevant statistics, such as method’s names, signatures, starting and ending lines and variables within methods. The tools have been developed in Java and Python and a comparison of the results in terms of execution time has been shown.