This paper represents a cross-language and cross-tool investigation of tools and techniques for static code analysis focused on basic software complexity metrics such as: McCabe Cyclomatic Complexity, Halstead, and Maintainability Index. The main goal of the research is to determine whether metric diferences between multilingual implementation codes exist and how disparate the outcomes of used static tools are. Furthermore aim to learn about the reasons for the discovered diferences. To perform this experiment, we use similar multilingual code samples of the RossetaCode project and identical multilingual code samples (SumProd) developed as cloning scenarios for the evaluation of cross-language clone detection. Because of the limited language support by tools and metrics, our experiment uses widespread C and Java programming languages. Our study will benefit from the selection of these two input languages as they have very similar syntax which enables us to write and pick equivalent code snippets in the two languages and their easier comparison. Our research is based on commercial and open-source static analysis tools. The final results provide divided opinions in cross-language analysis and agreement about the existence of absolute inconsistency in cross-tool metric values. More precisely, on the observed heterogeneous multilingual RossetaCode samples C code appears to be more complex, while observed homogeneous multilingual SumProd samples suggest that implementations in Java are more complex. The crosstool review releases absolute inconsistency of metric values over all covered open-source and commercial static tools. Finally, a manual check of in-tool outcomes signalizes that the roots of inconsistencies are in the tool implementations that do not appropriately follow metric calculation formulas and algorithms.
The design of a software system represents the most challenging and important phase in a software
life-cycle. It influences other aspects like the complexity of development organization and necessary
maintainability factors. Software metrics provide a countable and comparable way of measuring the
quality of software systems. There are two main approaches to calculating metric values, static analysis
of written code without its execution and dynamic analysis during program execution [
The goals of our research can be formulated in three questions: • RQ1. How two programming language implementations are disparate in terms of metric values? • RQ2. How metric outcomes of diferent static analysis tools are disparate?
The paper is organized into the following sections: The second section briefly introduces a reader to the history of this research field and its influence on our motivation in the form of a short related work. The section on the research design introduces the selection of code samples, tools, and metrics, as well as a description of the applied method. The fourth section presents the experimental results and noticed metric relationships. The last two sections state the experimental limitations, conclusions and suggestions for future work.
One of the open research questions in the software engineering field is related to applicability of
software metrics in practise and weaknesses of software metrics techniques and tools for their wider
usage [
While source code metrics are well defined, their thresholds among languages are compared and derived [10], and are often used for evaluation and comparison of various aspects of diferent programming languages (where support for these languages is available) [11, 12, 13, 14, 15, 16], reliability of these measurements and their consistency among languages and tools remains as an unsolved concern.
Early negotiation of diferences in interpretation and implementation of software metrics calculation algorithms starts with [17]. Here, the authors showed that diferent authors implement metrics diferently. This results in variation in metric results across tools even within a single language. A similar investigation towards an evaluation of metric tools supporting .NET languages [18] confirmed the variation in results for supported languages.
The problem grows with multilingual and cross-lingual software projects. Our early investigation of
code analysis in such projects starts with the identification of problems in the systematic application
of static metrics in practice [
Identified inconsistencies have opened a completely new space for research and innovation, with numerous reviews, surveys and investigations in the same or a similar direction. One of the most detailed field overview studies is described in [ 19], while there are also available studies on tools comparison and their precision [20] still without deeper analysis and understanding of reasons for imprecision and discrepancies.
In this study we are interested in observing diferences in calculated metric values. Starting from two syntactically similar languages (Java and C) that are, at the same time widely supported by available metric tools, and from small datasets to enable also manual inspection of diferences roots we aim for learning about the reasons for inconsistencies between metric values between tools and input languages.
This section describes the research design by introducing the datasets and methods used. It defines criteria for static tools and code sample selection, as well as a description of the well-known software complexity metrics and experimental setup.
The research includes equivalent implementations of the selected standard problems and toy examples implemented in Java and C programming languages [21]. This enables a mutual comparison of the code implementation languages themselves. The selected codebases provide multilingual implementations of defined problems, which enables, among other things, a mutual comparison of the code implementation languages themselves.
An overview of alternative datasets of multilingual implementations can be found on the RossetaCode webpage [22]. Many available online codebases don’t provide enough examples or consistency in solving the same problem in multiple programming languages. Considering the previously mentioned, the RossetaCode database was singled out as the only adequate bigger source for this research. Multilingual examples in RossetaCode semantically solve the same problems, but their syntax doesn’t have to be identical. The mentioned code bases are a good starting point for this study.
• RosettaCode is a collection of solutions for standard problems usually implemented by contributors who are developers in the language in which they contribute to the collection. An overview of alternative datasets of multilingual implementations can be found on the RossetaCode webpage [22]. Many available online code bases do not provide enough examples or consistency in solving the same problem in multiple programming languages. Considering the previously mentioned, the RossetaCode database was singled out as the only adequate bigger source for this research. Examples in RossetaCode solve the same problems in a semantically equivalent way, but their syntax is diferent. • SumProd is a collection of cloning scenarios, for all four types of clones, going from renaming a variable, across a reordering statement, to replacing while loop with a do one or with an if branching. It is used with small modifications, on the original code version, to achieve maximum uniformity of codes between languages. It should provide a small set of identical codes to be used as a control point to conclusions of bigger and divergent codes of the RossetaCode sample.
It should also be noted that the datasets don’t provide specific information about the program code. This means that we cannot gain insight into whether the code was written by the same developer, a diferent one, or the level of their knowledge of the implementation language. The initial step of the research included selecting adequate examples from the mentioned datasets, which implies their availability in both included programming languages and their implementation in one file (to avoid complicated compensation of program code modularity and implementation versions).
The covered metrics in this research are fundamental metrics of software code design complexity and quality such as McCabe, Halstead, and Maintainability Index. They include other basic metrics such as lines of code, and the number of operators and operands. The primary criteria for selecting metrics are their availability in considered tools (Section 3.3).
A short explanation of caught metrics [
Physical lines of code: count every single line of code Logical lines of code: consider diferent aspects of the code which overcome coding style dependency • McCabe: Cyclomatic Complexity (CC) of control-flow structures as number of linearly independent paths (V) through a control-flow graph (CFG) representation of a method/function 1 source code.
( ) = ( . )˘( . ) + 2 • Halstead: Product size and complexity metrics set based on the number of operators and operands.
1 : .
2 : .
1The limitation to method/function represented by a CFG is based on the limitation the the graph should be strongly
connected and have one entry and exit point (see [
1 : . 2 : .
ℎ : = 1 + 2 : = 1 + 2 : = : = * 2
1 : = 2 1 2 2 * 2 : = * • Maintainability Index: Relative ease of maintaining the code.
Original formula (0-100+): = 171− 5.2* ln − 0.23* − 16.2* ln Microsoft version (0-100): 100 = (0, * 171 )
A further step requires the selection of tools known as static analyzers which will automatically analyze the code examples. We initiated a search for existing tools used and recommended by real-life developers. Among numerous lists, we identified the most complete source[23].
Only open-source tools and commercial tools that provide a "free" or "trial edition" license are included in this survey. Popular commercial tools like SonarQube [24], Embold [25], PVS-Studio [26], DeepSource [27], and some others were excluded from the research because they did not provide adequate language or metrics coverage in the mentioned releases.
The review of the static analyzers market concludes that tools mainly focus on spotting and reporting code errors, security flaws, and code design irregularities rather than providing quantifiable metric values. For example, the SonarQube tool in the "free plan" does not support the C programming language, while Embold and DeepSource in the mentioned edition provide only descriptive metrics of their arrangement. Among the commercial tools, the research has included FrontEndART SourceMeter in the form of the SonarQube plug-in [28], CoderGears CppDepend/Jarchitect [29], and M Squared Technologies’s Resource Standard Metrics [30].
Some of the commercial tools are available in the form of SonarQube plugins like SourceMeter, CppDepend, and JArchitect. We used SourceMeter in the mentioned plugin form because it provides easier access to output results.
Extensive research into the open-source static analysis tools, via the GitHub portal, has been done in multiple revisions to achieve consistent and comparable metric values as mentioned between the languages and the tools. Certain tools could not successfully report metrics on the source code and therefore were not included in this research. On the other hand, some tools required minor corrections to display the appropriate parameters of the results or considerable upgrades of the logic due to its incompleteness.
The previously mentioned is shown in Table 1. The original names of the tools have been changed in the research (clearly indicated by their links) for easier distinction due to existing overlaps. Adoption of open-source tools has been involved following changes: • C-Cyclomatic-Halstead - working with a whole directory of source files instead of file-by-file • C-Static-Analyzer - display total no. of operators/operands for provided source file • Java-4-Metrics - debug of runtime-errors during the execution of RossetaCode examples The survey did not include popular languages like JavaScript and Python, due to the small number of static analyzer tools dedicated to them that could provide consistent and comparable results.
C C-Static-Analyzer (output adopted [32]) C_cyclomatic _halstead (output adopted [33] ) Commercial Tool Website CppDepend (2023.1) https://www.cppdepend.com JArchitect (2023.1) https://www.jarchitect.com SourceMeter (10.0.0) https://sourcemeter.com ResourceStandard Metrics (7.70) https://msquaredtechnologies.com https://github.com/sajedjalil/C-Static-Analyzer/tree/cbc5e5b https://github.com/AhmedEssam17/
CyclomaticComplexity-and-HalsteadMetrics/tree/1c2b9de
The research provides a comparison of static analysis tools over multilingual datasets in terms of software metric values. To make this experiment more replicable, we provide all used code samples online available in the repository at [21]. Some tools (like SourceMeter, CppDepend, and JArchitect) have been required to create whole project structures of given source code. The necessary steps for running the specific tool on a particular dataset have also been issued in detail in the GitHub repository. As earlier mentioned, our research includes a manual check (based on previously listed metric formulas) of each tool’s output results. Manual checking of the results proved especially necessary with opensource tools, but also missing metrics on some tools are manually calculated to make their outcomes mutually more comparable.
This section presents the metric results of static analysis on the selected codebase. Metric results are available on our GitHub repository [21]. It should be noted again that the given RossetaCode examples do not represent identical codes for diferent languages but rather they represent similar implementations of the same problems in diferent languages. For the analysis of identical multilingual samples, we used the SumProd dataset. The whole experiment is based on Java and C programming languages.
The results of the Resource Standard Metrics tool, shown in Table 2 and Table 3, indicate that the examples written in the C programming language contain more extensive implementation in terms of the lines of code and program control flow, as expected based on nature of the language [ 34]. Considering previous facts, they express greater code complexity as measured by Cyclomatic Complexity (Cyclo Vg). They also encourage the appearance of more "defects in the code which are not covered by the compiler" (Notice).
Despiting a larger number of methods, parameters, and return values, the examples written in the Java language reflect lower complexity but higher values of Functional Points (derived from the number of lines of code).
The same static tool in Table 3 reveals that C language dominates in several used program constructs (keywords). Also, it demonstrates inequality between two language implementations of RossetaCode samples.
The values in Table 4 of the SourceMeter tool demonstrate a similar trend in terms of number of Lines of Code and Cyclomatic Complexity. The remaining two Halstead Length & Vocabulary parameters (based on the total and unique number of operators and operands) are, as expected, higher for Java language examples.
Contrary to the above observations, the CppDepend and JArchitect cross-tool report on RossetaCode (Table 5) indicates that C examples express higher values of Halstead metrics. Other metric values like LOC, McCabe, and Maintainability Index are compatible with the previously mentioned tools.
Averages of multiple open-source tools in the same languages are concordant with previous CppDepend and JArchitect conclusions except for Halstead Vocabulary Size and Maintainability Index. It must be noticed that outcome values are dependable on the selection of the covered open-source tools.
Despite well-known metric formulas, static code analyzers provide uneven results. Because of that, this experiment doesn’t provide judgment about the amount of cross-language diferences. This is the reason that induces us to perform further cross-tool analysis. Answer to RQ1: Based on agreement between covered static tools outcomes, we can evidence that C language RossetaCode samples are larger in terms of LOC and more complex according to McCabe values. The majority of tools also agree that Halstead metrics and Maintainability Index are worse for C examples as covered in Figure 1.
The standard deviation among the results of the open-source tools (Table 6) directly demonstrates the mentioned deviations in the “cross-tool” values of the same parameter. Comparing outcomes in Tables 2, 4, 5, 6, and 9, it becomes obvious that all tools provide disparate values. Among all parameters, the McCabe seems to be the most uniform metric, but also it doesn’t manifest identical values across all considered static tools.
Observing the above defect and considering earlier mentioned changes in open-source tools, it suggests that the way of interpretation and parsing program constructs should have the biggest impact on the final outcomes of tools. There is no consensus among tools about the uniform way of treatment program code constructs. Comparison in Table 7 showed that open-source tools are proven as very divergent in the possibility of distinguishing between program constructs. Further, previous observations and comparisons with commercial tools in Tables 5 and 9 bring the same conclusion.
Interestingly, manual checking of the results of the open-source tools (Table 8) and commercial tools (Table 5) also demonstrates “in-tool” inconsistencies between Halstead values provided by the tool and those manually recalculated with well-known metric formulas. Considered recalculations are based on the information available in the results of the tool about the total and unique number of operators and operands (Table 7 and 5).
Parallel examination of Table 4, 5, and 6, release that open-source tools output higher values than commercial tools. Again, open-source tools utilize less restrictive approaches to program constructs assessment which is clearly shown in Table 5, 7, and 9. Answer to RQ2: “Cross-tool” analysis exhibits disparity in the majority of output values. Moreover, there is no one metric value that is consistent between all static tools. Also, “intool” analysis reveals that some tools don’t correctly follow well-known metric formulas. This distinction characterizes both open-source and commercial tools as demonstrated in Figure 2.
This section focuses on strictly comparing the results of identical program codes, to the mentioned shortcomings of the RossetaCode example. For this purpose, the SumProd examples were used.
The file changes themselves and the intermediate results of the static tool showed the expected absence of the impact of gaps and comments in the code on outcome metrics value. The confirmation of the identity of the constructs of the program codes discussed in this section is clearly shown in Tables 2, 9, and 10.
On the other hand in Table 2, the Resource Standard Metrics tool indicates higher function point values, line of code metrics, and notices (errors not covered by the compiler) in Java examples. Considering that the tool doesn’t include the Halstead metrics of code complexity, identical program codes can’t show diferences in terms of the cyclomatic complexity itself. Higher values of function points for Java examples are only concordant observations between SumProd and RossetaCode datasets.
The results of the SourceMeter tool on the SumProd dataset in Table 4 indicate that the implementation of Java examples is more complex in terms of Halstead and Maintainability Index. Examples of both languages are identical concerning McCabe and Physical LOC metrics, but C language is slightly complicated regarding only Logical LOC. Again, we can see that Halstead complexity measures (operators/operands quantity) prove more sensitive against McCabe metric (control-flow paths). Comparing with RossetaCode results of the SourceMeter tool, we can generally conclude that Java language is more complex regarding Halstead values while C language tends to be robust in terms of Logical LOC. In the context of average open-source tool results (Table 6), complexity metrics except the Maintainability Index are higher for Java examples. The outcomes of SumProd and RossetaCode code samples are exactly the opposite for open-source tools. Although the code samples are almost identical, outcomes in Table 9 suggest that the CppDepend tool has higher values of operators and operands followed by LOC, Halstead, and Maintainability Index. Higher values of LOC, Maintainability Index, and Halstead metrics match between SumProd and RossetaCode datasets. Its results on the SumProd case are opposite to SourceMeter and open-source outcomes.
Answer to RQ3: Contrasting to RQ1, the majority of static tools on the SumProd code suggest that Java examples manifest more complexity compared to C code samples as shown in Figure 3.
Cross-tool observations on the SumProd case in Figure 4 reveal the same conclusions as RQ2.
Our research exposes some restrictions of its scope. The analysis doesn’t provide cross-platform testing between tools on diferent operating systems. Also, our experiment only covers two programming languages because of the availability of static analysis tools which are included only in its “free-form”. As discussed earlier, the size and diverseability of used code datasets and software metrics are also bounded. Because of subjectivity and the unavailability of used metric definitions by tool designers, research doesn’t deeply consider approaches to their calculating.
As another limitation one may consider selection of small datasets based on classical algorithms and toy examples of cloning scenarios. However, this selection enables us to manually investigate calculated and expected values and learn about the reasons for imprecision and discrepancies.
The paper investigates the inconsistency between diferent static analysis tools for calculation of traditional and widely used LOC, Halsetad and McCabe CC metric, on similar and identical code samples implemented in Java and C programming languages.
Opinions are divided over the cross-language static analysis. The results on heterogeneous multilingual RossetaCode samples reveal that C codes are more complex implementations as expected. On the other hand, metric observations on homogeneous multilingual SumProd samples suggest that implementations of Java codes are more demanding. The reason for this inconsistency is hidden in the fact that SumProd implementation does not involve object-oriented design and avoiding complexity on code level based on design complexity. Based on these conclusions we can discuss a need for a complexity metric to consider complexity of an implementation taking into account both: source code and the implementation design inbuilt in it (as statically as possible having in mind dynamic nature of the design).
The cross-tool review releases absolute inconsistency of metric values over all covered open-source
and commercial static tools. Moreover, a manual check of in-tool outcomes reveals that some static
analyzers do not coherently follow well-known metric formulas. The primary reason for the result
distinction may be the well-known subjectivity in metric definition and program constructs interpretation
involved by the tool author. This is an obvious open space for future work - filling the gap between
metric algorithm implementations to consistently support diferent input languages as proposed by [
Asian J. Soc. Sci. Manag. Technol 4 (2022) 1–12. [16] K. A. Onyango, J. Kamiri, G. M. Muketha, A comparative study of the lexicographical complexity of java, python and c languages based on program characteristics, Journal of Innovation, Technology and Sustainability 1 (2023) 42–67. [17] R. Lincke, J. Lundberg, W. Löwe, Comparing software metrics tools, in: Proceedings of the 2008 international symposium on Software testing and analysis, 2008, pp. 131–142. [18] J. Novak, G. Rakić, Comparison of software metrics tools for .net, in: Proc. of 13th International
Multiconference Information Society-IS, volume A, 2010, pp. 231–234. [19] Z. Mushtaq, G. Rasool, B. Shehzad, Multilingual source code analysis: A systematic literature review, IEEE Access 5 (2017) 11307–11336. [20] V. Lenarduzzi, F. Pecorelli, N. Saarimaki, S. Lujan, F. Palomba, A critical comparison on six static analysis tools: Detection, agreement, and precision, Journal of Systems and Software 198 (2023) 111575. URL: https://www.sciencedirect.com/science/article/pii/S0164121222002515. doi:https://doi.org/10.1016/j.jss.2022.111575. [21] Repositorium of the study samples and results, Accessed on 1.7.2024. URL: https://github.com/ zaricu22/Multi-Lang_Metrics/. [22] Alternative collections, Accessed on 1.7.2024. URL: https://rosettacode.org/wiki/Help:Similar_Sites. [23] Top 40 static code analysis tools, Accessed on 1.7.2024. URL: https://www.softwaretestinghelp.
com/tools/top-40-static-code-analysis-tools. [24] SonarQube, Accessed on 1.7.2024. URL: https://www.sonarsource.com/products/sonarqube/. [25] Embold, Accessed on 1.7.2024. URL: https://embold.io/. [26] PVS-Studio, Accessed on 1.7.2024. URL: https://pvs-studio.com/en/pvs-studio/. [27] DeepSource, Accessed on 1.7.2024. URL: https://deepsource.com/. [28] FrontEndArt SourceMetter, Accessed on 1.7.2024. URL: https://github.com/FrontEndART/
SonarQube-plug-in. [29] JArchitect, Accessed on 1.7.2024. URL: https://www.jarchitect.com/. [30] Resource Standard Metrics, Accessed on 1.7.2024. URL: https://msquaredtechnologies.com/
Resource-Standard-Metrics.html. [31] AcuType, Java Code Analyzer, Accessed on 1.7.2024. URL: https://github.com/AccuType-911/
Java-code-analyzer/pull/1. [32] C Static Analyzer, Accessed on 1.7.2024. URL: https://github.com/sajedjalil/C-Static-Analyzer/pull/ 1. [33] CC and Halsead Metrics, Accessed on 1.7.2024. URL: https://github.com/AhmedEssam17/
CyclomaticComplexity-and-HalsteadMetrics/pull/1. [34] N. Patakia, Z. Porkolába, E. Csizmásb, Why code complexity metrics fail on the c++ standard template library, in: Proc. of the 7th International Conference on Applied Informatics (ICAI 2007), Eger, Hungary, volume 2, Citeseer, 2007, pp. 271–276.