With the evolution of object-oriented languages such as C#, new code constructs that originate from the functional programming paradigm are introduced. We hypothesize that a relationship exists between the usage of these constructs and the error-proneness. Measures de ned for this study will focus on functional programming constructs where object-oriented features are used, this often a ects the purity of the code. Built on these measures we try to de ne a metric that relates the usage of the measured constructs to error-proneness. To validate the metric that would con rm our hypothesis, we implement the methodology presented by Briand et al. [BEEM95] for empirical validation of code metrics. The results of this research granted new insights into the evolution of software systems and the evolution of programming languages regarding the usage of constructs from the functional programming paradigm in object-oriented languages.
The growing popularity of multi-paradigm language Scala [Car] and the introduction of functional programming (FP) features in signi cant object-oriented (OO) languages such as Java [Ora] and C# [Mic], code Copyright © by the paper's authors. Copying permitted for private and academic purposes. evaluation for multi-paradigm languages have become more signi cant. Landkroon has shown [Lan17] that metrics from the OO paradigm and the FP paradigm can be mapped to the multi-paradigm language Scala. However, the integration of OO and FP features introduces artifacts that are neither covered by OOinspired code metrics nor by FP-inspired code metrics. These constructs, such as usage of mutable class variables in lambda functions whose execution might be deferred, are not possible in the FP languages but are used in a functional manner. To evaluate these patterns, the metrics that are proposed for the OO paradigm [BBM96, HKV07, McC76] and FP paradigm [RT05, VdB95] might be unsuitable to give a valuable indication of quality regarding the usage of these combined constructs.
Measures that indicate complexity might have an intuitive relationship with error-proneness. But this does not have any concrete meaning and usefulness since you can not substantiate a predication by just intuition. Therefore evidence must be provided that a measure is useful, this can be done by proving there is a relationship to an external attribute such as errorproneness [BEEM95].
We follow the use of the term `measure' as used by Briand et al. [BEEM95]. Where the term `measure' refers to an assessment on the size of an attribute of the code.
The purpose of this research is to explore the relationship between the usage of the FP-inspired constructs and the error-proneness of the classes. Our approach is de ning measures that will cover the usage of these constructs and empirically relate them to the errorproneness of the corresponding class. 2
Since version 3.0 a set of features were added to C#, that are inspired by FP. Functions are now rstclass constructs, which enables higher-order functions.
Lambda functions provide a concise way of describing an anonymous function. Pattern matching allows concise and rich syntax for doing switch-case statements. The concept of lazy evaluation, which comes along 1 with the LINQ1 query syntax, which was previously 2 only possible by using the Lazy<T>-keyword [Mic]. 3 LINQ introduced syntax for list operations such as 4 map, filter and sort, which are basically higher order 5 functions as known from functional languages. This 6 syntax enables list mutations with concise syntax in C# as shown in Listing 1.
Enumerable.Range(1, 10) .Where(i => i % 2 == 0) //filter .Select(i => i * 10) //map .OrderBy(i => -i); //sort // ["100", "80", "60", "40", "20"]
3
In the following code snippets, two basic implementations are presented which have the functionality to get a list with vehicles that that start with `Red'. For the rst implementation (Listing 2) an imperative approach is chosen. The snippet has a Source Lines of Code (SLOC) count of 11 and a Cyclomatic Complexity of 3 since there are two branching points. This is how the general complexity of the snippet translates back into the values returned by the metrics. List<string> vehicles = new List<string>() {"Red Car", "Red Plane", "Blue Car"}; List<string> redVehicles =
new List<string>(); for (int i = 0; i < vehicles.Count; i++) { if (vehicles[i].StartsWith("Red")) {
redVehicles.Add(vehicles[i]); } }
The second implementation (Listing 4) LINQ library is used, which encourages the use of lambdaexpressions. The snippet has a SLOC count of 5 and a cyclomatic complexity of 1 since there are no branching points. Even though the functionality and the log1Language Integrated Query, an uniform query syntax in C# to retrieve data from di erent sources [Wag17] ical complexity are the same with both snippets, the cyclomatic complexity and SLOC di er drastically. List<string> vehicles = new List<string>() {"Red Car", "Red Plane", "Blue Car"}; List<string> redVehicles = vehicles .Where(t => t.StartsWith("Red")) .ToList();
Listing 3: C# With LINQ int Foo(int x, int y) { if (x < 0)
x = 0; if (y > 5)
y = 2; return y - x; } 4 4.1
Lambda functions in the context of OO languages are a concise way to write anonymous functions inline. Compared to a regular method, the parameter type, return type can all be omitted. This might introduce constructs which are harder to understand. An example for this scenario is given in Listing 5.
To calculate the value for this measure, we traverse the AST (abstract syntax tree). For each SyntaxN ode which has the type LambdaExpression, we raise the counter for this measure.
List<int> numbers = new List<int>()
{ 1, 2, 3 }; IEnumerable biggerThan2 = numbers
.Where(x => x > 2);
Listing 5: An Example Of A Lambda Expression 4.2
Where simple lambda expressions might not be extra information to reason about the execution, once the lambda expression becomes most complex this might not be the case. In listing 6 an example is given with a multiline lambda expression. As curly braces are taken included in the `source lines of code'-measure [HKV07], we also include these curly braces when calculating the 1 span of the lambda expression. Therefore, the snippet 2 in Listing 6 has an `Source Lines of Lambda'-count of 3 1 + 1 + 4 = 6. The density of the usage of lambda functions in a class can give an indication of how functional a class is. Our hypothesis for this measure is, that a relationship exists between how functional a class is and the error-proneness of the class. We calculate this lambda density with Equation 1. Sometimes it might be hard to predict when a lambda function is executed, therefore, it might be hard to reason about what value for the mutable eld will be used. An example for this scenario is given in Listing 7.
To calculate the value for this measure, we traverse the AST. For each variable inside a lambda expression, we check if the variable is non-constant and eld scoped by using the semantic data model (SDM) of the class. If this test passes, we increase the counter for this measure.
Func<int, bool> biggerThanY =
x => x > _y;
Listing 7: An Example Of A Lambda Expression With A Reference To A Mutable Field Variable 4.5
Sometimes it might be hard to predict when a lambda function is executed, therefore, it might be hard to reason about what value for the mutable local variable will be used. An example for this scenario is given in Listing 8.
To calculate the value for this measure, we traverse the AST. For each variable inside a lambda expression, we check if the variable is non-constant and local scoped by using the semantic model of the class. If this test passes, we increase the counter for this measure. void F() { } int y = 2; Func<int, bool> greaterThanY =
x => x > y; Listing 8: An Example Of A Lambda Expression With A Reference To A Mutable Local Variable 4.6
We think that the combination of side e ects in lambda functions with e.g. parallelization or lazy evaluation is dangerous because it can be hard to reason about when these side e ects occur. An example for this scenario is given in Listing 9.
To calculate the value for this measure, we traverse for each class its AST. For each variable inside a lambda expression, we check if local or eld variables are being mutated. By not terminating a collection query, it will be hard to reason when the query will be executed. Since these collection queries may contain functions that contain side e ects and use outside scope variables, the execution at di erent run-times can yield di erent and unexpected results. An example for this scenario is given in Listing 10.
To calculate the value for this measure we traverse the AST and count how many IEnumarable<T> are initiated.
List<int> nmbs = new List<int>()
{ 1, 2, 3 }; int y = 2; IEnumerable biggerThanY = numbers
.Where(x => x > y); Listing 10: An Example Of A LINQ-Query That Is Not Evaluated/Terminated 5
To empirically validate a proposed metric according to Briand et al [BEEM95] describes three assumptions that should be satis ed namely:
external attribute A2 . The hypothesized relationship between attribute A1 and A2 can be tested if assumption 2 and assumption 3 are assumed, by nd a relationship between X1 and X2.
A1. Measure X1 will measure de ned attributes of the code such as mutable external variables used in lambda functions. This measure X1 will be assumed to measure A1, A1 will the internal attribute such as purity of the lambda usages.
tribute A2. Measure X2 will measure the errorproneness A2 of a given class. The measure X2 used will be if the class contains a bug yes or no.
In Section 7 we elaborate on our approach to satisfying the above assumptions. 6
For this study we analyzed the following projects: • CLI The .NET Core command-line interface (CLI) is a new cross-platform toolchain for developing .NET applications. The CLI is a foundation upon which higher-level tools, such as Integrated Development Environments (IDEs), editors, and build orchestrators, can rest [Fou15b]. Analyzed version: bf26e7976 • ML Machine Learning for .NET is a crossplatform open-source machine learning framework which makes machine learning accessible to .NET developers. ML.NET allows .NET developers to develop their own models and infuse custom machine learning into their applications, using .NET, even without prior expertise in developing or tuning machine learning models [Fou18]. Analyzed version: b8d1b501 • AKK Akka.NET is a community-driven port of the popular Java/Scala framework Akka to .NET. Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications. Akka is the implementation of the Actor Model. [Akk14]. Analyzed version: bc5cc65a3 • ASP ASP.NET Core is an open-source and cross-platform framework for building modern cloud based internet connected applications, such as web apps, IoT apps and mobile backends. ASP.NET Core apps can run on .NET Core or on the full .NET Framework [Fou15a]. Analyzed version: 5af8e170bc • IS4 IdentityServer is a free, open source OpenID Connect and OAuth 2.0 framework for ASP.NET Core [Fou15c]. Analyzed version: da143532 • JF Jelly n is a Free Software Media System that puts you in control of managing and streaming your media [Jel13]. Analyzed version: d7aaa1489 • ORA OpenRA is an Open Source real-time strategy game engine for early Westwood games such as Command & Conquer: Red Alert written in C# using SDL and OpenGL [For09]. Analyzed version: 27cfa9b1f • DNS dnSpy is a debugger and .NET assembly editor. You can use it to edit and debug assemblies even if you don't have any source code available [0xd16]. Analyzed version: 3728fad9d • ILS ILSpy is the open-source .NET assembly browser and decompiler. [ics09] Analyzed version: 72c7e4e8 • HUM Humanizer meets all your .NET needs for manipulating and displaying strings, enums, dates, times, timespans, numbers and quantities. [Kha12] Analyzed version: b3abca2 • EF EF Core is an object-relational mapper (O/RM) that enables .NET developers to work with a database using .NET objects. It eliminates the need for most of the data-access code that developers usually need to write [Fou14]. Analyzed version: 5df258248
7.1
Investigating the relationship between code metrics to error-proneness is commonly done by creating a prediction model for error-proneness based on code metrics [BBM96, BEEM95, GFS05, BMW02].
The logistic regression classi cation technique [HJLS13] is often used to create such a predication model [GFS05, Lan17, BBM96, LV97].
With a logistic regression model trained with the data from our analysis framework, which processes repositories, we explore the relationship between our measured constructs to error-proneness. 7.1.1
With a univariate logistic regression model, we can evaluate, in isolation the predication model for errorproneness based on the measured constructs. Using the Equation 2 we construct a prediction model.
P (f aulty = 1) = (2) e 0+ lXl 1 + e 0+ lXl Where lXl is the coe cient multiplied with the value of the added measure. 7.1.2
To show that our measures are useful regarding errorproneness prediction, their inclusion must yield better results than metrics that are being used in the industry. This set of metrics will de ne the baseline for this study. We use the union of a set of general code metrics with a set of object-oriented metrics. For general code metrics we take Source Lines of Code (SLOC) [NDRTB07], Cyclomatic Complexity (CC) [McC76] and Comment Density [Son19].
The 00 metric suite used for this study was dened by Chidamber [CK94]. For our baseline, we implemented the metrics which showed any signi cance in the study. The metrics suite contains the following metrics Weighted Methods per Class (WMC), Dept of Inheritance Tree (DIT), Response For a Class (RFC), Number of Children of a Class (NOC), Coupling between Object Classes (CBO), Lack of Cohesion of Methods (LCOM). 7.1.3
Besides looking if our univariate logistic regression model gives an indication of good performance, we can use multivariate logistic regression to test if we can improve the model with the in-place OO metrics. The baseline for the evaluation of the model will be a multivariate logistic regression model based on our baseline set of metrics. To see if we can achieve an increased performance compared to our baseline model, we substitute baseline dependent variable with candidate measures as l.
P (f aulty = 1) =
e 0+ 1Xi+ 2X2+:::+ nXn+ lXl 1 + e 0+ 1Xi+ 2X2+:::+ nXn+ lXl (3) 7.1.4
We choose to validate our model using crossvalidation, which is commonly used for the validation of prediction models [Sto74, K+95]. We use the Holdout method for cross-validation. By default, holdout cross-validation separates the data set into a train set and a smaller test set. To compensate for the randomness of the division, we run the model tting with multiple di erent selections of the training set and average the results and assess the standard deviation. Based on a classi cation report created for the hold-out set, we assess the performance of the model. We use the F1-score, which calculates the harmonic mean of the precision and recall (F1-score) to assess our model performance with Equation 4.
Since our data set is unbalanced, as seen in Table 1 one could choose to calculate the micro-average between the F1-scores for the `faulty-classes'-class and the `non-faulty-classes'-class, where the support for each class is weighted. However, since we want good prediction performance in both classes we use the macro-average instead which calculates the harmonic mean between the two F1-scores [SL09]. syntax node in depth- rst order. Where needed we can request additional data from the SDM during the traversal, such as, what is the type and the level of scope for a given variable. Using this method we can calculate the values for all of our candidate measures. To make an estimation on how error-prone a class is, we make the assumption that if a class during the lifetime of a project was updated by a bug x, the class is error-prone. Unfortunately, the GitHub API does not provide an easy way to identify bug- xing commits. From a GitHub repository, we can request all the issues that were created regarding a bug. With this information, we identify all commits that close an issue by searching for issue closing keywords as described in [Git19]. All commits that mention an issue that was identi ed as a bug related issue, are marked as bug- x commits. We then extract the a ected lines from the metadata of the commit. Then derive with which classes the a ected lines intersect in the parent version of the bug- x commit by parsing the AST for the updated le. We use the parent version of the bugx commit since this is the version where the bug existed. Each of these intersected classes will be marked as error-prone. 8
In Table 1 descriptive stats are shown for the output of our static analysis. The test projects have been excluded for this our analysis since they are likely to be modi ed in a bug- xing commit to detect the bug if it would occur again.
CLI [Fou15b] 328 54 ML [Fou18] 1404 27 AKK [Akk14] 1621 171 ASP [Fou15a] 3212 99 IS4 [Fou15c] 331 40 JF [Jel13] 1420 81 ORA [For09] 1990 227 DNS [0xd16] 5345 ?* ILS [ics09] 1011 95 HUM [Kha12] 124 23
EF [Fou14] 1432 975 * During the runtime of research the labels in the repository of dnSpy were deleted. Therefore, we are unable to derive the bug- x count.
Notable, the relationship of bug- xing commits on faulty classes can be either positive or negative. This can be explained by that a set of the commits are only updating con guration or non-csharp code les, therefore, jBugF ixesj > jF aultyClassesj is possible. The alternative scenario, one commit is able to modify multiple classes, therefore, jBugF ixesj < jF aultyClassesj is possible. The variance in the ra
Classes tio of F aultyClasses between project is also notable, this can be partly attributed during the lifetime of a project. However, some projects contradict this hypothesis. Where ILSpy is one of the older projects that were analyzed, the ratio is not higher than e.g. the AKKA.NET project. Even though, the ILSpy project is more than twice as old.
To evaluate the measures in isolation, we t and evaluate a prediction model using univariate regression. In gure 1 we can see the macro-average F1-score for each project in combination with each candidate measure.
In Figure 1 we see m1 as described in Section 4.1 perform well on the projects: ILSpy and JellyFin. The univariate regression models created for the Entity Framework project, perform relatively bad compared to the other projects. Notable, the SLOC measure performs the best as an isolated measure in the Entity Framework project. Looking at the raw input for the project, we see that only 51 of the classes use lambda expressions, compared to other projects e.g. AKKA.NET 31 . The fewer usage of lambda expressions could in uence the usability of our measures. The Humanizer project seems to score an almost stable 0.48. When looking into the raw output data from our static analysis we see only 19 classes in this project uses lambda expressions. Therefore, this project might not be functional enough for our measures to yield any value.
To evaluate the value of our candidate measures compared to our baseline, we create a multivariate regression model based on K-Best features for each of the projects. In Table 8 is shown how many out of 11 K-best models included the corresponding measure as a feature.
LSc LC SLOL LMLV LMFV LSE UTQ # Models 3 6 9 4 6 0 0
Most notable is that 9 out of 11 projects include the SLOL-measure as described in Section 4.2. The one project where SLOL was excluded in the K-Best, was the Humanizer project. As described earlier, the project does not use a lot of the FP inspired constructs and therefore, is not suitable for our measures. The measure LSE counting the numbers of side-e ects in lambdas and UTQ, counting the number of unterminated collection queries, both do not seem to yield an interesting result. Even though these FP inspired constructs do occur in almost all projects, the amount of occurrences is too limited to yield good value.
To do a comparison between our baseline model and the selection of the K-best features model, the projects are plotted in Figure 2.
Looking at Figure 2 one can see that the worstperforming project did not gain improvement in performance. However, the rst quartile had a performance increase of 0.02. The best performing project also has a 0.02 increase in performance. 9
Threats to validity • Generalizability At this point in the research, only 11 open-source projects were analyzed. For the results of the research to be useful, substantiated claims need to be made about the generalizability of the results. Since we only processed 11 projects, we are unable to make any claims regarding that aspect. By analyzing a bigger corpus of projects, one could make an easier distinction on what di erent types of projects our proposed measures yield value. • Bug xes vs Bugs To make an estimation on how error-prone a class is, the assumption was made that bug- xes made in a class measure the error-proneness of the class. However, there is no way to guarantee that a class that was never updated by a bug- x is bug-free. Bugs might have not been identi ed yet, or maybe bugs that were never xed by a bug- x commit were accidentally xed during a refactoring. 10
Uesbeck [USH+16] did a control experiment where the impact of lambdas in C++ on productivity, compiler errors, percentage of time to x the compiler errors. The results show that the use of lambdas, as opposed to iterators, on the number of tasks completed was signi cant. "The percentage of time spent on xing compilation errors was 56.37% for the lambda group while it was 44.2% for the control group with 3.5 % of the variance being explained by the group di erence.". Where the groups consisted of developers with di erent amount of programming experience.
The increased time of xing compiler error where lambda functions were used, which seems likely to be the result of lambda expressions being harder to reason about. Which strengthens our hypothesis. Finifter [FMSW08] shows how veri able purity can be used to verify high-level security properties. By combining determinism with object-capabilities a new class of languages is described that allows purity in largely imperative programs. 11
We investigated the evolution of the OO language C# and what features inspired by the FP paradigm are added. The development and introduction of the FP inspired features seem to be going rapid and there is no indication of the development slowing down. This new declarative syntax enables more concise code constructs. Therefore, enables the software engineer to write more functionality with less code. However, these constructs are introduced without the constraints that would be present in FP languages. Therefore, impure usages of concepts that were designed pure, exist in the OO language C#. To cover the new type of complexity introduced by these FP inspired constructs and impure usages of these constructs, we de ned measures. The de ned measures cover the following FP inspired constructs: lambda expression usages and impure usages, in which the expression its evaluation is a ected by or a ects the outside state. Furthermore, we de ned a measure to report the unterminated collection queries.
The candidate measure SLOL yielded promising results when used in a univariate prediction model for all of the projects where FP inspired constructs were actively used. To assess if we can improve our baseline model, we swapped out the weaker metrics from the baseline model with stronger metrics based on our set of de ned measures. We were able to achieve a marginal improvement (F1-score 0.0-0.02) with respect to di erent projects. For some projects, we were able to achieve a small improvement in the prediction model. On the contrary, the projects where a low amount of FP inspired constructs was used, the candidate measures did not yield value.
So we did nd a correlation between our measures and error-proneness. But the presence of this correlation is too uncertain to make claims about causality.
As described earlier in the introduction the signi cant OO languages seem to adopt more features from the FP paradigm. Our hypothesis is that the set of FP inspired features will become bigger and receive a more FP-like syntax. The increase of performance in our prediction models found in this research seems marginal for now. But we hypothesize that their relevance will increase in the future, based on the ongoing evolution of OO languages and the increasing adoption by developers of these FP inspired features. 0xd4d. dnspy. https://github.com/ 0xd4d/dnSpy, 2016.
AkkaDotNet. Akka.net. https: //github.com/akkadotnet/akka.net, 2014.
Victor R Basili, Lionel C. Briand, and Walcelio L Melo. A validation of objectoriented design metrics as quality indicators. IEEE Transactions on Software Engineering, 22(10):751{761, 1996.
Lionel Briand, Khaled El Emam, and Sandro Morasca. Theoretical and empirical validation of software product measures. International Software Engineering Research Network, Technical Report ISERN-95-03, 1995.
Lionel C Briand, Walcelio L. Melo, and Jurgen Wust. Assessing the applicability of fault-proneness models across object-oriented software projects. IEEE transactions on Software Engineering, 28(7):706{720, 2002.
Pierre Carbonnelle. PYPL. http://
pypl.github.io/PYPL.html.
Shyam R Chidamber and Chris F Kemerer. A metrics suite for object oriented design. IEEE Transactions on software engineering, 20(6):476{493, 1994.
Matthew Finifter, Adrian Mettler, Naveen Sastry, and David Wagner.
Veri able functional purity in java. In Proceedings of the 15th ACM conference on Computer and communications security, pages 161{174. ACM, 2008.
Chris Forbes. OpenRA. https:// github.com/OpenRA/OpenRA, 2009. [0xd16] [Akk14] [BBM96] [BEEM95] [BMW02] [Car] [CK94] [FMSW08] [For09]
Tibor Gyimothy, Rudolf Ferenc, and Istvan Siket. Empirical validation of objectoriented metrics on open source software for fault prediction. IEEE Transactions on Software engineering, 31(10):897{910, 2005.
GitHub. Closing issues using keywords. https://help. github.com/en/articles/ closing-issues-using-keywords, 2019.
David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. Applied logistic regression, volume 398. John Wiley & Sons, 2013.
Ilja Heitlager, Tobias Kuipers, and Joost Visser. A practical model for measuring maintainability. In Proceedings of the 6th International Conference on Quality of Information and Communications Technology, pages 30{39. IEEE, 2007. icsharpcode. OpenRA. https:// github.com/icsharpcode/ILSpy, 2009.
Jelly n. Jelly n. https://github.com/ jellyfin/jellyfin, 2013.
Ron Kohavi et al. A study of crossvalidation and bootstrap for accuracy estimation and model selection. In Ijcai, volume 14, pages 1137{1145. Montreal, Canada, 1995.
Mehdi Khalili. dnspy. https://github. com/Humanizr/Humanizer, 2012. [Lan17] [LV97] [McC76] [Mic]
Erik Landkroon. Code quality evaluation for the multi-paradigm programming language Scala, 2017. University of Amsterdam.
Filippo Lanubile and Giuseppe Visaggio.
Evaluating predictive quality models derived from software measures: lessons learned. Journal of Systems and Software, 38(3):225{234, 1997.
Thomas J McCabe. A complexity measure. IEEE Transactions on Software Engineering, (4):308{320, 1976.
Microsoft. C# update notes.
https://docs.microsoft.com/
en-us/dotnet/csharp/whats-new/
csharp-version-history.