Having highly maintainable software decreases the time spent on development. Although various research e orts show that the names of identi ers play a large role in the readability and maintainability of code, code quality assessments often do not take these names into account. Although developers can usually quickly assess the quality of a name, the abstract nature of names makes a fully automated assessment di cult. This research investigates the creation of a general naming quality model. Our proposed model assesses: a) the syntactic quality of Java method names, b) how well a method body matches its name semantically. We assess this using 1) a set of guidelines from literature, 2) a machine learning algorithm trained on AST representations of method bodies.
Initial results show that the combination of a rule-based approach and a deep learning model can correctly indicate what names need attention. By inspecting the names agged as a violation by both approaches we found that the combination of syntactic and semantic information yields better results than either of them by themselves. Further validation experiments on a Github commit dataset show that the model can distinguish between good 1
Software quality plays a crucial role in the eld of software engineering [ISO10]. Higher quality software enables developers to implement new features and changes faster, lowering the development cost [HKV07]. Maintainability is one of the factors that de nes the quality of the software, which is further divided into modularity, re-usability, analysability, modi ability, and testability [ISO10]. This research will be related to the analysability and modi ability factors of the source code. Although source code properties that de ne these factors have been described [HKV07], the names used in the code are not yet used to in uence these properties, although they play a signi cant role in it.
Making code easily understandable by using proper names in uences the analysability and modi ability of code, and therefore the maintainability of a codebase [Mar08; McC04]. Every variable, class, argument, method, le and directory is named by the developer. Well thought names are the rst step in aiding the developer to gain an understanding of the code [Mar08; McC04]. If a developers chooses bad identi er names the functionality of the code is heavily obfuscated. For example, if a method is named `b', no developer can guess the intention of it. When it is named `calculateMedian' however, its functionality is clear from just the name.
The importance of naming is supported by Lawrie et al. [Law+06], who state that next to the comments, the developer is almost exclusively dependent on the naming of identi ers to understand the code. According to Murphy-Hill et al. renaming is the most commonly occurring refactoring done in codebases [MPB12]. Research has also shown that low quality identi ers can be related to lower quality code [But+10; But+09; Law+06; LFB06; LFB07; Rel04]
Despite the relevance of naming quality there is currently a lack of maintainability models and code quality analysis tools that take identi er naming into account. To overcome this we investigate the creation of a naming quality model that can improve maintainability models and that can be used in quality analysis software.
Names are harder to validate than other code metrics, since good names must be accurate abstraction of the source code it represents (its context). This abstraction of context re ects the meaning of the code, which we will refer to as the semantics of a name. To verify whether a name matches its context some understanding of the source code is necessary, which is easy to assess for a human but di cult in an automated assessment. In contrast to this are the syntactic qualities of a name, characteristics of the name itself, such as spelling and structure, which are more easily veri ed.
To do naming quality assessment we will employ a combination of a ruleset and machine learning model learned on code representations, which combines the syntactic and semantic aspects a name should adhere to.
Syntactic aspects of a name are well-researched and guidelines to which good names should adhere to can be found in literature [Rel04; But+09; Abe+09]. However, as stated before, to improve comprehensibility of code it is also essential that the method name correctly represents its method contents.
To achieve this, machine learning is employed to assess the semantics of the name. To train this model, it receives a large number of method names and their bodies as input. The names are represented as natural language, and method bodies as an Abstract Syntax Tree (AST). From these examples, the model learns code embeddings, which are numeric vector representations, for both the names and bodies. This results in similar method bodies having similar vector values, which also applies to method names.
We can then infer from this model what method body embeddings are often found with what method name embeddings, and base a prediction on what name would t a body on this [APS16; Pen+15; Liu+19; Alo+19; ALY18]. Comparing this name to the actual name can inform us how well the real name ts the method. This semantic check can then be used in combination with a syntactic check, which will result in a more complete naming quality assessment. 1.1
In this study we build upon existing research in naming quality to close the gap in software quality analysis by including a naming quality assessment. To achieve this goal we propose a model that uniquely combines syntactic and semantic information. Syntactic information is used since the guidelines and regulations code should adhere to are a tested and proven contributor to naming quality [But+09; But+10]. We improve on these known syntactic guidelines by introducing semantic information into the quality analysis model, which allows us to assess how well a method name ts the body. Semantic information has been extracted from code before but it has not been used to assess naming quality.
We validate this proposed naming quality model by building a proof-of-concept implementation that uses these concepts to generate a quality overview for a project.
Not only do we create a naming quality model and a proof of concept implementation, we also relate the assessment of a codebase using this model to other code metrics showing the relevance of the model and naming in general. Lastly, we contribute to a line of research combining expert knowledge with statistical machine learning approaches.
Initially this research will target method identi er quality, as this is the smallest abstraction unit with signi cant context, and is essential for understanding the code. Also, we build our tool to work for Java as a start since it one of the most widely used programming languages.
Ideally, the nal model can be employed by developers to quickly assess the general naming quality of their codebase and compare it to other codebases. Additionally, it allows them to nd occurrences of names that need attention to improve the maintainability of their code. The quality assessment performed by our proof-of-concept model will be validated by inspection, comparing it to other code metrics such as readability, and by running it on datasets with justi ed naming quality.
In this research we aim to answer the following research questions, focused towards investigating, validating, and improving the proposed model.
RQ1: How well does the proposed model perform.
RQ2: How does the syntactic correctness checker compare against the semantic correctness checker. RQ3: Can we improve the semantic model using the syntactic knowledge.
To create a naming quality model we have to be able to infer if the name itself is syntactically correct, and if it accurately re ects the context the identi er is used in (semantics), which for method names we de ne as the method body. We rst analyse research conducted on assessing the syntactic quality of names in code, and then dive into research using a combination of natural language processing and machine learning to analyse source code, which is what we will use to assess the semantic quality of names. The section ends with an overview of how a naming quality model could be validated based on related approaches. 2.1
A common approach to checking identi er syntax is to de ne rules that de ne what good quality identi ers need to adhere to [DP06; But+11; Esh+11; CT99; Abe+09]. These rules usually only focus on checking the structure of the name itself.
An example of using rules to de ne naming quality can be found in the studies by Butler et al. [But+10; But+09]. They used a list of twelve identi er guidelines aimed at the contents of the name itself, such as maximum words used in the identi er, or encodings in the name. They show that adherence to these guidelines can be related to less bugs. In contrast, Lawrie et al. [Law+06; LFB07] de ne good identi er quality based on the quality of just the words the name consists of. They collected a dictionary of English words, which had known abbreviations added, and argue that if the words in the name are not in this dictionary, the name does not have a clear meaning.
A di erent approach focuses on a set of consistency, conciseness, and composition rules [DP06]. Although useful in some cases, these rules are only applicable on 10% of all identi er names according to the results. To work well they also need developers to keep an identi er dictionary during development, containing all identi ers. Their rules are therefore hard to include in a fully automated and general assessment of naming quality and has a limited application area.
Other work has been done on investigating the identi ers used in the code. These attempts are however aimed on a very small syntactic aspect of naming (e.g. only that not both CamelCase and snake case are being used, or only detecting homonyms), do not take semantic context into account, or do not do a quality assessment. [DP06; CT99; CT00; Abe+09; AT13; Arn+10; Esh+11; H 09; All+14; All+15; Liu+19].
The most complete overview of what encompasses naming quality was de ned by Abebe et al. [Abe+09] who de ned so called lexicon bad smells, which, according to the authors, are indications of deeper code problems, just as normal code smells are. From this research, as well as the other studies, we collected a set of guidelines on the syntax of method names to use in our syntactic checker. 2.2 To compare a name and its context semantically in an automated assessment, both the name and the context need to be represented in the same space. These representations can then be compared against each other to calculate similarity and possibly indicate where names di er greatly from the context they represent. One method of matching an identi er and its context focuses on treating the words in both the identi er name and the context around it as English sentences [DDO11; Sri+10; AT10]. Representing both a method and a name as an English sentence e ectively places the problem in the Natural Language space. This method assumes that: a) the vocabulary used in both the name and its context are similar; b) the structure of the code is not necessary to accurately assess the context of a name. The advantage of this languagebased method is that we can lean on extensive research in comparing the semantics of English sentences. However, a lot of code speci c information is lost when translating code to an English sentence, making this method of representing code and names un t for our purposes.
Another method of representing information, which is widely used, is representing things as a numeric vector called an embedding. The idea is that semantically similar objects are mapped to similar vectors, which allows for comparison.
The rst approaches in this area were on natural languages, not on source code [Mik+13]. These approaches use neural networks to create word embeddings based on the words that are around the word of interest, e ectively representing a word as a vector of the words it co-occurs with. These embeddings are then used for a plethora of applications, such as machine translation.
New approaches then tried to create these embeddings from source code to try and leverage generalisable concepts from code. It is hypothesised that since software is made by humans, the language used has similar statistical properties as natural languages. We can use this approach to represent both the name and the method as a vector so they can be compared. Transforming input into a vector is usually done by training a machine learning model on a large amount of data, which then learns which input data corresponds to what vector. Research has shown that is is possible to train probabilistic models from existing code that can reason about code not seen before [All+18].
Some approaches transform the source code into a token stream and use it as input for the machine learning model [All+15; APS16], where others attempts build an Abstract Syntax Tree (AST) out of the source code and use that as input [Pen+15; Liu+19; Alo+19; ALY18]. This input is then fed into a machine learning model to create numeric vector representations. These approaches were tested with a method name prediction task, where a model performs good when it predicts the exact name of a method body based on its vector. The second approach (AST based) achieves better results in terms of precision and recall on this task.
To evaluate whether method bodies and names match, we can use these vector representations as created by a machine learning model to compare them. 2.2.2
After transforming both the method name and body into a similar representation, there are multiple ways to use them to do a naming quality assessment. The simplest approach is calculating the similarity between a name and a body and using that as a measure.
A language-based approach to this is taken by Arnaoudova et al. [Arn+10] who take identi ers and comments from the body of a method and calculate the textual similarity between them. De Lucia et al. [DDO11] take it one step further and convert the extracted words in a vector representation so the vectors spaces of both methods and names be compared. Sridhara et al. [Sri+10] use a Software Word Usage Model (SWUM) to extract the action, theme and arguments of a given Java method. They are then able to generate a single English sentence that re ects what the method does. Such a method can be used to extract a sentence that should represent the method body, and compare it with the actual name. The biggest problem with these approaches in relation to our envisioned model is the fact that we only have a very limited amount of text when we look at method identi ers. The similarity between a large amount of text extracted from an entire method including comments is not that accurately comparable to just a name.
Other approaches make use of machine learning to train a model to recognise similar representations. Advantages of this are that more complex statistical relations can be modelled.
Liu et al. [Liu+19] use paragraph vectors and convolutional neural networks to extract embeddings of method names and bodies that can be used as input to train a deep learning model. For each method name and body name, their model selects all other names and bodies that are close in the vector space of embeddings. When the name and the body have no overlap in their respective space, they are considered inconsistent, and names of other methods that are close (i.e. that are very similar to the snippet of code) are suggested.
Although their goal is similar to ours, no quality assessment of any kind is being performed, which is the area we want to apply the models knowledge.
We can however perform a quality assessment by using a model that was introduced by Alon et al. [Alo+19]. They trained a neural machine learning model to predict the name of a method based on ASTbased representations. Since the code2vec model by Alon et al. is completely open source, it can be employed by us to take it one step further and relate the output of the model to naming quality. 2.3
To assess the performance of our model, we want to validate whether the names indicated by our model as low quality decrease the maintainability of the code, and that the names not agged as violations do not decrease it. This is applicable to both the syntactic and semantic approach. On the one hand we need to relate our syntactic guidelines to improved code quality, and on the other hand we want to relate our semantic quality assessment to improved code quality.
To show the e ect of low-quality identi ers Butler et al. have done two di erent empirical studies that show that syntactically awed identi ers have a negative inuence on code quality [But+10; But+09]. Although their guidelines are very basic and do not re ect the entirety of naming quality, they have shown that already neglecting these basic guidelines relates to more code quality issues reported by a static code analysis tool called FindBugs. To test whether guidelines help developers create better names, Relf [Rel04] identi ed a set of twenty-one guidelines and did a small study with developers. In this study developers were asked to accept or discard the guidelines provided. All guidelines except one were con rmed by the developers as helpful in creating more descriptive identi ers. These two ndings combined show that guidelines do indeed lead to improved naming quality.
When looking at the semantic assessment, we need some other model or code metric to compare the output to. General code readability models only take structural aspects into account [Dor12; PHD11; BW10]. Structural aspects for example are number of comments or the length of a line, in other words syntactic aspects. Scalabrino et al. [Sca+16] show that adding textual aspect such as contents of comments and identi ers would improve those models. To do this, they add multiple textual aspects to the model. These aspects are comments and identi ers consistency, the number of dictionary terms in identi ers, hypernym density, textual coherence, comments readability, and number of meanings of a snippet.
The metrics used by both Scalabrino et al. [Sca+16] and Buse et al. [BW10] to de ne code readability can be used by us to compare the assessed naming quality. They also have both created an annotated dataset of methods with readability scores. This readability score was obtained by letting computer science students and developers go over les and awarding them a score. Although this score is clearly not a perfect truth, signi cant similarity between the scores from the assessors and the scores from our model can indicate our model approaches comprehensibility in a correct way. The biggest problem is that the assessment is done on readability of the code snippets, and not on the quality of the names. The scores are given for readability of the code, not the names. Even though naming is a big part of readability the scores are not exclusively a naming quality metric.
Beside relating the model to readability scores one could also evaluate a model by comparing it to a dataset of known good and bad names. Unfortunately, there is no gold standard of high-maintainable and readable code to compare a model against. To evaluate their model, Liu et al. [Liu+19] collected a dataset of methods whose names were changed in a GitHub commit whereas the body remained the same. The names before the change are considered inconsistent with their body, and the names after the commit consistent. This dataset can be employed by us to test whether our model also ags the pre-commit names as being low quality more often. 3
In this study we assess naming quality in two ways, syntactic correctness and semantic correctness. The model we porpose consists of two di erent parts, both having the objective to assess naming quality. The rst part does a syntactic quality analysis based on literature guidelines, whereas the second part does a semantic analysis of the method name and body. The output of both parts, as well as the complete model, is mapped to a binary assessment (\violation"/\no violation") per name so that the results are comparable.
Our intuition is that the combined model will achieve better results in detecting low quality names than both approaches by themselves can. Next steps in combining the two approaches are described in more detail in Section 7.1.
An overview of the proposed model can be found in Figure 1. Each of the following sections highlights a part of the setup. 3.1
To create a naming quality model we rst need to be able to extract the relevant pieces of information from the source code. With the use of JavaParser1 we create an AST representation of the source code which we can traverse to collect the info we needed. For the syntactic analysis, we parse the source code into a Java HashMap containing the le a method is in, the method name, and the return type.
For the semantic analysis, we need the output of the parser to match the input the code2vec neural models expects. Code2vec takes an AST as input, and generates all possible paths and their embeddings for it, see also Section 3.3. For this reason, we pass the entire AST to both approaches. 3.2
We analyse the information retrieved from the parser using two di erent approaches. The rst approach uses a rule-based model that checks the method name itself for violation of guidelines. As we concluded in our background section, research has been done that assesses the source code lexicon and indicates what high-quality identi ers should adhere to to increase readability and understandability. This research shows that a syntactic quality assessment is possible and bene cial. From the literature we collected a method naming ruleset that method identi ers should adhere to to be considered of high-quality, as shown in Table 1.
To check the source code for adherence to these guidelines, the output of the Java Parser is written to a le and read by a Python script. This script goes over all method names to check their adherence to the guidelines. The result is an overview of which method violates which guideline, and a total number of violations per guideline. Thereby laying the foundation of further experiments relating syntactic to semantic assessments.
Based on preliminary experiments some adjustments were made in the guidelines. We removed guideline two: \A method name should be composed of words found in the dictionary, and abbreviations and acronyms that are more commonly used than the unabbreviated form" since every new project introduced new abbreviations that were clear in their context, and as stated before, when an abbreviation is 'known' or more commonly used than its non-abbreviated form is hard to assess. 3.3
The second approach to assess naming quality revolves around learning a neural model on the extracted source 1https://javaparser.org/ code data, and allowing it to assess what name would best t a code snippet based on learned experience. This predicted name is then compared to the actual name.
Recent work by Alon et al. [Alo+19] resulted in a neural attention model that maps a method body onto a method name. This model takes an AST representation of a method body as input and returns an estimation of the name it predicts the body has, with di erent predictions and the con dence for each prediction. The attention mechanism of their model allows us to inspect how the model comes to its predictions and could teach us how to improve the model.
From the AST representation, each possible path through the AST is saved in a so called path-context containing the starting token, the terminal token, and nodes between these two tokens (e.g \x = 7" would become: hx; (N ameExpr " AssignExpr # IntegerLiteralExpr); 7i [Alo+19]). As seen in the example, the nodes are saved as a simpli ed representation, and arrows between the nodes denote if the next node is a parent or child node.
An example of the models output can be found in Figure 2 and 3, with the rst example showing an example where the model correctly predicts the real name, and a second example where the model has more trouble predicting what the name should be.
We adapted the code2vec model to generate a large number of expected names for method bodies, and compared these expected names to the real names using natural language similarity methods.
To detect names that \violated" the ML model, we calculated the textual similarity between the guessed name and the real name using the Jaro-Winkler distance, putting more emphasis on the start of the name since this is de ning for its functional intention. If the similarity was lower than .6 we considered the name to be too far o from the prediction, and counted it as a violation.
As described above we use this machine learning approach to validate the semantic integrity of a method name and the body it represents. In this study we compare this approach to the syntactic approach using guidelines from literature, as well as combine them to validate the structural and syntactic integrity of a name. Together they form the basis of our naming quality model. To assess the performance of the model and answer RQ1, we need an oracle that allows us to evaluate the output. This oracle should tell us how often the model is correct and how often it is not. Having this will also allow us to set a baseline to compare future improvements and iterations of the model to. To achieve this, we performed two experiments to try and validate the performance of the model.
As a rst validation attempt, we looked into the dataset made by Scalabrino et al. [Sca+18] and the dataset made by Buse et al. [BW10]. Both datasets contain a variety of Java code snippets with readability assessment scores of that snippet done by students and developers. For each snippet, we let both models check for violations and compared it to the average readability score given by the assessors.
If we can relate it to the readability assessments this could show that by our model highly-evaluated snippets are also considered highly readable by their model. Every new snippet can then be validated as either high or low quality considering their names based on the readability score. This could then represent our oracle that we base the models performance on.
Next to their datasets, we can use to readability score calculation tool as developed by Scalabrino et al. [Sca+18] that uses textual aspects to calculate a readability score for a snippet. If these score can be related to the amount of violations found by the model, we can use it as an oracle to de ne whether the model is correct. 4.1.2
As a second validation, we looked into the approach taken by Liu et al. [Liu+19] as explained in Section 2.3. They propose a dataset of method names that were changed in a commit where the body remained the same. They reason that since the name is changed while the body is not this indicates a change was needed to better re ect the body, meaning the old name did not re ect the content of the body. Therefore the names before the commit are seen as awed names, and the names after the commit as not awed.
From their dataset we removed duplicate method names which would skew the results, since the names all either do violate or do not violate the guideline, as the bodies are nearly equal. If the model happens to perform well in those occasions the overall performance seems better than it would on a more varied dataset. The resulting dataset contains 1597 methods.
For our experiment, we compare the assessment of our model on these names and their respective bodies to this quali cation. Ideally, all pre-commit names should be indicated by our models as awed, whereas all after-commit names should not be indicated as such. The underlying assumption made here is that no pre-commit names were already of high-quality and that every post-commit name is of high-quality.
Comparing the two di erent approaches tells us how they di er, what bene ts and disadvantages each approach has, and how each approach performs. To answer RQ2, we investigated the output of both parts of the model both by looking at the number of agged violations. We also performed an in-depth analysis of the models outputs to see what names they ag as violations and what names they do not.
A rst experiment to compare the syntactic checker to the semantic checker was done by running the model over di erent Java projects and outputting the number of methods that each approach ags as a violation. This gives us insight in the di erence between the semantic and the syntactic model, allowing a rst answer to RQ2.
For this analysis the codebases HadoopMapReduce2, Presto3 and Cassandra4 were used.
We report the number of methods that each model ags as a violation. The rule-based model outputs whether or not a name violates at least one of the guidelines. For the neural model we used the textual similarity between the actual name and the predicted name as explained in Section 3.3 to ag a name as a violation or not.
Besides plain numbers an in-depth qualitative analysis of the output will be reported, which further answers RQ2. 5
The following section shows the results from the experiments explained in Section 4, and what we can learn from the resulting data. We also relate the results back to the research questions, evaluating whether a semantic analysis is possible and how the results di er from a syntactic analysis. 5.1
To evaluate the performance of the model, we performed two experiments with di erent datasets to see how well they are usable as a validation. 5.1.1
The results of this experiment are shown in Table 2, where \at least one violation" indicates that there was at least one method in the snippet that violated both models. The snippets were overall small of size and mostly only contained a single method, so at least one 2https://github.com/apache/hadoop 3https://github.com/prestodb/presto 4https://github.com/apache/cassandra method in most occasions is identical to all methods in the snippet.
The results show that there is no clear relation between the readability scores given be the assessors and the violations indicated by the model. This can be explained by the assessors not taking the name itself too heavily into account when assessing the readability.
The readability calculation tool from Scalabrino et al. [Sca+18] was also used to assess whether these scores could be related to the naming quality model. This tool uses linguistic metrics of the source to calculate a readability score for a code snippet. It was however found that the impact of names on the readability tool was not in line with standard coding guidelines of our own tool. As an example we take the code snippet as shown in Listing 1, but with di erent names. Changing the name from 'median' to 'b' changes the readability from 0.508 to 0.531, indicating an increase in the readability according to the readability model. Changing it again to 'calculateMedian' gave a readability of 0.466, lowering the readability score even more. This shows that a change in the name of a method does not translate that well to the readability measurements.
These readability assessments therefore were deemed un t to base our validation on. These results also again show that validating the quality of a name used for a method is a di cult thing to do. public Integer median() f if(p1.size()==p2.size())
return (p1.peek() + p2.peek())/2; return p1.size()>p2.size() ?
p1.peek() : p2.peek(); g Listing 1: Code snippet uses to assess the in uence of the name on the readability score as calculated by the tool from Scalabrino et al. 5.1.2 The result of the analysis can be found in Table 3.
Total Violated both models - similarity score <0.6 - violated at least one rule Violated at least one rule Similarity score <0.6
Number of methods Pre-commit Post-commit name name 1597 1597
From this data we can see that the number of old names that violated the model is signi cantly higher than the number of new names. If both the dataset and the model were perfect however, then the result should be 100% and 0%. Not every name in the dataset is actually improved after the change, or is improved but still awed. Examples can be found in ``read self" being changed into ``self read" or ``get current size" into ``size", and new names being ``forcibly refresh all cluster state slow" or ``point num bytes". These entries in the dataset result in the model agging some new names as awed, which is not surprising as these names can be considered awed. This does however not make up for the entire di erence between ideal results and achieved results, as the model is also not perfect. These results set a baseline for this version of the model, which we can compare further iterations of our model to. An improvement on the assessment done on the dataset indicates that the model is better capable at distinguishing between good and awed method names, which allows us to answer RQ3 in the future. 5.2
The results of experiment 2 are shown in Table 4.
Cassandra 13684 149 389 328 2508 6360 5250 4177 8078 7005
As seen in the table, both approaches nd around the same number of methods violating the set restrictions. From all names detected as awed by each approach, about half of each is also agged by the other. The absence of a ground truth makes it di cult to assess the number of false positives and negatives. The low overlap is in part caused by the distinct nature of both approaches. Where the rule-based model assesses the syntactic integrity, the neural model focuses on semantic similarity.
These results indicate that improvements can be achieved when combining the semantic and syntactic information instead of using either. Almost half of the violations found by each checker are not agged by the other. This shows that using only a guideline based approach achieves di erent results than a semantic analysis, giving an initial answer to RQ2, although the exact way they di er is not yet clear. Additionally, we see a signi cant di erence in the percentage of methods found violating both checkers, showing that a difference in naming quality between projects could be indicated. 5.2.1
To gain a feel for what kind of names each approach assessed as low quality, where they di er and whether semantic analysis is possible, we performed a manual qualitative inspection. This inspection gives us more insight into RQ2. In this analysis, we went over the output of the model consisting of method names and bodies, the predicted name and the similarity score, and the list of guideline violations.
When looking at the syntactic analysis, we found that guidelines 1, 4, 5 and 6 were often violated (>10% of names), where the other guidelines were only violated sporadically (<1%).
Among the names inspected that violated both models, no names were found that according to our opinion should not be improved. For example, the name ``capacity" was predicted by the ML model as ``get starts" (similarity: 0.3166) and violating both rule 1 and 5. This indicates that both the syntactic and semantic quality were low.
Names that violated only the guidelines were often single-word method name, which violate guideline 1, but were not found by the ML model. For example method name \handle": was predicted to be \handle and thus had a similarity score of 1, even though the name is not very informative. This is likely caused by pollution of the training data of the code2vec model [Alo+19].
Names that violated only the ML prediction seemed to be a false positive in at least one in four inspected violations. This was caused by the predicted names being of low quality themselves. A correct example is method name ``double who am i" which was predicted to be named ``test login". ``double who am i" is not a very descriptive name and even though ``test login" is not perfect either, the fact that they are very di erent is important for our assessment. Other violation indications were less correct, as for example method name ``decode initial context token" was predicted to be ``put". Although their textual similarity is low and therefore should indicate a name not correctly describing the intention, the original name is actually quite descriptive and looks right, while the predicted name is uninformative and even violates the set guidelines. Investigating what causes this might lead us to being able to improve to model and prevent these mistakes from happening.
We found no names that violated neither of the models but could still be considered low quality names. Although this does not proof the model had no false negatives, it shows promising indications that the model can detect low-quality names using a combination of both approaches, without missing out on many low-quality names. 6
The results indicate that a naming quality model that uses both syntactic content and semantic context to assess the quality of identi ers is possible for the case of Java methods. We can positively answer RQ1 since from the output we see that the model seems to be able to detect what names need attention. The model also nds far more violations in pre-commit names compared to post-commit names, which also should be of higher quality.
When looking at the results of Experiment 2, we can see that the syntactic and semantic approach di er in what names they ag as violations. This provides an answer to RQ2, as well as indicate that there is bene t to be gained by combining both approaches.
The results also show that the model has room for improvement. As stated in Section 5.2.1, the machine learning model sometimes predicts names that are in con ict with the guidelines itself, which shows that sometimes a disparity between the predicted name and the real name does not tell us much about the quality of the name. The model is also sometimes simply not capable of accurately estimating the name the method should have.
These awed predictions can be due to the fact that machine learning model used to assess the similarity between a method's name and its contents is based on the assumption that the names in the training data are consistent with the body they represent. If the training data is awed and contains a lot of names that do not re ect their body well, the output of the model on new data will be in uenced by it. Since we have no large dataset of veri ed high-quality and low-quality names, this is unavoidable. A perfect assessment therefore seems not achievable, although our model does produce useful ndings. 6.1
Although the initial results are promising, there are some issues we would like to address.
It was mentioned earlier that the dataset based on commits as described in Section 5.1.2 is not a perfect truth. The degree to which a disparity between the expected ideal output of our model and the actual output can be addressed to either is hard to assess. Names could be changed in a commit due to other reasons than it not re ecting the body, such as adherence to naming conventions. To validate the correctness of the dataset, a manual evaluation by a group of professional developers could help, but a perfect yes/no golden dataset cannot be achieved since a 'good' name is not a binary metric. This also shows why a good naming quality model would be valuable.
The number of names indicated by the semantic model to be inconsistent with their body, which are not actually inconsistent, form a big threat to the models performance. This namely means that the machine learning model is not always predicting high-quality names itself, which is an assumption the quality assessment is based on. We address this issue Section 6.2.
Another point we want to address is that the qualitative analysis was done by the authors, instead of a group of volunteers, which might have led to bias in the interpretation of the models output. Another validation of the model done with a group of volunteers should help remedy this threat, although subjective opinion will always play a role in manual assessments.
The last issue is the fact that there is exists large number of naming guidelines, which only a subset of is employed by this research. Although these guidelines have been validated by other research, including other guidelines might in uence the results. A complete list of all guidelines in existence is not feasible, but a more extensive list than currently employed could be benecial in the future to achieve better results. 6.2
Although the current implementation of the proposed model shows promising results, there are di erent possible changes that can be made that might lead to better performance, which are addressed in this subsection.
The way the similarity between the predicted name and the actual name is calculated can be done in many di erent ways, and investigating the impact of di erent approaches could lead to increases performance. At the moment, a Jaro-Wrinkler textual similarity is being used, whereas one could also use code or word embeddings to calculate how similar both vectors are in their embedding space.
There are other machine learning approaches that also generate a descriptive name or sentence based on the contents of a method body. These approaches for example use a token stream instead of an AST-based approach. Although the code2vec model by Alon et al. [Alo+19] achieved the best recall and precision, this was based on predicting the correct names, while for quality assessment other approaches might work better, making it worthy of investigation.
Even though the model achieves good results for assessing the quality of Java method names, one could also include other abstraction units such as classes in our assessment. This extra context knowledge could lead to better prediction of names and less falsepositives.
It is also interesting to investigate how well the model applies to other programming languages. In theory, the model is generally applicable and can be extended to other languages, but it is worthy of investigation to see how the results di er.
So far, a golden dataset with annotated naming quality has not been found, making precision and recall no viable option for accuracy measurements. Finding one would make more extensive validation possible, so the search for one is still bene cial. Additionally, manual assessment of a the quality of a large part of methods by a group of developers could be done to create one ourselves. 7
To automatically assess the quality of identi er naming to enhance maintainability assessments, this research proposed a naming quality model that employs both a naive ruleset to check the syntactic quality of method names, and a Neural Language Model to check the semantic quality.
To answer RQ1: \How well does the proposed model perform." we employed readability scores, which were inconclusive, and the Github commit dataset, which seemed to be useable for validation. This second experiment gave insight into the performance of the model, showing that the model nds less violations in the pre-commit names compared to the post-commit names, which is in line with the expectation that precommit names are of lower quality.
For RQ2: \How does the syntactic correctness checker compare against the semantic correctness checker." we compared the output of the semantic model to a syntactic one. From the names agged as violations by the semantic model, about half are also agged by the syntactic quality model, and vice versa, giving a rst answer to RQ2. A more in depth analysis of the output has shown that a combination of both approaches seems to lead to the best results, as names agged by both approaches were found to be of lowquality. By itself the semantic model su ers from a lot of false positives where the original name is a good one and the predicted name is not. This results in names being agged as a non-descriptive while they actually are descriptive. By combining the semantic and syntactic checker the number of false positives seems to go down. This does lead to less low-quality names being found, but does indicate what names most need attention.
Initial validation attempts by relating it to other metrics show that a combination of both models seems to catch a lot of names that need attention, but the number of false positives and negatives is still significant. Validating the quality of a name is a di cult task, but our current approach is a step in a promising direction. 7.1
As explained above the machine learning model is not always predicting high-quality names. This is an assumption the quality assessment is based on, and vital for the performance. To overcome this and to answer RQ3: \Can we improve the semantic model using the syntactic knowledge." we will try to infuse the neural model with rule-based knowledge.
Rules in combination with a deep learning model show results that exceed using only either one [Vil+11; GAS16]. We could use our ruleset to instead of just checking the name beforehand, allow it to in uence the training of the neural model. The model would then adjusts its prediction based on the guidelines and optimise towards high-quality names as well; instead of just predicting the right name. This could help the predicting of low-quality names that itself violate the guidelines set by literature. This deeper integration of the guidelines would theoretically decrease the false positives from the neural model, and answer RQ3. This will be our rst step towards attempting to improve the models capability of distinguishing between good and awed method names. Uniquely this would combine literature-based approaches with deep learning approaches to assess the quality of names used in a project. [Abe+09] Surafel Lemma Abebe et al. \Lexicon bad smells in software". In: Reverse Engineering, 2009. WCRE'09. 16th Working Conference on. IEEE. 2009, pp. 95{99. [All+14] [All+15] [All+18] [Alo+19] [ALY18] [APS16]
ural coding conventions". In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM. 2014, pp. 281{293. [DP06] [AT10] [AT13]
In: Program Comprehension (ICPC), 2010 IEEE 18th International Conference on. IEEE. 2010, pp. 156{159.
\Automated identi er completion and replacement". In: Software Maintenance and Reengineering (CSMR), 2013 17th European Conference on. IEEE. 2013, pp. 263{ 272. [But+09] Simon Butler et al. \Relating identi er naming aws and code quality: An empirical study". In: Reverse Engineering, 2009.
WCRE'09. 16th Working Conference on.
IEEE. 2009, pp. 31{35. [But+10] Simon Butler et al. \Exploring the in uence of identi er names on code quality: An empirical study". In: Software Maintenance and Reengineering (CSMR), 2010 14th European Conference on. IEEE. 2010, pp. 156{165. [But+11] Simon Butler et al. \Mining java class naming conventions". In: Software Maintenance (ICSM), 2011 27th IEEE International Conference on. IEEE. 2011, pp. 93{ 102. [BW10] [CT00] [CT99] [DDO11]
In: IEEE Transactions on Software
Engineering 36.4 (
omen: Analyzing the language of function identi ers". In: Reverse Engineering, 1999. Proceedings. Sixth Working Conference on. IEEE. 1999, pp. 112{122.
and Rocco Oliveto. \Improving source code lexicon via traceability and information retrieval". In: IEEE Transactions on Software Engineering 37.2 (2011), pp. 205{227.
ability model". In: MCS Thesis available from (http://www. cs. virginia. edu/~ weimer/students/dorn-mcs-paper. pdf ) (2012).
Florian Deissenboeck and Markus Pizka. \Concise and consistent naming". In: Software Quality Journal 14.3 (2006), pp. 261{ 282. [Esh+11] Laleh M Eshkevari et al. \An exploratory study of identi er renamings". In: Proceedings of the 8th Working Conference on Mining Software Repositories. ACM. 2011, pp. 33{42. [GAS16]
Murray Shanahan. \Towards deep
symbolic reinforcement learning". In: arXiv
preprint arXiv:1609.05518 (
Binkley. \Quantifying identi er quality: an analysis of trends". In: Empirical Software Engineering 12.4 (2007), pp. 359{388.
Refactor Inconsistent Method Names". In: 41st ACM/IEEE International Conference on Software Engineering (ICSE). IEEE. 2019.
Robert C. Martin. Clean Code: A Handbook of Agile Software Craftsmanship. 1st ed. Upper Saddle River, NJ, USA: Prentice Hall PTR, 2008. [Mik+13] [Pen+15] [PHD11] [Rel04] [Sca+16] [Sca+18] [Sri+10] [Vil+11]