English. Source code comments provide useful insights on a codebase and on the intent behind design decisions and goals. Often, the information provided in the comment of a method and in its corresponding implementation may be not coherent with each other (i.e., the comment does not properly describe the implementation). Several could be the motivations for this issue (e.g., comment and source code do not evolve coherently). In this paper, we present the results of a manual assessment on the coherence between comments and implementations of 3; 636 methods, gathered from 4 Java open-source software. The results of this assessment has been collected in a dataset that we made publicly available on the web. We also sketch here the protocol to create this dataset.
Italiano. I commenti al codice sorgente forniscono informazioni utili sull’implementazione del codice e sulle intenzioni relative alle decisioni e agli scopi del progetto. Spesso, le informazioni presenti nel commento di un metodo e nella sua implementazione possono non essere coerenti (nel senso che il commento non da` una descrizione adeguata dell’implementazione). Ci possono essere diverse spiegazioni per questo (ad esempio, commenti e codice sorgente non sono stati modificati in modo coerente). In questo articolo, presentiamo i risultati di una valutazione manuale della coerenza tra commenti e implementazione di 3; 636 metodi, raccolti da 4 applicazioni open source in Java. I risultati di questa valutazione sono stati raccolti in un dataset che abbiamo pubblicato sul web. Accenniamo anche al protocollo seguito per la preparazione del dataset.
Natural language is used in different ways in the
development process of a software system and
therefore techniques of natural language
processing (NLP) and information retrieval (IR) are more
and more frequently integrated into software
development and maintenance tools. Many
automated or semi-automated techniques have been
proposed to aid developers in the
comprehension and in the evolution of existing systems,
e.g.,
Natural language information is provided in
source code comments and in the name of
identifiers. In the former case, standard natural language
is usually adopted, although quite technical.
Comments are written in English, even when
developers have different mother-tongues. On the other
hand, identifiers are typically constructed by
composing multiple terms and abbreviations.
Therefore, more sophisticated techniques are necessary
to extract the lexical information contained in each
identifier
Most of these techniques assume that the same
words are used whenever referring to a
particular concept
Therefore, more sophisticated models are necessary to determine if there is coherence between the lexicon provided in comments and in its corresponding source code. Hence, there exists coherence between a lead comment of a method and its source code (also simply coherence, from here on) if that comment describes the intent of the method and its actual implementation.
In this work, we focus on the lead comment of
methods. This kind of comments precedes the
definition of a given method and is supposed to
provide its documentation and details about the
implementation. We discuss here a dataset we made
publicly available on the web.1 It contains
annotations about the coherence of 3; 636 methods
collected from 4 implementations of 3 open source
projects written in Java. The defined protocol used
for its creation is also sketched, to give researchers
the opportunity to possibly extend it. Further
details on this protocol can be found in
For the assessment of the quality of the
annotation with special focus on computational
linguistics applications, a few indexes have been
considered
We expect that making freely available this dataset could give impulse to the research for approaches to assess the coherence between the implementation of a method and its lead comment (simply coherence, from here on). In fact, although no approach has been yet proposed in this regard, they could be of great help for software maintenance and evolution activities.
The paper is structured as follows. In Section 2, we discuss the methodology used to create our dataset. A description of the main characteristics of the dataset is given in Section 3, while in Section 4 the annotation is assessed. Some final considerations conclude the paper.
1www2.unibas.it/gscanniello/coherence/
To create our dataset, we adopted the
perspectivebased and the checklist-based review
methods
shared our experience with the community
in
The construction of the dataset has been completed in two main consecutive phases by using an ad-hoc web system implemented for the purpose: Verify coherence. Annotators verify by means of a checklist the coherence between the lead comments of a set of methods and their corresponding implementation.
Resolve conflicts. The intervention of experts is required whenever the judgements of the annotators differ. In our case, two of the authors, with a background in software engineering, assumed the role of experts and examined the problematic cases. For each conflicting method, the experts should reach an agreement about the coherence or the noncoherence. Methods on which experts do not get a consensus are automatically discarded. 3
Some descriptive statistics (e.g., number of classes and methods) of the software systems in our dataset are shown in Table 1.
CoffeeMaker2 is a software to manage inventory and recipes and to purchase beverages. We chose this software because it has 2agile.csc.ncsu.edu/SEMaterials/ tutorials/coffee_maker/ a simple and clear design (being it developed for educational purposes).
JFreechart3 is a Java tool supporting the visualization of data charts (e.g., scatter plots and histograms). We included two versions of this software. As reported in Table 1, both these versions contain almost the 80% of methods with lead comments. This suggests an extensive use of comments, which is the main reason why we decided to include this software in the dataset.
JHotDraw4 is a framework for technical and
structured graphics. Even if the source code
of JHotDraw is scarcely commented (see
Table 1), it is well-known in the software
maintenance community due to its good
ObjectOriented design
In Figure 1, we report the implementation and the lead comment of setTickLabelsVisible (extracted from JFreeChart ver. 0.7.1, included in our dataset). According to our definition of coherence, we can assert that this method is coherent. On the other hand, the save method reported in Figure 2 provides a very poor and inadequate description of the design intent of the method, thus reflecting a lack of coherence with the underlying implementation.
Three annotators were involved in the dataset creation process. Two of them hold a Bachelor degree in Computer Science, and have very similar technical backgrounds. On the other hand, the third annotator can be considered more experienced than the other two since he holds a Master degree in Computer Science. We distributed the effort among the annotators so that each software 3www.jfree.org/jfreechart/ 4www.jhotdraw.org/ /** * Sets the flag that determines whether or not * the tick labels are visible. * Registered listeners are notified of a * general change to the axis. * * @param flag The flag to set. */ public void setTickLabelsVisible(boolean flag) { if (flag!=tickLabelsVisible) { tickLabelsVisible = flag; notifyListeners(new AxisChangeEvent(this));} } } would be separately evaluated by at least two annotators. This allowed us to have multiple judgements for each method in the dataset, and to calculate the rate of agreement among annotators. 4
The whole dataset creation process occurred from January, 15th 2014 to June, 20th 2014, for a total of 800 man-hours. This gives an estimation of the effort required to conduct the study presented in this paper, and provides an indication to the researcher interested in extending our dataset.
The annotators provided indications on the coherence of methods by assigning them one out of three following possible values: NonCoherent, Don’t Know, and Coherent.
In this scenario, we use the kappa index
The classical formulation of the kappa index
considers a binary classification problem (e.g.,
Non-Coherent or Coherent). However in our case,
the neutral judgement (i.e., Don’t know) is also
allowed. Therefore, possible disagreements
include the case where one of the two answers is
the neutral one. In this case, it is possible to
differently weigh the possible disagreements among
annotators. In fact, disagreements due to the
neutral answers are less serious than disagreements
where judgments are totally divergent (i.e.,
Coherent and Non-Coherent, in our case). To this
end, Cohen
The agreement between annotators is good on the first three systems, and acceptable for JHotDraw. However, on this system the difference between the values for UK and WK is large, thus providing a more accurate indication on the agreement of the evaluations on this software.
At the end of the first step of the dataset creation process, the number of methods on which annotators did not agree was 302, corresponding to the 8:3% of the total number of methods from all the systems in the dataset. Most of these methods are those in JHotDraw, as suggested by the kappa index values (see Table 2). These methods were reviewed by two of the authors. An agreement was reached on all of these methods, which were then included in the dataset. The total number of methods in the dataset is reported in Table 3 (i.e., 2; 883). 5
In this paper, we have presented the early steps of our research on the coherence between the lead comment of methods and their implementations. In particular, we have provided a description of the problem settings, along with the experimental protocol defined to create our dataset. We made it publicly available on the web. We also sketched the results of quantitative analysis conducted on a codebase of 3; 636 methods, gathered from 4 different open-source systems written in Java.
There could be many possible future directions
for our research. For example, it would be
interesting to conduct an empirical study to investigate
the effect of maintenance operations on the
coherence for multiple versions of the same system. As
a step forward in this future research direction, we
already included in the dataset two versions of the
JFreeChart system. Our results and those by Fluri
et al.