An empirical analysis is conducted to investigate the association of the content of commit messages and technical debt. The analysis is based on 33 open-source Apache JAVA projects. Structural Topic Modelling, a recently developed text mining technique is employed for sophisticated analysis. The result shows that the certain content of commit messages such as empty messages are potentially associated with Technical Debt.
1.1
Commit messages writing plays an important role in software development for
it records, or documents the changes in natural languages during the progress of
the project. It is believed that well-written commit messages can provide useful
information to other developers in the life-cycle and eventually contribute to
enhancing the software quality in the end ([
However, to the author's best knowledge, there has not been an empirical
study based on real-world projects which challenges or con rms the above
mentioned \common sense". Besides, many related issues still haven't been properly
discussed, for instance, does empty message in uence the software quality? Or,
what kind of content in the commit messages are really helpful to the software
quality? The purpose of this research is to provide suggestions to developers
based on empirical analysis while writing commit messages .
Technical Debt (TD) [
Instead of analyzing the software with human e orts e.g. code reviews, to
nd out the potential TD issues, there are several Technical Debt measurement
tools available such as Better Code Hub 1, Coverity Scan 2 and SonarQube 3.
Among those mentioned tools SonarQube is one of the most commonly used
tool in both industry and research community [
TD has attracted considerable attentions in research communities (e.g. [
When it comes to text analysis, one common approach is topic modelling ([
Based on the above mentioned assumption, in each document, topics are occurring with di erent strength (prevalence), so that some documents may be composed mostly of a particular topic whereas others are a mixture of several other topics.
Thus, topic modelling is capable of not only capturing the underlying topic
content but also modelling the topic prevalence among a set documents. This
research employees STM [
There have been research works applying topic modelling to analyze commit
messages. For example, Hindle et al. [
Hu et al. [
However, although the above mentioned works have use topic modelling techniques, they have not investigated the relationships between the extracted topics and the software quality measurements especially TD that might be potentially associated with the commit messages.
There is another group of works ([
In this research. The Technical Debt Dataset [
Using the \reliabilityRemediationE ort" variable , the \debt change" of each commit is gained by calculating the di erence between the \reliabilityRemediationE ort" value before and after the commit. Moreover, an \accumulated debt" of a committer has contributed to a project can be obtained by summing up all the debt change values under a certain committer of the project. 3.2
The empty commit messages can potentially in uence the software quality, and they also re ect the developers' style while writing commit messages. To evaluate the impact and take it into account in the topic model, if the the commit message is empty (575 out of in total 128375 commit messages), the message is labeled as \emptymessage". The \emptymessage" is then treated as a special vocabulary. All the commit messages generated by a certain commiter under a speci c project are gathered as a document. There are in total 1083 documents. To focus on the contribution of a speci c developer to a project, the same developer in two di erent projects are considered two di erent developers.
The text are transformed into lower-cases and extra white space are removed. Note that, to better preserving the sophisticated writing hobbies in the commit messages, the text does not undergo some standard processing steps such stemming or lemmatizing.
The previously mentioned it the commit messages contribute to TD. Documents having positive \accumulated debt" are categorized as \debt contributed", otherwise \no debt contributed". 3.3
The Structural Topic Model (STM) [
In the model selection process, for each topic number from 6 to 20, 10 random
initialized models with randomly selected 50 % of the training documents are
built. The topic number with the highest average held-out likelihood value on
the 50% of the testing documents is selected. After deciding the topic number,
50 models are built and the one with the best semantic coherence [
Among 1083 documents (developer-project pairs), 868 of them are \no debt contributed", 215 of them are \debt contributed". The proportion of \debt contributed" documents is around 20%.
The result of model selection is shown in Figure 1. The x-axis (K) represents the number of topics and the corresponding held-out likelihood is shown on yaxis. Since the K = 18 reaches the highest held-out likelihood, it is thus selected.
Top words of the the extracted 18 topics can be found in Table 1. Apperently the common words such as \the", \and", \ x", \for" can be found in most of the topics. The relationship between topic prevalence and TD can be observed in Figure 2. The Topics 5, 6, 8, 9 are leading to not adding TD to the projects whereas the Topics 1, 3, 13, 15, 16 are leading to adding TD to the projects.
The prevalence of common words in the extracted topics are similar to the results
of the previously mentioned related works ([
When focusing on the no debt contributing Topics 5, 6, 8 and 9, one observation worth noticing is that is seems that detailed-oriented messages can somehow reduce the TD. For example Topic 5 has the word \version", \common", Topic 6 has the word \line", \code" and \here". Besides, they both have the word \this". It is likely that the Topic 8 is related to nal stage of the software development (\build", \ le", \tag" and \now") so the TD have to be removed. The Topic 9 contains only common words, the reason why it reduces the TD need further analysis it can be that it does not add or reduce the TD, and it is categorized into the \no debt contributed" group.
Most of the debt contributing Topics 1, 3, 13, 15 and 16 include the term \ x". It can be due to that some code smells or TD issues are generated by over-editing the code. Topic 13 contains terms \submitted", \reviewed" and \obtained" could be related the commits in some certain stages of the software development that generates TD issues and the "emptymessage" potentially shows that empty commit messages could result in TD issues. Topic 3 and 16 could be developer or project related.
The discovered topics related to TD issues are somehow explainable, however, some more detail-oriented analysis are still required to draw a more comprehensive picture.
Another limitation of this research is that the type project and the type of the changed le of a commit is not taken into consideration. In theory, the di culty and complexity can a ect the TD. Beside, the type of changed le should a ect the content of commit messages, e.g. changing \.html" les leads to more web developing related content whereas changing \.README" les is more likely to generate use user guideline messages. One practical di culty in this analysis task is that some commits are related to not just one changed le e.g. the changed les contain \.README" les and \.html" les. Therefore the e ect of the type of le is not taken into considerations. The e ect of the above mentioned factors can be taken into account in future works. 6
This research conducted an empirical text analysis focusing on the related commit messages and TD issues. The analysis uses a sophisticated topic modelling technique to analyze a collection comprises 128375 commit messages and the corresponding estimated TD measurement across 33 di erent real-world open access JAVA projects. The ndings show that 1. Some topics extracted from the commit messages and TD potentially associated. 2. In general, writing detailed-oriented commit messages have negative association to TD. 3. Empty commit message has positive association to TD.
However, more details and mechanisms regarding to the discovered associations still require further investigations.
Topic Number Top 6 Words Topic 1 the, and, for, block, xed, from, use, bug, xing Topic 2 x, add, for, remove, and, update, bug, use, support Topic 3 alexantonenko, for, via, contributed, and, x, shwethags, the, atlas Topic 4 the, for, closes, and, this, add, mavenreleaseplugin, prepare, from Topic 5 the, and, for, from, build, version, adding, commons, this Topic 6 cvs, this, the, then, from, here, and, has, line Topic 7 the, for, jaimin, and, add, from, x, ncole, task Topic 8 the, for, and, added, tag, build, le, ant, now Topic 9 add, javadoc, for, update, and, x, from, the, use Topic 10 the, and, for, added, xed, from, test, with, connection, not, and, wizard Topic 11 aonishuk, for, not, atkach, ababiichuk, onechiporenko, page, make, and Topic 12 not, the, with, use, was, from, samples, make, and Topic 13 from, submitted, reviewed, obtained, the, for, added, and, emptymessage Topic 14 yusaku, via, srimanth, for, the, and, not, should, page Topic 15 the, added, that, for, and, patch, test, xed, with Topic 16 via, for, ambari, swagle, not, the, upgrade, dsen, and Topic 17 the, dlysnichenko, akovalenko, for, and, from, created, jluniya, moe Topic 18 added, the, for, xed, updated, removed, and, javadoc, test