Traceability in software engineering is the ability to connect diferent artifacts that have been built or designed at various points in time. Given the variety of tasks, tools and formats in the software lifecycle, an outstanding challenge for traceability studies is to deal with the heterogeneity of the artifacts, the links between them and the means to extract each. Using a unified approach for extracting keywords from textual information, this paper aims to compare the concepts extracted from three software artifacts: source code, documentation and tests from the same system. The objectives are to detect similarities in the concepts emerged, and to show the degree of alignment and synchronisation the artifacts possess. Using the components of three projects from the Apache Software Foundation, this paper extracts the concepts from 'base' source code, documentation, and tests (separated from the source code). The extraction is done based on the keywords present in each artifact: we then run multiple comparisons (through calculating cosine similarities on features extracted by word embeddings) in order to detect how the sets of concepts are similar or overlap. For similarities between code and tests, we discovered that using pre-trained language models (with increasing dimension and corpus size) correlates to the increase in magnitude, with higher averages and smaller ranges. FastText pre-trained embeddings scored the highest average of 97.33% with the lowest range of 21.8 across all projects. Also, our approach was able to quickly detect outliers, possibly indicating drifts in traceability within modules. For similarities involving documentation, there was a considerable drop in similarity score compared to between code and tests per module - down to below 5%.
Software traceability is a fundamentally important task in software engineering: for some
domains, traceability is even assessed by certifying bodies [
The contribution of this paper is the notion of concept similarity, in the context of the traceability between source code, documentation and tests of a software unit (class, module or system). The book definition of “concept" is a principle or an idea 1 that is abstract or generic in nature 2. In software engineering, we add on to this definition to include the features in software components, where concepts are presently identified through source code analysis and manipulation. For our dataset, we use the modules present in three actively managed Java open source projects from the Apache Software Foundation (ASF): Apache’s Dubbo3, Skywalking4 and Flink5. Although the sample is small, our aim is to showcase the methodology behind the data extraction: the approach is straightforward and scalable, so it will be possible to analyse a larger sample with minimal efort. We extracted the concepts of each project from three sources: the source code, documentation and tests. The documentation was extracted from the README file, which serves as the first point of entry for project stakeholders. The tests of each project were identified through regular expressions of filenames, while the remaining Java files constitute the ‘base’ source code for these projects. We extracted the concepts emerging from the keywords used in each of these sources, and we run multiple similarity measurements to determine the similarity of these artifacts, answering the following:
RQ1: How similar (syntactically and semantically) are the three software artifacts: source code, documentation and tests?
RQ2: How does feature extraction (through word embeddings) perform when comparing textual similarity between the source code, documentation and tests?
This paper is structured as follows: Section 2 summarises the related work, and Section 3 describes the unified approach used to extract the concepts. Section 4 shows the results and discusses the findings. Section 5 concludes.
In [12], the authors present a roadmap of the research on software traceability, based on the topics that researchers had focused on so far. According to that roadmap, our work: • is based on vertical traceability relations (i.e., it includes the relations between diferent artifacts); 1https://dictionary.cambridge.org/dictionary/english/concept 2https://www.merriam-webster.com/dictionary/concept 3https://github.com/apache/dubbo 4https://github.com/apache/skywalking 5https://github.com/apache/flink • includes overlap relations (i.e., we study whether C1 and C2 6 refer to common concepts of a system); • targets the automatic generation of traceability relations (as compared to manual or semi-automatic approaches); • aims to analyse traceability for the purpose of software validation, verification and testing, insofar as ‘traceability relations may be used to check the existence of appropriate test cases for verifying diferent requirements and to retrieve such cases for further testing ’ [12].
The traceability field is most often associated with requirements traceability [ 13]: prepackaged, automated tools like TraceLab [14] have often been preferred to trace diferent artifacts, or versions of the same artifact. Although it became clear over the years that traceability of software artifacts is essentially an information retrieval problem [15], combining software traceability with semantic information of software artifacts was shown to be a promising technique. Prospective traceability tools [16] have been developed with both the architecture and semantic modelling in mind. This empirical approach to recover traceability links through topic modelling was later adapted and improved through integrating orthogonal information retrieval methods [17].
Whilst most of the traditional literature on traceability has focused on requirements, the research on traceability of open source systems has focused on other artifacts (source code, user documents, build management documents etc [18, 19]). When requirements are considered, they are not traditionally elicited through customer feedback, but just-in-time and termed feature requests [20], or elicited using various other artifacts (e.g., CVS/SVN commit logs and Bugzilla reports [21]). In the open source context, the extraction of requirements is often considered as a long term view, for instance in the context of impact analysis [22].
In the context of deriving semantic value from software artifacts, extracting topics from source code has been previously presented in [23] by demonstrating that the Latent Dirichlet Allocation (LDA) technique has a strong applicability in the extraction of topics from classes, packages and overall systems. This technique was also used in a later paper [24] where experts were consulted to assign software system in accordance to its domain. The keyword terms derived from this technique were also compared in terms of text similarity using various word embeddings in [25].
• Corpus keyword (term) – given the source code contained in a class, a token or term is any item that is contained in the source code. We do not consider as a term any of the Java-specific keywords (e.g., if, then, switch, etc.). Additionally, the camelCase or PascalCase notations are first decoupled in their components (e.g., the class constructor InvalidRequestTest produces the terms invalid, request and test). 6“C1" and “C2" are arbitrary concepts • Topic – this refers to the clusters of keywords extracted with the Latent Dirichlet Allocation (LDA) technique, and weighted by their relative relevance. For this paper, we will concatenate all the keywords sans weights as topic keywords. • Concept – the set of ‘source code concepts’ is the union of the (i) corpus keywords set and (ii) topics set, as extracted from the source code. These ‘concepts’ are derived from the lexicon used in the code. The ‘test concepts’ have a similar definition, using the sets from the test batch. • / SIM – Concept similarity between and , where and are software artifacts, and ̸= .
Table 1 shows the details of our sample dataset. We are fully aware that the analysis of three systems (instead of hundreds, or thousands) does not allow to draw any conclusion for any other system. The empirical study that we present below focuses on top rated systems (representing the quality of code developed by the ASF community, adhering strictly to established coding standards7) rather than promoting the width of representativeness. We will further discuss about the implications of our choice in the threats to external validity.
The extraction of the concepts and similarity measurements was executed in Python via a Jupyter notebook8. The extraction was carried out to build the class corpus for each project. Results were then compared and analysed.
For any project’s source code and tests, we extracted all the class names and identifiers that were used for methods and attributes. Additionally, inline comments were extracted as well, but the Java keywords were excluded9, along with the project names (such as apache and dubbo for P1) to provide a more accurate representation. This results in an extraction that comprehensively represents the semantic overview of concepts whilst minimising noise from code syntax. The final part of the extraction is the lemmatisation of the terms using SpaCy’s10 token.lemma_. Lemmatising is deriving the base from the terms (also called their dictionary form, or ‘lemma’), thus enabling more matches when we compare from the diferent
sources (e.g., best -> good). An excerpt of the complete corpus from the source code of P1’s dubbo-common module is shown at figure 1.
Extending this to all the modules in the three projects, we look at the textual similarity of corpus keywords extracted for (i) source code and (ii) tests per module.
activate reference service metadata colon separator service key service version registry config config available serial...
For the documentation extraction, we looked at the README file and ran through a similar cleaning pipeline. All non-English characters were disregarded during the exercise by checking if the character falls within the ASCII Latin space. Figure 2 shows a simplified diagram of our concept similarity (i.e., traceability) between artifacts.
For the CODE/DOC SIM and TEST/DOC SIM, we run topic modelling with Latent Dirichlet Allocation (LDA) due to the vast diference in token count between the README file and the code. We cluster the extracted class corpus into groups; an unsupervised learning approach to tag groups of terms to a topic based on their semantic context, then identifying the overarching theme of each cluster through their topics. Using Gensim11 LDA, we identify the topic clusters present. Next, we concatenate all the topic keywords from all the modules from code and tests respectively. Figure 3 shows an example of topic keywords that emerge from P1’s dubbo-common module source code.
These are used to compare with the topic keywords emerging from documentation. In Section 4, we will look at the results of proportion in overlap of these concepts extracted between code and documentation, and tests and documentation.
We apply multiple vectorisation techniques to represent corpus keywords as vectors in a vector space, and then we run cosine similarity against these vectors to address RQ1 and RQ2. Cosine similarity is a “distance" metric, irrespective of orientation and magnitude: the lower the angle between the vectors, the higher the similarity. Table 2 shows the diferent vectorisation techniques used with cosine similarity for measurement.
For space limitations we have made available online the result summary of Code and Test similarity (CODE/TEST SIM)12, and the results of Code and Documentation similarity (CODE/DOC SIM) and Test and Documentation similarity (TEST/DOC SIM)13 across all modules in each project.
Figure 5 shows the box plot distribution of CODE/TEST SIM for the vectorisation techniques across all modules for P1, P2 and P3 (RQ2). Similarly to the trends found in [25], the analysis of the three Apache projects confirms that the baseline TF-IDF measurement has the widest ranges of 0.62 to 0.85, and lowest mean scores of 0.34 to 0.66. At the same time, the role of pre-trained embeddings is central in deriving semantic context to concepts: contextual similarity (helped by the three trained datasets) is linked to a higher cosine similarity when comparing corpus keywords in software artifacts. This is expected, since the Java syntax closely mirrors the words already present in the pre-trained vector spaces.
With pre-trained embeddings trained on a wide vector space (e.g, an English vocabulary across all domains), the range of similarity score gets even narrower. We see this range decrease 12https://github.com/zakipauzi/benevol2021/blob/main/benevolcodetestsimsummary.csv 13https://github.com/zakipauzi/benevol2021/blob/main/benevolcodetestdocsimsummary.csv as we run CODE/TEST SIM with StackOverflow data (SO W2V), expanding vocabulary beyond just within the scope of our current artifacts (TF-IDF), and further decrease with SpaCy and FastText embeddings. In other words, the wider the vector space of embeddings, the narrower the range and the higher the mean. Ultimately, we know that a high score does not necessarily denote a better similarity, but it is worth exploring further on how we can continue to balance the role of pre-trained embeddings with accurately representing similarity that is domain-specific, which ideally relates to our traceability reconstruction solution through concept similarities.
A key advantage of using this approach (particularly with pre-trained embeddings) is that the outliers can be easily detected in figure 5, indicating drifts in traceability via similarity measurements. For P1, we can see outliers such as modules dubbo-container, dubbo-filter and P3’s flink-streaming-scala: these show that the concepts emerging from their code and tests are vastly diferent and need looking into.
As for our similarity measurements: CODE/DOC SIM and TEST/DOC SIM, comparing module to the documentation in its entirety will not be accurate, thus we ran topic modelling to represent the hierarchical structure of the syntax more efectively, a similar approach to [ 24]. Topics emerging from corpus keywords represent clusters that are a level higher than the corpus keywords, bridging the gap between per module syntax with README corpus. The results for both similarities (using Jaccard Index on topic keywords) are well below CODE/TEST SIM: >5%. Further work will need to be done to establish the accuracy of this result, such as incorporating weights to topics and identifying traceability beyond the syntax of keywords. across P1 modules scored lower than baseline whereas it difers for some P2 modules.
The approach that we have shown has some threats and limitations that we have identified. • External tests and documentation not included. Other than the basic unit and logic tests, we do have other tests that may involve other software and systems, which are not included (i.e., integration tests). • Construct: definition of concepts as a construct . Our definition looked at both facets – the corpus keywords and the derived topics. Moreover, our dataset assumes on the notion that top rated projects from an open source body with established coding standards (such as ASF) represent good code quality in artifacts and hence, we expect that the concepts emerging from the sources are aligned, as in [25]. Our approach ensures that artifacts are treated similarly, and drifts between artifacts are clearly captured by outliers. • Conclusion: non uniform identifiers and code smells . From the analysis of the systems, we observed that the structure of code has some degree of non-uniformity in the way identifiers are used to represent meaning.
In this paper we explored the triangulation of textual similarity via diferent techniques between the concepts extracted from the source code, documentation and tests of Java software systems. The aim was to assess the traceability between the three sources, and to put it in the context of concept overlap. There is great potential in the results for further development and analysis in semantic traceability for software artifacts (e.g., establishing connections between topics derived outside of lexical intersection, exploring diferent metrics to detect traceability). Also, expanding our dataset to include projects of various domains, languages and categories. Moving forward, we would extend this solution to include other artifacts such as architecture diagrams, bug reports and functional requirements. We would also want to look at ways to adopt this approach to a supervised automated logic of domain categorisation. [12] G. Spanoudakis, A. Zisman, Software traceability: a roadmap, in: Handbook Of Software Engineering And Knowledge Engineering: Vol 3: Recent Advances, World Scientific, 2005, pp. 395–428. [13] R. Torkar, T. Gorschek, R. Feldt, M. Svahnberg, U. A. Raja, K. Kamran, Requirements traceability: a systematic review and industry case study, International Journal of Software Engineering and Knowledge Engineering 22 (2012) 385–433. [14] E. Keenan, A. Czauderna, G. Leach, J. Cleland-Huang, Y. Shin, E. Moritz, M. Gethers, D. Poshyvanyk, J. Maletic, J. H. Hayes, et al., Tracelab: An experimental workbench for equipping researchers to innovate, synthesize, and comparatively evaluate traceability solutions, in: 2012 34th International Conference on Software Engineering (ICSE), IEEE, 2012, pp. 1375–1378. [15] M. Borg, P. Runeson, A. Ardö, Recovering from a decade: a systematic mapping of information retrieval approaches to software traceability, Empirical Software Engineering 19 (2014) 1565–1616. [16] H. U. Asuncion, A. U. Asuncion, R. N. Taylor, Software traceability with topic modeling, in: 2010 ACM/IEEE 32nd International Conference on Software Engineering, volume 1, 2010, pp. 95–104. doi:10.1145/1806799.1806817. [17] M. Gethers, R. Oliveto, D. Poshyvanyk, A. D. Lucia, On integrating orthogonal information retrieval methods to improve traceability recovery, in: 2011 27th IEEE International Conference on Software Maintenance (ICSM), 2011, pp. 133–142. doi:10.1109/ICSM. 2011.6080780. [18] H. Kagdi, J. I. Maletic, B. Sharif, Mining software repositories for traceability links, in: 15th IEEE International Conference on Program Comprehension (ICPC’07), IEEE, 2007, pp. 145–154. [19] H. Kagdi, J. Maletic, Software repositories: A source for traceability links, in: International Workshop on Traceability in Emerging Forms of Software Engineering (GCT/TEFSE07), 2007, pp. 32–39. [20] P. Heck, A. Zaidman, Horizontal traceability for just-in-time requirements: the case for open source feature requests, Journal of Software: Evolution and Process 26 (2014) 1280–1296. [21] N. Ali, Y.-G. Guéhéneuc, G. Antoniol, Trustrace: Mining software repositories to improve the accuracy of requirement traceability links, IEEE Transactions on Software Engineering 39 (2012) 725–741. [22] M. Gethers, B. Dit, H. Kagdi, D. Poshyvanyk, Integrated impact analysis for managing software changes, in: 2012 34th International Conference on Software Engineering (ICSE), IEEE, 2012, pp. 430–440. [23] A. Kuhn, S. Ducasse, T. Gírba, Semantic clustering: Identifying topics in source code,
Information and Software Technology 49 (2007) 230–243. [24] A. Capiluppi, N. Ajienka, N. Ali, M. Arzoky, S. Counsell, G. Destefanis, A. Miron, B. Nagaria, R. Neykova, M. Shepperd, et al., Using the lexicon from source code to determine application domain, in: Proceedings of the Evaluation and Assessment in Software Engineering, 2020, pp. 110–119. [25] Z. Pauzi, A. Capiluppi, Text similarity between concepts extracted from source code and documentation, in: Intelligent Data Engineering and Automated Learning – IDEAL 2020 - 21st International Conference, 2020, Proceedings, 2020, pp. 124–135. doi:10.1007/ 978-3-030-62362-3_12. [26] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830. [27] V. Efstathiou, C. Chatzilenas, D. Spinellis, Word embeddings for the software engineering domain, in: 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), 2018, pp. 38–41. [28] explosion, en_core_web_md, https://github.com/explosion/spacy-models/releases/tag/en_ core_web_md-3.1.0, 2021. [29] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, A. Joulin, Advances in pre-training distributed word representations, 2017. arXiv:1712.09405.