A mapping between a system's implementation and its software architecture is mandatory in many architecture consistency checking techniques. Creating such a mapping manually is a non-trivial task for most complex software systems. Machine learning-based text classification may be an highly efective tool for automating this task. How to make use of this tool most efectively has not been thoroughly investigated yet. This article presents a comparative analysis of three classifiers applied to map the implementations of five open-source systems to their architectures. The performance of the classifiers is evaluated for diferent extraction and preprocessing settings as well as diferent training set sizes. The results suggest that Logical Regression and Support Vector Machines both outperform Naive Bayes unless information about coarse-grained implementation structures cannot be exploited. Moreover, initial manual mappings of more than 15% of all source code files, or 10 files per module, do not seem to lead to a significantly better classification.
which are hence discouraged [
In some cases, architectural documentation
describSoftware architecture degradation is the phenomenon ing the relationship between architecture and
implemenof the implementation of a software system diverging tation can help create this mapping. More often than
from the intended software architecture [
One approach to combat software architecture degra- tects and designers in a study by Ali et al., creating the
dation is software architecture consistency checking [
Most approaches have in common that some kind of These approaches show promising results. The quesmapping between architectural units, e.g. modules, and tion arises though whether the full potential of machine implementation units, such as source code files, is re- learning for text classification in this context has already quired. Reflexion modelling, for example, exploits this been tapped. Several text classification algorithms that mapping to detect source code dependencies that are not perform well in diferent contexts have not yet been incovered by dependencies in an architectural model and vestigated. The question of how to optimally extract and preprocess source code for classification has not yet been exhaustively explored either.
The goal of this article is to shed some light on the performance, i.e. the predictive capability, and other properties of several machine learning-based text classifiers for BytesRef.java ifnes allowed / prohibited, or expected / discouraged dependencies between architectural modules. In order to check whether the dependencies present in source code conform with those, i.e. to compare code with architectural dependencies, the architectural modules to which sources and targets of code dependencies are mapped, need to be known. Fig. 1 shows a cut-out of a so-called reflexion model of one of the systems (Lucene) used in the experiments presented in this paper.
It depicts two architecture modules as boxes. Dashed lines (as opposed to solid lines) indicate that dependencies between these modules are architecturally discouraged in either direction; they are, however, present
the described task. We present a comparative analysis of three classifiers that were applied to map the code of five diferent systems to their specified architectures.
The contribution of the paper is a set of findings that may guide further research and use of these classifiers for the task of interest. These guidelines, on the one hand, give advice for the selection of an appropriate classifier based on assumptions regarding the alignment between architecture and modular implementation elements like packages. On the other hand, they provide rules of thumb for the recommended size of an initial, manual mapping required to train the classifiers.
The remaining article is organized as follows. The following section describes relevant technical background as well as related work. Section 3 explains the experimental setup of the comparative analysis. In Sec. 4, we summarize the results, which are discussed in Sec. 5. The article is concluded in Sec. 6.
Natural Language Processing [21]. The goal of text
classification is to assign a text written in natural language
to one or more predefined categories. Applications of
this activity include, for example, sentiment analysis or
spam detection.
2. Background The central idea of applying machine learning to the
task of text classification is to train a classification model
2.1. Architecture Consistency Checking based upon text samples for which the assigned
categories are already known. Fig. 2 depicts the typical steps
Techniques for checking the consistency between the in training a text classification model and using it to
software architecture of a system and its implementation predict its label/category, i.e. to classify new text. For
come in various forms. Several authors provide exhaus- learning, a set of text documents for which their labels,
tive overviews of available approaches and tools [
Many techniques have in common that they require learning algorithm that is applied. It can then be used to an association between elements of the architecture predict the label of a new text that was preprocessed and and elements of the implementation for many typical brought into its numerical representation. consistency constraints. The most fundamental con- The overall, general procedure can be transferred to sistency constraints are related to dependencies. The the specific context of mapping code to architecture quite intended architecture of a software system often de- easily. The documents to be classified are the aforemeng n i n i a r T n o i t c i d e r P data extraction & preprocessing feature extraction machine learning algorithm text documents preprocessed text documents
numerical representation classification model predicted label tioned source code entities, like source code files. Ar- mapping for the remaining source code entities. The inchitectural modules are represented by labels—for yet formation used for classification is extracted from the unlabelled source code entities, a classification model compiled source code and consists of package names, should propose the correct module. For training, we re- file/class names, and attribute and variable identifiers. quire a suficiently large set of source code entities for Compound words, like indicated through camel-casing, which their labels—the modules they are mapped to—is are split and the resulting texts are stemmed. The reknown. sulting documents are complemented by terms reflecting dependencies. This way structural information can 2.3. Related Work be considered in the classifier without the need to integrate a separate dependency analysis approach. The The studies by Christl et al. were among the first to in- authors show that this approach outperforms HuGMe vestigate techniques for automating the mapping step significantly; if module descriptions are available, though, needed in architecture consistency checking [14, 15]. InMap performs slightly better [19]. They developed a technique, called HuGMe, for interac- The focus of the work by Link et al. is slightly diftive, human-guided mapping and compared two diferent ferent yet related [20]. In their approach called RELAX, attraction functions, CountAttract and MQAttract, mea- code entities are not mapped to architectural modules suring how well a code entity will map to an architectural but concerns which are potentially reusable across sysmodule based on structural properties. tems. Any document of a system considered being part
Bittencourt et al. presented a technique based on infor- of an architectural concern can be fed into the training mation retrieval, thus addressing the mapping problem process of a Naive Bayes classifier to categorize new docfrom an textual analysis angle instead [16]. They devel- uments according to their textual content. The approach oped an attraction function based on Latent Semantic is compared with two other clustering approaches for Indexing and evaluated it separately as well as a hybrid architecture recovery as this is the main scenario that the approach with both CountAttract and MQAttract. The authors target. For five out of eight case study systems, best results were achieved by integrating their novel at- RELAX is shown to perform best in comparison. The traction function and CountAttract. study does neither include details of the preprocessing of
Both approaches require a set of manually mapped documents nor a replication package such that technical source code entities as a foothold for the applied tech- details of how information is extracted from source code niques. Sinkala and Herold instead exploit textual de- remain unclear. scriptions of the modules of intended architectures to provide their information retrieval-based technique called InMap with initial information for recommending map- 3. Experimental Design pings [18, 19].
Olsson et al. developed and analysed a technique based 3.1. Research Questions on machine learning [17]. Taking an initial, manually created mapping of a portion of the source code, a Naive Bayes classifier is trained and then used to predict the
tural modules. As Sec. 2.3 shows, the focus of related ap- reason and their good accuracy outperforming Naive proaches so far has been a single classification algorithm Bayes in comparative studies [24]. Logistic Regression as and less a comparison of classifiers or an investigation of the final classifier has been shown to perform at similar their performance properties when applied in the context performance levels as SVM and was hence selected for of interest. comparison, too [25].
The envisaged scenario for the usage of machine learning technique in this context is that a classifier is first 3.3. Experiment 1: Comparing Extraction trained with an initial set of manually created mappings based on the textual content of source code files. After and Preprocessing Variants that, the trained classification model is used to predict the mappings for the remaining source code files.
We therefore break the main motivating question down into two research questions: The goal of the first experiment is to address RQ1. In a ifrst step, we developed a list of elements in (Java) source code that we believed to potentially carry architecturally relevant information w.r.t. the required mapping. We judged the following elements to be potentially relevant: • RQ1: How does the selection of source code elements during preprocessing afect the performance of these classifiers? • RQ2: How is the performance of diferent classiifers afected by the size of the training set size, i.e. the number of code entities that need to be mapped manually initially?
For training and evaluating text classifiers for the task at hand, a set of systems is required for each of which a) the source code is accessible and b) the mapping between an intended architecture and the source code is known. We explored two data sources for identifying systems that fulfil these prerequisites: the SAEroCon repository 1 and the repository of the s4rdm3x tool [22]. Five open source software systems from these two repositories as listed in Table 1 were selected for this study. They are all written in Java.
Three commonly used machine learning-based classifiers were selected for the study. Naive Bayes for text classification was selected as the most relevant related work is built on it (see Sec. 2.3) and because of its good performance with even little training data [23]. Support Vector Machines (SVM) were selected for the same 1https://github.com/sebastianherold/SAEroConRepo/wiki • Package declarations: Indicate containment relationships that might match course-grained architectural structures. • Import declarations: Elements of the same architectural module often share the same dependencies. • Class declarations: Types defined in the same module might share the same (part of the) domain vocabulary as expressed in their names. • Public methods: Same rationale as for class declarations. • Comments: May refer to architectural aspects and decisions, parts of the domain vocabulary, etc., beyond what is being expressed in code
ing steps that could be activated or deactivated for each of the above elements in a source code file:
Customer Id. 2. Stemming: reduce inflected word to their stem,
e.g. notification or notify become notif. 3. Transform to lower case 4. Removing single characters 5. Removing stop words, such as the, and or of 6. Removing Java keywords, such as class or public A complete investigation of all combinations of preprocessing steps in a fixed order would lead to 27 options per extracted source code element. For extracting all of the above elements alone, this would lead to 235 combination which we considered infeasible. Instead, we experimented with several settings in an exploratory pre-study from which we concluded to activate the preprocessing steps 3 to 7 per default for all code elements as deactivating them lead to decreased performance in the explored alternatives.
In the same pre-study two diferent feature representation techniques were compared, bag-of-words and tf-idf [26, 27]. We noted that bag-of-words outperformed tf-idf on average and hence chose the former for the experiments.
For each combination of subject systems, classifier, and combination of code extraction and active preprocessing steps, we trained and validated ten models following a Monte-Carlo cross-validation scheme [28]. The training set ratio was kept constant at 0.2 and stratified sampling was applied. The latter ensures that the proportion of the classes (i.e., modules) in the overall dataset is kept in both training and testing sets during cross-validation. This ensures that both sets are representative for the overall dataset.
The performance of the models were evaluated in terms of accuracy, i.e. the relative frequency of correct classifications, and averaged over all subject systems.
7. Tokenization of words: chopping the stream of is defined as the precision/recall per class (module) dicharacters that the document consists of into ac- vided by the number of classes. The weighted average tual tokens based on separators such as spaces, precision/recall takes the proportions of classes into accolons, etc. count and weights the individual precision/recall scores accordingly. The weighted average recall is equal to the accuracy of a classification model 2.
Practically speaking, this experiment corresponds roughly to a situation in which a software architect/designer can estimate the number of code entities that should be mapped to each of the architectural modules. The experiment could ofer advice regarding the relative number of entities she should map per module in order to get a suficiently accurate automated mapping for the rest of the system.
This scenario, however, is not always realistic as module sizes may be unknown or estimations may be wrong.
For that reason, we repeated the experiment described above with diferent absolute training set sizes, expressed as absolute number of files per modules that should enter the training set. Obviously, this way of sampling is not stratified; the number of splits and metrics for evaluation remain the same as for comparing based upon relative training set sizes.
The replication package, including the scripts for preprocessing the data, training and evaluating the classifiers is available at https://github.com/sebastianherold/ml-forarchitecture-mapping.
3.4. Experiment 2: Measuring the Efect As described in Sec. 3.3, we explored the accuracy of all classification algorithms for diferent data extraction of Training Set Sizes and preprocessing settings in the first experiment. Fig. 3 The goal of the second experiment is to address RQ2. summarises the findings per combination of extracted Based on the results of the first experiment, one of the source code elements. All three classification algorithms best performing combinations of extraction and prepro- scored best when the data extracted from the code files cessing settings was selected for each classification al- was limited to package declarations and class declaragorithm. The code files for each system were extracted tions. Logistic regression and SVM achieved accuracies and preprocessed accordingly and represented as bags- of 0.93 each, outperforming Naive Bayes by 0.07. SVM’s of-words. and Logistic regression’s accuracy drop significantly to
We then trained models for each of the three classifi- 0.68 and 0.73 at maximum, respectively, when package cation algorithms at diferent training set sizes expressed declarations are not included in the data. Naive Bayes as fractions of the overall datasets, i.e. relative number of drops to 0.75 at extracting everything else but package available mappings between source code files and archi- declarations, performing more accurately than SVM and tectural modules. Per combination of system, training set Logistic Regression in this scenario. size of interest, and classifier, we trained and evaluated according to a Monte Carlo cross-validation with 100 splits and stratified sampling. 2The recall of each individual class is weighted by ||
In order to evaluate the resulting models, we computed with being the total number of data points. Since || = several precision and recall averages per system, train- + , each term for the weighted average recall turns ing set size, and classifier. The average precision/recall into / which summed up over all classes is equivalent to the definition of accuracy.
The role of comments also changed with the inclu- 0.15, in particular for JabRef, ProM, and Teammates. The sion of package declarations. With package declara- curves show similar behaviour for the weighted average tions included, adding comments, while keeping inclu- precision. Unweighted averages keep a steeper slope in sion/exclusion of the other code elements unchanged, comparison even beyond training set sizes of 0.15 which seem to rather decrease the accuracy of the classifiers. shows that the performance for smaller modules benefits Without package declarations, including comments lead from increasing the training set size. to accuracy improvements of up to 0.07. In Fig. 6, the results of evaluating classification perfor
It should be noted that the mapping onto modules mance for diferent numbers of files per module in the aligned quite well with the package structure in all five training set are shown. Most curves across all metrics systems, which might explain the impact of including the show a sharp increase in performance that slows down at package declaration. We excluded the results for only ex- 10 files per module. This is less pronounced, sometimes tracting package declarations as we believe that the very hardly visible, for Naive Bayes as compared to SVM and good scores (beyond 0.98) of those models were overfit- Logistic Regression. The results for the weighted averting and heavily biased towards the selected systems. ages seem more similar to their unweighted counterparts
Fig. 4 illustrates the standard deviation for each ex- in this experiment. SVM and Logistic Regression outpertraction setting and classifier. The standard deviation is form Naive Bayes in almost all settings and systems in below 0.02 in 80% of the cases, exceeding 0.05 slightly this experiment, too. in only one case. These results show that the variable preprocessing settings, stemming of words and splitting of compound words, afect accuracy only slightly. 5. Discussion
The results related to classification performance over training set size as relative fraction of the overall number 5.1. Findings regarding RQ1 of source code files are visualised in Fig. 5. They confirm that SVM and Logistic Regression perform better than Naive Bayes in accuracy, precision and recall in almost all settings3. The improvement in accuracy decreased for all systems and classifiers beyond a relative training size of in which the only diference is to not consider package are well-aligned but mapped to more than one package declarations is common across the results. in such systems, identifying the relevant packages for a
This seems quite natural as the mappings for the sys- module can still be tedious. tem used for training are largely aggregating source code An interesting question in the light of the first finding elements along several subtrees of the package hierarchy is whether the approaches by Olsson et al. and Link et al. instead of individual classes from unrelated packages. could benefit from using a diferent classifier than Naive Only in the mapping of JabRef exist cases of packages Bayes [17, 20]. While the alignment with source code whose contained classes/interfaces, i.e. and correspond- structures is largely unclear for Link et al., Olsson et al. ing files, are mapped to diferent modules, and which applied their approach to the same, well-aligned systems these diferent mappings do not align with the subpack- used in this study. This finding also suggests that their age/subdirectory structure. It does hence not surprise approach could be further tuned to only use package and that settings including package declarations and only type information as compared to including also variable few other pieces of information score best. In our experi- identifiers. In use cases, in which the slightly slower ments, class declarations seem to complement package training of SVM and Logistic Regression is an issue, Naive declarations best. Since Naive Bayes does not perform as Bayes might be the better alternative. well as the other classification algorithms, we formulate It is common that the mappings are not that wellour first finding as aligned and straight-forward [29]. Furthermore, some programming languages do not declare any containment Finding 1. In settings, in which the architectural mod- relationships equivalent to packages declarations. The ule structure can be assumed to align well with macro- tested systems do not represent this scenario properly. structures declared in the system’s implementation, these We therefore looked at the performance of the classifiers declarations and type information should be extracted. without considering package declarations as approximaSVM and Logistic Regression provide more accurate results tion of their behaviour if we did not have that informathan Naive Bayes. tion or considered it useless. In this setting, Naive Bayes Note that a straight-forward alignment does not neces- exploiting import declarations, class declarations, and sarily imply that a mapping can easily be constructed comments, showed the best accuracy (on a par with admanually without the need for automation in the first ditionally including declarations of public methods). place. In large-scale systems, structures of hundreds of packages are not uncommon. If architectural modules Finding 2. If alignment with any macro-structures declared or derived from source cannot or should not be as- 15% of the overall dataset (equal to the total number of sumed, Naive Bayes trained based on declarations of types, source code files) and above. Enhancing the initial mapimports, and comments should be used. ping beyond this point may therefore turn out infeasible. Even in the relatively small sample systems of this study The standard deviation within groups of identical extrac- like JabRef, increasing this mapping by 5% of the overall tions regarding diferent preprocessing settings is very number of code files means to map more than 40 addilow. This indicates that the impact of stemming and tional files. This may possibly not pay of, in particular for splitting of compound words does not have a significant larger systems, if the gain in classification performance impact on the resulting accuracy of any of the tested is minimal. We therefore state: classification algorithms.
Finding 3. The selection of parts to be extracted for classifier training and mapping prediction appears to be more important than the selection of the preprocessing steps considered optional in this study.
The results suggest that in many cases the additional gain in accuracy, precision, and recall slackens at around Finding 4. If the number of files supposed to be mapped to each module can be estimated, mapping around 15% of that number in the initial mapping may be a good rule-ofthumb for training an eficient classifier.
The results of experimenting with absolute training set size complement these result for scenarios in which it is not possible or desirable to estimate the number of ifles mapped to each module. For the tested systems, the gain in accuracy, precision, and recall flattens out at 10 ifles per module in the initial mapping which leads us to our final finding: Finding 5. An initial mapping of at least 10 files per module may lead to a satisfactorily performing classifier.
Again these findings can only properly compared to Olsson et al. as Link et al. do no report details about training 6. Conclusion set sizes [17, 20]. The results reported by Olsson et al., suggest that an initial mapping of ca. 20% lead to satisfying performance. Those results, however, are measured The results of the presented study indicate that there is as average over even imbalanced initial mappings that do no silver bullet classifier. The choice of an optimal clasnot take proportions of modules into account. We there- sifier and elements to be extracted from source code is fore think that our findings recommending a slightly influenced by system characteristics like the alignment of smaller relative size of the, however, stratified mapping macro-structural elements with the assumed architecture. is in line with those results. To identify more of such distinguishing characteristics or scenarios seems to be an interesting objective of future 5.3. Validity research. It will be particularly relevant to investigate whether the recommendations regarding the size of iniSeveral factors limit the external validity of this study. tial mappings hold in practice and if they apply for larger Firstly, although the subject systems are anything but systems, too. trivial, they certainly do not represent large-scale soft- Last but not least more classifiers wait to be tested for ware systems. Further research in particular to confirm their ability to automate code-to-architecture mapping. or refine the findings regarding training set sizes is re- For these as well as for those tested in this study, difquired. Moreover, the results may not accurately reflect ferent preprocessing techniques should be investigated the behaviour of classifiers if package or equivalent dec- more deeply and the improvement that hyperparameter larations are considered but the architecture does not tuning might achieve should be explored. Such a more align with them. A further threat to external validity is exhaustive comparative study might also need to take the the scoping to systems written in Java. This is due to the performance and the resource demands of the training limited availability of systems for which the architecture process into account. as well as the source code is available. The systems identtiufiendeapthpeeahryepdearlpltaorabmeeJatevras-bbaustedo.nMlyortoeouvcehre,dweupdoidnntohtis References non-exhaustively in the before-mentioned exploratory pre-study. This might be considered a limitation as well as a threat to external validity as the results might difer for classification models with diferent hyperparameters.
The experiments aim at identifying causal relationships between independent variables (extraction/preprocessing settings and training set sizes, respectively) and dependent variables (performance measures).
We are pretty confident that the internal validity is high as all other identified parameters were kept constant throughout the experiments. A potential threat are of course bugs in the scripts and software used to extract and preprocess data as well as for training and evaluating the classifiers. We consider this risk to be low though as established software libraries were used for this purpose and any self-written code (largely produced by the first and second author) was carefully reviewed by the third and fourth author.
We consider the selected method for cross-validation as main threat to construct validity. Inappropriate train/test splits may lead to biased classification models that might not reflect a classifier’s performance properly. We believe though that the chosen repetitions for the cross-validation in the experiments was suficient to mitigate this risk.