Background. A systematic literature review is a process in which all relevant available research about a research question is identified, evaluated, and interpreted through individual studies. The workload required for this process may bias the evaluation of the studies, affecting the result. Aim. Creating a decision support architecture to assist participants of a systematic review in the selection process of the individual studies and quality assessment of these studies, possibly improving the execution time and reducing the evaluation bias. Method. Improving the primary studies selection and quality assessment processes by using text mining techniques and ontologies to construct a decision support architecture. We will also conduct experiments to evaluate the proposed architecture. Contribution. Improve the primary studies selection and quality assessment processes, reducing its workload, and lowering the evaluation bias in systematic literature reviews.
The process of primary studies selection is a free interpretation of
the criteria of whom is leading the SLR, hence, the large number
of papers retrieved during the search process and the poor quality
of their abstracts [2] makes the completion of the selection
process a hard task and sometimes inaccurate. After the selection
process is carried out to evaluate the quality of selected papers to
increase the reliability and importance of SLR results, and to
perform this task, there are several guidelines, which are usually
not properly followed and the use, in general, is not justified by
the authors [
There are studies that talk about automatic selection of primary
studies using text mining techniques, in which the primary studies
are classified by the similarity of texts [
The semi-automatic evaluation of the quality and the
inclusion/exclusion criteria semi-automatic search should support
the evaluation of primary studies in SLR process. With this, we
aim to reduce the execution time of the SLR, due to the primary
study reading procedure, decreasing the primary studies
evaluation bias that may occur due to the evaluation process
subjectivity, increasing the assurance that the outcome of the SLR
is not being compromised [
There are several guidelines available for quality assessment of
primary studies, such as [
There is still not a standard process for primary studies quality assessment in software engineering, several authors have suggested several ways to estimate the validity of the studies. Probably due to this lack of standardization, in our research were not found tools that automate the process, or part of this process.
The selection of primary studies is the most time-consuming task
for an SLR, which can be affected by the titles and abstracts that
Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes
do not reflect well the content of the work [
Automatic classification of primary studies, indicating its
inclusion/exclusion can be found in some works [
Therefore, this research proposes a semi-automated approach, leaving to the researcher the final decision of the inclusion or not of these studies. It is likely that when comparing our methodology and the automatic classifiers, there will be a loss of time taken to perform the task by using the technique proposed in this work.
However, it is possible that the final result of the selection process, using our proposal, is more satisfactory and significantly faster than the manual approach.
According to the aim of this study, our guiding research question (RQ) is “How can we improve the process of evaluating primary studies by automating parts of the process?”. This question can be decomposed into: RQ1. How can we improve the selection process by automating parts of the process? RQ2. How can we improve the quality assessment process by automating parts of the process? An initial version of the proposed Decision Support Architecture (DS Architecture) is presented in Figure 1.
The knowledge about the SLR will be represented by an ontology
(1) based on the protocol designed for the execution of the SLR
and possible research conductor's refinements. This approach is
proposed by Biolchini et al [
These criteria should be used as input to a text mining algorithm,
which should be available as a tool component (DSTool) and will
be able to identify whether the selection criteria have been met
and which quality criteria can be identified inside the primary
study text. The interest points of the text, where the tool will be
based to respond, will be shown to the research conductor, which
will decide whether the criteria were actually achieved. The
inclusion/exclusion criteria will be provided by the conductor in
the protocol creation, and the quality criteria will come from one
of the guidelines available in the literature, after to perform a
systematic literature review that should indicate the best guideline
to be implemented by the text mining algorithm. The main idea to
provide a previous guideline is to improve the robustness and
accuracy of the algorithm, which does not prevent that other
criteria, which are not present in the selected guideline, may be
used. Another input to the text mining algorithm is the ontology
built upon the protocol (3), many text mining techniques use
ontologies as a knowledge base, one of which is the
ontologybased question answering system [
Based on the system architecture proposed by Bo and Yunqing
[
The questions that will serve as input to the algorithm will be
provided by the protocol, at this time the criteria for
inclusion/exclusion and the quality criteria will be provided one
by one to the segmentation process (1). At this point, the question
will be broken in terms, and the keywords will be extracted (2).
As, for example, the following quality criteria: “Is there a clear
statement (definition) of the aims (goals, purposes, problems,
motivations, objectives, questions) of the research?” [
To evaluate the effectiveness of the Decision Support Architecture
(DSArch) a controlled experiment [
RQ1. Does DSArch decrease the selection time of primary studies? RQ2. Does DSArch increase consensus between individuals of the same pair including/excluding studies? RQ3. Does DSArch decrease the quality assessment time of the selected primary studies?
The null hypotheses are presented below: H0,RQ1. There is no difference in the execution time in the selection process with or without the use of DSArch. H0,RQ2. There is no difference in the consensus among peers with or without the use of DSArch. H0,RQ3. There is no difference in the quality assessment time of the selected primary studies with or without the use of DSArch.
To examine the hypotheses the following dependent variables will be used:
Selection Time. The time to complete the process for selecting primary studies. Consensus. Measured by inter-rater agreement coefficient. Cohen´s Kappa will be used, due to the decision be taken by pairs. Quality Time. The quality assessment time of the selected primary studies.
The experiment factor is the process for evaluating the primary studies, by selecting these studies and quality assessment of the selected primary studies. The factor alternatives (treatments) are executing the inclusion/exclusion process and quality assessment using DSArch and the other is using only the protocol criteria manually, without tool support for the alternatives.
To perform the experiment a subset of primary studies drawn from an SLR will be provided to participants. The protocol created for the SLR will be provided and both the search process and the visualization of the primary studies will be conducted by REviewER1. It belongs to our research group and it is a tool that gives support to search process in some databases (ACM library, Engineering Village, IEEExplorer, Science Direct, Scopus, and Springer Link) in a automated way and gives support to the primary studies selection process. A version of REviewER will be developed, containing the DS Architecture, to be used in the evaluation. However, only the DSArchictecture should be assessed. 1 http://sites.google.com/site/eseportal/tools/reviewer
A training will be conducted with the participants to present the reviewer tool and how one should analyze the DSArch. For this training some primary studies, chosen from a subset of the primary studies set obtained from the SLR search process, will be selected and participants should evaluate them using the factor alternatives.
To measure the dependent variables two tasks will be realized, which requires no prior knowledge on SLR from the subjects, hence, we aim to facilitate the process of selection of these participants. The tasks to be performed in the experiment are the selection of primary studies from a subset of the primary studies set obtained from the SLR search process and the quality assessment of these selected studies. No other SLR procedures will be performed.
At the end of the execution, the participants must provide a list of accepted primary studies, the time taken for completion of the selection process and the time taken for completion of the quality assessment process. A questionnaire will be performed after the experiment to evaluate the experiment itself and what the participants thought about the proposed architecture.
The experiment will take place in a lab with the presence of all
participants at the same time. Participants will be divided into
pairs following the proposed by Kitchenham et al [
Latin squares have an analysis procedure very similar to the factorial experiments (multiple factors). In the case of factorial experiments, we can consider the lock variable as a factor. For the Latin square, we have two lock variables and a factor of interest. Also, there is another complicating factor: the experiment design has multiple replications. This leads to a replicated Latin square design with equal columns (processes) and different lines (participants).
A possible statistical test for the analysis is the ANOVA, as
proposed in [
Some possible threats to validity of the experiment are already being assessed.
The completion of the training can generate an apprenticeship in relation to DSArch, which may influence the evaluation process of the primary studies when participants are not using the DSArch. However, if there is influence, it will be in favor of the null hypothesis.
The results may not be generalizable to all the researches that perform SLRs, because we do not sample from the population of SLRs researchers, but we intend to make a satisfactory outcome that will bring evidence of the effectiveness of DSArch.
The use of the DSArch might reduce the effort to complete SLR, which is one of the major problems encountered in conducting this type of research. Another problem to be addressed by using DSArch is the primary studies subjective selection, which is biased, by automating part of the process. Another gain by automating part of the process is decreasing the number of conflicts generated during the evaluation of studies within the pairs. An expectation for the proposed algorithm is that it can be used to assist in the SLR data analysis process, since, the question answer principle can also be used at this stage, however, this analysis is out of the context of this thesis.
So far, it was conducted an ad-hoc literature review on the proposed theme, as well a first design of the controlled experiment that will be conducted to evaluate the DSArch. The planned next steps are: Execute a systematic literature review of techniques for assessing the quality of primary studies, aiming at choosing a technique to be semi-automated; Select and implement an ontology to represent the prior knowledge on the SLR; Development of the DSArch; Refine the plan and execute the controlled experiment to assess the DSArch; Analyze the obtained results; Write the Thesis.
The main points where advices are needed:
The proposed architecture is consistent with the problem found? The experimental design can evaluate the proposed architecture? The statistical test, ANOVA can evaluate the data generated by the experiment?
[1] Kitchenham B. A. and Charters S.. Guidelines for performing systematic literature reviews in software engineering, Technical Report: 2007. [2] Brereton P., Kitchenham B. A., Budgen D., Turner M. and Khalil M.. Lessons from applying the systematic literature review process within the software engineering domain. at