Numerous combinatorial testing techniques and tools are available for generating test cases. However, many of them are never used in practice. Why is it so difficult to introduce software testing research into testing practice? In this paper, by means of a panel conducted at 25th International Conference on Advanced information Systems Engineering, we attempt to provide an answer to this question by identifying mainly obstacles and challenges from two different perspectives: Industry and Academy.
Though research in Software Testing has been up to more than 56 years [
However, although in the last decade, experimental studies are the most widelyused and reported, the majority of them have significant limitations with respect to the artifacts and subjects utilized; even, when no human intervention is required in a study, the preparation of artifacts is expensive (e.g. automated environment for test execution), which limits to use only toy artifacts, like small numbers of programs, versions, or faults. Moreover, most of the subjects are usually students, given that practitioners are reluctant or do not have sufficient free time to participate in such empirical studies.
In order to gain a deeper understanding of the problem to conduct industrial case studies within a real industrial environment, with real people and real systems, we organized a panel at the 25th International Conference on Advanced information Systems Engineering (CAiSE’13) in Valencia.
The objective of the panel was to bring together senior researchers and practitioners to discuss why it is so difficult to investigate the applicability of testing tools and techniques (e.g. cost effectiveness) in industrial contexts. A deeper understanding of this issue should contribute to better testing practice and more relevant research for practitioners.
Among the questions that the panelists were asked to address were the following: Industrial perspective.
What obstacles did you perceive to adopt any software testing research results into practice? What are the current challenges you are facing during your current software testing processes? How are you searching for ways to overcome these challenges? What did you do to overcome these obstacles? And what do you think academia should do to overcome these obstacles? How could the testing practitioners reach the software testing research community? Academic perspective What obstacles did you encounter when evaluate testing tools? What did you do to overcome these obstacles? And what do you think industry should do to overcome these obstacles? Does software testing research address the real issues arising in practice? How could the software testing research community reach testing practitioners? 2
This section summarizes the discussion of the panel, that was moderated by Ricardo Baeza-Yates (Yahoo).
Obstacles. Testing is viewed as an very expensive activity. In most companies, it causes two main consequences: i) software testers are much less technical than development team. Thus, this limited technical background make more difficult the effective communication with researchers. ii) Software testers are constantly under pressure to produce test cases in short term. This reduces the possibility of exploring new techniques that could be more costeffective than traditional ones.
On the other hand, academic panelist consider that if there are no enough funds for training, the introduction of new testing techniques, tools is fruitless. This training should not be provided only by industry, but also by the university. However, few Spanish universities provide up-to-date testing courses at an advanced undergraduate level .
Considering that software applications become more complex, and current testing techniques are not detecting relevant defects, researchers aim to automate the testing process, with the purpose of reducing the costs and increase the effectiveness of test cases.
From practitioner perspective, panelists consider that the only way to increase the budget for testing activities is to quantitatively demonstrate the applicability of testing tools in other similar contexts. However, as negative results are usually more difficult to be published, empirical evidence is available only partially, which can affect to make important decisions at organizational level. Panelist team agreed in fostering continuous collaboration between industry and academy from the beginning of the research cycle: identification of research problem. A prerequisite to succeed in the close collaboration is that industry must understand and accept that researchers can address areas with industrial challenges; and researchers do not solve short-term problems. 3
We would like to thank Ricardo Baeza, Ywan van der Kleijn, Maximiliano Mannise, and Maria Jose Scalona for their contribution as panelists of the CAiSE’13 industrial track. We also acknowledge the partial support from EU grant ICT-257574 (FITTEST).