User Adaptive Systems (UASs) are futile without software. Moreover, integrating user modeling component into software system may add bugs if not tested properly. However, the evaluation of UASs does not intersect with software evaluation as commonly defined in Software Engineering. We suggest adopting the common software engineering practices, changing the community's practice and methods by integrating software testing as an integral part of any study involving software development. That will allow win-win situation for both: the researcher and community, since the code will be bug-free and hence easily reproducible/reusable by other members of the community. • Information systems → Recommender systems • Software and its engineering → Software testing and debugging by that reduces a frustration of people that often struggle with the guessing an author's intention. Albeit everyone agrees with the fact that it is essential to improve the quality of research algorithms by at least making their code publicly available and elaborating on the tools one had used during the research, nobody reports on any testing done to ensure that a system provides correct, consistent results. Once researchers get good results in terms of accuracy/coverage/novelty/serendipity or any other predefined evaluation metric, they hasten to share their findings with the community. However, without testing whether they have bug in their implementation or even inconsistency in the chain of actions according to the paper they have published, the results may be questionable. Hence we inquire the reader:” How do you know whether your User Adaptive system's results are correct? Have you tried to verify your results by writing new code and reproducing the steps that were defined in your paper?” By following such strategy, you can kill two birds with one stone: first, test that your method/experiment is reproducible and secondly prove that it works correctly by testing results with other implementation. The goal of this position paper is to examine existing evaluation methods in both UASs and Software Engineering and suggest testing approach that will strengthen this research field's outcome in the future. Evaluation metrics are commonly used to determine both the quality and performance of UASs [2]. Most frequently used are statistical evaluation methods, personality tests, accuracy, RMSE, A/B Testing and Benchmarking. Albeit all these metrics appraise UASs on their performance, accuracy and statistical significance, none of them addresses software testing for code, that is used to implement these UAS.
User-Adaptive Systems (UASs) are interactive systems that adjust their functionality to individual users according to the user model that was built by learning user behavior, inference, or decision making [13]. Over the last decade, a wide variety of different UASs has been introduced. Indeed, they incentivize researchers to publish their results by writing papers that often have the same structure: They contain of few common essential parts, like introduction, related work, experiment/method, evaluation and conclusions. The evaluation part allows us, as researchers, to decide whether this specific system lives up to both: the scientific community and the end user expectations in terms of quality and performance [16]. However, no one, as far as we know, reported on testing these UAS for their correctness. This is, primarily, because performing a study without testing the software properly first is the reality in our community. As Valentino Rossi maintained once:” Once the races begin it's more difficult and there is never that much time for testing”[9].
Let us consider other researchers’ expectations within the same
scientific community when they read a relevant paper and are eager
to replicate the same experiment again but with a different setup.
There are two primary aspects in such scenario: reproducibility and
correctness of the method/algorithm/system that introduced in the
paper. Nowadays, the reproducibility is a hot topic [
Let us peruse two kinds of applications to show the differences in
the testing process. Web application, a client-server software
application, differs from other applications in few ways. Indeed, it
can be accessed by a wide number of users, from different parts of
the world. Since each one of them uses different hardware, OS,
Web browser and etc., such application should be able to run on
heterogeneous execution environments. Moreover, the ability to
respond the user input in real time is essential [
As can be seen above software testing is an essential part of software development. All applications require such approach, with no exception and depending on their functionality and the goal, various methods for testing are chosen.
In this section, we claim that since user models and user modeling
components should not be separated from the software, testing
techniques that are applicable in software development should also
suit User Modeling research. The question only is what testing
techniques can be applied from Software Engineering field.
After scrutinizing various factors categories [
As a further matter, in order to be able to rely on the results of the system, there is a need to ensure its testability and it needs to be tested at least for accuracy. Moreover, in order to enable replication of the experiments and software reuse, both portability and reusability need to be tested as well. We believe that approaching testing in UASs is going to benefit both the researcher and the community. As Burt Rutan maintained:” Testing leads to failure, and failure leads to understanding”. [8] http://www.brainyquote.com/quotes/quotes/b/burtrutan39455 6.html [9] http://www.brainyquote.com/quotes/quotes/v/valentinor3012 35.html [10] https://en.wikipedia.org/wiki/Mission_critical [11] https://en.wikipedia.org/wiki/Portability_testing [12] https://en.wikipedia.org/wiki/Web_application [14] Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R.
M. (2009). Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery, 18(1), 140-181. [16] Said, A., & Bellogín, A. (2015, September). Replicable Evaluation of Recommender Systems. In Proceedings of the 9th ACM Conference on Recommender Systems 363-364. ACM.