Two of the fundamental sources of data for requirement gathering for system upgradation are usage monitoring data and feedback data. Work has been done to acquire both individually and collectively. This work aims at validating the existing methods for combining feedback and monitoring data on a large scale and taking it a step further by developing a recommender system for requirement elicitation and prioritization. We have developed prototypes for data gathering agents built in JavaScript and connected to our server through REST APIs so that we don't affect the original code base of deployed applications. Once done with testing locally we will take it to production and proceed further. Validation will be done on SAKAI learning management system implementations in educational institutions. The software requirements prioritization process will be evaluated before and after applying the recommender system. Causal-comparative/quasi experimental research method will be used to evaluate the system developed.
Software evolves and adapts continually according to the user’s ever-changing requirements.
Requirements are elicited from the stakeholders continuously so that their needs concerning the
particular software are met [
Monitoring and Feedback data collection frameworks are developed with a goal of continuous improvement in requirement elicitation. Such systems have components related to acquisition and storage of data. Further enhancement is made to these components by incorporating a methodology where monitoring data will further be aggregated using matrices along with runtime feedback in order to enable clustering and grouping of feedback statistics. This improves the quality of information presented to the requirements engineer, enhancing the elicitation process. Furthermore to enhance the prioritization process there is a need to develop a recommender system.
Frameworks developed for continuous requirements elicitation through combining both usage monitoring and user feedback have been tested on a limited data where data set was small and fragmented. We want to draw conclusion on quantitative benefits of using the above mentioned approach based on a sizable data set. Components for storing and combining data need to be developed, incorporating advance aggregation and clustering techniques. Based on the results gathered from this clustering and aggregations new requirements should be elicited and prioritized. There is a need for developing a recommender system that prioritizes requirements based on combination of user feedback and monitoring data.
Literature review identifies that internal sources for user feedback, tools should be implemented to automate user feedback processing; third, user feedback to requirements should be related through reference points; requirements validation for user feedback should be enhanced along with establishing developer–user communication [14]. Data driven approach towards requirement engineering has also been presented [15].
Several researchers have advocated the usage of monitoring data alongside feedback data as a way
of eliciting new requirements. As an example, in order to facilitate elicitation process there are
approaches [
In contrast, a method [
Few solutions exist that use monitoring and feedback data together for other purposes. QoE probe [8] is its example. It’s a mobile app that capitalizes on monitoring and feedback data for verification and validation of requirements.
Probe requests feedback from time to time and continuously aligns it with system logs. Usage data is recorded with timestamps. It’s recorded when a feature usage started and how long it ended and how the user interacted with the system. Whenever an interaction is completed user is presented with a dialogue box to provide feedback. This gives a thorough insight about user acceptance level and how much they resonance with user needs. This comprises of feedback and monitoring data but lacks analysis component [8].
MyExperience [9] is a similar tool combining monitoring and feedback data. This system facilitates studies regarding human health as health metrics are populated through IOT devices and directly took feedback about how they felt. The concepts presented in this work haven’t been technically implemented.
FAME framework was presented, with an objective to facilitate requirement engineers in collecting and analyzing the monitoring and feedback data together with respect to a software system. User feedback is provided by end users using feedback modules while the real time events are monitored through monitoring agents in the host application that are then sent to a data lake. Data from all sources is combined through ontology. The requirement engineer is presented the combined data used to elicit requirements [10]. In this work data pre-processing and incorporation of data storage and clustering and aggregation of feedback and monitoring data is left as future work. It also lacked grouping of feedback and monitoring data and calculating metrics rather than just recording run time events.
An effort has been made to let the system elicit user requirements on its own by using monitors that detect certain situations occurring while system use. On such occurrences a system generated request is sent. Feedback is then correlated to monitoring data at that instant. This was implemented in a smart city project and positive results were achieved by combing real time monitoring with preemptive feedback mechanisms. The techniques implemented showed enhancement in the requirement elicitation process and improved the end-user experience [11]. This paper lacked the statistical significance and the study was specific to IOT systems.
A systematic literature review done in 2020 concludes that more machine learning and repositorymining based research for requirement prioritization is needed, there is a need for addressing match between software project contextual settings and methods, it is required to evaluate and compare requirement prioritization methods in real-world [16].
The review of literature done above establishes the interest in eliciting requirements from feedback and monitoring data. Validating the frameworks presented in the literature on a sizable dataset and using advanced techniques for clustering and aggregation is a future work consideration [10] as indicated in the above mentioned literature.
A need for a requirement prioritization recommender system that combines user feedback and usage monitoring data is established from literature.
We will be testing the combined effect of utilizing both feedback and monitoring data onto a large real-time user base. We will be taking the data through tools that would not affect the original code base and will work as an extension. We would be developing a recommendation system that prioritizes software requirements gathered through the combination of user feedback and usage monitoring data.
A detailed literature review was done to establish the relevance of research. Search terms used were: user feedback data, usage monitoring, combination of usage monitoring and user feedback data, and recommendation system for requirement prioritization, upon literature review it was established that there is a need for a recommendation system that prioritizes software requirements gathered through the combination of user feedback and usage monitoring data. The validation has to be done in real-time production environment.
Following steps will be performed for developing the above mentioned system
Step 4: Clustering and aggregation on feedback and monitoring data Step 5: Eliciting and prioritizing requirements based on the processed data
Step 6: Quantitative and qualitative comparison of elicitation process prior and after applying the framework
If step 6 is successful, we will perform correlation analysis that includes data reduction and correlation coefficient calculation. Then we will develop a recommender system using correlation based collaborative filtering and similarity score calculations. We should be getting elicited and prioritized requirements as an output. This framework will be validated on SAKAI learning management system implemented in universities.
By far we have shortlisted feedback and monitoring metrics. We have developed agents to monitor following metrics: • • • • • • • •
Sessions per active users Time spent on a feature Activity abandonment Activity completion Crash instances Bug reporting
Feedback provision • • • • • • • • • •
Assignments portal Academic and social alerts Calendar and scheduling Feedback Area Forums Gradebook Management Lessons Management Syllabus Management
Tests & Quizzes Initially these metrics will be applied to evaluate following modules of SAKAI Causal-comparative/quasi experimental research method will be used to evaluate the system developed. We will evaluate the requirement prioritization process before and after implementation of the recommender system.
We will apply the recommender system to the SAKAI learning management system in three universities in Pakistan. The end users are thousands in number. We will collect data regarding software prioritization before and after implementation of the system from the system developers and we will analyze the data collected through the system. Usage data will be collected before and after taking feedback and monitoring data will be collected to access how usage was effected after prioritization through our recommender system.
By far we have developed an understanding around SAKAI architecture and deployment and we have reached a conclusion that we will be adding our agents to the front end of the application on through JavaScript code and will link this code to our server through REST APIs. We have developed the basis for collecting monitoring data. This will give us scalability and the code base of the production application won’t be disturbed. We have developed a prototype of relevant JavaScript code and REST APIs and currently we are in process of integration with different modules of the SAKAI application. Once locally tested we will get these agents installed in production.
This project timeline indicates current progress and future plan for this research
January 2020-January 2021 • Selecting the domain • Background Study & research topic selection • Literature Review- State of the Art • Defining the Problem Statement • Proposing a methodology • Implementation of environment setup for data collection • Synopsis Document February 2021-June 2021 • (Tool Development) • Designing scenarios for data collection • Executing scenarios for data collection • Data Preprocessing July 2021-December 2021 • Formulation of Controlled Experiment • Checking Insights • Results Analysis January 2022-June 2022 • Post validation Survey • Evaluation • Extended Literature Review • Compilation of Thesis • Final Defense
[8] Fotrousi, F., & Fricker, S. A. (2016). QoE probe: A Requirement-monitoring Tool. In REFSQ Workshops.
[9] Froehlich, J., Chen, M. Y., Consolvo, S., Harrison, B., Landay, J. A., Street, S. B., & Mateo, S. (2007). MyExperience : A System for In situ Tracing and Capturing of User Feedback on Mobile Phones. 57–70.
[10] Oriol, M., Stade, M., Fotrousi, F., Nadal, S., Varga, J., Seyff, N., & Schmidt, O. (2018, August). FAME: supporting continuous requirements elicitation by combining user feedback and monitoring. In 2018 IEEE 26th International Requirements Engineering Conference (RE) (pp. 217-227). IEEE.
[11] Wüest, D., Fotrousi, F., & Fricker, S. (2019, March). Combining monitoring and autonomous feedback requests to elicit actionable knowledge of system use. In International Working Conference on Requirements Engineering: Foundation for Software Quality (pp. 209-225). Springer, Cham.
[12] Glinz, M., & Roots, A. T. (2019). CrowdRE : Achievements, Opportunities and Pitfalls. 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), 172–173 [13] Maalej, W. (2015). Bug Report , Feature Request , or Simply Praise ? On Automatically Classifying App Reviews. 116–125.
[14] Johanssen, J. O., Kleebaum, A., Bruegge, B., & Paech, B. (2019). How do Practitioners Capture and Utilize User Feedback during Continuous Software Engineering ? 2019 IEEE 27th International Requirements Engineering Conference (RE), 153–164.
[15] Franch, X., Ayala, C., López, L., Martínez-fernández, S., Rodríguez, P., Gómez, C.,Rytivaara, V. (2017). Data-driven Requirements Engineering in Agile Projects : The Q-Rapids Approach.
[16] Bukhsh, F. A., Bukhsh, Z. A., & Daneva, M. (2019). A systematic literature review on requirement prioritization techniques and their empirical evaluation. Computer Standards & Interfaces, 103389.