Modelling and Facilitating User-Generated Feedback for Enterprise Information Systems Evaluation Nada Sherief Faculty of Science and Technology, Bournemouth University, UK nsherief@bournemouth.ac.uk Abstract. Most enterprises operate within a complex and ever-changing context. Users understand the software as a means to meet their requirements and needs, thus, giving them a voice in the continuous runtime evaluation of software would naturally fit this level of abstraction to ensure that requirements keep pace with changing context. However, this evaluation knowledge is often provided in an ad-hoc manner, which endures a great deal of impression and ambiguity leading to another problem, which is how engineers can extract meaningful and useful information from such feedback to inform their maintenance and evolution decisions. This doctoral work is novel in providing classifications of users’ feedback constituents and how they could be structured, which can be employed for a formal feedback acquisition method. Also, capturing structured feedback using systematic means can aid engineers in obtaining useful knowledge for evaluating enterprise information systems in order to maintain and evolve their requirements. Keywords: Users’ Feedback; Feedback Analysis; User Involvement; Requirements Models; Enterprise Information Systems Evaluation; 1 Research Problem and Motivation Requirements management is still one of the most challenging fields in software development [1] [2], has the most impact on project success, and is a major issue for decision makers in enterprises. Requirements are gathered from a diverse group of users; and they are basically volatile in nature. These issues are worsened by the problem that users still typically provide their feedback on the fulfilment of their requirements in a natural language and in an ad-hoc manner, which introduces a great deal of imprecision and ambiguity. To cope with such a lack of precision, a range of semi-automated techniques have been suggested to handle such user data (this includes techniques such as text mining and/or human facilitator). These techniques may be used to gather, interpret, aggregate, and revise what users say, partly to mitigate for such issues as bias and subjectivity in their textual responses. However, more effective results can be reached if the feedback is written in a structured manner. Structured feedback text would, arguably, allow approaches, such as text processing, to provide more accurate results within less time and with fewer human interventions. Also, if text is structured the S. España J. Ralyté, P. Soffer, J. Zdravkovic and Ó. Pastor (Eds.): PoEM 2015 Short and Doctoral Consortium Papers, pp. 117-124, 2015. 118 N. Sherief requirements extraction process can be more systematic, eliminating complexity and ambiguity found in natural language, and requiring less effort. Presently, the design and conduct of feedback acquisition are heavily dependent on engineer’s creativity. To maximize the expressiveness of users’ feedback and still be able to efficiently analyze it, feedback acquisition should be designed with that goal in mind. Hence, the need to provide foundations to develop systematic approaches is needed for the structuring and use of users’ feedback [3, 4], and supporting engineers with appropriate tools for evaluating requirements and thus making appropriate maintenance and evolution decisions. 2 Research Aim The aim of this research is twofold that is 1) to explore common feedback structures and their pillars so that acquisition methods can be provided, which maximize quality without hindering users experience, and 2) to explore how to support requirements engineers in analyzing and transcribing end users feedbacks into well-defined requirements. This will lead to a more effective management and richness of the users’ role as evaluators. Also, it provides a systematic means for requirements engineers to capture and analyze and prioritize feedbacks and thus requirements too. 3 Research Questions RQ1) What are the concepts that constitute the feedback structure? How can they be modelled? And utilized in a feedback acquisition method? RQ2) How can requirements knowledge be extracted from the collected end-user feedback to help engineers in evaluating requirements? 4 Research Objectives Objective 1 - Background Search and Literature Review: the first objective is to review the relevant work done in the literature in order to analyze what peers have reached regarding the definition of users feedback, how it is analyzed, and utilized. Thus, the gaps can be identified and new methods proposed that can move the research field forward. This objective will be accomplished in parallel with other objectives in the PhD, because it is an ongoing task of analysis and criticism to the relevant topics, which should be continuously maintained throughout the research. Objective 2 - Developing a new classification of feedback components and types: the aim is to come up with a new classification and definition of feedback types, the elements that constitute each feedback type, and the details that users employ to describe their feedbacks. Modelling and Facilitating User-Generated Feedback 119 Objective 3 - Developing a Structured Feedback Acquisition Method: our goal is to provide a systematic means that is able to automatically classify users’ feedback, be able to validate this feedback and store it in a structured and interrelated manner. Objective 4 - Developing new evaluation templates for the engineers that can be used to support software maintenance and evolution: the analysis of classified structured feedback will provide the engineers with a unique set of cases that carry important knowledge that can inform the evaluation process. Our goal is to design a combination of cases that will deliver a novel set of evaluation templates that contain concrete and formal instances of inter-related feedback that can help engineers in different decision-making situations they encounter in runtime evaluation. Objective 5 - Verification and validation of the effectiveness of the approach: At the end, the proposed approach will be verified and validated through a case study to: investigate whether it is successful in providing engineers with useful/ meaningful instances of the feedbacks, and in helping them in taking evolution and maintenance decisions. 5 Literature Review Mainly this research literature is divided into three main streams. The first stream is user-centered approaches in which general topics were reviewed regarding how users are involved in traditional approaches and how enterprises benefit from user involvement to communicate problems and enhance their overall process. There are several paradigms where the role of users is central such as User centered design [5], User Experience [6], Agile methodology [7], Usability Testing [8]. These techniques can aid the design of enterprise information systems, but they are expensive and time consuming when used for highly variable software designed to be used by a large number of users in contexts that are hardly predictable at design time. Furthermore, this research is similar to End-user Computing [9] in the motivation of involving users and enabling them to change in the system itself to meet their requirements and needs. However, this research relies on users to provide feedback in order to decide on maintenance and evolution decisions rather than taking actions. . Recent research has been directed towards involving users in evaluating and modelling evolving requirements for large enterprise software. Authors in [10], main contribution is a theoretical understanding of user involvement as a key success factor in implementing and maintaining business intelligence solutions. Moreover, in [11], authors suggest users involvement in developing Business Process Management projects. Their modelling approach involves using User Requirements notation that integrates goals and usage scenarios, from which requirements can evolve. Additionally, in [12] the authors present how strategy maps can be augmented by consumer values to include goals reflecting consumer values, which can be used as requirements for new solutions. All the above work supports the importance of users in driving the enterprise business process as a lifelong activity. However, their work operates on the management of requirements at a rather strategic level to ensure goal satisfaction, and business strategy implementation. In contrast, this research aims to 120 N. Sherief provide engineering approach with concrete constructs to model and acquire feedback and enable their role to take place. Also, in the last decade there has been a lot of interest in the area of engineering runtime self-adaptive systems [13] [14]. In spite of its importance, the role of users in supporting and tailoring the adaptation process and decisions is still unclearly presented. The involvement of users as partners with the adaptation process amplifies its potential and range of applications. Second, more focused topics were reviewed to study the work that peers have reached regarding the definition of users’ feedback, how it is communicated, analyzed, and utilized. Authors in [15], extract the main topics mentioned in the feedback, along with some sentences demonstrative to those topics using sentiment analysis. Also in [16], have defined a simple domain ontology consisting of generic broad types of feedback and associations. They cluster feedback messages according to the entities they refer to, use natural language parsing and heuristic filtering that can match the detected keywords to domain ontology. Moreover, in [17], the research aims on providing an elicitation approach that can offer new opportunities for users to support them in documenting their needs using a mobile tool. In contrast, and instead of analyzing given feedback, e.g. through forums and social networks, this research contributes to forward engineer the acquisition process itself making the analysis more efficient. Finally, facilitating paradigms and platforms have been studied such as requirements models [18] [19] [20], ontologies [21] [22], controlled natural languages [23] and recommender systems [24] with the intension to employ them in the proposed solutions. 6 Research Methods To achieve objective 2: a two-phase empirical study was designed. In the first phase study a two sessions focus group study was conducted, which is a popular technique of qualitative research in software engineering [25]. The Grounded Theory [26] was one of the most appropriate approaches to take. The Grounded Theory allows researchers to discover as much as possible variations in people’s behaviors, issues and/or concerns about the problem rather than depending on prior hypotheses. The focus groups’ sessions lasted 2 hours and 52 minutes. Both sessions were audio recorded and transcribed with consent from participants. The goal was to collect insights and experience from users who have actually given feedback before. Also, both junior and senior software engineers were invited to understand how more high- tech users give feedback and how they think a good feedback should be structured in order to be easily understandable and analyzed. The main areas to explore were: RQ1) How users would like feedback to look like, and the criteria that judge whether the feedback is meaningful and useful? RQ2) How users would like to be involved in the process of providing feedback, and what encourages them to act as evaluators? The research aim necessitates building a more concrete description for feedback structures. So in order to get the elaborated view, the second phase in-depth study was Modelling and Facilitating User-Generated Feedback 121 conducted, which involved the analysis of three actual enterprise systems’ -online forums where people give feedback on business software. Two hundred feedback from twenty different sources found on enterprise software forums, which are Microsoft’s TechNet, WordPress, and SAP were analyzed. The main areas to explore in the second phase study (i.e. the three forums analysis) were: RQ1) What are the main concepts that constitute the feedback structure? RQ2) What are the designs of the identified feedback concepts? Enterprise business software was targeted, as normally users tend to give a more serious and focused feedback, because of the social norms in such kind of forums. These three forums were chosen in order to target different types of business users with diverse technical capabilities. Actual users’ feedback was studied through observation and analysis of their posts and responses on forums. Forums provide a considerable amount of feedback that was analyzed using thematic analysis [27] with the intention to come up with the main concepts that constitute a feedback, and the outlines of the identified concepts. To achieve objective 3: the main method that will be used to ensure formalism is to build ontology [21] of feedback concepts in order to reach a common definition of the structure of feedback and the rules and relationships that govern its use. Ontologies include machine-interpretable definitions of basic concepts in the domain and relationships among them. Also, it can be easily maintained and extended. Furthermore, the ontology will be validated by an ontology reasoner to ensure its efficiency and consistency. To achieve objective 4: the interviewing technique [28] will be used to gather information from engineers. This involves providing them with examples of new validated feedback linked to mainstream RE models such as goal models to represent the stakeholders’ goals. This can also be related to the feature model to represent both the functional and non-functional requirements of the system. By relating the structured feedbacks to the feature model, engineers can propagate through the interconnections between them to determine different levels of evaluation information. Moreover, this technique will help to gain information about the RE methods and models they use, its drawbacks and the possible advantages they can reach when using the suggested methods, and models. This will assist in gaining a deep understanding of how the instances of formalized feedbacks entered by users can be utilized to create new cases that constitute evaluation templates that map to the actual needs of engineers in real situations of enterprise software evaluation. To achieve objective 5: a case study [28] will be performed to describe that particular case of users providing feedback, its acquisition, validation, and utilization in enterprise software evaluation in detail. Also, it will be investigated whether it is successful in providing engineers with useful/ meaningful instances of the feedbacks, and in helping them in taking evolution and maintenance decisions, and take learning from that to develop theory. 122 N. Sherief 7 Results Achieved The focus groups were analyzed using the thematic mapping approach [27]. The results of the focus groups analysis were initially introduced in [29] and further elaborated in [30]. This study provided a good level of understanding of users’ feedback aspects. The resulted thematic areas can be viewed from two different perspectives. In the first perspective, participants gave several insights regarding the structure of the feedback and what are the characteristics they think make their feedback meaningful and useful. These ideas are covered in the environment and structure thematic areas. In the second perspective, participants gave their perceptions regarding what they expect from a feedback acquisition method. How it can support, motivate and value their feedback. These ideas are covered in the engagement and involvement thematic areas [30]. This contributes to the literature by providing an initial set of thematic areas, where each can be studied and elaborated more to move the research field forward. Our research motivation and aim directed us to further study and observe components and types of user feedback in business context in more details. The two studies are linked through taking partial initial results from the focus groups’ themes, enhancing and expanding them through in-depth enterprise systems’ forums analysis and observation, which resulted in a novel classification and definition of feedback types and level of details used to describe them as elaborated in [30]. Also a set of conclusions regarding each feedback type’s elements was derived from the analysis of the forums’ feedback threads that will serve as a basis for providing formal definitions of the resulted classification (i.e. to achieve objective 3). 8 Next Steps To realize Objective 3: 1) to formalize the definition of user feedback elements the ontology [21] will be developed that classifies these elements using a set of rules; 2) to improve clarity and enable consistent automated semantic analysis of the feedback, a feedback controlled natural language [23] can be employed as an acquisition method for users to provide their feedback. It will restrict the user by general rules such as keeping sentences short and only use reserved keywords to define textual blocks; 3) a workflow integration layer will be designed to orchestrate the workflow between controlled natural language engine, the ontology reasoner to validate users’ feedbacks, and relate feedbacks to a feature model that will provide further systematic assistance for the engineers in extracting problems related to certain features, or determining which features are more problematic. To realize objective 4: new templates can be derived that combine multiple feedbacks and feedback types to form new cases that can inform the engineers by giving them a detailed view of the software’s evaluation status from the users’ point of view. This can be accomplished by: 1) defining the template building blocks through combining the feedback types that match together to form a concrete, useful and meaningful set of information (i.e. case). This will be designed by ontology rules that will contain the rules that govern the case identification; 2) describing a set of Modelling and Facilitating User-Generated Feedback 123 rules for extracting information, and building templates that can support engineers in taking maintenance and evolution decisions from the numerous users’ feedbacks that are continuously filled in the ontology knowledge base; 3) designing a workflow integration layer that will manage the interaction between the ontology reasoner for case definitions component, the template extraction rules component, relating them to the system’s goal model, and the component for handling the utilization of recommender systems [24] that will be applied on the users’ feedbacks to help eliminate extra time and effort and produce more accurate templates. 9 Conclusions This paper has presented the current results and ongoing research on modelling and facilitating user feedback for enterprise information systems’ evaluation. Moreover, an explanation was provided of how the findings can be employed to develop a collaborative acquisition method that utilizes the ontology’s formalism and the controlled natural language to validate and store structured feedbacks for representing problems in a systemized way where the risks resulting from human interventions are minimized. Finally, high level view architecture was suggested that will inform the construction of evaluation templates, which will help the engineers in taking the maintenance and evolution decisions. Therefore, the current research results serve as a foundation step for a holistic approach for the structuring and use of users’ feedback for enterprise software requirements evaluation. Acknowledgments. I would like to thank my supervisors Dr. Raian Ali, Prof. Keith Phalp, and Assoc. Prof. Walid Abdelmoez for their invaluable feedback and support. The research was supported by an FP7 Marie Curie CIG grant (the SOCIAD Project) and by Bournemouth University – Fusion Investment Fund and the Graduate School PGR Development Fund. References 1. Jarke, M., Loucopoulos, P., Lyytinen, K., Mylopoulos, J., & Robinson, W.: The brave new world of design requirements. Information Systems, vol. 36(7), pp. 992-1008, (2011). 2. Cleland-Huang, J., M. Jarke, L. Liu, and K. Lyytinen.: Requirements Management–Novel Perspectives and Challenges. Dagstuhl Reports 2, vol. 10, pp. 117-152, (2013). 3. Almaliki, M., Ncube, C., and Ali, R.: The design of adaptive acquisition of users feedback: An empirical study. In: RCIS'14. IEEE, Morocco (2014). 4. Almaliki, M., Ncube, C. and Ali, R.: Adaptive software-based Feedback Acquisition: A Persona-based design. In RCIS'15. IEEE, Greece (2015). 5. Vredenburg, K., Mao, J. Y., Smith, P. W., and Carey, T.: A survey of user-centered design practice. In: CHI'02, pp. 471-478. ACM, Minneapolis, Minnesota, USA (2002). 6. Law, E. L. C., and Van Schaik, P.: Modelling user experience - An agenda for research and practice. In: Interacting with computers, vol. 22(5), pp. 313-322. (2010). 7. Dybå, T. and Dingsøyr, T.: Empirical studies of agile software development: A systematic review.In: Information and Software Technology, vol. 50(9–10), pp. 833-859. (2008). 8. Adikari, S. and McDonald, C.: User and Usability Modeling for HCI/HMI: A Research Design. In: ICIA'06, pp. 151-154. IEEE, (2006). 124 N. Sherief 9. Doll, W.J. and Torkzadeh, G.: The measurement of end-user computing satisfaction. In: MIS quarterly, pp. 259-274. (1988). 10. Yeoh, W. and Koronios, A.: Critical success factors for business intelligence systems. In: Journal of computer information systems, vol. 50(3), pp. 23-32. (2010). 11. Pourshahid, A., Amyot, D., Peyton, L., Ghanavati, S., Chen, P., Weiss, M., and Forster, A. J.: Business process management with the user requirements notation. In: Electronic Commerce Research, vol. 9(4), pp. 269-316. (2009). 12. Svee, E.-O., Giannoulis, C. and Zdravkovic, J.: Modeling business strategy: A consumer value perspective. In: The Practice of Enterprise Modeling, pp. 67-81. Springer, (2011). 13. Ali, R., Solis, C., Omoronyia, I., Salehie, M., and Nuseibeh, B.: Social adaptation: when software gives users a voice. In: ENASE’12. Poland (2012). 14. Ali, R., Solis, C., Salehie, M., Omoronyia, I., Nuseibeh, B., and Maalej, W.: Social sensing: when users become monitors. In: ESEC/FSE’11, p. 476-479. Hungary (2011). 15. Galvis Carreño, L. V., and Winbladh, K.: Analysis of user comments: an approach for software requirements evolution. In: ICSE'13, pp. 582-591. IEEE Press, CA, USA (2013). 16. Schneider, K.: Focusing spontaneous feedback to support system evolution. In: RE'11. IEEE, Italy (2011). 17. Seyff, N., Graf, F., and Maiden, N.: Using Mobile RE Tools to Give End-Users Their Own Voice. In: RE'10, pp. 37-46. IEEE Computer Society, Sydney, Australia (2010). 18. Yu, E.S.: Social Modeling and i*. In: Conceptual Modeling: Foundations and Applications, pp. 99-121. Springer Berlin Heidelberg, (2009). 19. Kang, K. C., Kim, S., Lee, J., Kim, K., Shin, E., and Huh, M.: FORM: A feature-; oriented reuse method with domain-; specific reference architectures. In: Annals of Software Engineering, vol. 5(1), pp. 143-168. (1998). 20. OMG, B.P.M.N., Version 1.0. OMG Final Adopted Specification, Object Management Group. (2006). 21. Noy, N.F. and D.L. McGuinness: Ontology development 101: A guide to creating your first ontology. Stanford knowledge systems laboratory technical report KSL-01-05 and Stanford medical informatics technical report SMI-2001-0880. (2001) 22. Siegemund, K., Thomas, E., Zhao, Y., Pan, J., Assmann, U.: Towards ontology-driven requirements engineering. In Workshop semantic web enabled software engineering at 10th international semantic web conference (ISWC), Bonn. (2011). 23. Kuhn, T.: A Survey and Classification of Controlled Natural Languages. Computational Linguistics, vol. 40(1), pp. 121-170. (2014). 24. Adomavicius, G. and Tuzhilin, A.: Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering, vol. 17(6), pp. 734-749. (2005). 25. Kontio, J., Lehtola, L., and Bragge, J.: Using the focus group method in software engineering: obtaining practitioner and user experiences. In Proceedings of the International Symposium on Empirical Software Engineering. IEEE, USA. (2004). 26. Creswell, J.W.: Qualitative Inquiry And Research Design: Choosing Among Five Approaches, Sage Publications, Inc. (2006). 27. Braun, V. and Clarke, V.: Using thematic analysis in psychology. Qualitative research in psychology, vol. 3(2), pp. 77-101. (2006). 28. Berg, B.L.: Qualitative research methods for the social sciences. Pearson Boston. Vol. 5. (2004). 29. Sherief, N., Jiang, N., Hosseini, M., Phalp, K., and Ali, R.: Crowdsourcing software evaluation. In: EASE'14, pp. 19. ACM, London (2014). 30. Sherief, N., Abdelmoez, W., Phalp, K., Ali, R.: Modelling Users Feedback in Crowd- Based Requirements Engineering: An Empirical Study, in 8th IFIP WG 8.1 working conference on the Practice of Enterprise Modelling, Valencia, Spain. (2015).