Advancing RE Education: Towards a Mapping Scheme for Benchmarking Students’ Specification Skills Anthea Moravánszky1 and Samuel A. Fricker, 2 1 University of Szeged, Doctoral School of Education, Hungary & University of Applied Sciences of the Grisons, Pulvermühlestrasse 57, 7000 Chur, Switzerland 2 University of Applied Sciences and Arts Northwestern Switzerland, Bahnhofstrasse 5, 5210 Windisch, Switzerland Abstract In Requirements Engineering (RE) Education, aligning students' requirements engineering skills with industry demands is one of the primary aims for educators. RE curricula vary between higher education institutes, but how they differ is difficult to describe. This paper proposes a novel mapping scheme to determine students' specification skills and, consequently, their industry readiness. This paper describes and motivates the scheme based on established bodies of knowledge in requirements engineering, syllabi for education and certification, and requirements specification standards. A preliminary validation of the scheme is provided by applying it to results produced by students of two RE courses taught at two universities of applied sciences, with the respective lecturer reviewing the accuracy and precision of its application. This paper describes the scheme and its development, outlines the preliminary validation process, and discusses its potential use for mapping RE education practices. Keywords Requirements Specification, Specification Skills, Mapping Scheme1 1. Introduction The significance of preparing students for industry becomes evident in the requirement specification skills discourse [1]. Given the diverse expectations on RE skills across various industries and ever-new challenges addressed by changing technologies for designing solutions, educators must continuously update their RE curricula [2]. Benchmarking is a strategic approach to describing and comparing such practices and discussing the evolution of these practices. A scheme for benchmarking is an instrument for structured and repeatable documentation. In RE education, there is a scarcity of schemes for benchmarking. There are syllabi, such as [13], that could be used for such benchmarking. While the intent of teaching such practices is easy to determine in course descriptions, it is unclear how evidence about the students’ learned ability to apply these practices would be documented so that these abilities can be identified In: D. Mendez, A. Moreira, J. Horkoff, T. Weyer, M. Daneva, M. Unterkalmsteiner, S. Bühne, J. Hehn, B. Penzenstadler, N. Condori-Fernández, O. Dieste, R. Guizzardi, K. M. Habibullah, A. Perini, A. Susi, S. Abualhaija, C. Arora, D. Dell’Anna, A. Ferrari, S. Ghanavati, F. Dalpiaz, J. Steghöfer, A. Rachmann, J. Gulden, A. Müller, M. Beck, D. Birkmeier, A. Herrmann, P. Mennig, K. Schneider. Joint Proceedings of REFSQ-2024 Workshops, Doctoral Symposium, Posters & Tools Track, and Education and Training Track. Co-located with REFSQ 2024. Winterthur, Switzerland, April 8, 2024 anthea.moravanszky@fhgr.ch (A. Moravánszky), samuel.fricker@fhnw.ch (S. A. Fricker) Copyright © 2024 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings and compared between education providers. Today, there is only a limited understanding of the RE practices being learned, and discussions between education providers regarding best teaching practices and their improvement hardly occur. This paper aims to narrow this gap by proposing an evidence-based scheme for mapping students' demonstrated specification skills. While specification skills are only a subset of the requirement engineering practices being taught, they can be evaluated based on concrete results that are created by the students. 2. Background In software engineering, requirements specification documents are created to document the results of RE activities. These documents contain software requirements and supplementary details such as a vision, solution concept, reference documents, context, acceptance criteria, and glossaries [3, 15]. Several works have addressed requirements specification and benchmarking specification skills in the educational context. Fricker et al. [4] surveyed industry projects, focusing on specification techniques. Their findings revealed diversity in the techniques and methods employed, with natural language emerging as the predominant notation, often complemented by integrating UML diagrams. Virodula and Fortino [1] assessed students' requirement specification skills based on an Industry Body of Knowledge within a STEM graduate program, aligning the curriculum with the International Institute of Business Analysts (IIBA) standards. While successfully assessing theoretical knowledge, the study did not evaluate students' practical application of requirements engineering (RE) specification skills. Anil and Moiz [5] introduced a comprehensive rubric for evaluating students' software requirements specifications. While providing a foundational tool for educators to assess Software Requirements Specifications (SRS), this rubric may require substantial modifications for universal application across different universities, diverse SRS templates, or a combination of single specification artefacts. The Requirements Engineering Ontology (REO) by Saito, Iimura, and Aoyama [6] maps requirements engineering techniques based on three RE Bodies of Knowledge (SWEBOK, BABOK, and REBoK). However, it lacks sufficient depth in outlining recommended specification techniques for benchmarking students' specification skills. The primary goal of this work is to promote comparability between higher education institutions. We do so here by assessing the specification skills of RE students. To achieve this, we propose a mapping scheme for application across higher education RE courses. The scheme focuses on mapping specification skills of students because it showcases their proficiency in generating distinct requirements-related artefacts and applying techniques suitable for various aspects of requirement specifications. We believe these specifications demonstrate that students possess a robust understanding of RE, comprehend the interconnections between different RE artefacts, and are capable of performing specific specification techniques, rendering them 'industry-ready.' 3. Methodological Approach Addressing this gap involves a comprehensive examination encompassing Bodies of Knowledge, model curricula, syllabi, and standards for identifying relevant RE specification practices and requirements specification document templates. Our methodological approach is based on the systematic mapping process outlined by Petersen et al. [7]. We selected our knowledge sources based on REE literature recommendations [8]. We included the Software Engineering Body of Knowledge SWEBOK [9] (version 3.0, 2014, since SWEBOK 4.0 is still under review) and the guide to the Business Analysis Body of Knowledge (BABOK) [10]. Furthermore, we also included the RE Body of Knowledge (REBoK) [11]. However, as the REBoK itself is only available in Japanese, we utilised a descriptive presentation of the REBoK authored in English instead. We also included standards such as IEEE 29148:2018 [12], which replaced earlier specifications. Additionally, the International Requirements Engineering Board (IREB) syllabus [13] is a valuable resource, particularly the syllabus of the CPRE FL certification. To address agile specification types, we incorporate Requirements Artifacts of the Rational Unified Process [14], enhancing our mapping to encompass a comprehensive range of RE specification practices. The screening process began with keywords acquired from requirement types and specification techniques and structures in the 'specification' section of Fricker et al.'s [4] work on 'common requirements engineering practices. From there, we mapped all our findings in the scheme and enhanced it with additional findings from the screened sources. In each of the mentioned works, the respective chapters were first read by one author, and then the entire work was subsequently searched again for keywords by the same author. A second author reviewed the results and conducted random checks via keyword searching. 4. Mapping Scheme for RE Specification Practices The foundation of our mapping scheme was derived from Fricker et al.'s specification section [4], which included "Requirement Types" and "Notations" as subparts. While we retained the subsection on requirement types, our mapping results led us to separate notations to distinguish specification techniques. Additionally, we introduced a new subsection, 'Requirements-related Artefacts and Document Structures,' to provide a more nuanced and detailed section. The decision not to include Storage was based on the understanding that it does not lend itself to direct mapping from a student's specification. Fig. 1 shows the developed mapping scheme. It is designed to comprehensively address various aspects of Requirements Engineering (RE) specifications, offering versatility in its applications. A key feature is its ability to capture specification variations, encompassing not only the count of specific elements but also their percentage representation to illustrate different class sizes of students. The intended use of our mapping scheme is versatile and can serve multiple purposes: 1) For Characterization of a Single Specification: Educators can employ the scheme to comprehensively characterise and understand the nuances of individual students' specification artifacts or entire specification documents. 2) For Characterization of Demonstrated Learning Outcomes: The mapping scheme can be applied to assess and articulate the demonstrated learning outcomes of a course in RE. 3) For Comparison Between Courses or Institutes: Educators should be able to compare the demonstrated learning outcomes between different courses or institutes of higher education, providing valuable insights for curriculum development and improvement. Figure 1: Basic mapping scheme 5. Early Validation with two Universities of Applied Sciences Our mapping scheme underwent validation in two RE modules at different universities of applied sciences offering bachelor's degree courses. Course A focuses on fundamental RE practices such as requirements elicitation, analysis, validation, and management in software development projects. It covers terminology, vision development, business analysis, prototype workshops, and reviews of requirements specifications. Students explore requirements specification languages and templates like Shall-, User Story-, and Use Case templates. Learning objectives include understanding RE concepts, creating visions and requirements for software systems, and quantifying quality requirements. The course emphasises developing release plans within staged and agile software development lifecycles and includes group projects for practical application. Course B is aligned with the syllabus of IREB CPRE FL and should prepare students to pass the certification. Agile methods are excluded as they are part of a subsequent course. Evaluating students' assignments reveals differences and commonalities. Most students use Use Case Diagrams and Specifications, with familiarity with activity and class diagrams. Functional and quality requirements are present and expressed through various methods. While elements like vision and goals are recognised, other document structures, except for the product backlog, were not recognised clearly. Lecturer A agreed with the mapping scheme's assessment, recognising its reflection of taught requirements specification techniques and student choices. However, Lecturer A suggested enhancing the scheme's sensitivity to detect more techniques in student documents by establishing more explicit links between scheme categories and providing precise detection guidelines for stakeholder descriptions. Regarding the assessment of Course B, Lecturer B noted that students were assigned three specific tasks: 1.) writing functional and non-functional requirements; 2.) modeling a use case diagram and writing a use case specification for one of the use cases of the diagram; 3.) modeling an activity diagram and class diagram based on a description) instead of creating a coherent requirement specification document, as for Course A. This resulted in a limited view and detected coverage of specification skills and the demonstrated variety. Figure 2: Summarized Results of the Mappings of Student Specification Assignments 6. Discussion Our work builds upon established knowledge frameworks to develop a mapping scheme to assess students' specification skills in REE. Leveraging these frameworks, we aimed to create a universal tool for mapping individual specification artefacts, thereby enabling the benchmarking of courses or institutes by comparing students' specification skills demonstrated in course assignments. The practical application of our mapping scheme uncovered its effectiveness in characterising specification documents. However, it also revealed limitations, particularly in supporting additional artefacts like implemented functional prototypes. Challenges persisted in identifying specific templates, especially phrase-based and document- template-based approaches. The flexibility and adaptability of students' expression of requirements were evident, but the nuanced task of differentiating between templates necessitated further consideration. One potential threat to the validity of our evaluation was the reliance on our works for testing the mapping scheme, introducing the risk of bias. To mitigate this, further validations on a diverse set of works are imperative to ensure the independence and generalizability of the mapping scheme. Our mapping scheme primarily reflects students' performance based on given assignments, potentially limiting its ability to entirely capture acquired specification skills from the course. The nature of the task assignment constrained the scope of skills showcased by the students, underscoring the importance of considering the broader curriculum context when interpreting the results. By providing insights into the nuances of students' performance and assignment tasks, the paper contributes not only to the benchmarking of RE education but also offers a pathway for continuous improvement in curriculum development. It encourages educators to consider the broader implications of the mapping scheme, fostering further exploration and adaptation to meet the industry's evolving demands and the diverse landscape of Requirements Engineering education. References [1] T. Virodula and A. Fortino, "Assessment of Systems Requirements Specification Skills Based on an Industry Body of Knowledge," IEEE Integrated STEM Education Conference (ISEC), Princeton, US, 2021. [2] M. Daun, A. M. Grubb, V. Stenkova, and B. Tenbergen, "A systematic literature review of requirements engineering education," Requirements Engineering, vol. 28.2 (2023) 145-175. [3] C. Rupp, Requirements-Engineering und-Management: Das Handbuch für Anforderungen in jeder Situation, Carl Hanser Verlag, 2020. [4] S. A. Fricker, R. Grau, and A. Zwingli, "Requirements engineering: best practice," in Requirements Engineering for Digital Health, Springer, 2014. [5] G. R. Anil and S. A. Moiz, "A holistic rubric for assessment of software requirements specification," 5th National Conference on E-Learning & E-Learning Technologies (ELELTECH), Hyderabad, India, 2017. [6] S. Saito, Y. Iimura, and M. Aoyama, "REO: Requirements Engineering Ontology Spectrum Analysis of Requirements Engineering Knowledge and Its Practical Application," IEEE 39th Annual Computer Software and Applications Conference, Taichung, Taiwan, 2015. [7] K. Petersen, S. Vakkalanka, and L. Kuzniarz, "Guidelines for conducting systematic mapping studies in software engineering: An update," Information and Software Technology, 64 (2015) 1-18. [8] S. Ouhbi et al., "Requirements engineering education: a systematic mapping study," Requirements Engineering, 20.2 (2015) 119-138. [9] The Software Engineering Body of Knowledge, 2024-02-06, URL: https://www.computer.org/education/bodies-of-knowledge/software-engine. [10] IIBA BABOK, Leitfaden zur Business Analyse Version 3.0, Götz Schmidt, 2017 [11] M. Aoyama, "Requirements Engineering Based on REBoK (Requirements Engineering Body Of Knowledge)," 2011, URL: http://www.nise.org/eng/REBoK/REBoK-APSEC2011- Tutorial-2011-12-05.pdf. [12] ISO/IEC/IEEE 29148:2018 Systems and software engineering. Life cycle processes. Requirements Engineering, 2018, URL: https://ieeexplore.ieee.org/document/8559686. [13] IREB CPRE FL Syllabus, 2021, URL: https://www.ireb.org/content/downloads/21-cpre- advanced-level-re-agile-syllabus/syllabus_cpre_al_re%40agile_en_v2.0.pdf. [14] P. Kruchten, The Rational Unified Process: An Introduction, Addison-Wesley Professional, 2004. [15] H. Kittlaus, S. A. Fricker, Software Product Management, Springer, 2017.