The significance of preparing students for industry becomes evident in the requirement specification skills discourse [1]. Given the diverse expectations on RE skills across various industries and ever-new challenges addressed by changing technologies for designing solutions, educators must continuously update their RE curricula [2]. Benchmarking is a strategic approach to describing and comparing such practices and discussing the evolution of these practices. A scheme for benchmarking is an instrument for structured and repeatable documentation.
In RE education, there is a scarcity of schemes for benchmarking. There are syllabi, such as [13], that could be used for such benchmarking. While the intent of teaching such practices is easy to determine in course descriptions, it is unclear how evidence about the students' learned ability to apply these practices would be documented so that these abilities can be identified and compared between education providers. Today, there is only a limited understanding of the RE practices being learned, and discussions between education providers regarding best teaching practices and their improvement hardly occur.
This paper aims to narrow this gap by proposing an evidence-based scheme for mapping students' demonstrated specification skills. While specification skills are only a subset of the requirement engineering practices being taught, they can be evaluated based on concrete results that are created by the students.
In software engineering, requirements specification documents are created to document the results of RE activities. These documents contain software requirements and supplementary details such as a vision, solution concept, reference documents, context, acceptance criteria, and glossaries [3,15]. Several works have addressed requirements specification and benchmarking specification skills in the educational context. Fricker et al. [4] surveyed industry projects, focusing on specification techniques. Their findings revealed diversity in the techniques and methods employed, with natural language emerging as the predominant notation, often complemented by integrating UML diagrams. Virodula and Fortino [1] assessed students' requirement specification skills based on an Industry Body of Knowledge within a STEM graduate program, aligning the curriculum with the International Institute of Business Analysts (IIBA) standards. While successfully assessing theoretical knowledge, the study did not evaluate students' practical application of requirements engineering (RE) specification skills. Anil and Moiz [5] introduced a comprehensive rubric for evaluating students' software requirements specifications. While providing a foundational tool for educators to assess Software Requirements Specifications (SRS), this rubric may require substantial modifications for universal application across different universities, diverse SRS templates, or a combination of single specification artefacts. The Requirements Engineering Ontology (REO) by Saito, Iimura, and Aoyama [6] maps requirements engineering techniques based on three RE Bodies of Knowledge (SWEBOK, BABOK, and REBoK). However, it lacks sufficient depth in outlining recommended specification techniques for benchmarking students' specification skills.
The primary goal of this work is to promote comparability between higher education institutions. We do so here by assessing the specification skills of RE students. To achieve this, we propose a mapping scheme for application across higher education RE courses. The scheme focuses on mapping specification skills of students because it showcases their proficiency in generating distinct requirements-related artefacts and applying techniques suitable for various aspects of requirement specifications. We believe these specifications demonstrate that students possess a robust understanding of RE, comprehend the interconnections between different RE artefacts, and are capable of performing specific specification techniques, rendering them 'industry-ready.'
Addressing this gap involves a comprehensive examination encompassing Bodies of Knowledge, model curricula, syllabi, and standards for identifying relevant RE specification practices and requirements specification document templates. Our methodological approach is based on the systematic mapping process outlined by Petersen et al. [7].
We selected our knowledge sources based on REE literature recommendations [8]. We included the Software Engineering Body of Knowledge SWEBOK [9] (version 3.0, 2014, since SWEBOK 4.0 is still under review) and the guide to the Business Analysis Body of Knowledge (BABOK) [10]. Furthermore, we also included the RE Body of Knowledge (REBoK) [11]. However, as the REBoK itself is only available in Japanese, we utilised a descriptive presentation of the REBoK authored in English instead. We also included standards such as IEEE 29148:2018 [12], which replaced earlier specifications. Additionally, the International Requirements Engineering Board (IREB) syllabus [13] is a valuable resource, particularly the syllabus of the CPRE FL certification. To address agile specification types, we incorporate Requirements Artifacts of the Rational Unified Process [14], enhancing our mapping to encompass a comprehensive range of RE specification practices. The screening process began with keywords acquired from requirement types and specification techniques and structures in the 'specification' section of Fricker et al.'s [4] work on 'common requirements engineering practices. From there, we mapped all our findings in the scheme and enhanced it with additional findings from the screened sources. In each of the mentioned works, the respective chapters were first read by one author, and then the entire work was subsequently searched again for keywords by the same author. A second author reviewed the results and conducted random checks via keyword searching.
The foundation of our mapping scheme was derived from Fricker et al.'s specification section [4], which included "Requirement Types" and "Notations" as subparts. While we retained the subsection on requirement types, our mapping results led us to separate notations to distinguish specification techniques. Additionally, we introduced a new subsection, 'Requirements-related Artefacts and Document Structures,' to provide a more nuanced and detailed section. The decision not to include Storage was based on the understanding that it does not lend itself to direct mapping from a student's specification.
Fig. 1 shows the developed mapping scheme. It is designed to comprehensively address various aspects of Requirements Engineering (RE) specifications, offering versatility in its applications. A key feature is its ability to capture specification variations, encompassing not only the count of specific elements but also their percentage representation to illustrate different class sizes of students.
The intended use of our mapping scheme is versatile and can serve multiple purposes: 1) For Characterization of a Single Specification: Educators can employ the scheme to comprehensively characterise and understand the nuances of individual students' specification artifacts or entire specification documents. 2) For Characterization of Demonstrated Learning Outcomes: The mapping scheme can be applied to assess and articulate the demonstrated learning outcomes of a course in RE. 3) For Comparison Between Courses or Institutes: Educators should be able to compare the demonstrated learning outcomes between different courses or institutes of higher education, providing valuable insights for curriculum development and improvement. Evaluating students' assignments reveals differences and commonalities. Most students use Use Case Diagrams and Specifications, with familiarity with activity and class diagrams. Functional and quality requirements are present and expressed through various methods. While elements like vision and goals are recognised, other document structures, except for the product backlog, were not recognised clearly.
Lecturer A agreed with the mapping scheme's assessment, recognising its reflection of taught requirements specification techniques and student choices. However, Lecturer A suggested enhancing the scheme's sensitivity to detect more techniques in student documents by establishing more explicit links between scheme categories and providing precise detection guidelines for stakeholder descriptions.
Regarding the assessment of Course B, Lecturer B noted that students were assigned three specific tasks: 1.) writing functional and non-functional requirements;
Our work builds upon established knowledge frameworks to develop a mapping scheme to assess students' specification skills in REE. Leveraging these frameworks, we aimed to create a universal tool for mapping individual specification artefacts, thereby enabling the benchmarking of courses or institutes by comparing students' specification skills demonstrated in course assignments. The practical application of our mapping scheme uncovered its effectiveness in characterising specification documents. However, it also revealed limitations, particularly in supporting additional artefacts like implemented functional prototypes. Challenges persisted in identifying specific templates, especially phrase-based and documenttemplate-based approaches. The flexibility and adaptability of students' expression of requirements were evident, but the nuanced task of differentiating between templates necessitated further consideration. One potential threat to the validity of our evaluation was the reliance on our works for testing the mapping scheme, introducing the risk of bias. To mitigate this, further validations on a diverse set of works are imperative to ensure the independence and generalizability of the mapping scheme.
Our mapping scheme primarily reflects students' performance based on given assignments, potentially limiting its ability to entirely capture acquired specification skills from the course. The nature of the task assignment constrained the scope of skills showcased by the students, underscoring the importance of considering the broader curriculum context when interpreting the results. By providing insights into the nuances of students' performance and assignment tasks, the paper contributes not only to the benchmarking of RE education but also offers a pathway for continuous improvement in curriculum development. It encourages educators to consider the broader implications of the mapping scheme, fostering further exploration and
adaptation to meet the industry's evolving demands and the diverse landscape of Requirements Engineering education.