This technical report introduces the results of sentiment analysis of artifacts in requirements engineering phase. These artifacts contain descriptions of requirements and functions for the development target such as software product and solution. The descriptions of requirements reflect user needs and problems are described based on the analysis of users' dissatisfaction with the current situation and their expectations. On the other hand, the description of functions describes the behavior of the system and the interaction between the system and humans. Therefore, we apply sentiment analysis to requirements artifacts which are created in an exercise for university students. We, then, investigate how the sentiment of the descriptions in the artifacts are changed. Sentiment analysis is performed using Google Cloud's Natural Language API on the descriptions included in the artifacts such as customer journey maps and user story mappings. From the results of the application, we confirmed that the sentiment score of each artifact was different.
Sentiment Analysis is a method for measuring and understanding the feelings of individuals from text data such as reviews on the web, blog posts and SNS posts, and is used in various situations such as understanding customer product satisfaction and checking employee stress. Sentiment Analysis determines whether an opinion is positive, negative or neutral from text data including phrases, words and expressions contained in sentences.
Sentiment analysis is also widely used in various research fields in software engineering. In the field of software repository mining, efforts have been reported to apply sentiment analysis to textual data extracted from developers' discussions (e.g. ticket comments, commit messages) in order to predict defects in source code and interruptions in OSS projects [1] . In addition, efforts to predict support ticket escalation by performing sentiment analysis on support tickets that represent issues raised by customers and combining it with machine learning has been reported [2] [3]. In the field of requirements engineering, efforts to acquire requirements by applying sentiment analysis to ratings and review comments on products have also been reported [4].
The main data handled in software engineering can be roughly classified into two categories: data obtained from the development and operation process and data obtained from the development artifacts (product). In addition to the application of sentiment analysis in software engineering to the development and operational process data mentioned above, there are also efforts targeting development artifact data. For example, in the paper [5], they took the Software Requirements Specification (SRS), which is one of the final products of the requirements definition process, and applied sentiment analysis to the text data obtained from the SRS, and found that They report that almost all sentences in the SRS (about 96%) were neutral.
On the other hand, there are no reports of sentiment analysis on artifacts in requirements engineering phase. In the requirements engineering phase, the problem awareness, needs and expectations of stakeholders (users, operators, etc.) are analysed and the functions and performance that satisfy these needs are defined. Goal models have traditionally been used to analyse problems and the consistency between problems and solutions. In initiatives that integrate design thinking and requirements engineering, personas, customer journey maps, etc., are created [6]. In these artifacts, it is recommended to describe the image of stakeholders (users, operators, etc.) and realistic images of the system's usage scenario. Therefore, it is conceivable that the emotional tendencies measured in the deliverables created during the requirements engineering phase may differ from the emotional tendencies of the SRS described above.
In this paper, we analyse the tendency of measured emotions in the artifacts created in the requirements engineering phase (refer to Figure 1). At this time, the analysis approach that has been used for a long time is called the classical approach, while the analysis used in design thinking is called the modern approach.
In this paper, we set the following Research Question (RQ).
In order to answer the above-mentioned research question, we analyse and evaluate the artifacts created based on two approaches (classical and modern) as university exercises task of the requirements engineering phase in software development.
The composition of this paper is as follows: Section 2 describes the content of the artifacts to be analysed; Section 3 describes the analysis methods and results; Section 4 considers the results of the analysis; and finally Section 5 provides a summary. In this paper, we target several artifacts created by the students in an exercise of a lecture on the upstream process of software development (part of the requirements definition and external design process) at a university (refer to Table 1). The number of students taking the lecture was 54, and more than 90% of them were third-year undergraduate students in science and engineering. The students have already taken lectures on programming and modelling (UML, etc.) and have basic knowledge of software development.
In the exercises, after the teacher explained the contents of the artifacts, each student independently created all nine artifacts in the order shown in Figure 1.In the first stage of the requirements definition process, they assume users' opinions and requests for the ideas of services provided by the teacher, and describe them in writing. In the subsequent exercises in the requirements definition process, they create artifacts based on the Classical Approach (2 types) and the Modern Approach (4 types). The creation of artifacts by several people and third-party reviews of the created artifacts are not carried out. Therefore, a series of artifacts for 54 students were created. In advance, we obtained permission to use the artifacts for this study from the students who produced the analysed artifacts. We targeted artifacts that contained a certain amount of natural language descriptions for sentiment analysis. Specifically, there are 6 artifacts in total: users' opinions and requests, persona requirement, service scenario, customer journey map and user story mapping, which are the artifacts created using the modern approach, and goal model, which is the artifact created using the classical approach. We exclude the business flow, which is a typical artifact created using classical approach from the sentiment analysis. This is because the business flow also include natural language descriptions in the labels of activities and flows, but the amount of it is small. Similarly, we excluded the screen prototypes created in the external design process from the sentiment analysis. Similarly, we excluded the screen prototypes created during the external design process from the analysis. In the following we will explain the content of
Users' opinions and requests are created in order to verbalise their opinions and requests for services. In this artifact, 2 opinions or requests such as 'This kind of service would be useful' or 'This kind of service is disappointing' are described for each of the three services listed below.
• A service wanted to enrich your learning (lessons, research, etc.) at university • A service wanted for self-development during long holidays (summer holidays, etc.) • A Service wanted to enjoy daily life (housework, entertainment, meals, etc.)
A persona is a fictional character that represents a typical user of the product or service to be developed. Details of the character, such as its specific profile and requirements, are set. A Persona are used in the persona scenario method to devise and design services and systems that satisfy the defined persona, as well as for the characters in the artifacts to be created later. Setting a persona helps developers to develop user-centered services centered on the persona, rather than on the developer's self-indulgent services. First, one service is selected from the services considered in 'Users' opinions and requests'. Then, a persona is determined, assuming the person who must be satisfied with the service. The persona is then made detailed by adding not only the basic profile (name, age, height and weight), but also the place of work, place of residence and hobbies and preferences. After the detailed information of the persona is determined, what the persona wants for the selected service (persona requirements) is described in 350 characters or more. In this paper, we only analyse the description of persona requirements among the persona.
The description of service scenarios is one of the processes in the persona scenario method, and is created to assume how the main persona will use the service in his/her daily life. Specifically, it is described in a scenario format with 6 or more steps, when, how, in which situations, and what operations are performed by the main persona to realize the service.
A customer journey map (hereafter CJM) is a visual representation of the predicted actions and emotions that a pre-defined persona will take until using a service or product, arranged in chronological order. This is created to vividly imagine the user experience after determining the target user profile and the key process to focus on when considering the service. Creating this can help developer visualize how persona feel so they can avoid potential issues ahead of time, increase persona retention, and discover key information to make the best decisions for development.
We show an example of a CJM in Figure 2. A CJM consists of 6 elements: 'Persona Requirements', 'Specific Scenes', 'Scenes Name', 'Persona Actions', 'Persona Emotions' and 'Insights (from persona's actions and emotions)'. In the 'Specific Scenes', write a concrete sentence that enables the reader to imagine the situation in which the persona's requirements are generated. Then, in the 'Scenes Name', describe the specific scene in chronological order by dividing it into four or five scenes. In the 'Persona Actions', describe the actions the persona is likely to take in each scene, and in the 'Persona Feelings', describe the feelings and thoughts of the persona in each scene, including positive and negative ones, in text form. Then, organise the actions and feelings and describe in the 'Insights' why they act that way, why they feel that way, the solutions, etc.
A user story mapping (hereafter USM) is a visual representation of the values (functions) that a service wishes to realize in chronological order and in order of priority, based on the actions of personas. After the persona and CJM have been created and the image of the service has been established, a USM is created to concrete the image of the service. Creating this can help the entire development team organize persona behavior and the value the service will bring so they can understand the value of the service, and determine development priorities.
We show an example of USM in Figure 3. A USM consists of 5 elements: 'Persona Problem', 'Service Value', 'Activity Overview', 'Narrative Flow' and 'User Stories'. In the 'Persona Problem', describe the persona's problem obtained from CJM, and in the 'Service Value', describe how the service defined with the persona scenario method solves the persona's problem. In the 'Activity Overview', describe the implementation overview of the service provided, and in the 'Narrative Flow', describe the stories of the persona using the services provided with reference to the CJM, in chronological order. In the 'User Stories', the user stories required to experience the elements of the 'Narrative Flow' are arranged in such a way that the essential services with the highest priority are at the top, and the optional services with the lowest priority as you move down. The user story is a requirement for the realisation of a service. The service is composed of a set of user stories. It does not describe about the system, but the requirements and goals of the persona to use the service. It is written as 'The user wants to ∼ (so that ∼)'.
A goal model is a representation in a tree structure of the persona's goals and the ways to achieve them in the system to be developed. Creating this can help developer organize the requirements regarding the system so they can avoid creating gaps between the user's requirements and the system.
We show an example of a goal model in Figure 4. A goal model is a tree structure, in which the higher goals are the objectives of the lower goals. The top goal of the tree structure is the desired situation when the problem of the persona defined in the USM is solved. The top goal is then decomposed and detailed to create subgoals. The subgoals are decomposed and detailed in the same way, and this process is repeated to finally derive the functional and non-functional requirements.
We extracted the only texts described by the students from the 6 artifacts described in chapter 2, except for the elements names. Then, we split the extracted texts with symbols such as punctuation marks, periods, exclamation points, and question marks, as well as with spaces and line breaks. We obtained 2639 texts from all artifacts. We show the number of extracted texts for each artifact type in Table 2.
We performed a sentiment analysis on these texts.
In this paper, we use Google Cloud's Natural Language API [7] for sentiment analysis of text. Natural Language API is a service by Google Inc. that provides natural language processing techniques such as sentiment analysis, entity analysis, entity sentiment analysis, content classification, and syntactic analysis using machine learning, and is available for free for a certain number of times. In sentiment analysis, given a text, we obtain a score, which indicates the polarity of the overall sentiment of the text, and a magnitude, which indicates the intensity of the sentiment, based on word meanings, etc. score indicates the emotion of the text and has values from -1.0 (negative) to 1.0 (positive).
I wants to look good when I turn the camera ON, even in a first period non-face-to-face class on a very busy day in the morning.
A day when I overslept and woke up 30 minutes before the start of class. It happens to be a day with a full morning of classes, so I'd like to have a light breakfast. But I also want to put on some makeup in case the camera is turned on, and I don't want to be slammed into the computer right before first period. User wants to register his/her phone information with the service so that he/she can set the time from his/her phone.
1 User wants to be informed of days when automatic PC startup is not required, based on past PC startup times and class attendance.
User wants to start up his/her PC at a time other than a set time, even from a remote location. 2 User wants to register a time with the service when his/her PC will automatically start up in the morning.
User wants to register a mascot with the service so that he/she wants his/her PC to be started up by his/her favorite mascot. In this paper, we use the score obtained from the sentiment analysis of each sentence, and analyze them in units of artifact and artifact type. We show an example of texts extracted and split from artifacts, and the score obtained by sentiment analysis on the texts in the Table 3.
We analyzed the scores obtained by sentiment analysis for the texts by artifact type in terms of two aspects: the ratio of texts without emotional expression and the range of the emotions.
We analyze the percentage of texts without emotional expression by artifact type. First, texts with absolute scores of 0.25 or less were considered neutral (neutral texts without emotional expression), and texts with other scores were considered emotional (texts with emotional expression). Then, we calculated the percentage of texts without emotional expression and the percentage of texts with emotional expression by artifact type for all 30 students. We show the result of this analysis in Figure 5. As shown in Figure 5, the percentage of neutral tends to be higher in artifacts created later in the process, such as in CJM and USM, than in artifacts created earlier in the process. However, the percentage of texts with emotional expressions in the artifacts created using both classical and modern approaches is more than 30 percent. Especially in artifacts such as users' opinions and requests, persona requirements, and CJM, the percentage of texts with emotional expressions is more than 50 percent, which means that texts with emotional expressions are more frequent.
We analyzed the range of emotions in artifacts by artifact type. First, texts with score greater than 0 were defined as positive, and texts with other score were defined as negative.
Then, we calculated the maximum value from the positive score and the minimum value from the negative score for each artifact. Also, we calculated the average of the maximum positive score and of the minimum negative score by artifact type. We show the result of this analysis in Figure 6. As shown in Figure 6, the range of emotions is larger for CJM and USM created using the modern approach, and smaller for the service scenario and the goal model created using the classical approach.
The result of the analysis of the percentage of texts without emotional expressions in the session 3.2.1 showed that emotions were measured in about 30 % or more of the texts for all types of artifacts. In particular, artifacts created using modern approaches such as persona requirements and CJM were found to have emotional expressions in more than half of the texts on average.
Therefore, the answer to the Research Question RQ: Are the emotional expression measured from texts in the requirements engineering artifacts created using classical and modern approaches neutral? is that the emotional expression measured from almost all texts in the artifact created using the both approaches is not only neutral, but also negative and positive. Also, the artifacts created using the modern approach except for service scenarios tend to have a higher percentage of texts with emotional expressions than artifacts created using the classical approach. This is different from the tendency, reported in the paper [5], of emotional expression measured from texts in the SRS. We believe that the modern approach mainly requires to describe the persona's expectations and dissatisfaction, so that the sentences are more likely to have emotional expressions in artifacts created using modern aproach. For the service scenario, the functional descriptions such as the operations performed by the persona to realize the service and the accompanying system behavior are mainly required, so the percentage of texts without emotional expressions may have increased compared to the artifacts created by the other modern approaches.
On the other hand, the classical approach mainly requires to describe the functional and non-functional requirements of the system. In the goal model, functional and non-functional requirements for the system are derived by detailing the goals from the higher-level goals to the lower-level goals. In the detailing process, the top goal described the requirements for the persona, such as the desired situation when the persona's problem is solved, so it is assumed that emotions were measured from the sentences of the some high-level goals. Now that we have confirmed that texts in artifacts in requirements engineering phase often contain emotional expressions, we will discuss the results of the section 3.2.2 analysis of the range of emotions. The result of this analysis confirmed that the range of emotions in CJM is particularly large. This suggests that many artifacts of the CJM tend to contain both strongly positive and strongly negative texts. This is because the CJM include a direct verbal description of what the persona is feeling, such as I'm happy!, " "Good, " "My tension is low" etc., in the "Persona Emotions" item, and thus it is easier to measure strong emotions from such descriptions, and we believe that we were able to measure strong emotions from many of the CJMs. Thus, not only the appearance frequency of text with emotional expressions but also the range of emotions that emerge differs depending on the type of artifact, and in particular, artifacts that directly describe emotions and artifacts that describe dissatisfaction and expectations are likely to have a large range of emotions.
From these results, we confirmed that artifacts in requirements engineering phase often contain texts with emotional expressions, and that some types of artifacts tend to contain strong emotions. We believe that sentiment analysis of the texts in artifacts and extraction of texts with large score will facilitate understanding of the stakeholders' dissatisfaction and expectations, and the scenes in which these feelings are held. On the other hand, We believe that by extracting neutral (texts without emotilnal expressions), it will be possible to extract descriptions of functional and non-functional requirements for the system from the artifacts. In addition, if the results of sentiment analysis of an artifact (e.g., CJM), which should reflect stakeholders' expectations and dissatisfaction, show a small percentage of text with emotional expressions or a small range of sentiment, we believe that the artifact may not have successfully acquired or extracted stakeholders' demands. Therefore, we believe that performing sentiment analysis on artifacts can be used to measure the degree to which artifacts are acquiring demands. Thus, sentiment analysis of artifacts will facilitate the extraction of descriptions of stakeholder sentiments and functions, and will measure the success of artifacts in extracting and obtaining stakeholder requirements, thereby supporting the efficiency of system development, and so on.
In this paper, we reported the results of sentiment analysis on artifacts in requirements engineering phase of software development, which were created using two approaches, classical and modern. Specifically, we conducted sentiment analysis using Google Cloud's Natural Language API on the descriptions in six artifacts, such as customer journey map and goal model, and analyzed emotion scores obtained by artifact type. The results showed that the percentage of text with emotional expressions in all types of artifacts created using the two approaches was more than 30 percent, and especially in the persona requests and customer journey maps created using the modern approach, the percentage of text with emotional expressions was more than 50 percent. From this, as an answer to the research question, "Are the emotional expression measured from texts in the requirements engineering artifacts created using classical and modern approaches neutral?", it was confirmed that emotional expressions measured from texts in artifacts created using both approaches were not only neutral, but also negative and positive. In addtion, it was confirmed that artifacts created using modern approach tended to have a higher percentage of texts with emotional expressions than artifacts created using classical approach. This may be due to the fact that the modern approach is more likely than the classical approach to describe requirements that persona has. It was also confirmed that the range of emotions differed depending on the type of artifact. This is because the required descriptions differ depending on the artifact, and the range of emotion is considered to be larger for artifacts that directly describe emotions and those that describe dissatisfaction and expectations.
we believe that sentiment analysis of artifacts can be used to measure the degree to which artifacts are acquiring demands. Thus, sentiment analysis of artifacts will facilitate the extraction of descriptions of stakeholder sentiments and functions, and will measure the success of artifacts in extracting and obtaining stakeholder requirements, thereby supporting the efficiency of system development, and so on.
In this report, we analyzed the percentage of texts without emotional expressions and the range of emotions in each artifact type, but in the future we would like to conduct more detailed analysis of the characteristics of emotions in artifacts by analyzing artifact units and analyzing other factors besides the range of emotions. We would also like to investigate the relationship between the emotion of the artifact and the quality of that artifact and the quality of the artifacts (e.g., screen prototypes) that are created behind the process. In addition, we would like to confirm whether similar trends can be obtained using other artifact sets.