An important question with regard to new systems is whether they can be trusted by human users, especially when they are safety-critical. This problem should already be addressed in the course of requirements engineering for such systems. However, trust is actually a complex psychological and sociological concept. Thus, it cannot simply be taken into account as a desired or needed property of a system. We propose to learn from the Human Factors discipline on trust in technical systems. In particular, we argue that both undertrust and overtrust need to be avoided. The challenge is to determine system properties and activities in the course of requirements engineering for achieving that. We conjecture that both actual properties like safety and subjective assessment like perceived safety will be important, and how they will have to be balanced for avoiding undertrust and overtrust.
higher levels of automated driving functions. Regarding users' and societal acceptance, one of the mentioned challenges is trust. There are no hints, however, on how to address it.
At the recent RE'18 conference, Cysneiros et al. [CRdPL18] suggested that self-driving cars that demonstrate transparency in their operations will increase consumer trust, and they investigated transparency as a NonFunctional Requirement. However, they did not take the work on trust in automation by the Human Factors community into account, which applies psychological and physiological principles to the (engineering and) design of products, processes, and systems. The goal of Human Factors is to reduce human error, increase productivity, and enhance safety and comfort with a speci c focus on the interaction between the human and the thing of interest, see https://en.wikipedia.org/wiki/Human_factors_and_ergonomics.
For a better understanding of the issues involved in requirements engineering regarding trust, we propose to learn from the eld of Human Factors. 2
Hence, let us sketch some previous work on Human Factors that may inform future work on requirements engineering. The signi cance of trust is not limited to the interpersonal domain; trust can also de ne the way people interact with technology. Thus, the concept of trust in automation has been the focus of substantial research over the past several decades.
In their review of related work after 2002, Ho and Bashir [HB15] surveyed many studies where trust in automation was transferred from trust relationships between humans. The truster is human, the trustee is an automated system.
Earlier work on similarities and di erences between human{human and human{automation trust has been reviewed already in [MW07b]. In a nutshell, a major insight appears to be that anthropomorphizing, i.e., ascribing human features to an automated system, reduces the di erences. This is con rmed by a more recent study with potential (end-)users of autonomously driving cars, which assumed that trust is directly linked to perceived safety [HvBPB17]. Using a car simulator without any aspect of actual safety involved, they showed that trust in a self-driving car was directly linked in their human factors study to the perceived safety, which was highest in the case of anthropomorphic visualization through a chau eur avatar. It reacts to the same events as a world in miniature visualization that presents the car's perception of the surroundings, its interpretation and its actions, but is more human-like and potentially associated with more feelings.
Wang et al. [WJH09] studied the appropriateness of reliance depending on \reliability disclosure," which refers to whether participants were explicitly informed of the reliability of the feedback (as estimated by the system itself). More precisely, they provided meta-information on the reliability of feedback together with the feedback itself. Overall, the informed group seemed to resolve the changing reliability of the feedback better than the uninformed group. Instead of verbally informing, Neyedli et al. [NHJ11] studied displaying this meta-information in di erent ways, pie displays and mesh displays, respectively (in real time). They showed that this can change reliance on the automation. 3
We conjecture that both actual properties like safety and subjective assessment like perceived safety will be important, and how they will have to be balanced for avoiding undertrust and overtrust. The challenge for requirements engineering is to determine system properties and activities in the course of requirements engineering for achieving a good balance between actual and perceived safety.
As illustrated in Figure 1, this means to avoid both the region of undertrust and the region of overtrust. An open question is what can be done to avoid undertrust so that, e.g., a safe system (with a very low probability of causing injuries or even fatalities) would not be used by people because they do not trust it. Conversely, another open question is what can be done to avoid overtrust so that, e.g., drivers using current autopilots of cars would not simply hand-over control but still take care of the tra c situation and stay ready for taking over.
We propose to study these questions by considering objectively determined properties like safety and to determine adequate anthropomorphism for disclosing its level. For evaluations, studies like those established in the eld of Human Factors will most likely be necessary for assessing the balance.
However, based on the very recent observations in [FWCR19], it seems we cannot just rely upon the social sciences to take into account growing computer science research. It is critical for us to phrase the engineering questions in a way that they can be included in social science research, and to nd a way of incorporating the results in the engineering solutions.
Besides the explicit relationship between safety and trust in safety-critical systems, the importance of trust is becoming more and more evident in privacy and security-critical systems. For example, it was observed that privacy violations are having signi cant impact on trust [Mar18], and the security of the online systems is a necessary prerequisite for trust in such systems. This is especially evident in a new generation of blockchain-based systems [ARS18].
In order to achieve a proper understanding of how to handle undertrust and overtrust, we will have to incorporate studies with humans into requirements engineering using a combination of standard social science research methods: case study, survey, observational, correlational, experimental, and cross-cultural methods. However, in order for these studies to be e ective, it will be of critical importance that we learn how to pose the research questions in a way that the results of the social studies can be engineered into technical systems to be trusted.
Finally, let us propose a short list of research questions. We believe it will be of critical importance to handle properly both methodological and technical questions. The most pressing methodological questions that we see are:
How do we map research questions and methods from Requirements Engineering to Human Factors, and vice versa? How do we make engineering questions properly included in Human Factors research, and how can the produced results be e ectively engineered into the systems to be trusted? How do we apply computational social science methods in both Requirements Engineering and Human Factors?
The technical questions regarding trust, undertrust, and overtrust in Requirements Engineering will become even more pressing and probably best tackled in the context of emerging application areas of Arti cial Intellgence, autonomous adaptive systems, and blockchain:
How do we relate trust/undertrust/overtrust and non-functional requirements? How do we relate trust regarding safety/security/privacy? How do we develop evaluation techniques for trust regarding actual vs. perceived safety? How do we reduce trust manipulations in autonomous adaptive systems through techniques such as anthropomorphization? How do we de ne trust in fully open decentralized blockchain systems that store data and run code supplied by any anonymous entity?
How do we ensure ongoing trust in evolving systems with emerging behavior and machine learning? [ARS18]
Israa Alqassem, Iyad Rahwan, and Davor Svetinovic. The anti-social system properties: Bitcoin network data analysis. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018. [ES17] [HB15] [Mar18] [MW07a] [MW07b] [NHJ11] [WGM00] [WJH09]
EPoSS ERTRAC and ETIP SNET. Ertrac automated driving roadmap. ERTRAC Working Group, 7, 2017.
Kevin Anthony Ho and Masooda Bashir. Trust in automation: Integrating empirical evidence on factors that in uence trust. Human Factors, 57(3):407{434, 2015.