Heuristic-based Evaluation for Socio-technical Systems Felix Thewes1, Thomas Herrmann 1 and Marina Konrad1 1 Ruhr-University Bochum, Universitätsstr. 150, Bochum, 44780, Germany Abstract This paper reports on the development of a socio-technical evaluation method based on an interactive questionnaire and its application in six case studies. Expanding on the socio- technical heuristics by Herrmann et al. [1] the mixed-method interview questionnaire was developed for comprehensive evaluation of the intertwinement between social and organizational practices with technical artifacts and their usage. The interactive questionnaire enables organizations to perform evaluations without employing evaluation experts or the need for extensive training. A semi-structured participative identification of problems allows for a broad examination of socio-technical principles, which then can be assessed with corresponding guided interview passages. This approach was applied by students in diverse companies as project within a course on “design of socio-technical information systems”. The results of six evaluated teams of employees demonstrate the wide range of identified issues as well as the similarities between these companies. Keywords 1 Socio-technical system, socio-technical evaluation, work systems, mixed-methods interview, interactive questionnaire 1. Introduction The field of socio-technical systems design provides criteria for designing and re-designing work environments. Methods for evaluating socio-technical systems are either time-consuming or rely heavily on expert knowledge. Existing methods often investigate a specific issue or socio-technical themes. To examine socio-technical systems broadly, without prior knowledge of the existing issues, a method was needed to identify issues and problematic areas for further investigation. Herrmann and Nierhoff [2] combined methods from six different approaches presented in socio-technical literature to compile socio-technical heuristics. The challenge presented in this paper is the expansion on these heuristics by transforming the condensed content into an interactive questionnaire. Frist hand experience of the evaluated socio-technical system is utilized through interviews while evaluators are guided to investigate the system holistically. This is intended to lower the barrier of entry for consideration of complex socio-technical perspectives and open socio-technical evaluation to a wider range of applicants. Quick results are achieved by first identifying issues on a broad spectrum of aspects and perspectives to narrow thorough evaluation to the revealed critical issues. We present results from six case studies conducted by students. The socio-technical questionnaire was applied by students as part of a computer science master’s course “design of socio-technical information systems”. The results are used to investigate the following three research questions. R1: How far does the proposed procedure of socio-technical, heuristic-based evaluation help to detect relevant problems from the employees’ view - with respect to variety and quantity? R2: Which differences can be observed between the various cases that have been inspected? R3: How far are the novice evaluators able to address socio-technical intertwinement between technical and organizational issues? Proceedings of the 7th International Workshop on Socio-Technical Perspective in IS Development (STPIS’21) , October 11–12, 2021, Trento, Italy EMAIL: felix.thewes@rub.de; thomas.herrmann@rub.de ORCID: 0000-0002-8436-8106 (F. Thewes); 0000-0002-9270-4501(T. Herrmann) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 158 2. Background Our understanding of socio-technical systems focusses on the intertwinement of social and organizational practices with the infrastructure of technical artifacts. Initially this understanding was based on the observation that the development of organizations cannot be understood without referencing the technology used [3], [4]. Later, scholars emphasized that organizations should not predominantly be considered as work like technical systems [5]. Equal weight should be given to social as well as technical issues [6]. The practical side of the socio-technical perspective pursues an improvement of technical systems in areas such as ubiquitous computing[7], health care [8], maintenance repair [9], and others. Early approaches to support and evaluate socio-technical design [10], [11] have been renewed, e.g. by MEESTAR in the context of nursing [12], or by Imanghaliyeva et al. [13] who propose a synthesis of socio-technical principles. Shin and colleagues argue that fairness, transparency, and accountability are the most relevant issues to be satisfied in the context of socio-technical systems [14], [15]. On the theoretical side, the socio-technical view is also present, for example in the discourse on sociomateriality [16], where social subsystems (including roles, hierarchies, communication networks, and others) are distinguished from technical subsystems as imbrications of human (social) agency and material agency are differentiated. Scholars referring to Orlikowski [17] use the term entanglement for the intertwinement of the social and the material. In our view, the concept of entanglement does not necessarily imply a symmetry [18] between the social and the material as proposed, by contrast to Actor-Network-Theory [19], [20]. Once a technical artifact is integrated in a certain context (e.g., organization, communities, social practice), the social practices and the technical artifacts merge and cannot be separated anymore. This intertwinement varies from one system to another. We suggest that these differences in the intertwinement of social practices and technical artifacts to be a highly relevant subject of evaluation and potential improvement. Related work points to challenges in evaluation approaches. Several authors propose means or methods for socio-technical evaluation; for example, Shin[7], Krenn [21] and Nelles et al. [22], [23]. These methods do not address socio-technical systems in general but each of them has a certain focus such as human-robot collaboration or internet of things. These approaches, however, do not systematically consider the social and organizational practices, in which technical artifacts are embedded, as a subject of evaluation and redesign. By contrast, the methods for socio-technical evaluation mainly refer to the quality of technical artifacts and infrastructure in the context of their utility and usability for work tasks. In summary, current methods cover some aspects of the socio- technical intertwinement, but neglect to aim in the improvement of the organizational context, social practices, or social dynamics [24]. Other methods, e.g. ETHICS [25] or participatory design approaches such as MUST [26], that consider the fit of technical artifacts into an organization, mainly in the context of labor, focus on the phase of designing sociotechnical systems rather than evaluating them in the phase of usage. Hence, we focus on guiding the evaluation of the sociotechnical intertwinement instead of principles that aim at supporting design before use [13]. To deal with the complexity and manifold relationships within socio- technical intertwinement we propose the usage of heuristics as a basis of evaluation where we suggest that “Heuristics are rules of thumb for reasoning, a simplification, or educated guess that reduces or limits the search for solutions in domains that are difficult and poorly understood” [27]. Heuristics are useful mainly to quickly detect the most serious problems and help to draft design recommendations [28]. For example in the context of usability, experts or users inspect the features of an interactive system step-by-step by applying a list of items (heuristics) as proposed by Nielsen [29] or the International Standard Organization [30]. From the perspective of advanced HCI research and socio- technical design, it is reasonable to extend the evaluation of the usability of technical artifacts by including the social practice and broader organizational context, as proposed by [2], [31]. They propose eight heuristics that widens the focus from a pure view on technology and its usability to a broader view by also evaluating the context of organizational conditions and social practices. The eight heuristics are derived from criteria, principles, guidelines etc. as they are discussed in six different fields: human- computer interaction (HCI) and usability; computer-supported cooperative work (CSCW)/groupware; 159 process (re)design; socio-technical design; principles of job design, and privacy. Each of these eight heuristics covers both dimensions - the social and the technical - as well as their intertwinement. One challenge is to evaluate real practices of people who interact with technology in a socio-technical context and to understand the quality of the intertwinement between social and organizational practices with technical infrastructure. The field of job re-design is most advanced in providing detailed analysis of the employees’ situation; for instance a method called KOMPASS is provided [32]. However, applying this method requires a lot of time and needs the involvement of competent evaluation experts in the field of industrial psychology. Thus, we see the need to propose a procedure that applies the eight heuristics in a way that a) allow novice evaluators who are – however – familiar with some issues in the practical field where socio-technical intertwinement takes place b) is efficient enough to be repeated from time to time to support an agile series of re-design cycles. A more detailed description of the socio-technical description and the underlying socio-technical understanding can be found in appendix A. 3. A questionnaire-based, two-stage evaluation process for socio-technical systems The main objective of this method is to enable organizations and teams to evaluate their work environments independently from external experts. The distinctive knowledge of their respective work environments should be applied by novice evaluators to identify problems and potentials for improvement. Experts in systems design are not able to advise every necessary change in process or technology for specific contexts. The interactive socio-technical questionnaire utilizes condensed conclusions from socio-technical literature to substitute the expertise from system design experts. Herrmann and Nierhoff [2] conducted extensive literature research in six research domains relevant for socio-technical systems. These findings were condensed into a set of eight socio-technical heuristics which aid evaluators in evaluating complex socio-technical systems. The broad scope of the heuristics can be used to identify issues directly or to uncover areas of interest, that were not considered as critical prior to the evaluation. These areas of interest contain several small issues, obstacles in workflows or other effects that cause discontent. These areas can be based on activity (i.e. communication, knowledge management, distribution of tasks, customer interaction), but also context (i.e. predictive maintenance, artificial intelligence, product development, etc.) or can be related to supportive technical infrastructure. Early research on the socio-technical heuristics shows a necessity to provide more tangible information for novice evaluators [33]. While exhaustive information can be drawn from the ST- heuristics if evaluators have extensive experience or domain knowledge, novices only utilize a small portion without specialized training. Bendel[34] utilized the socio-technical heuristics as a tool for reflection and elaboration in a digitization project. Participants used the socio-technical heuristics and derived guided questions for the discussion of conceptualized socio-technical processes and developing improvements. Table 1 Socio-technical heuristics (Herrmann et al. [1]) 1. visibility about task handling and feedback about its success; 2. flexibility for variable task handling leading to a participatory evolution of the system; 3. communication support for task handling and social interaction; 4. Purpose-orientated information exchange for facilitating mental work; 5. balance of effort and experienced benefit by organizational structuring of tasks; 6. compatibility between requirements, development of human competencies, and the system’s features; 7. efficiency-oriented allocation of tasks for pursuing holistic goals; and 8. supportive technology and resources for productive and flawless work Note: The underlined keywords are used to identify the heuristics in what follows 160 The approach presented in this paper extends the socio-technical heuristics into an interactive questionnaire to identify and investigate problems in socio-technical systems. We propose a two-stage evaluation process for the identification and analysis of possible issues. This process divides the questionnaire into four interview sessions, each investigating 2 of the 8 socio-technical heuristics (see Table 1). As shown in Figure 1, each session contains a preliminary interview phase, a questionnaire to identify problems, passages of guided in-depth interviews and a follow-up discussion. These segments are described further in the following sections. Figure 1: Evaluation process for each interviewee (for the modelling notation see [35]) Structure of questionnaire. One of the challenges of the socio-technical approach is the selection of critical areas for closer evaluation. One solution to focus within the evaluation is to address a specific process or workflow. This approach is especially reasonable if a process is already identified as suboptimal or critical for the organization in general. Problems on the intersection of processes or inherently independent of specific processes are difficult to identify. In broad explorative evaluations, evaluators have to pay special attention to participants to pick up small clues indicating large issues. It is hard to distinguish between individual notions and deep-rooted problems that are grudgingly accepted and not questioned[36], [37]. Another challenge is the identification of root causes for identified issues. Issues are often identified by their symptoms. Understanding the underlying causes and the context of issues is crucial to their resolution. Exploring and documenting issues separately from the initial recording is often difficult. Establishing the context of the issue or even recalling the thought process that led to its identification can be problematic if the documentation is not substantial [28]. The developed questionnaire combines the identification of problems and concerns with the identification and investigation of the root issue. We advanced the set of socio-technical heuristics by transforming the condensed subject matter into a detailed questionnaire. Each heuristic was split into a series of single issues. Explicit and implicit interconnections between several aspects were also added as single issues. This expanded the set of evaluation aspects extensively. The extent of the generated collection of issues is a trade-off for making the condensed and partly implicit information contained in the heuristics more accessible. While the socio-technical heuristics each create an extensive multidimensional scope for evaluation, the socio- technical questionnaire is intended as a checklist that guides the evaluation of complex socio-technical systems. In order to construct a specific questionnaire, we analyzed the content of each of the socio-technical heuristics to gather explicit and implicit aspects of considerations on socio-technical systems. Two people formulated a list of specific issues for each heuristic individually. These issues then were compared and discussed to find a holistic view of the heuristics and to formulate one aggregated list of 161 issues for each heuristic. We derived an accumulated list of 113 specific, easily understandable statements for the questionnaire. All statements are idealistic descriptions of the preferable state according to a well-designed socio-technical system. Interviewees are intended to compare their work environments to the idealistic statements and indicate to which degree the statement is accurate. The statements are designed to be applicable in most domains and work environments, while being precise enough to verify the socio-technical requirement. An excerpt of the interactive socio-technical questionnaire can be seen in Table 2. Additional information and the complete questionnaire can be found at [38]. For each statement we developed a guided in-depth interview passage to investigate if a disagreement concerning a certain statement within the heuristics hints to a problem within the socio- technical system. Each in-depth interview passage is specifically designed to explore the context of the situation regarding the statement. The interview passages have three major components: Exploration of the situation and context, identification of reasons for objection and suggestion for improvements. Depending on the scrutinized statement, multiple aspects are investigated by iterating these components several times or by a combined inquiry from different angles. The guided interviews are designed to investigate and distinguish between multiple causes for the objection to the ideal statement. Table 2 Excerpt from interactive socio-technical questionnaire Heuristic 3: Communication support for task handling and social interaction S2) Various options are available for conversations or sharing information (appropriate rooms, phone, video, text messaging, or other). Optional questions if selected for further elaboration: • Can you share more about this? • What places or rooms can you use for conversations? • Are the places appropriate for conversations? What is good or bad about the places? • What are your tech options for talking to someone or sharing information? • How well does technical communication work? What are the advantages and disadvantages? • When do you use technology to communicate with someone, when do you prefer to meet in person? Why? • What should change? Heuristic 7: Efficiency-oriented allocation of tasks for pursuing holistic goals S13) The technology and organization are getting increasingly better. You can work better and better by enhancing your work environment. Optional questions if selected for further elaboration: • Can you elaborate more on the background? • How is your work environment changing? • How is the technology you use evolving? When will the existing technology be replaced or upgraded? • Where do you see opportunities for improvement? • How has your work changed in the last year? • How has the organization of the work and your team changed? • How does the change in work match the change in the work environment? 162 • What suggestions do you have for what could improve? • How satisfied are you? What should change? During the development of the socio-technical questionnaire, the structure and integrity of the socio- technical heuristics (see Table 1) was maintained, e.g. with respect to a systemic covering of social and technical aspects. Each heuristic is represented by a list of between 9 and 23 statements (see Table 3). An example of two statements is shown in Table 2. To identify possible issues or important areas of interest for improvement, the statements are presented to interviewees. Participants are able to mark their agreement or disagreement whether their work environment matches the idealized statement on a seven-point Likert scale. This allows for a quick overview of concerns. If the interviewee disagrees with an idealized statement (i.e. numeric answer is below midpoint) it should be explored further. Not all participants feel comfortable to openly disagree and transpose their answer-spectrum to sound more positive. This can result in strong disagreement only being denoted in the middle of the scale. To account for this effect, answers below the mean value for the individual participant should also be selected for further exploration. Additionally, outliers within a heuristic indicate special circumstances regarding the theoretical aspect of the statement. Regardless whether the circumstances have positive or negative effects on the socio-technical systems, it is critical to identify the circumstances and their reasons for a comprehensive evaluation. Social tensions can be identified by observing significant deviations in answers inside a team or socio-technical system. These can also indicate differing expectations or perceptions, which could lead to problems in the future. For statements that are already identified as critical for underlying problems during previous interviews with other employees within the same team, different perspectives should be ascertained. Even if the individual participant does not indicate the respective statement as abnormal. If participants indicate special circumstances regarding a statement not by the value of their answer but by comment, the respective statement should also be considered for closer inspection of underlying issues. Analysis through passages of guided in-depth interviews. For each statement an individual collection of guided interview questions was created. To facilitate the later development of possible solutions, the context, scope and effects for identified issues have to be elicited. The wide selection of investigated issues requires a customized guided interview to query the required information with direct and specific questions. All guided interviews start with a request to explain the general situation leading to the disagreement with the respective statement. The following questions inquire background information for specific aspects of the statement. Utilized approaches are to request descriptions of typical micro-tasks or description of the reasoning process in the specific situation. These questions are designed for the participant to recall similar experienced situations and evaluate memorable instances instead of generalized abstract situations. One of the possible follow up questions for these scenarios is what obstacle or unmet requirements exist in these types of situations. Instead of focusing on one specific instance, these questions invite the participant to explore their experiences and recount multiple relevant instances. These deliberations allow a comprehensive understanding of the investigated concepts and their implementation in the socio-technical system. For some statements the sentiment towards a situation is one of the possible concerns. The guided interviews for these statements inquire about the personal attitude for this aspect of the specific situation or the socio-technical system in general. If the respective statement consists of multiple aspects, the described types of questions are repeated as necessary. All guided interviews end with a request for suggestions for improvement. This repeated request for suggestion and participation imprints itself on the participants and induces a thought process to think about possible improvements, even between interviews. Preliminary talk and follow up discussion. The main instrument for conducting the evaluation is the socio-technical questionnaire. To fully utilize its potential, we implement preliminary talks and follow up discussions directly before and after the main interview respectively. The preliminary talk allows the participants to acclimate to the interview. Participants can present their work environment from their perspective, while the interviewer gathers important information about the perceived social structures, procedures and individual tasks about the participant. This information is critical to probe 163 specific aspects during the main interview and as examples for inquiry if the participants are unsure how to answer questions in the questionnaire. A joint (verbal) exploration of the personal work environment also conveys the participant-centered approach - participants are not an accessory to the evaluation, they are the main actors. During the preliminary talk already known issues are mentioned by the participant and should be revisited during the questionnaire or the follow-up discussion. The follow up discussion of the interview presents an opportunity to reflect on the socio-technical system outside the guided questionnaire. Issues, suggestions and comments, that did not fit within the questionnaire are added. This includes issues that appear in the socio-technical system but are not subject of the current interview session or are additional issues occurring in situations that were already discussed. Additionally, during the follow up discussion as well as the preliminary talk participants can share reflections on their work environments they had between interview sessions. Iterative testing and improvement. We tested the approach of using statements and guided interviews in the two-stage evaluation process of the questionnaire in a pretest with an IT-administration team within our department. This enabled a closer look on the method within the questionnaire, since the socio-technical system is already prominently known and gaps or misrepresentations could be identified. Each interview was conducted using the statements of the questionnaire from two heuristics and was coupled with the preliminary talk and a follow up discussion. While the guided interviews were useful in analyzing the individual employees' situation and to understand the socio-technical system, interviewees indicated that some of the statements within the questionnaire were hard to assess. We used the feedback and our observations from the interviews to iteratively improve the questionnaire. A prominent challenge was to query complex and interrelated socio-technical aspects in simple terms that are easily and correctly understood by laymen. Statements were broken up to reflect different facets of the same socio-technical issue. The resulting statements and questions were more comprehensible and precise. Additionally, for each heuristic specific questions for the preliminary talk and follow up discussion were developed to explore the context necessary to investigate the socio-technical system. While the pretest was successful in finding problems and shortcomings within the analyzed socio- technical system and refining the interactive questionnaire, the method needed further application to evaluate its usefulness for novice evaluators and its application on other socio-technical systems. 4. Six case studies testing the questionnaire-based, two-stage evaluation process To test our evaluation process, we asked teams of students to apply the method to various work teams in industry. This task was part of the course “design of socio-technical information systems” to gather insights on the application by novice evaluators in real-world scenarios. All participants chose this elective as part of their Master of Science degree in Applied Computer Science. The students were familiar with software development and software engineering beforehand. Some of the participants had taken an elective on “groupware and knowledge management”. During the lectures, students received an introduction on the history of socio-technical systems and various concepts for their evaluation. The socio-technical heuristics and the interactive socio-technical questionnaire were presented and discussed. As part of their practical training, they were asked to evaluate socio-technical systems using the interactive socio-technical questionnaire. Groups of two to four students chose different socio-technical systems to evaluate. Each team of students evaluated a selected team inside a larger organization which had to handle a shared task or process. These processes were modelled prior to the evaluation. The evaluation consisted of four interviews per interviewed employee. Topics for each interview were selected as shown in table 3. Every student had to conduct at least four interviews which each consisted of the preliminary talk, discussion of the questionnaire for two heuristics, and the follow up discussion (also see Figure 1). All interviews were conducted within a span of 8 weeks. The student groups had to write a final report on the socio-technical system they evaluated. The reports contained the procedure of the evaluation, as well as the problems found with the questionnaire. In addition, the students developed proposals for solutions to the problems and prepared a reflection on the project. 164 Table 3 Combination of socio-technical heuristics for interviews of every case with number of statements per heuristic Interview 1 Interview 2 Interview 3 Interview 4 STH1 Visibility STH2 Flexibility (9) STH5 Balance (18) STH6 Compatibility &Feedback (13) (19) STH4 Information STH3 Communication STH8 Supportive STH7 Efficiency (13) exchange (11) (9) Technology (21) We analyzed six reports to find out how far the approach of using the interactive questionnaire within the two-stage evaluation process supports the identification of problems within socio-technical systems. The qualitative analysis was done with an exploratory approach coding with MAXQDA. The focus of the analysis lied on the problems instead of the suggested solutions by the students to evaluate if the students as novice evaluators can detect socio-technical problems. We coded the evaluated problems based on their content, which heuristic they were assigned to, and with which heuristic they were found in. Additionally, we tried to cluster the problems based on their content to find themes within all reports. The six analyzed socio-technical systems can be described as follows: 1. Web development: A team of software developers who worked within a web development company. The main tasks of the team include the development process of the internal products as well as project- and product management and customer care. Here, potential tools supporting the developing activities as well as the project coordination represent the technical dimension while the conventions for managing the project as well as the accompanying communication processes are part of the social aspect. The relevance of both sides as well as their interplay are significant for what we consider a socio-technical system or process. 2. IT service provider: a software development team at an IT service provider developing software for master data management for the chamber of commerce and industry. Especially during the shift to remote work due to the Covid-19 pandemic, the team relies on different tools and software for the development process. Whereas the coordination within the team members (via Scrum) as well as customer support during the development processes are organizational challenges. 3. Car part manufacturer: multiple employees with different roles and responsibilities within the shopfloor management of a car part manufacturer. Shopfloor management is an organizational system for managing manufacturing processes, which is supported by technical solutions. This case study focusses on the coordination of different roles within the team and organization of the processes. 4. Public transportation company: a complaint management department of a public transportation company. The team works on virtual customer support, thus does not have any personal contact with customers. Organization within different team members is necessary to resolve these complaints because singular complaints are not allocated to singular team members. The organization and exchange of complaints and their management are supported by an online complaint management system and additional technical solutions. 5. University Examination office: An administrative team within a University Examination Office. The intertwined socio-technical components include a software for listing courses, documenting grades and providing records as well as regulations for examinations, required information exchange, and well-established conventions for handling examinations. The interviewees included students, lecturers and the examination office to analyze all roles involved within the system. 6. Rail transport solutions: Employees of the HSSEQ, Sale, Customer Service and IT departments within a company that provides rail transport solutions for freight and passenger transportation. The focus within this social-technical system was on the software Zedas which is the main operating system for transport and logistics management. 165 Additionally, consultations and organization within the different teams using the software are examined. Table 4 gives an overview over the constellations in the different cases. Table 4 Characteristics of the cases Case- abbreviation number of number of number of Number of students interviewees interviews problems 1 Web development 2 9 20 24 2 IT service provider 4 5 20 16 3 Car part manufacturer 4 5 20 57 4 Public transportation 4 5 20 90 company 5 University Examination 3 16 16 23 office 6 Rail transport solutions 4 4 16 14 We chose to include various reports based on their quality and length to ensure a broader scope of application of the developed method. In total, nine groups participated in the course. Reports of three groups omitted information relevant to this analysis. To ensure the reliability, these three reports were not included in the following analysis. While the six reports differed in length and detail it is important to note that all teams of students were able to analyze the respective socio-technical system efficiently and applied the method properly. 5. Results Altogether the students identified 224 problems. The number of problems within each report ranged from 14 to 90 problems; the average number of problems per case is 37.66. Table 5 provides an overview on the number of problems in each report, as well as the number of problems categorized within each heuristic. It is important to note that many of the problems were associated with multiple heuristics as they concerned complex issues. This explains why the categorization within the heuristics exceeds the total number of problems within one case study. Table 5 Number of problems for each case study by heuristics Heuristic Indicator of case and number of problems 1 2 3 4 5 6 ∑ Rank STH1 Visibility &Feedback 5 3 8 9 9 3 37 5 STH2 Flexibility 5 1 6 4 2 2 20 7 STH3 Communication 1 3 6 4 3 2 19 8 STH4 Information exchange 0 2 3 17 6 2 30 6 STH5 Balance 5 4 15 21 6 1 52 3 STH6 Compatibility 2 7 10 27 4 3 53 2 STH7 Efficiency 7 6 13 16 9 5 56 1 STH8 Supportive Technology 0 5 20 12 6 4 47 4 Total 24 16 57 90 23 14 224 Within five out of the six case studies, problems covering all of the eight heuristics were identified. Case study 1 focused on only four heuristics (STH1, STH2, STH3 and STH7) within their analysis, due to their team only consisting of two students. Overall the method provided a range of identified problems within all of the applied heuristics. 166 While analyzing the reports of the case studies, we found that multiple problems across different case studies were similar. We decided to cluster the problems of all case studies based on their content. In the following, we list the 20 types of problems which we were able to identify and one example within each type to underline the content. Table 6 Number of problems of a certain type for each case Number of problems per case Type of problems Related heuristics 1 2 3 4 5 6 ∑ 1. Problems with customers, i.e. language barriers with 6 3 3 customers; 2. Problems of employees developing new skills, i.e. 5 1 1 2 monotone tasks in day-to-day-business; 3. Missing information from other departments, i.e. 4 1 3 2 6 limited knowledge of projects within other departments; 4. Processes not optimized, i.e. multiple rounds of 7 5 5 2 12 reporting results from tasks; 5. Problems with the personal motivation of employees, 5 1 11 1 13 i.e. limited proactive engagement; 6. Personal stress factors and burdens of employees, i.e. 5 2 1 4 8 2 17 overload of tasks; 7. Privacy concerns, i.e. insufficient reconnaissance in 4 2 1 1 4 the data collection of employees; 8. Feedback loops, i.e. feedback is not provided quickly, 1 1 6 2 1 10 directly or regularly; 9. Organization of tasks, i.e. limited flexibility in 2 4 1 5 10 executing tasks; 10. Disturbed communication and reachability, i.e. 3 3 7 12 5 27 limited reach of supervisors; 11. Technical problems, i.e. loading time of servers and 8 1 2 3 5 9 3 23 software; 12. Organizational problems, i.e. under staffing of 6 6 9 8 1 2 26 teams; 13. Use of multiple tools/ missing intertwinement of 4, 1 3 2 6 different applications, i.e. information is only saved locally; 6 14. Missing training and further education, i.e. missing 6 1 3 1 1 6 training when using new technological solutions; 15. Efficiency problems, i.e. inefficient execution of 7 1 1 2 4 tasks; 16. Time problems, i.e. spontaneous delegation of tasks 2, 1 1 3 5 which leads to overtime; 7 17. Knowledge management, i.e. missing upkeep of 4 2 7 8 2 19 databases and information on processes 18. Problems which arose during the Covid-19 - 9 1 2 5 2 19 pandemic, especially regarding a shift to mainly working from home 19. Individual conflicts, i.e. personal conflicts between - 1 2 2 5 employees; 20. Missing digitalization, i.e. some information is 8 1 2 3 6 3 15 collected on paper and later transferred into a database; Table 6 presents the number of problems within the 20 types for each case study. The second column proposes heuristics that can be assigned to the type of problems; question marks indicate that there is no heuristic that could be assigned evidently. Grey shadowed cells indicate the dominant type(s) of 167 problems per case. Case studies 3 and 4 identified problems within a broad range of themes, while the problems of case studies 1,2,5 and 6 were narrowly focused on specific themes. Within every case study one or two themes were prominently discussed as problems by the respective employees. Multiple smaller problems could be connected to one issue or were raised by multiple employees. In what follows we shortly characterize the problems found in each case: 1. The most prominent problems within the web development team (case study 1) concerned the shift to working from home due to the Covid-19 pandemic. The issues within home office work covered missing technical equipment, missing possibilities for informal exchange with other colleagues, the use of new software, additional time required for tasks and problem solving, missing information on availability of colleagues, maintaining a work-life balance and a disrupted work environment depending on personal circumstances at home. Multiple interviewees were concerned with the shift to working from home and described the situation as more exhausting and stressful. Overall, the team still thought that the switch to working from home was successful and were satisfied with the arrangement. 2. Within the software development team at the IT service provider (case study 2) the majority of the problems concerned organizational issues. This included the communication of decision making, descriptions of tasks, insufficient planning and implementation of company targets, a mismatch between technical language with the customers’ language. Furthermore, difficulties with planning and organizing work due to incompatible working methods within the company (the cycles of the agile software development team is not coordinated with classic project management teams) were observed. This mismatch leads to interferences with other teams within Scrum cycles that cause disruptions within sprints and a mismatch of delegation of tasks for individual employees. 3. The main problems within the shop floor management of the car part manufacturer (case study 3) were also of organizational nature. Employees raised the following problems: the understaffing of teams, corporate goals which were only attainable if the production line had no stops, a high noise level at the open plan offices, an overload of meetings as well as calls and reports, insufficient break rooms, missing practicability of top-down decisions, changes are implemented across a wide range of departments instead of gradually, missing specified training periods for new systems within the production line, and the purchase of cheaper equipment instead of durable equipment. Most of these problems were only raised by single employees. However, this can be explained by the explorative nature of the selection of interviewees. Within this case, one employee of each level of managerial oversight was interviewed. 4. The main problem types within the complaint management team of the public transportation company (case study 4) were twofold. The first main type concerned disturbed communication and reachability, and included inefficiency of information exchange, missing announcements of software updates and process changes, no informal exchange with team leaders and employees, no disclosure of important information, misunderstandings with other departments, supervisors who are not easily accessible when needed, service providers who cannot be reached on the phone, and missing information on whom to contact regarding specific issues. The other main category concerns problems of personal motivation which include team leaders who feel in need to check whether tasks have been completed, employees withholding information from each other, employees not helping their colleagues if a mistake occurs, employees disliking the digitalization and change processes and doing tasks inefficiently to slow down the process, employees not participating in decision making processes, management not being able to motivate employees, a reluctancy to learn new tasks and a general motivation problem of the team. 5. The main issues which occurred within the examination office (case study 5) were technical problems of interacting with the system that is used to register and organize exams and courses. These problems included a lack of being able to enter information into the software, incompatibility between the system’s structure and exceptional types of examination, out of date information on the system, incorrect automatic exam registrations of students, problems with the control mechanism when entering grades of students, problems with uploading exam grades into the software, an unclear user interface, missing interfaces of different programs 168 which lead to transferring data manually and a general overload of used programs within the processes. 6. Within case study 6 at the company providing rail transport solution no prominent type of problem could be identified, the most pressing problems included technical problems of and missing training for the main operating system, a fear of employees to discuss problems freely, monotone tasks and inefficient delegation of tasks, problems with the organization of meetings and low motivation of employees. Within the reports of the six case studies, we observed differences in quality and quantity of the problems. Some of the reports described the problems in more detail and connected different aspects which were raised by different employees to one problem, other reports did not inspect the problems in detail but focused on descriptions of the socio-technical system and proposals for solutions. One notable difference between the case studies was the selection of interview participants. Some case studies focused on a singular team working together and interviewed multiple employees within one team (case studies 1, 2 and 4) while others interviewed one or multiple employees from different departments (case studies 5 and 6) or employees from multiple levels of management (case study 3). The case studies focusing on one team inspected more problems which were raised by multiple employees. The problems of the other cases were more individual as interviewees are doing different tasks within their respective roles. However, within the cooperation of different teams or employees, problems that interfere with multiple people or systems can be observed. As described above, similar problems were found in different case studies but overall, the socio-technical systems and problems within each case are individual. The depth and clarity of the described situations within the reports of the cases vary, as well as the quantity and quality of detected problems. This can have a multitude of causes: the willingness of the employees to discuss and disclose issues at work to the students, the experience of the students as interviewers, the care and time invested in the final report, the complexity of the socio-technical system as only small teams could be analyzed in the case studies or the suitability of the questions to inspect a statement within the questionnaire. With the small sample size of each case study the scope and results are only limited to the respective teams and cannot be applied to broader scopes of the companies. In general, although the interviews raised problems for the respective interviewees, in most cases the employees were satisfied with their work environments. Furthermore, we found that the problems that have been identified by the students can be differentiated with respect to three categories: 1. Social or organizational (s): Problems that do not refer to any technology being involved 2. Socio-technical (st): Problems that address the interplay between technology and social behavior or organizational settings 3. Technical (t): Problems that exclusively deal with technical issues. We take the following examples of problems from the case “Car Part manufacturer” to demonstrate the difference between the three categories: 1. S: At first, both the feedback emanating from superiors and from colleagues was criticized as irregular and insufficient. As a result, it was also often stated that one's own contribution and also the contribution of colleagues to the success of a project or manufacturing process is not clear at all. 2. ST: For example, data and information such as key figures or even problems are currently still recorded with pen and paper and only digitized later. Important information is also often printed out in the morning in order to have it available throughout the day. Employees therefore have to go to the office every time to get this information, or print it all out in the morning. 3. T: In the opinion of the employees, many technical solutions are undersized, which is associated with the fact that the company always weighs up "favorable price vs. durability" in favor of the favorable price. Table 7 presents the distribution of these 3 categories between the cases. The displayed numbers help to discuss how far the computer science students and their evaluation were oriented to problems that have not a pure technical background but refer to accompanying issues and to the intertwinement between technology and social practices. 169 Table 7 Differentiation between social (s), socio-technical (st) and technical (t) issues per case Category Cases s st t ∑ 1. Web development 17 71% 6 25% 1 4% 24 2. IT service provider 9 56% 6 38% 1 6% 16 3. Car part manufacturer 32 56% 22 39% 3 5% 57 4. Public transportation company 52 58% 31 34% 7 8% 90 5. University Examination office 2 9% 20 87% 1 4% 23 6. Rail transport solutions 5 36% 7 50% 2 14% 14 ∑ 117 52% 92 41% 15 7% 224 6. Discussion With an average of 37.66 per case and a minimum and maximum from 14 to 90, it becomes obvious that the methods helps to detect sociotechnical problems. In all investigated cases, the novice evaluators were able to identify relevant problems in the respective socio-technical systems. Table 5 reveals that all eight heuristics were relevant with respect to the detected problems. However, the number of assignments varies between a maximum of 56 (efficiency) and a minimum of 19 (communication support). This might be seen as an indicator for a varying relevance of the heuristics. However, the study [1] where the heuristics were developed applied them to a data base of 42 problems with a maximum of 21% of assignments to visibility and 7% to technical support; efficiency got the second lowest percentage of assignment. Therefore, we suggest that the variances between the number of assignments to heuristics does not depend on the general relevance of the heuristics but on the characteristics of the cases. As seen in table 4 the students identified problems corresponding to every investigated heuristic in all case studies. Only in one case study, no problems corresponding to heuristics 4 and 8 were identified. However, the questionnaires to identify problems in heuristics 4,5,6 and 8 were not applied in this case. The fact that problems related to heuristics 5 and 6 were identified despite not explicitly being investigated supports the goal of investigating a broad, generally applicable spectrum of socio-technical issues. Participants are guided to reflect on their work environment and the existence of hindrances or problems in regard to known socio-technical issues. The statistical and qualitative results suggest that not only the specifically prompted issues were reflected upon, but also situations that are connected to these issues (e.g. Covid-19 related problems). While problems identified in this manor predominantly correspond to different issues of the same heuristics, a significant amount are related to several different heuristics. This overlap of heuristics is intended as the heuristics, as well as the questionnaire, are designed to identify issues in complex socio-technical systems[2]. Both approaches focus on identifying existing issues from various socio-technical perspectives while not being highly selective between these perspectives. Furthermore, problems were not necessarily identified by individual questions, but by sequences of questions which in combination yielded specific problems. This effect was especially intensified by consolidation of answers from multiple participants. Connected issues were also identified by the sequence of statements and questions corresponding to the overarching socio-technical principle. With respect to our categorization of the problems, as documented with table 6, in all case studies at least eight types (case 2 and 6) of problems were identified which each consist of several distinct problems. These clusters of distinct problems associated with the same type of problem suggest the existence of a critical problem or inherent flaw of the socio-technical system that influences multiple workflows and therein produces subsequent problems. Case four addresses 19 of 20 types of problems. Five categorized types of problems are consistent over at least five of the six case studies. This suggest that the questionnaire enables the identification of problems regarding to stress, technical equipment, organization, missing digitization and also problems related to the specific situation caused by the Covid-19 pandemic reliably. Additionally, the questionnaire supports identification of problems 170 specific to the individual work environment. It is remarkable that the set of heuristics triggers the identification of problems that are not directly related to this set, such as the problems being assigned to the Covid-19 pandemic. While some groups identified numerous problems in every heuristic, other groups found a higher concentration of problems in selected heuristics and only few problems in others. Additionally, the total number of problems varies widely between groups. As described in section 5 this can have multiple reasons ranging from number of evaluators to complexity of evaluated socio-technical system and level of dedication of participants and evaluators. While the method proved to be successful to understand and evaluate socio-technical systems and their shortcomings, the assigning of the shortcomings and problems to the heuristics was rather difficult for novice evaluators. The categorization of identified problems into problem-types and the attribution to related heuristics was not main focus of this project. It is, however, an often-required step for subsequent resolution of identified problems or similar improvement efforts to order and prioritize identified issues. As the categorization of problems provides a quick overview of the relevance of individual heuristics for the evaluated socio-technical system it is a useful information for subsequent evaluations or reviews. For the identification of problem areas and critical issues a categorization of problem topics for the individual socio-technical system proved useful. This overview supports the identification of cohesion and interconnection of identified problems regardless of the theoretical classification. The analysis of the content of the problems reveals that in every case a series of problems were found that hinder the handling of tasks such as insufficient • Technical equipment, • Training and preparation of employees • Communication support and information exchange • Motivation We conclude that these problems are highly relevant and oppose an efficient task handling (as indicated by the high number of efficiency related problems, see table 5). Thus, we suggest that the detected problems are relevant with respect to task handling and eventually with respect to the employees’ job-satisfaction (as indicated by the high number of problems assigned to the balance between effort and benefits, see table 5). With respect to the three categories social/organizational vs. socio-technical vs. technical, only in the cases of the University Examination Office and the Rail Transport Solutions, the st-problems dominate (see Table 7). Actually, we expected that the questionnaire instrument supports a stronger focusing on socio-technical issues. By contrast, the numbers of table 7 reveal that the social/organizational category found the strongest consideration. Therefore, we can conclude that the evaluation method helped the computer science students to shift their attention from the technical to the organizational issues. However, the influence of technology and its intertwinement with social or organizational problems is not sufficiently investigated. This might have been caused by the way of how the interviewees framed the problems they have experienced. It can be seen as a particular challenge to investigate whether the causes of the problems are predominantly attributed to human behavior or can also be seen in relation to a lack of technical support. The interactive questionnaire was designed to support the interviewers with guidance for socio-technical aspects outside of their (primary) expertise. This attempted balance might have resulted in an overemphasis on social perspectives. Only in the case of the University Examination Office, the socio-technical aspects dominate exceedingly since most of the employees’ tasks and problems are directly related to the usage of an exam- management-system that the interviewers chose as a main focus. Consequently, we assume that the interviewers should be oriented more explicitly on the challenge of investigating whether a problem can be related to a lack of technical support. For instance, the problem that we gave as an example for a mainly organizational problem could have been accompanied by further questions by the interviewers. With these questions they could have tried to understand whether the tasks for which regular feedback was solicited were supported by an information system that also covers reporting to the management and whether a function might be missed or reasonable that reminds the mangers to give feedback. Thus, we conclude that the evaluation instrument includes hints of how to go deeper into the consideration of socio-technical intertwinement. 171 The counter-intuitive approach to investigate intertwined socio-technical issues in complex work- environments by zooming in to elementary aspects proved effective. The structure of the interactive questionnaire allowed participants and interviewers to divide the complex environments and to identify specific situations and problem-areas. These situations and problems were then closely examined using guided interviews. While the discussed elementary categorization of issues is especially useful for development and evaluation of the questionnaire itself, it is only of limited use for the improvement of the socio-technical work environment. However, the categorization of issues to heuristics and problems indicate a comprehensive evaluation of identified situations and problem-areas (see Table 5). Issues were identified according to the individual requirements and needs of the participants (see Table 6), even if these requirements were not specifically included in the questionnaire (e.g. problems related to COVID-19). The structure of the interactive questionnaire was designed to enable unrestricted exploration of the work environment while leaving the execution of the exploration to the cooperation between interviewee and interviewer. This exploration is guided by prompts for investigation of specific socio-technical aspects to facilitate a comprehensive evaluation and to balance different levels of expertise. Repeated identification of problems with identical underlying issues in separate sessions and heuristics indicates examination not on the elementary level but rather a broad and deliberate investigation. While the case studies show an over-emphasis on social perspectives, the approach to divide socio- technical system and socio-technical perspectives and to combine them on an elementary level was successful. This divide and conquer approach enabled the support of novice evaluators to perform comprehensive investigations of complex environments and to identify specific issues and their effects on the work environment. However, this study did not investigate how this level of guidance and support is effective or obstructive for intermediate and expert evaluators. Further research is needed to determine how different expertise and experience levels effect evaluation with the interactive socio- technical questionnaire. 7. Conclusion The research questions can be shortly answered as follows: R1: A broad scope of different, relevant problems (20 types) were found with a means of 37,33 problems per case. With respect to efficiency of our method, 112 interviews were needed to find 224 problems; by average 2 unique problems were found per interview. Issues of socio-technical intertwinement covered 41% of the problems found. R2: The cases differ with respect to the focus of problem categories and with respect variety of problem types they address. Issues of socio-technical intertwinement dominated only 2 of the 6 cases. R3: The novice evaluators were able to shift the view from their technical educational background to organizational issues. Socio-technical intertwinement was addressed but not as dominantly as being intended by the set of heuristics. Limitations: By contrast to many other studies with heuristic-based evaluation [29], [39], our method does not refer to the severity of the problems found. We suppose that it is hard for novice evaluators to make valid assumptions about the impact of the problems found. Further studies could check whether it is helpful to ask the interviewees for a final ranking of the problems found in the environment. Furthermore, the evaluation was mainly focused on its effectiveness and less on its efficiency and on aspects for its improvement. As part of a student project, the novice evaluators had to focus on several challenges at once. While their experience in evaluating socio-technical systems is comparable with novice evaluators in work scenarios, the students also had to focus on different academic courses that were not related to this project. Therefore, an extensive familiarity with the evaluated sociotechnical system could not be reached. In the investigated teams, not all team members were interviewed. Further research has to investigate if a more comprehensive investigation enhances the derived problem collection in a substantial and uniform way. Additionally, a more detailed evaluation could investigate how the count of identified critical and shallow problems scale with the number of interviewed participants. 172 In further research we aim to integrate collaboration support to facilitate information exchange between evaluators and foster a shared and more complete understanding of the evaluated system. As every participant has their own perspective and perceived experiences, a collective understanding between evaluators can support the investigation of identified problems and their relevant contexts. This extends to the analysis of problems and their interconnections. In small groups evaluators are able to exchange relevant information and preliminary interpretations quickly. This helps to investigate identified problems from the perspective of other participants. In larger groups or constraining collaboration environments this proved to be challenging. While the presented research focused on the identification of problems, further research on the support of documentation and resolution of identified problems is necessary. Special attention must be paid to the information necessary to resolve socio-technical problems. It is not yet determined how far the explorative questions of the interactive socio-technical questionnaire include all aspects necessary for comprehensive understanding and resolution of problems and how this can be supported further. 8. References [1] T. Herrmann, I. Jahnke, and A. Nolte, “A problem-based approach to the advancement of heuristics for socio-technical evaluation,” Behav. Inf. Technol., pp. 1–23, 2022, doi: 10.1080/0144929X.2021.1972157. [2] T. Herrmann and J. Nierhoff, “Heuristiken zur Evaluation digitalisierter Arbeit bei Industrie-4.0 und KI-basierten Systemen aus soziotechnischer Perspektive,” FGW, vol. Digitalisierung von Arbeit 16, Feb. 2019, [Online]. Available: http://www.fgw-nrw.de/studien/industrie16.html [3] E. Trist and H. Murray, “Introduction to Vol. II.,” in The Social Engagement of Social Science. A Tavistock Anthology, Philadelphia: University of Pensylvania Press, 1993, pp. 36–60. Accessed: Oct. 12, 2020. [Online]. Available: https://www.upenn.edu/pennpress/book/1928.html [4] E. Trist, “The relations of social and technical systems in coal-mining,” Br. Psychol. Soc. Ind. Sect., 1950. [5] F. E. Emery, E. Thorsrud, and E. Thorsrud, Form and Content in Industrial Democracy : Some experiences from Norway and other European countries. Routledge, 2013. doi: 10.4324/9781315013695. [6] E. Mumford, “A Socio-Technical Approach to Systems Design,” Requir. Eng., vol. 5, no. 2, pp. 125–133, Sep. 2000, doi: 10.1007/PL00010345. [7] D. Shin, “A living lab as socio-technical ecosystem: Evaluating the Korean living lab of internet of things,” Gov. Inf. Q., vol. 36, no. 2, pp. 264–275, Apr. 2019, doi: 10.1016/j.giq.2018.08.001. [8] M. S. Ackermann, S. P. Goggins, T. Herrmann, M. Prilla, and C. Stary, Designing Healthcare That Works – A Socio-technical Approach. United Kingdom, United States: Academic Press, 2017. [9] D. Vinck, “Maintenance and Repair Work,” Eng. Stud., vol. 11, no. 2, pp. 153–167, May 2019, doi: 10.1080/19378629.2019.1655566. [10] J. H. E. Andriessen, Working with Groupware. London: Springer London, 2003. doi: 10.1007/978-1-4471-0067-6. [11] M. J. Muller and A. McClard, “Validating an extension to participatory heuristic evaluation: quality of work and quality of work life,” in Conference Companion on Human Factors in Computing Systems, New York, NY, USA, May 1995, pp. 115–116. doi: 10.1145/223355.223457. [12] J. Wutzkowsky and B. Böckmann, “Using MEESTAR to Identify Ethical and Social Issues Implementing a Digital Patient-Centered Care Platform.,” in eHealth, 2018, pp. 278–285. [13] A. A. Imanghaliyeva, P. Thompson, P. Salmon, and N. A. Stanton, “A synthesis of sociotechnical principles for system design,” in International Conference on Applied Human Factors and Ergonomics, 2020, pp. 665–676. [14] D. Shin and M. Ibahrine, “The socio-technical assemblages of blockchain system: how blockchains are framed and how the framing reflects societal contexts,” Digit. Policy Regul. Gov., vol. 22, no. 3, pp. 245–263, Aug. 2020, doi: 10.1108/DPRG-11-2019-0095. 173 [15] D.-H. Shin, H. Choo, and K. Beom, “Socio-Technical Dynamics in the Development of Next Generation Mobile Network: Translation Beyond 3G,” Technol. Forecast. Soc. Change, vol. 78, no. 3, pp. 514–525, Mar. 2011, doi: 10.1016/j.techfore.2010.04.017. [16] P. M. Leonardi, “Materiality, sociomateriality, and socio-technical systems: What do these terms mean? How are they different? Do we need them,” Mater. Organ. Soc. Interact. Technol. World, vol. 25, 2012. [17] W. J. Orlikowski, “Sociomaterial practices: Exploring technology at work,” Organ. Stud., vol. 28, no. 9, pp. 1435–1448, 2007. [18] L. Suchman, “Agencies in technology design: Feminist reconfigurations,” in workshop on Gendered Innovations in Science and Engineering, 2005, pp. 15–1. [19] R. G. Booth, M.-A. Andrusyszyn, C. Iwasiw, L. Donelle, and D. Compeau, “Actor-Network Theory as a sociotechnical lens to explore the relationship of nurses and technology in practice: methodological considerations for nursing research,” Nurs. Inq., vol. 23, no. 2, pp. 109–120, 2016. [20] Law, John, “Actor network theory and material semiotics,” Soc. Theory, p. 141, 2009. [21] B. Krenn, “Robot: Multiuse Tool and Ethical Agent,” in A Construction Manual for Robots’ Ethical Systems: Requirements, Methods, Implementations, R. Trappl, Ed. Cham: Springer International Publishing, 2015, pp. 11–29. doi: 10.1007/978-3-319-21548-8_2. [22] J. Nelles et al., “Best-Practice Approach for a Solution-Oriented Technology Assessment: Ethical, Legal, and Social Issues in the Context of Human-Robot Collaboration,” Berlin: Springer, 2017, pp. 1–14. doi: 10.1007/978-3-662-53305-5_1. [23] K. Behrenbruch, O. Kieselmann, M. Schuldt, M. Söllner, and L. Schmidt, “System EvaluationEvaluation,” in Socio-technical Design of Ubiquitous Computing Systems, K. David, K. Geihs, J. M. Leimeister, A. Roßnagel, L. Schmidt, G. Stumme, and A. Wacker, Eds. Cham: Springer International Publishing, 2014, pp. 297–319. doi: 10.1007/978-3-319-05044-7_17. [24] S. Bødker, “Third-wave HCI, 10 years later—participation and sharing,” interactions, vol. 22, no. 5, pp. 24–31, 2015. [25] E. Mumford, Designing human systems for new technology: the ETHICS method. Manchester: Manchester Business School, 1983. [26] F. Kensing, J. Simonsen, and K. Bodker, “MUST: A Method for Participatory Design,” Human– Computer Interact., vol. 13, no. 2, pp. 167–198, Jun. 1998, doi: 10.1207/s15327051hci1302_3. [27] Interaction Design Foundation, Heuristics and heuristic evaluation. 2021. Accessed: May 19, 2021. [Online]. Available: https://www.interaction-design.org/literature/book/the-glossary-of- human-computer-interaction/heuristics-and-heuristic-evaluation [28] C. Wilson, User interface inspection methods: a user-centered design method. Newnes, 2013. [29] J. Nielsen, “Enhancing the explanatory power of usability heuristics,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, Apr. 1994, pp. 152–158. doi: 10.1145/191666.191729. [30] DIN, ISO 9241-110: 2006–Ergonomics of human-system interaction–Part 110: Dialogue principles. 2006. [Online]. Available: https://www.iso.org/standard/38009.html [31] T. Herrmann, I. Jahnke, and A. Nolte, “Evaluating Socio-Technical Systems with Heuristics – a Feasible Approach?,” in Proceedings of STPIS16, http://ceur-ws.org/Vol-1604/, Sep. 2016, pp. 91–97. [32] G. Grote, C. Ryser, T. Wāler, A. Windischer, and S. Weik, “KOMPASS: a method for complementary function allocation in automated work systems,” Int. J. Hum.-Comput. Stud., vol. 52, no. 2, pp. 267–287, Feb. 2000, doi: 10.1006/ijhc.1999.0289. [33] F. Thewes, T. Herrmann, and A. Kluge, “Validating a Heuristic Evaluation Method - An Application Test,” in Proceedings of Mensch Und Computer 2019, New York, NY, USA, 2019, pp. 593–597. doi: 10.1145/3340764.3344465. [34] A. Bendel, “Arbeits- und prozessorientierte Digitalisierung in Industrieunternehmen: Über die Anwendung eines interventionsorientierten und soziotechnischen Forschungs- und Gestaltungsansatzes,” 2021, pp. 247–276. doi: 10.5771/9783748923046-247. [35] T. Herrmann, “SeeMe in a nutshell.,” Available Online, 2006, [Online]. Available: http://www.imtm-iaw.ruhr-uni-bochum.de/wp-content/uploads/sites/5/2011/09/Seeme-in-a- nutshell.pdf 174 [36] B. G. Freimut, “Developing and using defect classification schemes,” Fraunhofer IESE IESE- Rep., vol. No 72, Sep. 2001. [37] M. Leszak, D. E. Perry, and D. Stoll, “Classification and evaluation of defects in a project retrospective,” J. Syst. Softw., vol. 61, no. 3, pp. 173–187, Apr. 2002, doi: 10.1016/S0164- 1212(01)00146-7. [38] F. Thewes, M. Konrad, and T. Herrmann, “Socio-technical interactive questionnaire.” https://hi4.iaw.ruhr-uni-bochum.de/#!/questionnaire [39] R. Langevin, R. J. Lordon, T. Avrahami, B. R. Cowan, T. Hirsch, and G. Hsieh, “Heuristic Evaluation of Conversational Agents,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama Japan, May 2021, pp. 1–15. doi: 10.1145/3411764.3445312. [40] D. Shin, “A socio-technical framework for Internet-of-Things design: A human-centered design for the Internet of Things,” Telemat. Inform., vol. 31, no. 4, pp. 519–531, Nov. 2014, doi: 10.1016/j.tele.2014.02.003. [41] T. Herrmann, M. S. Ackermann, S. P. Goggins, C. Stary, and M. Prilla, “Designing Health Care That Works – Socio-technical Conclusions.,” in Designing Healthcare That Works. A Socio- technical Approach., United Kingdom, United States: Academic Press, 2017, pp. 187–203. [42] S. Amir and V. Kant, “Sociotechnical resilience: A preliminary concept,” Risk Anal., vol. 38, no. 1, pp. 8–16, 2018. [43] N. Luhmann and E. M. Knodt, Social Systems, 1st Edition. Stanford, Calif: Stanford University Press, 1996. [44] G. Fischer and T. Herrmann, “Meta-design: Transforming and enriching the design and use of socio-technical systems,” in Designing Socially Embedded Technologies in the Real-World, Springer London, 2015, pp. 79–109. [45] J. M. Carroll and R. L. Campbell, “Artifacts as psychological theories: The case of human- computer interaction,” Behav. Inf. Technol., vol. 8, no. 4, pp. 247–256, 1989. [46] C. Kirsch, P. Troxler, and E. Ulich, “Integration of people, technology and organization: the european approach,” in Symbiosis of Human and Artifact, vol. 20, Y. Anzai, K. Ogawa, and H. Mori, Eds. Elsevier, 1995, pp. 957–961. doi: 10.1016/S0921-2647(06)80337-0. [47] R. Kling, “What Is Social Informatics and Why Does It Matter?,” Inf. Soc., vol. 23, no. 4, pp. 205–220, Jun. 2007, doi: 10.1080/01972240701441556. A. Clarification of the socio-technical understanding Quotes from: Thomas Herrmann, Isa Jahnke & Alexander Nolte (2021)[1]: A problem-based approach to the advancement of heuristics for socio-technical evaluation, Behaviour & Information Technology, DOI: 10.1080/0144929X.2021.1972157 Socio-technical background “With the increasing emergence and adoption of applications, such as electronic health care, e-commerce, social networks, ubiquitous computing, and smart factories, the socio-technical perspective has gained more and more relevance. We characterise these emerging phenomena with an increasing complexity and contingency of intertwined social practices and technical artifacts. They cannot be fully analysed or modelled; once you start describing them, and before you come to an end, the intertwinement has already evolved and changed. This complexity arises from the different nature of multiple elements that interact with each other [40]. The dynamics of socio-technical intertwinement undergo a continuous evolution in a complex setting that is characterised by dynamic changes, uncertainty, and ambiguity [41].” (Herrmann et al. 2021, p. 1) “On the theoretical side, the socio-technical view is also present in approaches such as the concept of socio-technical resilience [42] or the discourse on sociomateriality [16]. Related to the latter, Leonardi differentiates between the social subsystem (including roles, hierarchies, communication networks, and others) and the technical subsystem that he characterises as imbrication of human (social) agency and material agency. We suggest that intertwinement should refer to the interplay between the social and technical sub-system as well as to the imbrication of social and material agency. Scholars referring to 175 Orlikowski [17] use the term entanglement for the intertwinement of the social and the material. We however perceive the concept of entanglement to not necessarily constitute a symmetry between the social and the material though, as proposed in the Actor-Network- Theory [19], [20]. Communities such as CSCW and HCI perceive social practices and technical artifacts as inevitably intertwined. Once the technology is launched and integrated in a certain context (e.g. organisation, communities, social practice), the social practices and the technical artifacts merge into a form of a system in which the two parts cannot be separated anymore; the one affects the other and vice versa. This intertwinement varies from one system to another– not all systems have the same intertwinement of social practices and technical artifacts. We suggest that these differences in the intertwinement of social practices and technical artifacts should be a subject of inspection and potential improvement. From our theoretical point of view, the intertwinement combines well-structured with less structured or formal with informal phenomena. The variety of these combinations leads to increased contingency of socio-technical systems in the sense of Luhmann [43], which is characterized by variability, particularity, mutability and uncertainty, and serves as a basis for continuous evolution [44].” (Herrmann et al. 2021, p. 3) “Figure 1 illustrates how we perceive the socio-technical intertwinement. Human–computer interaction plays a central role in this intertwinement of social and organisational practices with technical artifacts. From this perspective, socio-technical intertwinement can be described as intertwined organisational practices of task-handling in an ecology of tasks [45], [46] and the usage of technical artifacts or infrastructure. Such organisational practices are part of social practices that inevitably require human communication [43] between various roles [47] and include both formal and informal tasks [41]” (Herrmann et al. 2021, p. 3) Figure 1. Socio-technical intertwinement of social practice and technical artifacts through human- computer interaction. (From Hermann et al. 2021, p.4) Socio-technical Heuristics: “In order to prepare our literature search and the integrated set of new heuristics (see Section 4.1), we identified various relevant domains or disciplines that already have published sets of categories, criteria, principles, or guidelines. We then used these existing sets and synthesized them into a new set of socio- technical heuristics.” (Herrmann et al. 2021, p. 4-5) “We aimed to focus on the most influential research work in the domains. In the literature review, we first searched for already existent principles (Section 2.4.1-2.4.6) that we then used later in our empirical study. It is important to note that prior work does not always utilize the term heuristics. Instead, one will find terms such as principles, categories, design guidelines, or golden rules. They all refer to the same idea of heuristics in that they provide strategies to make decisions on how to improve the system where rational choices are possible [27]. In this study, we use them as synonyms for heuristics. All the heuristics that we refer to in the following sub-sections might serve a variety of purposes while we mainly consider them with respect to their potential contribution to socio-technical evaluation.” (Herrmann et al. 2021, p. 5) “We apply the term, heuristic, as it has been influenced by cognitive psychology. ‘Heuristics are rules of thumb for reasoning, a simplification, or educated guess that reduces or limits the search for solutions in domains that are difficult and poorly understood’ [27].” (Herrmann et al. 2021, p. 4) 176 # ST-Heuristics with short description 1 Visibility about task handling and feedback about its success Focused information is continuously offered about the progress of technical processes and, as far as permitted, about collaborative workflows. This helps to understand what further steps are possible or not and why as well as how well the expectations of others are met. 2 Flexibility for variable task handling leading to a participatory evolution of the system One can vary options of task handling and can flexibly decide about technology usage, time management, sharing of tasks, etc. Consequently, one can develop a wide range of competencies that support participation in the ongoing evolution of the whole system. 3 Communication support for task handling and social interaction By means of technical and spatial support for communication, a person can be reached to an influenceable extent for purposes of task handling and coordination This support is intertwined with negotiating duties and rights of roles, including values, so that reciprocal reliability can be developed. 4 Purpose-orientated information exchange for facilitating mental work To support task handling, information is purposefully exchanged via technical means, updated, kept available, and minimized. This implies technical linking of information and the emergence of personal profiles that must be visible and exchanged in compliance with privacy regulations. 5 Balance of effort and experienced benefit by organizational structuring of tasks Organizational structuring of tasks supports a proportional balance between individuals’ effort and experienced benefit. Tasks are assigned to people, pooled, and technically supported in a way that makes sense and is fun for people. Tasks comply with individuals’ technical, social, and physical competencies while also supporting health. Thus, a sustaining balance of efforts and personal benefits is pursued. 6 Compatibility between requirements, development of competencies, and the system’s features Technical and organizational features of the system are continuously adjusted to work with each other. Within clarified limits, they meet outside requirements in a way that is based on the development of competencies and proactive help for dealing with changing challenges. 7 Efficiency-oriented allocation of tasks for pursuing holistic goals By appropriate sequencing, integration, and distribution of tasks between humans and technology, seamless collaboration is supported. Unnecessary steps or wasting resources is avoided. If needed, an increase of efficiency can be realized. 8 Supportive technology and resources for productive and flawless work Technology and further resources support work and collaboration and consider the intertwining of criteria, such as technology acceptance, usability and accessibility for different users, avoiding consequences of mistakes and of misuse, security, and constant updating. Note. A more explicit description with examples is available for each heuristic (https://hi4.iaw.ruhr-uni-bochum.de/#!/manual) (From Herrmann et al. 2021, p. 15) 177