=Paper=
{{Paper
|id=Vol-3016/paper14
|storemode=property
|title=Heuristic-based evaluation for socio-technical systems
|pdfUrl=https://ceur-ws.org/Vol-3016/paper14.pdf
|volume=Vol-3016
|authors=Felix Thewes,Thomas Herrmann,Marina Konrad
|dblpUrl=https://dblp.org/rec/conf/stpis/ThewesHK21
}}
==Heuristic-based evaluation for socio-technical systems==
https://ceur-ws.org/Vol-3016/paper14.pdf
Heuristic-based Evaluation for Socio-technical Systems
Felix Thewes1, Thomas Herrmann 1 and Marina Konrad1
1
Ruhr-University Bochum, Universitätsstr. 150, Bochum, 44780, Germany
Abstract
This paper reports on the development of a socio-technical evaluation method based on an
interactive questionnaire and its application in six case studies. Expanding on the socio-
technical heuristics by Herrmann et al. [1] the mixed-method interview questionnaire was
developed for comprehensive evaluation of the intertwinement between social and
organizational practices with technical artifacts and their usage. The interactive questionnaire
enables organizations to perform evaluations without employing evaluation experts or the need
for extensive training. A semi-structured participative identification of problems allows for a
broad examination of socio-technical principles, which then can be assessed with
corresponding guided interview passages. This approach was applied by students in diverse
companies as project within a course on “design of socio-technical information systems”. The
results of six evaluated teams of employees demonstrate the wide range of identified issues as
well as the similarities between these companies.
Keywords 1
Socio-technical system, socio-technical evaluation, work systems, mixed-methods interview,
interactive questionnaire
1. Introduction
The field of socio-technical systems design provides criteria for designing and re-designing work
environments. Methods for evaluating socio-technical systems are either time-consuming or rely
heavily on expert knowledge. Existing methods often investigate a specific issue or socio-technical
themes. To examine socio-technical systems broadly, without prior knowledge of the existing issues, a
method was needed to identify issues and problematic areas for further investigation. Herrmann and
Nierhoff [2] combined methods from six different approaches presented in socio-technical literature to
compile socio-technical heuristics. The challenge presented in this paper is the expansion on these
heuristics by transforming the condensed content into an interactive questionnaire. Frist hand
experience of the evaluated socio-technical system is utilized through interviews while evaluators are
guided to investigate the system holistically. This is intended to lower the barrier of entry for
consideration of complex socio-technical perspectives and open socio-technical evaluation to a wider
range of applicants. Quick results are achieved by first identifying issues on a broad spectrum of aspects
and perspectives to narrow thorough evaluation to the revealed critical issues.
We present results from six case studies conducted by students. The socio-technical questionnaire
was applied by students as part of a computer science master’s course “design of socio-technical
information systems”. The results are used to investigate the following three research questions.
R1: How far does the proposed procedure of socio-technical, heuristic-based evaluation help to
detect relevant problems from the employees’ view - with respect to variety and quantity?
R2: Which differences can be observed between the various cases that have been inspected?
R3: How far are the novice evaluators able to address socio-technical intertwinement between
technical and organizational issues?
Proceedings of the 7th International Workshop on Socio-Technical Perspective in IS Development (STPIS’21) , October 11–12, 2021, Trento,
Italy
EMAIL: felix.thewes@rub.de; thomas.herrmann@rub.de
ORCID: 0000-0002-8436-8106 (F. Thewes); 0000-0002-9270-4501(T. Herrmann)
©️ 2021 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
158
2. Background
Our understanding of socio-technical systems focusses on the intertwinement of social and
organizational practices with the infrastructure of technical artifacts. Initially this understanding was
based on the observation that the development of organizations cannot be understood without
referencing the technology used [3], [4]. Later, scholars emphasized that organizations should not
predominantly be considered as work like technical systems [5]. Equal weight should be given to social
as well as technical issues [6].
The practical side of the socio-technical perspective pursues an improvement of technical systems
in areas such as ubiquitous computing[7], health care [8], maintenance repair [9], and others. Early
approaches to support and evaluate socio-technical design [10], [11] have been renewed, e.g. by
MEESTAR in the context of nursing [12], or by Imanghaliyeva et al. [13] who propose a synthesis of
socio-technical principles. Shin and colleagues argue that fairness, transparency, and accountability are
the most relevant issues to be satisfied in the context of socio-technical systems [14], [15].
On the theoretical side, the socio-technical view is also present, for example in the discourse on
sociomateriality [16], where social subsystems (including roles, hierarchies, communication networks,
and others) are distinguished from technical subsystems as imbrications of human (social) agency and
material agency are differentiated. Scholars referring to Orlikowski [17] use the term entanglement for
the intertwinement of the social and the material. In our view, the concept of entanglement does not
necessarily imply a symmetry [18] between the social and the material as proposed, by contrast to
Actor-Network-Theory [19], [20].
Once a technical artifact is integrated in a certain context (e.g., organization, communities, social
practice), the social practices and the technical artifacts merge and cannot be separated anymore. This
intertwinement varies from one system to another. We suggest that these differences in the
intertwinement of social practices and technical artifacts to be a highly relevant subject of evaluation
and potential improvement.
Related work points to challenges in evaluation approaches. Several authors propose means or
methods for socio-technical evaluation; for example, Shin[7], Krenn [21] and Nelles et al. [22], [23].
These methods do not address socio-technical systems in general but each of them has a certain focus
such as human-robot collaboration or internet of things. These approaches, however, do not
systematically consider the social and organizational practices, in which technical artifacts are
embedded, as a subject of evaluation and redesign. By contrast, the methods for socio-technical
evaluation mainly refer to the quality of technical artifacts and infrastructure in the context of their
utility and usability for work tasks. In summary, current methods cover some aspects of the socio-
technical intertwinement, but neglect to aim in the improvement of the organizational context, social
practices, or social dynamics [24].
Other methods, e.g. ETHICS [25] or participatory design approaches such as MUST [26], that
consider the fit of technical artifacts into an organization, mainly in the context of labor, focus on the
phase of designing sociotechnical systems rather than evaluating them in the phase of usage. Hence, we
focus on guiding the evaluation of the sociotechnical intertwinement instead of principles that aim at
supporting design before use [13]. To deal with the complexity and manifold relationships within socio-
technical intertwinement we propose the usage of heuristics as a basis of evaluation where we suggest
that “Heuristics are rules of thumb for reasoning, a simplification, or educated guess that reduces or
limits the search for solutions in domains that are difficult and poorly understood” [27]. Heuristics are
useful mainly to quickly detect the most serious problems and help to draft design recommendations
[28]. For example in the context of usability, experts or users inspect the features of an interactive
system step-by-step by applying a list of items (heuristics) as proposed by Nielsen [29] or the
International Standard Organization [30]. From the perspective of advanced HCI research and socio-
technical design, it is reasonable to extend the evaluation of the usability of technical artifacts by
including the social practice and broader organizational context, as proposed by [2], [31]. They propose
eight heuristics that widens the focus from a pure view on technology and its usability to a broader view
by also evaluating the context of organizational conditions and social practices. The eight heuristics are
derived from criteria, principles, guidelines etc. as they are discussed in six different fields: human-
computer interaction (HCI) and usability; computer-supported cooperative work (CSCW)/groupware;
159
process (re)design; socio-technical design; principles of job design, and privacy. Each of these eight
heuristics covers both dimensions - the social and the technical - as well as their intertwinement. One
challenge is to evaluate real practices of people who interact with technology in a socio-technical
context and to understand the quality of the intertwinement between social and organizational practices
with technical infrastructure. The field of job re-design is most advanced in providing detailed analysis
of the employees’ situation; for instance a method called KOMPASS is provided [32]. However,
applying this method requires a lot of time and needs the involvement of competent evaluation experts
in the field of industrial psychology. Thus, we see the need to propose a procedure that applies the eight
heuristics in a way that
a) allow novice evaluators who are – however – familiar with some issues in the practical field where
socio-technical intertwinement takes place
b) is efficient enough to be repeated from time to time to support an agile series of re-design cycles.
A more detailed description of the socio-technical description and the underlying socio-technical
understanding can be found in appendix A.
3. A questionnaire-based, two-stage evaluation process for socio-technical
systems
The main objective of this method is to enable organizations and teams to evaluate their work
environments independently from external experts. The distinctive knowledge of their respective work
environments should be applied by novice evaluators to identify problems and potentials for
improvement. Experts in systems design are not able to advise every necessary change in process or
technology for specific contexts. The interactive socio-technical questionnaire utilizes condensed
conclusions from socio-technical literature to substitute the expertise from system design experts.
Herrmann and Nierhoff [2] conducted extensive literature research in six research domains relevant
for socio-technical systems. These findings were condensed into a set of eight socio-technical heuristics
which aid evaluators in evaluating complex socio-technical systems. The broad scope of the heuristics
can be used to identify issues directly or to uncover areas of interest, that were not considered as critical
prior to the evaluation. These areas of interest contain several small issues, obstacles in workflows or
other effects that cause discontent. These areas can be based on activity (i.e. communication, knowledge
management, distribution of tasks, customer interaction), but also context (i.e. predictive maintenance,
artificial intelligence, product development, etc.) or can be related to supportive technical infrastructure.
Early research on the socio-technical heuristics shows a necessity to provide more tangible
information for novice evaluators [33]. While exhaustive information can be drawn from the ST-
heuristics if evaluators have extensive experience or domain knowledge, novices only utilize a small
portion without specialized training. Bendel[34] utilized the socio-technical heuristics as a tool for
reflection and elaboration in a digitization project. Participants used the socio-technical heuristics and
derived guided questions for the discussion of conceptualized socio-technical processes and developing
improvements.
Table 1
Socio-technical heuristics (Herrmann et al. [1])
1. visibility about task handling and feedback about its success;
2. flexibility for variable task handling leading to a participatory evolution of the system;
3. communication support for task handling and social interaction;
4. Purpose-orientated information exchange for facilitating mental work;
5. balance of effort and experienced benefit by organizational structuring of tasks;
6. compatibility between requirements, development of human competencies, and the system’s features;
7. efficiency-oriented allocation of tasks for pursuing holistic goals; and
8. supportive technology and resources for productive and flawless work
Note: The underlined keywords are used to identify the heuristics in what follows
160
The approach presented in this paper extends the socio-technical heuristics into an interactive
questionnaire to identify and investigate problems in socio-technical systems. We propose a two-stage
evaluation process for the identification and analysis of possible issues. This process divides the
questionnaire into four interview sessions, each investigating 2 of the 8 socio-technical heuristics (see
Table 1). As shown in Figure 1, each session contains a preliminary interview phase, a questionnaire to
identify problems, passages of guided in-depth interviews and a follow-up discussion. These segments
are described further in the following sections.
Figure 1: Evaluation process for each interviewee (for the modelling notation see [35])
Structure of questionnaire. One of the challenges of the socio-technical approach is the selection
of critical areas for closer evaluation. One solution to focus within the evaluation is to address a specific
process or workflow. This approach is especially reasonable if a process is already identified as
suboptimal or critical for the organization in general. Problems on the intersection of processes or
inherently independent of specific processes are difficult to identify.
In broad explorative evaluations, evaluators have to pay special attention to participants to pick up
small clues indicating large issues. It is hard to distinguish between individual notions and deep-rooted
problems that are grudgingly accepted and not questioned[36], [37].
Another challenge is the identification of root causes for identified issues. Issues are often identified
by their symptoms. Understanding the underlying causes and the context of issues is crucial to their
resolution. Exploring and documenting issues separately from the initial recording is often difficult.
Establishing the context of the issue or even recalling the thought process that led to its identification
can be problematic if the documentation is not substantial [28]. The developed questionnaire combines
the identification of problems and concerns with the identification and investigation of the root issue.
We advanced the set of socio-technical heuristics by transforming the condensed subject matter into
a detailed questionnaire. Each heuristic was split into a series of single issues. Explicit and implicit
interconnections between several aspects were also added as single issues. This expanded the set of
evaluation aspects extensively. The extent of the generated collection of issues is a trade-off for making
the condensed and partly implicit information contained in the heuristics more accessible. While the
socio-technical heuristics each create an extensive multidimensional scope for evaluation, the socio-
technical questionnaire is intended as a checklist that guides the evaluation of complex socio-technical
systems.
In order to construct a specific questionnaire, we analyzed the content of each of the socio-technical
heuristics to gather explicit and implicit aspects of considerations on socio-technical systems. Two
people formulated a list of specific issues for each heuristic individually. These issues then were
compared and discussed to find a holistic view of the heuristics and to formulate one aggregated list of
161
issues for each heuristic. We derived an accumulated list of 113 specific, easily understandable
statements for the questionnaire. All statements are idealistic descriptions of the preferable state
according to a well-designed socio-technical system. Interviewees are intended to compare their work
environments to the idealistic statements and indicate to which degree the statement is accurate. The
statements are designed to be applicable in most domains and work environments, while being precise
enough to verify the socio-technical requirement. An excerpt of the interactive socio-technical
questionnaire can be seen in Table 2. Additional information and the complete questionnaire can be
found at [38].
For each statement we developed a guided in-depth interview passage to investigate if a
disagreement concerning a certain statement within the heuristics hints to a problem within the socio-
technical system. Each in-depth interview passage is specifically designed to explore the context of the
situation regarding the statement. The interview passages have three major components: Exploration of
the situation and context, identification of reasons for objection and suggestion for improvements.
Depending on the scrutinized statement, multiple aspects are investigated by iterating these components
several times or by a combined inquiry from different angles. The guided interviews are designed to
investigate and distinguish between multiple causes for the objection to the ideal statement.
Table 2
Excerpt from interactive socio-technical questionnaire
Heuristic 3: Communication support for task handling and social interaction
S2) Various options are available for conversations or sharing information (appropriate rooms,
phone, video, text messaging, or other).
Optional questions if selected for further elaboration:
• Can you share more about this?
• What places or rooms can you use for conversations?
• Are the places appropriate for conversations? What is good or bad about the places?
• What are your tech options for talking to someone or sharing information?
• How well does technical communication work? What are the advantages and disadvantages?
• When do you use technology to communicate with someone, when do you prefer to meet in
person? Why?
• What should change?
Heuristic 7: Efficiency-oriented allocation of tasks for pursuing holistic goals
S13) The technology and organization are getting increasingly better. You can work better and
better by enhancing your work environment.
Optional questions if selected for further elaboration:
• Can you elaborate more on the background?
• How is your work environment changing?
• How is the technology you use evolving? When will the existing technology be replaced or
upgraded?
• Where do you see opportunities for improvement?
• How has your work changed in the last year?
• How has the organization of the work and your team changed?
• How does the change in work match the change in the work environment?
162
• What suggestions do you have for what could improve?
• How satisfied are you? What should change?
During the development of the socio-technical questionnaire, the structure and integrity of the socio-
technical heuristics (see Table 1) was maintained, e.g. with respect to a systemic covering of social and
technical aspects. Each heuristic is represented by a list of between 9 and 23 statements (see Table 3).
An example of two statements is shown in Table 2. To identify possible issues or important areas of
interest for improvement, the statements are presented to interviewees. Participants are able to mark
their agreement or disagreement whether their work environment matches the idealized statement on a
seven-point Likert scale. This allows for a quick overview of concerns. If the interviewee disagrees
with an idealized statement (i.e. numeric answer is below midpoint) it should be explored further. Not
all participants feel comfortable to openly disagree and transpose their answer-spectrum to sound more
positive. This can result in strong disagreement only being denoted in the middle of the scale. To
account for this effect, answers below the mean value for the individual participant should also be
selected for further exploration. Additionally, outliers within a heuristic indicate special circumstances
regarding the theoretical aspect of the statement. Regardless whether the circumstances have positive
or negative effects on the socio-technical systems, it is critical to identify the circumstances and their
reasons for a comprehensive evaluation. Social tensions can be identified by observing significant
deviations in answers inside a team or socio-technical system. These can also indicate differing
expectations or perceptions, which could lead to problems in the future. For statements that are already
identified as critical for underlying problems during previous interviews with other employees within
the same team, different perspectives should be ascertained. Even if the individual participant does not
indicate the respective statement as abnormal. If participants indicate special circumstances regarding
a statement not by the value of their answer but by comment, the respective statement should also be
considered for closer inspection of underlying issues.
Analysis through passages of guided in-depth interviews. For each statement an individual
collection of guided interview questions was created. To facilitate the later development of possible
solutions, the context, scope and effects for identified issues have to be elicited. The wide selection of
investigated issues requires a customized guided interview to query the required information with direct
and specific questions.
All guided interviews start with a request to explain the general situation leading to the disagreement
with the respective statement. The following questions inquire background information for specific
aspects of the statement. Utilized approaches are to request descriptions of typical micro-tasks or
description of the reasoning process in the specific situation. These questions are designed for the
participant to recall similar experienced situations and evaluate memorable instances instead of
generalized abstract situations. One of the possible follow up questions for these scenarios is what
obstacle or unmet requirements exist in these types of situations. Instead of focusing on one specific
instance, these questions invite the participant to explore their experiences and recount multiple relevant
instances. These deliberations allow a comprehensive understanding of the investigated concepts and
their implementation in the socio-technical system. For some statements the sentiment towards a
situation is one of the possible concerns. The guided interviews for these statements inquire about the
personal attitude for this aspect of the specific situation or the socio-technical system in general. If the
respective statement consists of multiple aspects, the described types of questions are repeated as
necessary. All guided interviews end with a request for suggestions for improvement. This repeated
request for suggestion and participation imprints itself on the participants and induces a thought process
to think about possible improvements, even between interviews.
Preliminary talk and follow up discussion. The main instrument for conducting the evaluation is
the socio-technical questionnaire. To fully utilize its potential, we implement preliminary talks and
follow up discussions directly before and after the main interview respectively. The preliminary talk
allows the participants to acclimate to the interview. Participants can present their work environment
from their perspective, while the interviewer gathers important information about the perceived social
structures, procedures and individual tasks about the participant. This information is critical to probe
163
specific aspects during the main interview and as examples for inquiry if the participants are unsure
how to answer questions in the questionnaire. A joint (verbal) exploration of the personal work
environment also conveys the participant-centered approach - participants are not an accessory to the
evaluation, they are the main actors. During the preliminary talk already known issues are mentioned
by the participant and should be revisited during the questionnaire or the follow-up discussion.
The follow up discussion of the interview presents an opportunity to reflect on the socio-technical
system outside the guided questionnaire. Issues, suggestions and comments, that did not fit within the
questionnaire are added. This includes issues that appear in the socio-technical system but are not
subject of the current interview session or are additional issues occurring in situations that were already
discussed. Additionally, during the follow up discussion as well as the preliminary talk participants can
share reflections on their work environments they had between interview sessions.
Iterative testing and improvement. We tested the approach of using statements and guided
interviews in the two-stage evaluation process of the questionnaire in a pretest with an IT-administration
team within our department. This enabled a closer look on the method within the questionnaire, since
the socio-technical system is already prominently known and gaps or misrepresentations could be
identified. Each interview was conducted using the statements of the questionnaire from two heuristics
and was coupled with the preliminary talk and a follow up discussion. While the guided interviews were
useful in analyzing the individual employees' situation and to understand the socio-technical system,
interviewees indicated that some of the statements within the questionnaire were hard to assess. We
used the feedback and our observations from the interviews to iteratively improve the questionnaire. A
prominent challenge was to query complex and interrelated socio-technical aspects in simple terms that
are easily and correctly understood by laymen. Statements were broken up to reflect different facets of
the same socio-technical issue. The resulting statements and questions were more comprehensible and
precise. Additionally, for each heuristic specific questions for the preliminary talk and follow up
discussion were developed to explore the context necessary to investigate the socio-technical system.
While the pretest was successful in finding problems and shortcomings within the analyzed socio-
technical system and refining the interactive questionnaire, the method needed further application to
evaluate its usefulness for novice evaluators and its application on other socio-technical systems.
4. Six case studies testing the questionnaire-based, two-stage evaluation
process
To test our evaluation process, we asked teams of students to apply the method to various work
teams in industry. This task was part of the course “design of socio-technical information systems” to
gather insights on the application by novice evaluators in real-world scenarios. All participants chose
this elective as part of their Master of Science degree in Applied Computer Science. The students were
familiar with software development and software engineering beforehand. Some of the participants had
taken an elective on “groupware and knowledge management”.
During the lectures, students received an introduction on the history of socio-technical systems and
various concepts for their evaluation. The socio-technical heuristics and the interactive socio-technical
questionnaire were presented and discussed. As part of their practical training, they were asked to
evaluate socio-technical systems using the interactive socio-technical questionnaire. Groups of two to
four students chose different socio-technical systems to evaluate. Each team of students evaluated a
selected team inside a larger organization which had to handle a shared task or process. These processes
were modelled prior to the evaluation. The evaluation consisted of four interviews per interviewed
employee. Topics for each interview were selected as shown in table 3. Every student had to conduct
at least four interviews which each consisted of the preliminary talk, discussion of the questionnaire for
two heuristics, and the follow up discussion (also see Figure 1). All interviews were conducted within
a span of 8 weeks. The student groups had to write a final report on the socio-technical system they
evaluated. The reports contained the procedure of the evaluation, as well as the problems found with
the questionnaire. In addition, the students developed proposals for solutions to the problems and
prepared a reflection on the project.
164
Table 3
Combination of socio-technical heuristics for interviews of every case with number of statements per
heuristic
Interview 1 Interview 2 Interview 3 Interview 4
STH1 Visibility STH2 Flexibility (9) STH5 Balance (18) STH6 Compatibility
&Feedback (13) (19)
STH4 Information STH3 Communication STH8 Supportive STH7 Efficiency (13)
exchange (11) (9) Technology (21)
We analyzed six reports to find out how far the approach of using the interactive questionnaire within
the two-stage evaluation process supports the identification of problems within socio-technical systems.
The qualitative analysis was done with an exploratory approach coding with MAXQDA. The focus of
the analysis lied on the problems instead of the suggested solutions by the students to evaluate if the
students as novice evaluators can detect socio-technical problems. We coded the evaluated problems
based on their content, which heuristic they were assigned to, and with which heuristic they were found
in. Additionally, we tried to cluster the problems based on their content to find themes within all reports.
The six analyzed socio-technical systems can be described as follows:
1. Web development: A team of software developers who worked within a web development
company. The main tasks of the team include the development process of the internal
products as well as project- and product management and customer care. Here, potential
tools supporting the developing activities as well as the project coordination represent the
technical dimension while the conventions for managing the project as well as the
accompanying communication processes are part of the social aspect. The relevance of both
sides as well as their interplay are significant for what we consider a socio-technical system
or process.
2. IT service provider: a software development team at an IT service provider developing
software for master data management for the chamber of commerce and industry. Especially
during the shift to remote work due to the Covid-19 pandemic, the team relies on different
tools and software for the development process. Whereas the coordination within the team
members (via Scrum) as well as customer support during the development processes are
organizational challenges.
3. Car part manufacturer: multiple employees with different roles and responsibilities within
the shopfloor management of a car part manufacturer. Shopfloor management is an
organizational system for managing manufacturing processes, which is supported by
technical solutions. This case study focusses on the coordination of different roles within
the team and organization of the processes.
4. Public transportation company: a complaint management department of a public
transportation company. The team works on virtual customer support, thus does not have
any personal contact with customers. Organization within different team members is
necessary to resolve these complaints because singular complaints are not allocated to
singular team members. The organization and exchange of complaints and their
management are supported by an online complaint management system and additional
technical solutions.
5. University Examination office: An administrative team within a University Examination
Office. The intertwined socio-technical components include a software for listing courses,
documenting grades and providing records as well as regulations for examinations, required
information exchange, and well-established conventions for handling examinations. The
interviewees included students, lecturers and the examination office to analyze all roles
involved within the system.
6. Rail transport solutions: Employees of the HSSEQ, Sale, Customer Service and IT
departments within a company that provides rail transport solutions for freight and
passenger transportation. The focus within this social-technical system was on the software
Zedas which is the main operating system for transport and logistics management.
165
Additionally, consultations and organization within the different teams using the software
are examined.
Table 4 gives an overview over the constellations in the different cases.
Table 4
Characteristics of the cases
Case- abbreviation number of number of number of Number of
students interviewees interviews problems
1 Web development 2 9 20 24
2 IT service provider 4 5 20 16
3 Car part manufacturer 4 5 20 57
4 Public transportation 4 5 20 90
company
5 University Examination 3 16 16 23
office
6 Rail transport solutions 4 4 16 14
We chose to include various reports based on their quality and length to ensure a broader scope of
application of the developed method. In total, nine groups participated in the course. Reports of three
groups omitted information relevant to this analysis. To ensure the reliability, these three reports were
not included in the following analysis.
While the six reports differed in length and detail it is important to note that all teams of students
were able to analyze the respective socio-technical system efficiently and applied the method properly.
5. Results
Altogether the students identified 224 problems. The number of problems within each report ranged
from 14 to 90 problems; the average number of problems per case is 37.66. Table 5 provides an
overview on the number of problems in each report, as well as the number of problems categorized
within each heuristic. It is important to note that many of the problems were associated with multiple
heuristics as they concerned complex issues. This explains why the categorization within the heuristics
exceeds the total number of problems within one case study.
Table 5
Number of problems for each case study by heuristics
Heuristic Indicator of case and number of problems
1 2 3 4 5 6 ∑ Rank
STH1 Visibility &Feedback 5 3 8 9 9 3 37 5
STH2 Flexibility 5 1 6 4 2 2 20 7
STH3 Communication 1 3 6 4 3 2 19 8
STH4 Information exchange 0 2 3 17 6 2 30 6
STH5 Balance 5 4 15 21 6 1 52 3
STH6 Compatibility 2 7 10 27 4 3 53 2
STH7 Efficiency 7 6 13 16 9 5 56 1
STH8 Supportive Technology 0 5 20 12 6 4 47 4
Total 24 16 57 90 23 14 224
Within five out of the six case studies, problems covering all of the eight heuristics were identified.
Case study 1 focused on only four heuristics (STH1, STH2, STH3 and STH7) within their analysis, due
to their team only consisting of two students. Overall the method provided a range of identified
problems within all of the applied heuristics.
166
While analyzing the reports of the case studies, we found that multiple problems across different
case studies were similar. We decided to cluster the problems of all case studies based on their content.
In the following, we list the 20 types of problems which we were able to identify and one example
within each type to underline the content.
Table 6
Number of problems of a certain type for each case
Number of problems per case
Type of problems Related heuristics 1 2 3 4 5 6 ∑
1. Problems with customers, i.e. language barriers with 6 3 3
customers;
2. Problems of employees developing new skills, i.e. 5 1 1 2
monotone tasks in day-to-day-business;
3. Missing information from other departments, i.e. 4 1 3 2 6
limited knowledge of projects within other departments;
4. Processes not optimized, i.e. multiple rounds of 7 5 5 2 12
reporting results from tasks;
5. Problems with the personal motivation of employees, 5 1 11 1 13
i.e. limited proactive engagement;
6. Personal stress factors and burdens of employees, i.e. 5 2 1 4 8 2 17
overload of tasks;
7. Privacy concerns, i.e. insufficient reconnaissance in 4 2 1 1 4
the data collection of employees;
8. Feedback loops, i.e. feedback is not provided quickly, 1 1 6 2 1 10
directly or regularly;
9. Organization of tasks, i.e. limited flexibility in 2 4 1 5 10
executing tasks;
10. Disturbed communication and reachability, i.e. 3 3 7 12 5 27
limited reach of supervisors;
11. Technical problems, i.e. loading time of servers and 8 1 2 3 5 9 3 23
software;
12. Organizational problems, i.e. under staffing of 6 6 9 8 1 2 26
teams;
13. Use of multiple tools/ missing intertwinement of 4, 1 3 2 6
different applications, i.e. information is only saved locally; 6
14. Missing training and further education, i.e. missing 6 1 3 1 1 6
training when using new technological solutions;
15. Efficiency problems, i.e. inefficient execution of 7 1 1 2 4
tasks;
16. Time problems, i.e. spontaneous delegation of tasks 2, 1 1 3 5
which leads to overtime; 7
17. Knowledge management, i.e. missing upkeep of 4 2 7 8 2 19
databases and information on processes
18. Problems which arose during the Covid-19 - 9 1 2 5 2 19
pandemic, especially regarding a shift to mainly working
from home
19. Individual conflicts, i.e. personal conflicts between - 1 2 2 5
employees;
20. Missing digitalization, i.e. some information is 8 1 2 3 6 3 15
collected on paper and later transferred into a database;
Table 6 presents the number of problems within the 20 types for each case study. The second column
proposes heuristics that can be assigned to the type of problems; question marks indicate that there is
no heuristic that could be assigned evidently. Grey shadowed cells indicate the dominant type(s) of
167
problems per case. Case studies 3 and 4 identified problems within a broad range of themes, while the
problems of case studies 1,2,5 and 6 were narrowly focused on specific themes. Within every case study
one or two themes were prominently discussed as problems by the respective employees. Multiple
smaller problems could be connected to one issue or were raised by multiple employees.
In what follows we shortly characterize the problems found in each case:
1. The most prominent problems within the web development team (case study 1) concerned the
shift to working from home due to the Covid-19 pandemic. The issues within home office work
covered missing technical equipment, missing possibilities for informal exchange with other
colleagues, the use of new software, additional time required for tasks and problem solving,
missing information on availability of colleagues, maintaining a work-life balance and a
disrupted work environment depending on personal circumstances at home. Multiple
interviewees were concerned with the shift to working from home and described the situation
as more exhausting and stressful. Overall, the team still thought that the switch to working
from home was successful and were satisfied with the arrangement.
2. Within the software development team at the IT service provider (case study 2) the majority
of the problems concerned organizational issues. This included the communication of decision
making, descriptions of tasks, insufficient planning and implementation of company targets, a
mismatch between technical language with the customers’ language. Furthermore, difficulties
with planning and organizing work due to incompatible working methods within the company
(the cycles of the agile software development team is not coordinated with classic project
management teams) were observed. This mismatch leads to interferences with other teams
within Scrum cycles that cause disruptions within sprints and a mismatch of delegation of tasks
for individual employees.
3. The main problems within the shop floor management of the car part manufacturer (case study
3) were also of organizational nature. Employees raised the following problems: the
understaffing of teams, corporate goals which were only attainable if the production line had
no stops, a high noise level at the open plan offices, an overload of meetings as well as calls
and reports, insufficient break rooms, missing practicability of top-down decisions, changes
are implemented across a wide range of departments instead of gradually, missing specified
training periods for new systems within the production line, and the purchase of cheaper
equipment instead of durable equipment. Most of these problems were only raised by single
employees. However, this can be explained by the explorative nature of the selection of
interviewees. Within this case, one employee of each level of managerial oversight was
interviewed.
4. The main problem types within the complaint management team of the public transportation
company (case study 4) were twofold. The first main type concerned disturbed communication
and reachability, and included inefficiency of information exchange, missing announcements
of software updates and process changes, no informal exchange with team leaders and
employees, no disclosure of important information, misunderstandings with other departments,
supervisors who are not easily accessible when needed, service providers who cannot be
reached on the phone, and missing information on whom to contact regarding specific issues.
The other main category concerns problems of personal motivation which include team leaders
who feel in need to check whether tasks have been completed, employees withholding
information from each other, employees not helping their colleagues if a mistake occurs,
employees disliking the digitalization and change processes and doing tasks inefficiently to
slow down the process, employees not participating in decision making processes,
management not being able to motivate employees, a reluctancy to learn new tasks and a
general motivation problem of the team.
5. The main issues which occurred within the examination office (case study 5) were technical
problems of interacting with the system that is used to register and organize exams and courses.
These problems included a lack of being able to enter information into the software,
incompatibility between the system’s structure and exceptional types of examination, out of
date information on the system, incorrect automatic exam registrations of students, problems
with the control mechanism when entering grades of students, problems with uploading exam
grades into the software, an unclear user interface, missing interfaces of different programs
168
which lead to transferring data manually and a general overload of used programs within the
processes.
6. Within case study 6 at the company providing rail transport solution no prominent type of
problem could be identified, the most pressing problems included technical problems of and
missing training for the main operating system, a fear of employees to discuss problems freely,
monotone tasks and inefficient delegation of tasks, problems with the organization of meetings
and low motivation of employees.
Within the reports of the six case studies, we observed differences in quality and quantity of the
problems. Some of the reports described the problems in more detail and connected different aspects
which were raised by different employees to one problem, other reports did not inspect the problems in
detail but focused on descriptions of the socio-technical system and proposals for solutions. One notable
difference between the case studies was the selection of interview participants. Some case studies
focused on a singular team working together and interviewed multiple employees within one team (case
studies 1, 2 and 4) while others interviewed one or multiple employees from different departments (case
studies 5 and 6) or employees from multiple levels of management (case study 3). The case studies
focusing on one team inspected more problems which were raised by multiple employees. The problems
of the other cases were more individual as interviewees are doing different tasks within their respective
roles. However, within the cooperation of different teams or employees, problems that interfere with
multiple people or systems can be observed. As described above, similar problems were found in
different case studies but overall, the socio-technical systems and problems within each case are
individual. The depth and clarity of the described situations within the reports of the cases vary, as well
as the quantity and quality of detected problems. This can have a multitude of causes: the willingness
of the employees to discuss and disclose issues at work to the students, the experience of the students
as interviewers, the care and time invested in the final report, the complexity of the socio-technical
system as only small teams could be analyzed in the case studies or the suitability of the questions to
inspect a statement within the questionnaire.
With the small sample size of each case study the scope and results are only limited to the respective
teams and cannot be applied to broader scopes of the companies. In general, although the interviews
raised problems for the respective interviewees, in most cases the employees were satisfied with their
work environments.
Furthermore, we found that the problems that have been identified by the students can be
differentiated with respect to three categories:
1. Social or organizational (s): Problems that do not refer to any technology being involved
2. Socio-technical (st): Problems that address the interplay between technology and social
behavior or organizational settings
3. Technical (t): Problems that exclusively deal with technical issues.
We take the following examples of problems from the case “Car Part manufacturer” to demonstrate the
difference between the three categories:
1. S: At first, both the feedback emanating from superiors and from colleagues was criticized as
irregular and insufficient. As a result, it was also often stated that one's own contribution and
also the contribution of colleagues to the success of a project or manufacturing process is not
clear at all.
2. ST: For example, data and information such as key figures or even problems are currently still
recorded with pen and paper and only digitized later. Important information is also often printed
out in the morning in order to have it available throughout the day. Employees therefore have
to go to the office every time to get this information, or print it all out in the morning.
3. T: In the opinion of the employees, many technical solutions are undersized, which is associated
with the fact that the company always weighs up "favorable price vs. durability" in favor of the
favorable price.
Table 7 presents the distribution of these 3 categories between the cases. The displayed numbers help
to discuss how far the computer science students and their evaluation were oriented to problems that
have not a pure technical background but refer to accompanying issues and to the intertwinement
between technology and social practices.
169
Table 7
Differentiation between social (s), socio-technical (st) and technical (t) issues per case
Category
Cases s st t ∑
1. Web development 17 71% 6 25% 1 4% 24
2. IT service provider 9 56% 6 38% 1 6% 16
3. Car part manufacturer 32 56% 22 39% 3 5% 57
4. Public transportation company 52 58% 31 34% 7 8% 90
5. University Examination office 2 9% 20 87% 1 4% 23
6. Rail transport solutions 5 36% 7 50% 2 14% 14
∑ 117 52% 92 41% 15 7% 224
6. Discussion
With an average of 37.66 per case and a minimum and maximum from 14 to 90, it becomes obvious
that the methods helps to detect sociotechnical problems. In all investigated cases, the novice evaluators
were able to identify relevant problems in the respective socio-technical systems.
Table 5 reveals that all eight heuristics were relevant with respect to the detected problems.
However, the number of assignments varies between a maximum of 56 (efficiency) and a minimum of
19 (communication support). This might be seen as an indicator for a varying relevance of the heuristics.
However, the study [1] where the heuristics were developed applied them to a data base of 42 problems
with a maximum of 21% of assignments to visibility and 7% to technical support; efficiency got the
second lowest percentage of assignment. Therefore, we suggest that the variances between the number
of assignments to heuristics does not depend on the general relevance of the heuristics but on the
characteristics of the cases.
As seen in table 4 the students identified problems corresponding to every investigated heuristic in all
case studies. Only in one case study, no problems corresponding to heuristics 4 and 8 were identified.
However, the questionnaires to identify problems in heuristics 4,5,6 and 8 were not applied in this case.
The fact that problems related to heuristics 5 and 6 were identified despite not explicitly being
investigated supports the goal of investigating a broad, generally applicable spectrum of socio-technical
issues. Participants are guided to reflect on their work environment and the existence of hindrances or
problems in regard to known socio-technical issues. The statistical and qualitative results suggest that
not only the specifically prompted issues were reflected upon, but also situations that are connected to
these issues (e.g. Covid-19 related problems). While problems identified in this manor predominantly
correspond to different issues of the same heuristics, a significant amount are related to several different
heuristics. This overlap of heuristics is intended as the heuristics, as well as the questionnaire, are
designed to identify issues in complex socio-technical systems[2]. Both approaches focus on identifying
existing issues from various socio-technical perspectives while not being highly selective between these
perspectives. Furthermore, problems were not necessarily identified by individual questions, but by
sequences of questions which in combination yielded specific problems. This effect was especially
intensified by consolidation of answers from multiple participants. Connected issues were also
identified by the sequence of statements and questions corresponding to the overarching socio-technical
principle.
With respect to our categorization of the problems, as documented with table 6, in all case studies
at least eight types (case 2 and 6) of problems were identified which each consist of several distinct
problems. These clusters of distinct problems associated with the same type of problem suggest the
existence of a critical problem or inherent flaw of the socio-technical system that influences multiple
workflows and therein produces subsequent problems. Case four addresses 19 of 20 types of problems.
Five categorized types of problems are consistent over at least five of the six case studies. This suggest
that the questionnaire enables the identification of problems regarding to stress, technical equipment,
organization, missing digitization and also problems related to the specific situation caused by the
Covid-19 pandemic reliably. Additionally, the questionnaire supports identification of problems
170
specific to the individual work environment. It is remarkable that the set of heuristics triggers the
identification of problems that are not directly related to this set, such as the problems being assigned
to the Covid-19 pandemic.
While some groups identified numerous problems in every heuristic, other groups found a higher
concentration of problems in selected heuristics and only few problems in others. Additionally, the total
number of problems varies widely between groups. As described in section 5 this can have multiple
reasons ranging from number of evaluators to complexity of evaluated socio-technical system and level
of dedication of participants and evaluators.
While the method proved to be successful to understand and evaluate socio-technical systems and
their shortcomings, the assigning of the shortcomings and problems to the heuristics was rather difficult
for novice evaluators. The categorization of identified problems into problem-types and the attribution
to related heuristics was not main focus of this project. It is, however, an often-required step for
subsequent resolution of identified problems or similar improvement efforts to order and prioritize
identified issues. As the categorization of problems provides a quick overview of the relevance of
individual heuristics for the evaluated socio-technical system it is a useful information for subsequent
evaluations or reviews. For the identification of problem areas and critical issues a categorization of
problem topics for the individual socio-technical system proved useful. This overview supports the
identification of cohesion and interconnection of identified problems regardless of the theoretical
classification.
The analysis of the content of the problems reveals that in every case a series of problems were
found that hinder the handling of tasks such as insufficient
• Technical equipment,
• Training and preparation of employees
• Communication support and information exchange
• Motivation
We conclude that these problems are highly relevant and oppose an efficient task handling (as
indicated by the high number of efficiency related problems, see table 5). Thus, we suggest that the
detected problems are relevant with respect to task handling and eventually with respect to the
employees’ job-satisfaction (as indicated by the high number of problems assigned to the balance
between effort and benefits, see table 5).
With respect to the three categories social/organizational vs. socio-technical vs. technical, only in
the cases of the University Examination Office and the Rail Transport Solutions, the st-problems
dominate (see Table 7). Actually, we expected that the questionnaire instrument supports a stronger
focusing on socio-technical issues. By contrast, the numbers of table 7 reveal that the
social/organizational category found the strongest consideration. Therefore, we can conclude that the
evaluation method helped the computer science students to shift their attention from the technical to the
organizational issues. However, the influence of technology and its intertwinement with social or
organizational problems is not sufficiently investigated. This might have been caused by the way of
how the interviewees framed the problems they have experienced. It can be seen as a particular
challenge to investigate whether the causes of the problems are predominantly attributed to human
behavior or can also be seen in relation to a lack of technical support. The interactive questionnaire was
designed to support the interviewers with guidance for socio-technical aspects outside of their (primary)
expertise. This attempted balance might have resulted in an overemphasis on social perspectives. Only
in the case of the University Examination Office, the socio-technical aspects dominate exceedingly
since most of the employees’ tasks and problems are directly related to the usage of an exam-
management-system that the interviewers chose as a main focus. Consequently, we assume that the
interviewers should be oriented more explicitly on the challenge of investigating whether a problem
can be related to a lack of technical support. For instance, the problem that we gave as an example for
a mainly organizational problem could have been accompanied by further questions by the interviewers.
With these questions they could have tried to understand whether the tasks for which regular feedback
was solicited were supported by an information system that also covers reporting to the management
and whether a function might be missed or reasonable that reminds the mangers to give feedback. Thus,
we conclude that the evaluation instrument includes hints of how to go deeper into the consideration of
socio-technical intertwinement.
171
The counter-intuitive approach to investigate intertwined socio-technical issues in complex work-
environments by zooming in to elementary aspects proved effective. The structure of the interactive
questionnaire allowed participants and interviewers to divide the complex environments and to identify
specific situations and problem-areas. These situations and problems were then closely examined using
guided interviews. While the discussed elementary categorization of issues is especially useful for
development and evaluation of the questionnaire itself, it is only of limited use for the improvement of
the socio-technical work environment. However, the categorization of issues to heuristics and problems
indicate a comprehensive evaluation of identified situations and problem-areas (see Table 5). Issues
were identified according to the individual requirements and needs of the participants (see Table 6),
even if these requirements were not specifically included in the questionnaire (e.g. problems related to
COVID-19). The structure of the interactive questionnaire was designed to enable unrestricted
exploration of the work environment while leaving the execution of the exploration to the cooperation
between interviewee and interviewer. This exploration is guided by prompts for investigation of specific
socio-technical aspects to facilitate a comprehensive evaluation and to balance different levels of
expertise. Repeated identification of problems with identical underlying issues in separate sessions and
heuristics indicates examination not on the elementary level but rather a broad and deliberate
investigation.
While the case studies show an over-emphasis on social perspectives, the approach to divide socio-
technical system and socio-technical perspectives and to combine them on an elementary level was
successful. This divide and conquer approach enabled the support of novice evaluators to perform
comprehensive investigations of complex environments and to identify specific issues and their effects
on the work environment. However, this study did not investigate how this level of guidance and support
is effective or obstructive for intermediate and expert evaluators. Further research is needed to
determine how different expertise and experience levels effect evaluation with the interactive socio-
technical questionnaire.
7. Conclusion
The research questions can be shortly answered as follows:
R1: A broad scope of different, relevant problems (20 types) were found with a means of 37,33
problems per case. With respect to efficiency of our method, 112 interviews were needed to find 224
problems; by average 2 unique problems were found per interview. Issues of socio-technical
intertwinement covered 41% of the problems found.
R2: The cases differ with respect to the focus of problem categories and with respect variety of
problem types they address. Issues of socio-technical intertwinement dominated only 2 of the 6 cases.
R3: The novice evaluators were able to shift the view from their technical educational background
to organizational issues. Socio-technical intertwinement was addressed but not as dominantly as being
intended by the set of heuristics.
Limitations: By contrast to many other studies with heuristic-based evaluation [29], [39], our
method does not refer to the severity of the problems found. We suppose that it is hard for novice
evaluators to make valid assumptions about the impact of the problems found. Further studies could
check whether it is helpful to ask the interviewees for a final ranking of the problems found in the
environment. Furthermore, the evaluation was mainly focused on its effectiveness and less on its
efficiency and on aspects for its improvement.
As part of a student project, the novice evaluators had to focus on several challenges at once. While
their experience in evaluating socio-technical systems is comparable with novice evaluators in work
scenarios, the students also had to focus on different academic courses that were not related to this
project. Therefore, an extensive familiarity with the evaluated sociotechnical system could not be
reached. In the investigated teams, not all team members were interviewed. Further research has to
investigate if a more comprehensive investigation enhances the derived problem collection in a
substantial and uniform way. Additionally, a more detailed evaluation could investigate how the count
of identified critical and shallow problems scale with the number of interviewed participants.
172
In further research we aim to integrate collaboration support to facilitate information exchange
between evaluators and foster a shared and more complete understanding of the evaluated system. As
every participant has their own perspective and perceived experiences, a collective understanding
between evaluators can support the investigation of identified problems and their relevant contexts. This
extends to the analysis of problems and their interconnections. In small groups evaluators are able to
exchange relevant information and preliminary interpretations quickly. This helps to investigate
identified problems from the perspective of other participants. In larger groups or constraining
collaboration environments this proved to be challenging.
While the presented research focused on the identification of problems, further research on the
support of documentation and resolution of identified problems is necessary. Special attention must be
paid to the information necessary to resolve socio-technical problems. It is not yet determined how far
the explorative questions of the interactive socio-technical questionnaire include all aspects necessary
for comprehensive understanding and resolution of problems and how this can be supported further.
8. References
[1] T. Herrmann, I. Jahnke, and A. Nolte, “A problem-based approach to the advancement of
heuristics for socio-technical evaluation,” Behav. Inf. Technol., pp. 1–23, 2022, doi:
10.1080/0144929X.2021.1972157.
[2] T. Herrmann and J. Nierhoff, “Heuristiken zur Evaluation digitalisierter Arbeit bei Industrie-4.0
und KI-basierten Systemen aus soziotechnischer Perspektive,” FGW, vol. Digitalisierung von
Arbeit 16, Feb. 2019, [Online]. Available: http://www.fgw-nrw.de/studien/industrie16.html
[3] E. Trist and H. Murray, “Introduction to Vol. II.,” in The Social Engagement of Social Science. A
Tavistock Anthology, Philadelphia: University of Pensylvania Press, 1993, pp. 36–60. Accessed:
Oct. 12, 2020. [Online]. Available: https://www.upenn.edu/pennpress/book/1928.html
[4] E. Trist, “The relations of social and technical systems in coal-mining,” Br. Psychol. Soc. Ind.
Sect., 1950.
[5] F. E. Emery, E. Thorsrud, and E. Thorsrud, Form and Content in Industrial Democracy : Some
experiences from Norway and other European countries. Routledge, 2013. doi:
10.4324/9781315013695.
[6] E. Mumford, “A Socio-Technical Approach to Systems Design,” Requir. Eng., vol. 5, no. 2, pp.
125–133, Sep. 2000, doi: 10.1007/PL00010345.
[7] D. Shin, “A living lab as socio-technical ecosystem: Evaluating the Korean living lab of internet
of things,” Gov. Inf. Q., vol. 36, no. 2, pp. 264–275, Apr. 2019, doi: 10.1016/j.giq.2018.08.001.
[8] M. S. Ackermann, S. P. Goggins, T. Herrmann, M. Prilla, and C. Stary, Designing Healthcare
That Works – A Socio-technical Approach. United Kingdom, United States: Academic Press,
2017.
[9] D. Vinck, “Maintenance and Repair Work,” Eng. Stud., vol. 11, no. 2, pp. 153–167, May 2019,
doi: 10.1080/19378629.2019.1655566.
[10] J. H. E. Andriessen, Working with Groupware. London: Springer London, 2003. doi:
10.1007/978-1-4471-0067-6.
[11] M. J. Muller and A. McClard, “Validating an extension to participatory heuristic evaluation:
quality of work and quality of work life,” in Conference Companion on Human Factors in
Computing Systems, New York, NY, USA, May 1995, pp. 115–116. doi:
10.1145/223355.223457.
[12] J. Wutzkowsky and B. Böckmann, “Using MEESTAR to Identify Ethical and Social Issues
Implementing a Digital Patient-Centered Care Platform.,” in eHealth, 2018, pp. 278–285.
[13] A. A. Imanghaliyeva, P. Thompson, P. Salmon, and N. A. Stanton, “A synthesis of sociotechnical
principles for system design,” in International Conference on Applied Human Factors and
Ergonomics, 2020, pp. 665–676.
[14] D. Shin and M. Ibahrine, “The socio-technical assemblages of blockchain system: how
blockchains are framed and how the framing reflects societal contexts,” Digit. Policy Regul. Gov.,
vol. 22, no. 3, pp. 245–263, Aug. 2020, doi: 10.1108/DPRG-11-2019-0095.
173
[15] D.-H. Shin, H. Choo, and K. Beom, “Socio-Technical Dynamics in the Development of Next
Generation Mobile Network: Translation Beyond 3G,” Technol. Forecast. Soc. Change, vol. 78,
no. 3, pp. 514–525, Mar. 2011, doi: 10.1016/j.techfore.2010.04.017.
[16] P. M. Leonardi, “Materiality, sociomateriality, and socio-technical systems: What do these terms
mean? How are they different? Do we need them,” Mater. Organ. Soc. Interact. Technol. World,
vol. 25, 2012.
[17] W. J. Orlikowski, “Sociomaterial practices: Exploring technology at work,” Organ. Stud., vol.
28, no. 9, pp. 1435–1448, 2007.
[18] L. Suchman, “Agencies in technology design: Feminist reconfigurations,” in workshop on
Gendered Innovations in Science and Engineering, 2005, pp. 15–1.
[19] R. G. Booth, M.-A. Andrusyszyn, C. Iwasiw, L. Donelle, and D. Compeau, “Actor-Network
Theory as a sociotechnical lens to explore the relationship of nurses and technology in practice:
methodological considerations for nursing research,” Nurs. Inq., vol. 23, no. 2, pp. 109–120, 2016.
[20] Law, John, “Actor network theory and material semiotics,” Soc. Theory, p. 141, 2009.
[21] B. Krenn, “Robot: Multiuse Tool and Ethical Agent,” in A Construction Manual for Robots’
Ethical Systems: Requirements, Methods, Implementations, R. Trappl, Ed. Cham: Springer
International Publishing, 2015, pp. 11–29. doi: 10.1007/978-3-319-21548-8_2.
[22] J. Nelles et al., “Best-Practice Approach for a Solution-Oriented Technology Assessment: Ethical,
Legal, and Social Issues in the Context of Human-Robot Collaboration,” Berlin: Springer, 2017,
pp. 1–14. doi: 10.1007/978-3-662-53305-5_1.
[23] K. Behrenbruch, O. Kieselmann, M. Schuldt, M. Söllner, and L. Schmidt, “System
EvaluationEvaluation,” in Socio-technical Design of Ubiquitous Computing Systems, K. David,
K. Geihs, J. M. Leimeister, A. Roßnagel, L. Schmidt, G. Stumme, and A. Wacker, Eds. Cham:
Springer International Publishing, 2014, pp. 297–319. doi: 10.1007/978-3-319-05044-7_17.
[24] S. Bødker, “Third-wave HCI, 10 years later—participation and sharing,” interactions, vol. 22, no.
5, pp. 24–31, 2015.
[25] E. Mumford, Designing human systems for new technology: the ETHICS method. Manchester:
Manchester Business School, 1983.
[26] F. Kensing, J. Simonsen, and K. Bodker, “MUST: A Method for Participatory Design,” Human–
Computer Interact., vol. 13, no. 2, pp. 167–198, Jun. 1998, doi: 10.1207/s15327051hci1302_3.
[27] Interaction Design Foundation, Heuristics and heuristic evaluation. 2021. Accessed: May 19,
2021. [Online]. Available: https://www.interaction-design.org/literature/book/the-glossary-of-
human-computer-interaction/heuristics-and-heuristic-evaluation
[28] C. Wilson, User interface inspection methods: a user-centered design method. Newnes, 2013.
[29] J. Nielsen, “Enhancing the explanatory power of usability heuristics,” in Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, Apr. 1994,
pp. 152–158. doi: 10.1145/191666.191729.
[30] DIN, ISO 9241-110: 2006–Ergonomics of human-system interaction–Part 110: Dialogue
principles. 2006. [Online]. Available: https://www.iso.org/standard/38009.html
[31] T. Herrmann, I. Jahnke, and A. Nolte, “Evaluating Socio-Technical Systems with Heuristics – a
Feasible Approach?,” in Proceedings of STPIS16, http://ceur-ws.org/Vol-1604/, Sep. 2016, pp.
91–97.
[32] G. Grote, C. Ryser, T. Wāler, A. Windischer, and S. Weik, “KOMPASS: a method for
complementary function allocation in automated work systems,” Int. J. Hum.-Comput. Stud., vol.
52, no. 2, pp. 267–287, Feb. 2000, doi: 10.1006/ijhc.1999.0289.
[33] F. Thewes, T. Herrmann, and A. Kluge, “Validating a Heuristic Evaluation Method - An
Application Test,” in Proceedings of Mensch Und Computer 2019, New York, NY, USA, 2019,
pp. 593–597. doi: 10.1145/3340764.3344465.
[34] A. Bendel, “Arbeits- und prozessorientierte Digitalisierung in Industrieunternehmen: Über die
Anwendung eines interventionsorientierten und soziotechnischen Forschungs- und
Gestaltungsansatzes,” 2021, pp. 247–276. doi: 10.5771/9783748923046-247.
[35] T. Herrmann, “SeeMe in a nutshell.,” Available Online, 2006, [Online]. Available:
http://www.imtm-iaw.ruhr-uni-bochum.de/wp-content/uploads/sites/5/2011/09/Seeme-in-a-
nutshell.pdf
174
[36] B. G. Freimut, “Developing and using defect classification schemes,” Fraunhofer IESE IESE-
Rep., vol. No 72, Sep. 2001.
[37] M. Leszak, D. E. Perry, and D. Stoll, “Classification and evaluation of defects in a project
retrospective,” J. Syst. Softw., vol. 61, no. 3, pp. 173–187, Apr. 2002, doi: 10.1016/S0164-
1212(01)00146-7.
[38] F. Thewes, M. Konrad, and T. Herrmann, “Socio-technical interactive questionnaire.”
https://hi4.iaw.ruhr-uni-bochum.de/#!/questionnaire
[39] R. Langevin, R. J. Lordon, T. Avrahami, B. R. Cowan, T. Hirsch, and G. Hsieh, “Heuristic
Evaluation of Conversational Agents,” in Proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems, Yokohama Japan, May 2021, pp. 1–15. doi:
10.1145/3411764.3445312.
[40] D. Shin, “A socio-technical framework for Internet-of-Things design: A human-centered design
for the Internet of Things,” Telemat. Inform., vol. 31, no. 4, pp. 519–531, Nov. 2014, doi:
10.1016/j.tele.2014.02.003.
[41] T. Herrmann, M. S. Ackermann, S. P. Goggins, C. Stary, and M. Prilla, “Designing Health Care
That Works – Socio-technical Conclusions.,” in Designing Healthcare That Works. A Socio-
technical Approach., United Kingdom, United States: Academic Press, 2017, pp. 187–203.
[42] S. Amir and V. Kant, “Sociotechnical resilience: A preliminary concept,” Risk Anal., vol. 38, no.
1, pp. 8–16, 2018.
[43] N. Luhmann and E. M. Knodt, Social Systems, 1st Edition. Stanford, Calif: Stanford University
Press, 1996.
[44] G. Fischer and T. Herrmann, “Meta-design: Transforming and enriching the design and use of
socio-technical systems,” in Designing Socially Embedded Technologies in the Real-World,
Springer London, 2015, pp. 79–109.
[45] J. M. Carroll and R. L. Campbell, “Artifacts as psychological theories: The case of human-
computer interaction,” Behav. Inf. Technol., vol. 8, no. 4, pp. 247–256, 1989.
[46] C. Kirsch, P. Troxler, and E. Ulich, “Integration of people, technology and organization: the
european approach,” in Symbiosis of Human and Artifact, vol. 20, Y. Anzai, K. Ogawa, and H.
Mori, Eds. Elsevier, 1995, pp. 957–961. doi: 10.1016/S0921-2647(06)80337-0.
[47] R. Kling, “What Is Social Informatics and Why Does It Matter?,” Inf. Soc., vol. 23, no. 4, pp.
205–220, Jun. 2007, doi: 10.1080/01972240701441556.
A. Clarification of the socio-technical understanding
Quotes from: Thomas Herrmann, Isa Jahnke & Alexander Nolte (2021)[1]: A problem-based approach to the
advancement of heuristics for socio-technical evaluation, Behaviour & Information Technology, DOI:
10.1080/0144929X.2021.1972157
Socio-technical background
“With the increasing emergence and adoption of applications, such as electronic health care, e-commerce,
social networks, ubiquitous computing, and smart factories, the socio-technical perspective has gained more
and more relevance. We characterise these emerging phenomena with an increasing complexity and
contingency of intertwined social practices and technical artifacts. They cannot be fully analysed or
modelled; once you start describing them, and before you come to an end, the intertwinement has already
evolved and changed. This complexity arises from the different nature of multiple elements that interact with
each other [40]. The dynamics of socio-technical intertwinement undergo a continuous evolution in a
complex setting that is characterised by dynamic changes, uncertainty, and ambiguity [41].” (Herrmann et
al. 2021, p. 1)
“On the theoretical side, the socio-technical view is also present in approaches such as the concept of
socio-technical resilience [42] or the discourse on sociomateriality [16]. Related to the latter, Leonardi
differentiates between the social subsystem (including roles, hierarchies, communication networks, and
others) and the technical subsystem that he characterises as imbrication of human (social) agency and
material agency. We suggest that intertwinement should refer to the interplay between the social and
technical sub-system as well as to the imbrication of social and material agency. Scholars referring to
175
Orlikowski [17] use the term entanglement for the intertwinement of the social and the material. We however
perceive the concept of entanglement to not necessarily constitute a symmetry between the social and the
material though, as proposed in the Actor-Network- Theory [19], [20]. Communities such as CSCW and
HCI perceive social practices and technical artifacts as inevitably intertwined. Once the technology is
launched and integrated in a certain context (e.g. organisation, communities, social practice), the social
practices and the technical artifacts merge into a form of a system in which the two parts cannot be separated
anymore; the one affects the other and vice versa. This intertwinement varies from one system to another–
not all systems have the same intertwinement of social practices and technical artifacts. We suggest that
these differences in the intertwinement of social practices and technical artifacts should be a subject of
inspection and potential improvement. From our theoretical point of view, the intertwinement combines
well-structured with less structured or formal with informal phenomena. The variety of these combinations
leads to increased contingency of socio-technical systems in the sense of Luhmann [43], which is
characterized by variability, particularity, mutability and uncertainty, and serves as a basis for continuous
evolution [44].” (Herrmann et al. 2021, p. 3)
“Figure 1 illustrates how we perceive the socio-technical intertwinement. Human–computer interaction
plays a central role in this intertwinement of social and organisational practices with technical artifacts. From
this perspective, socio-technical intertwinement can be described as intertwined organisational practices of
task-handling in an ecology of tasks [45], [46] and the usage of technical artifacts or infrastructure. Such
organisational practices are part of social practices that inevitably require human communication [43]
between various roles [47] and include both formal and informal tasks [41]” (Herrmann et al. 2021, p. 3)
Figure 1. Socio-technical intertwinement of social practice and technical artifacts through human-
computer interaction. (From Hermann et al. 2021, p.4)
Socio-technical Heuristics:
“In order to prepare our literature search and the integrated set of new heuristics (see Section 4.1), we
identified various relevant domains or disciplines that already have published sets of categories, criteria,
principles, or guidelines. We then used these existing sets and synthesized them into a new set of socio-
technical heuristics.” (Herrmann et al. 2021, p. 4-5)
“We aimed to focus on the most influential research work in the domains. In the literature review, we
first searched for already existent principles (Section 2.4.1-2.4.6) that we then used later in our empirical
study. It is important to note that prior work does not always utilize the term heuristics. Instead, one will
find terms such as principles, categories, design guidelines, or golden rules. They all refer to the same idea
of heuristics in that they provide strategies to make decisions on how to improve the system where rational
choices are possible [27]. In this study, we use them as synonyms for heuristics. All the heuristics that we
refer to in the following sub-sections might serve a variety of purposes while we mainly consider them with
respect to their potential contribution to socio-technical evaluation.” (Herrmann et al. 2021, p. 5)
“We apply the term, heuristic, as it has been influenced by cognitive psychology. ‘Heuristics are rules of
thumb for reasoning, a simplification, or educated guess that reduces or limits the search for solutions in
domains that are difficult and poorly understood’ [27].” (Herrmann et al. 2021, p. 4)
176
# ST-Heuristics with short description
1 Visibility about task handling and feedback about its success
Focused information is continuously offered about the progress of technical processes and, as far as permitted,
about collaborative workflows. This helps to understand what further steps are possible or not and why as well
as how well the expectations of others are met.
2 Flexibility for variable task handling leading to a participatory evolution of the system
One can vary options of task handling and can flexibly decide about technology usage, time management,
sharing of tasks, etc. Consequently, one can develop a wide range of competencies that support participation in
the ongoing evolution of the whole system.
3 Communication support for task handling and social interaction
By means of technical and spatial support for communication, a person can be reached to an influenceable
extent for purposes of task handling and coordination This support is intertwined with negotiating duties and
rights of roles, including values, so that reciprocal reliability can be developed.
4 Purpose-orientated information exchange for facilitating mental work
To support task handling, information is purposefully exchanged via technical means, updated, kept available,
and minimized. This implies technical linking of information and the emergence of personal profiles that must
be visible and exchanged in compliance with privacy regulations.
5 Balance of effort and experienced benefit by organizational structuring of tasks
Organizational structuring of tasks supports a proportional balance between individuals’ effort and
experienced benefit. Tasks are assigned to people, pooled, and technically supported in a way that makes sense
and is fun for people. Tasks comply with individuals’ technical, social, and physical competencies while also
supporting health. Thus, a sustaining balance of efforts and personal benefits is pursued.
6 Compatibility between requirements, development of competencies, and the system’s features
Technical and organizational features of the system are continuously adjusted to work with each other. Within
clarified limits, they meet outside requirements in a way that is based on the development of competencies and
proactive help for dealing with changing challenges.
7 Efficiency-oriented allocation of tasks for pursuing holistic goals
By appropriate sequencing, integration, and distribution of tasks between humans and technology, seamless
collaboration is supported. Unnecessary steps or wasting resources is avoided. If needed, an increase of efficiency
can be realized.
8 Supportive technology and resources for productive and flawless work
Technology and further resources support work and collaboration and consider the intertwining of criteria,
such as technology acceptance, usability and accessibility for different users, avoiding consequences of mistakes
and of misuse, security, and constant updating.
Note. A more explicit description with examples is available for each heuristic
(https://hi4.iaw.ruhr-uni-bochum.de/#!/manual) (From Herrmann et al. 2021, p. 15)
177