The paper gives a systematic view on speech assistants in the eld of technical service of industrial machines. We describe the results of a requirements analysis targeting companion technologies for service technicians and we report on a rst reference implementation. The effectiveness of the approach is evaluated in a diagnosis task scenario and preliminary results of a user study are discussed. knowledge-based diagnosis system intelligent personal assistant speech interaction dialog systems ?Supported by German BMWi Project Grant ZF4172703BZ7 (MARS project) Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Intelligent personal assistants for private use are implemented in many devices nowadays. With their availability in smartphones, watches, and TVs, they diffused into daily life and support users when, for instance, sending text messages, setting reminders, and starting apps or media channels. They simplify the use of existing technology by making it more intuitive and quicker in execution.
In general, an intelligent personal assistant provides a natural language
interface to take requests from the user and perform corresponding actions. Today,
many assistants interact with the user by a speech interface. In research, the
development of (smart) speech assistants was elaborated in many works, for
instance see [
Today's assistance systems for service technicians are often structured like search applications of the 2000er years: Providing simple textual search interfaces the technician need to manually formulate an appropriate query and click through the delivered results. Only a few number of systems provide a kind of semantic interface that is able to return relevant information bits not necessarily matching the original but the intended meaning of the query. Also, the technicians are required to actually hold and touch the assistance system's device. Speech assistants would enable the technicians to intuitively formulate the information request and also free them to touch a keyboard device while doing so. The aim of the present study is an user-centered development of a personal speech assistant for service technicians. 2.2
Norman and Draper [
Even though, the UCD process is well established in the HCI, context and
requirement analyses for speech assistance systems are rare. Dumas et al. [
Our use-context is the domain of the Technical Service, focusing particularly on the service technician. For this reason, we rst introduce the domain of Technical Service of industrial machines and then domain speci c tasks, environmental conditions, and experiences of a service technician. The use-context and rst implications for the development of a speech assistant were analyzed iteratively by three experts from the domains human-computer-interaction and technical services. The analyses of context result in four prototypical personas. We conclude the section with a description of the context-related requirements, that|in the view of experts|a targeted speech assistant need to meet. Figure 1 shows the key points of the analyzing phase in the present study (inner red circle, green area). 3.1
Advanced industrial machinery is one of the most driving domains in today's life. Machinery produces almost all consumer goods with high e ciency. In this case, it touches all aspects of personal life, for instance harvesting machines feeding the food production of animals and humans, paper making and printing machines to produce newspapers, and automotive factories to produce cars and trucks.
However, machinery needs to be safely operated and maintained for its optimal performance. In case of malfunction it need to be brought back to work quickly. Often, the standstill of such machines yield exceptional costs in the area of hundreds of euros and more|per minute. The Technical Service, and namely the service technicians, are responsible for the maintenance, performance optimization, diagnosis and repair of industrial machinery. 3.2
Domain Tasks As an overall goal, the service technician should make sure to maintain the machine state and sometimes even to optimize the machine performance. To archive these goals we distinguish the following sub-tasks: 1. Operation and monitoring of the machine performance 2. Disassembly/assembly of machine components (optimization/repair) 3. Diagnosis of faulty machine behavior 4. Maintenance operations for a machine 5. Documentation of accomplished work for commercial and knowledge management reasons
The present paper focuses on the task of diagnosis including the use of an information system with a guided troubleshooting on the one hand and the work around/in/on/under the machine during the diagnosis on the other hand. The latter involves the use of special tools or working gloves while the former requests two free hands for navigate through the di erent diagnosis steps. As consequences for the development of the speech assistance result the implications of the integration of the speech assistant in a diagnosis interface of a semantic information system on the one hand and the possibility of a hands-free interaction on the other hand.
Environmental Conditions The service technician accomplishes the tasks within changing environmental parameters. Basically, the experts distinguish the following main parameters that a ect the quality and speed of the technicians work: 1. Location The technician is working in a workshop or \in- eld". As consequences for the development of the speech assistant we see: { (Reduced) availability of required tools { (Reduced) support and knowledge provided by on-site colleagues { (Reduced) Internet access for remote communication 2. Supporting Information and Infrastructure considers the availability of information resources in breadth and depth, e.g., technical documentation, diagnostic systems, maintenance plans, communication channels, and mobile devices. As consequences for the development of the speech assistant result we see: { Required infrastructure for providing information resources (cloud systems and storage) { Di erent levels of information can be provided depending on the availability of mobile devices 3. Limited Time may stress the technician to complete the task. This primary limitation in uences the use of tools, the quality of work, and the type of possible tasks. As consequences for the development of the speech assistant result we see: { Speech assistants need to be adaptable with respect to the available time and mental workload of the user. This a ects the query patterns, the possible intents, and the (extensive) answers provided by the assistant. Domain Experience The performance of the overall service task heavily depends on the experience of the technician. In general, it is di cult to classify the individual experience, but some indicators can help to distinguish di erent experience levels: a) The number and type of training sessions completed in recent years and b) The years of work in the particular domain.
Some companies de ne speci c levels of experience based on the type of work the technician can complete individually, e.g., from Level 1 (Technician can safely operate the machine) to Level 4 (Technician can safely diagnose and repair faulty behaviour for which diagnostic knowledge is available) and up to Level 5 (Technician can safely diagnose and repair a previously unknown issue). Consequentially, a speech assistant should support the individual experience level of a service technician. 3.3
Based on the analysis of basic work-related characteristics of a service technician's tasks, environmental conditions, and domain experiences, the experts identi ed four typical personas, that represent di erent prototypes of service technicians. They focus in variations on domain experience and the available time to complete a given task. Thus, the four personas represent a typical crossselection of the domain of Technical Service.
e Brad iTm Brave Carl Conservative
Teresa Thorough Frank Frantic
Experience
In Figure 2, we see a classi cation of the identi ed personas: Brad Brave ("I am happy to learn new stu every day!") is innovation friendly, but a beginner of the domain. He has the motivation to use new things and invests time into the exploration and use of information systems. Carl Conservative ("I cannot spend extra time for this additional stu .") also has not much experience in the eld but has not motivation to invest time, e.g., in the exploration and use of an information system. Frank Frantic ("It is about the bolts not about the apps!") is very experienced in the domain, but shows no interest to explore new tools that possibly may improve his work. Theresa Thorough ("I love to improve every day!") has much experience in the domain and also is very innovation-friendly. She is interested to further improve her personal performance by trying and using advanced tools. 3.4
Following the context analysis, three experts analyzed speci c requirements for developing a speech assistant for service technicians. In summary, 28 requirements were identi ed and clustered in ve categories. Table 1 shows the ve clusters and examples of the corresponding speci c requirements.
In order to proof the relevance of the analyzed requirements, ten independent experts (age: 23 to 43 years, 9 male) from the eld of diagnostic dialogues revisited the 28 requirements and rated the importance on a ve-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).
Table 2 shows the requirement clusters and ratings of the second expert group. Symptom detection was rated as most important, followed by orientation within the dialog, speech recognition, and explanations. Trust in the speech assistant was rated as less important.
In addition to the 28 requirements identi ed by the rst expert group, the second expert group found 11 requirements. These 11 additional requirements t in the above presented requirement clusters and did not changed the importance pattern of the clusters.
In sum, the two expert groups identi ed 40 requirements clustered in ve groups with di erent importance for the development of a speech assistant for service technicians.
The rst conceptional design of the speech assistant targets on the prototypical persona of Brad Brave (see Figure 2), because he is innovation-friendly and invests time into the exploration and use of information systems. Due to his limited working experience (Level 3, i.e., needs guidance), his working performance and experience can be improved by companion technologies. From the identi ed requirements, Brad needs the following: { The assistant works o ine and online, because the availability is required not only in the workshop but also on{ eld with a probably worse internet connection. { Brad often needs both hands to accomplish the service work, thus the assistant should be able to operate hands-free. { The assistant needs to support the following tasks:
Basic information research for technical documentation
Support a diagnosis task by a guided diagnostic dialog
Following these requirements the assistant should be helpful for the remaining personas as well, at least in some situations. In the subsequent section, we introduce a concept of a speech assistant for the service technician Brad Brave, that takes the considerations from above into account. 5 5.1
The competence of a personal speech assistant can be characterized by the collection of intents it understands and can react on. Intents are formulated by the user in natural language and require one or more corresponding actions performed by the assistant. The requirements stated above yield the following main intents: a) Search for information in the available technical documentation. b) Provide diagnostic support for a speci c problem description.
User Interface
Automated
Speech Recognition
Image Recognition action query query
Action Execution
Intent Identification
update use
Context Query Formulation An advanced user interface of an assistant is able to capture the user query in di erent modalities, e.g., speech utterances, video images and basic touch/text input. In the context of this paper, we focus on query formulation using speech utterances. The query needs to be transparently elicited by the system, i.e., the user sees the entered input instantly in the system while actually providing it. This is trivial for keyboard entries but also the text of recognized speech input and captured video images should be displayed to the user for transparent tracability.
Deliver Action The user interface outputs the results of the actions derived by the assistant. In the simplest case, the assistant simply displays the document the user asked for. In the case of a guided diagnosis system, the action module delivers an interactive dialog with the user.
text (sound waves) into text input, so it can be later analyzed by the subsequent intent module.
Intent Identi cation The intent identi cation is responsible for the semantic interpretation of the recognized text. The module tries to identify the request of the user in order to nd an appropriate action for the request. It includes techniques from natural language understanding and question answering. Classic approaches are based on rules and patterns, but recently also statistical learning approaches were introduced.
Action Module Based on the recognized intents an appropriate action is selected. In our scenario, we refer to the tasks that a speech assistant should support (see Section 3.4): Support the documentation research by providing useful information and facts for a given question and the support for diagnostic questionnaires.
Context Update The assistant is able to track the work of the technician and uses this work context to support the intent identi cation and the action selection.
Typical task information elements are the previously stated queries, the selected actions, and the environment of the current use. 5.2 A prototypical implementation of the speech assistant was developed for the existing information system Service Mate (http://www.servicemate.de). Originally, the application serves as an information system for the research and consumption of technical service documentation. For the implementation of the diagnosis capabilities, the existing speech assistant was extended by an interactive speech dialog. The diagnostic dialog now can be started by simply stating "start diagnosis" into the speech input.
Figure 4 depicts an example dialog of the diagnosis system. Here, the malfunction of a bicycle is analyzed by a question{answer dialog. The gure shows a question asking for the wear of the wheel rim with possible answers "visible" and "not visible". The multi-modal interface of the system allows for simply touching the buttons to answer the question, but also the answer can be given by speech input. After answering the question the next best relevant question is asked in order to derive a possible cause for the observed fault of the bike.
The next section outlines the second evaluation phase. Note, that the complete user evaluation is not described in this paper due to space constraints. 6
In order to investigate whether the speech assistant indeed support the service technician, we compared the user experience of the developed speech interaction with the established touch interaction of the system as well as expectations previous the usage. Participants The study was conducted by 26 participants having a mean age M = 31.23 with standard deviation SD = 7.98, 4 females. The participants had medium experience with the information system.
Material and Procedure The empirical study use an exemplary but fully functional service system for a bicycle, i.e., the dBike system. Besides the technical documentation, a circuit diagram, and a 3D model the system also contains diagnosis routines for the most relevant functions of the bike. The experiments were conducted at a bike, a roughedized tablet computer was running the information system with the touch and speech interaction. Additionally, a separate speaker broadcasts the speech instructions. A further notebook was provided for answering the questionnaires of the experiment study. Participants had two tasks: (i) the diagnosis of faults with the gear shift and (ii) the diagnosis of a malfunction of the brake. The bike was specially prepared for each task. For the examination of the bike, the persons need to use the work gloves. Hence, for using the tablet computer, participants had to undress the gloves for touch interaction. In addition, some diagnostic steps request the usage of a special measuring tool.
Half of participants conducted the diagnosis using the speech assistant, the other half was using the touch interaction on the tablet (independent variable: mode of interaction). The groups only di er in the mode of interaction implying that participants using the touch interaction had to move between the bike and the laptop for each diagnostic step and had to undress the gloves for interaction on the one side. On the other side, participants using touch interaction saw pictures with instruction. If participants of the speech interaction need a visualization, they had to move to the touch laptop, too.
Previous to the interaction, all participants provided demographical information. Participants using speech interaction rated the expected usefulness and expected problems of speech interaction for the diagnosis process on a 5-point Likert scale ranging from 1 (not useful) to 5 (very useful), or 1 (many problems) to 5 (no problems), respectively. After their interaction with the speech assistant, we assessed qualitatively positive and negative experiences. While the participants using speech interaction assessed their expectation prior the experiment, participants using touch interaction assessed the expected usefulness and problems of speech interaction after the experiment on a 7-point Likert scale ranging from 1 (not useful) to 7 (very useful), or 1 (many problems) to 7 (no problems), respectively. Other questionnaires were applied, but are not reported in the present paper. 6.2
Prior the interaction with the speech assistant, participants rated speech assistance to support the diagnosis process as rather useful (M = 3.92, SD = .64, range between 1 and 5). The auditive interaction mode and the corresponding free-hand interaction were mentioned as most frequent reasons. But also problems were expected (M = 3.62, SD = .51, range between 1 and 5). Most problems were expected for the speech recognition. Further, some participants expected problems with the comprehension of diagnostic instructions. After the interaction with the speech assistant, most participants liked the hand-free interaction, the ease of use, and the intuitive control commands. On the other side, participants reported negative experience concerning the restricted use of language and the fact that they cannot interrupt the speech assistant when it was speaking.
After the interaction with the touch based assistant, participants also expected speech assistance as rather useful to support the diagnosis process (M = 5.58, SD = 1.31, range between 1 and 7) and as medium problematic (M = 4.83, SD = 1.40, range between 1 and 7). The reasons for their ratings were similar to those of the pre-experimental ratings. Free-hand interaction were mentioned most frequently for a useful application. Speech detection and the comprehension of diagnostic instructions were mentioned as most frequently problems. Further, participants using touch interaction reported positive experiences with pictures supporting the comprehensive of the corresponding diagnosis steps. Undressing the gloves for touch interaction was experienced as negatively. 6.3
The results revealed the potentials of speech assistance in the domain of technical services. Particularly, the possibility of hand-free interaction is expected as useful. However, the results also revealed concerns. The use of auditive guidance might lead to comprehensive problems and pictures support the comprehension of corresponding diagnosis steps. Thus, a multi-modal interaction providing hands-free speech interaction and the visualization of 3D-models might be the best support for service technicians. 7
In this paper we introduced speech assistants in the domain of Technical Service. Based on a user-centered requirements analysis for service technicians, we selected a number of relevant requirements to be implemented in an existing information system. We evaluated the research question, whether and how speech assistants can support a service technician. In the experiment we focused on the diagnosis task. In the future, we are planning to extend the reported user studies to a diverse range of speci c research questions. Particularly, a comparison of speech, touch, and multi-modal interaction is planned for future studies. Further, a eld study including real service technicians is planned.