Advanced Ubiquitous Monitoring Services for Workers in Automated Production Environments

Abstract

In so-called “smart factories”, overseeing highly automated manufacturing processes and preparing for human interventions are increasingly important tasks for production staf. In this paper, we outline our recent eforts towards eficient ubiquitous monitoring solutions for workers. We briefly introduce the overall architecture of our system and its current mobile and wearable app prototypes. Based on these buildings blocks, we present several ideas for advanced monitoring services such as context-aware and personalized notifications, worker-to-worker messaging, and decision-making support. CCS Concepts: • Human-centered computing → Ubiquitous and mobile devices. Additional Key Words and Phrases: smart factory, worker, monitoring, intervention

1 INTRODUCTION

Starting in the 18th century, factories always have been workplaces heavily affected by technological advances and continuous automation efforts. Following the industrial utilization of steam power, electrical energy, and computers and electronics, we currently observe a fourth industrial revolution characterized by networked machinery, smart devices, and AI-controlled processes - with major impact on the roles of factory staff, again.

The majority of previous HCI-related research in the context of such “smart factories” has either focused on advanced monitoring approaches for operators (cf. [ 12 ]) or assistive services for workers in concrete tasks (such as maintenance or assembly, cf. [ 10 ]). In contrast, we study worker-oriented approaches for keeping track of automated production processes and preparing for human interventions. Despite the continuously increasing level of automation, such interventions by human experts are required either in cases of failures (e.g., material accumulation) or as preventive measures to ensure smooth processes (e.g., refilling material early). Our goal is to create efficient monitoring apps for workers which allow for unobtrusively overviewing automated manufacturing processes, advanced decision-making, and reduced reaction times for interventions. a cloud platform (middle) for storage,

In this paper, we outline our recent work on a mobile and wearable monitoring system for production workers. We give an overview of our system’s architecture and current prototype apps. Furthermore, we identify several promising extensions and future research directions for eficiently monitoring and interacting with automated production processes.

2 MOBILE AND WEARABLE MONITORING SYSTEM

This section briefly introduces the overall system architecture of our current monitoring system and presents our two recent prototypes of monitoring apps for workers. We developed this system in cooperation with Geberit, a globally active manufacturer of sanitary products, and M&F Engineering AG, a software specialist for Industry 4.0 solutions.

2.1 Architecture

Figure 1 depicts the architecture of our current monitoring system. Major components are central building blocks of the lfexiot system (https://fle xiot.ch), a flexible cloud-based toolkit for IIoT (Industrial Internet of Things) applications. In our setup, machinery data is transmitted continuously through the OPC UA protocol [ 9 ] to the flexiot cloud for storage, analysis and secure access. In-depth remote monitoring is supported in Web browsers through a custom Web app. Customized data interfaces allow for providing client apps with machinery data. Using an extensible rule engine, we defined various rules for deriving high-level information from the machinery data and detect incidents. Using Firebase Cloud Messaging, the client apps are notified about incidents.

2.2 Smart Device Applications

We created two functional prototypes of mobile and wearable monitoring apps co-designed with production workers [ 3 ]. The mobile app (Figure 2, right) is a Web app built with Angular, the wearable app (Figure 2, left) is written for the Wear OS platform. Both apps provide the same functionality: Workers can subscribe for incidents of specific machines and then receive notifications about failures and warnings regarding these machines. Having resolved an incident, the workers are prompted for the reason of the failure as well as the measure for successful troubleshooting.

ADVANCED UBIQUITOUS MONITORING

In this section, we summarize several ideas for expanding our monitoring system by advanced features. 3.1

Context-Aware Notifications

Currently, notifications on incidents are sent to devices which subscribed to respective machines. However, gathering additional context information (e.g., a worker’s location or vital parameters) would allow the system to prioritize workers when sending out incident notifications. For example, notifying the worker closest to a failed machine could shorten walking paths and thus reduce reaction times.

While performing an intervention or maintenance task, getting interrupted by additional incident notifications not only increases the time to complete the tasks but also the error rate and annoyance [ 2 ]. To detect whether a worker is already completing a task could be realized by checking if an unresolved notification has been sent to the worker. Furthermore, acceleration sensors and gyroscopes of the wearable devices could be used to detect current activities performed by a worker [ 8 ]. Modern smartwatches would even allow to monitor vital parameters and stress, subsequently, which could be used as an additional data source to decide which worker a notification should be sent to. Consequently, context aware messaging could result in faster troubleshooting. 3.2

Personalization

Context-aware messaging could be further improved by also considering worker knowledge. Based on their experience, workers could be provided with diferent information about an incident [ 5 ]. For experienced workers, the name of a machine with an error code may be suficient while a new or inexperienced worker would need additional information. This additional information could include displaying a short text with an error description or providing a compact manual with suggestions on how to fix the problem. Such a personalized system could also be designed to support the learning process of new workers. For this purpose, workers choose their own taxonomy levels based on their experience.

3 Individual support might range from educational tutorial videos to short text instructions to no support beyond the actual notification at all. 3.3

Worker-to-Worker Communication

Besides rule-triggered notifications on incidents, mobile and wearable devices in a smart factory might also be used for worker-initiated messages. For example, integrating a customised or third-party instant messaging service would enable workers to communicate via free-text messages. However, reading and writing lengthy messages on mobile and wearable devices is cumbersome and would interfere with work tasks.

More eficient alternatives include predefined short messages which can be sent out to co-workers, in particular by the push of a button on a smartwatch. An example is informing workers responsible for the same machines about leaving for a break. An alternative approach to speed up the process of entering text are audio or speech-to-text messages [ 11 ]. This would provide greater freedom in composing the message content, while also not being as time-consuming as writing a text message. However, recording and listening to audio messages might not be feasible in noisy production environments. Another approach are predefined forms to fill in a summary of incidents for each machine at the end of a shift. Workers in the subsequent shift would then be able to browse an incident report by scanning a QR code at the respective machine. 3.4

Data Collection

The use of smartphones and smartwatches allows collecting and analyzing various contextual parameters for evaluating and improving respective work processes. For example, workers’ paths could be analyzed and optimized through suitable indoor localization techniques. This would also support the idea of context-aware messaging described in the corresponding section. However, instead of such a tracking of workers, the evaluation of built-in pedometers could be suficient to make statements about the movement eficiency of the workers and assess the impact of such mobile monitoring approaches.

Furthermore, modern smartwatches feature various health-related services, e.g. detecting and measuring its wearer’s stress. Since employee wellbeing at work has a positive impact on a workers performance [ 4 ], the evaluation of related data could provide inputs to a team leader as to which employees sufer from stress levels above average and their work load needs to be reduced, respectively. 3.5

Decision-Making Support

Using our current app prototypes (Figure 2), workers can capture the cause of failure after having resolved an incident. To keep data entering both eficient and simple, they are presented three options why the incident could have occurred, from which they choose the most suitable. Following this dialogue, workers are shown a list with possible solutions to the selected option from which they choose the one that was used to resolve the respective incident.

After an initial phase of collecting these data, a recommendation system could be implemented based on this database of “worker experience”. We assume that a simple solution would be suficient as no user preferences need to be taken in account as with more complex solutions [ 1 ]. Together with the incident notification, actions could be proposed or even particular workers with the corresponding knowledge and expertise could be targeted. This could be further expanded by adding support during an intervention based on worker experience previously captured. Especially new or inexperienced workers could benefit from such an extension which might positively afect their learning process. 4 3.6

Alternative and Custom Hardware

While discussing and co-designing suitable monitoring approaches with production staf, our workshop participants favored smartwatches and smartphones over data glasses and tablets [ 3 ]. Blazevski and Haslwanter [ 5 ], in contrast, found in their evaluation of smartphones and tablets in an assembly line, that the participants preferred the tablet over the smartphone.

Examples for custom hardware in a related monitoring scenario can be found intensive care. Cobus et al. [ 7 ] developed a custom head mounted display, based on the frame of safety glasses. They attached a micro controller and LEDs to display diferent colors and blinking patterns for informing ICU nurses about incidents through peripheral light. Data glasses were later used by Cobus, Busse, and Heuten [ 6 ] in intensive care to improve their first prototype. The authors used Google Glasses for developing a prototype with sound and peripheral light alerts with three diferent urgency levels. For our monitoring system, we see related opportunities for more lightweight and unobtrusive monitoring approaches. One example is a vibrating wristband with the purpose of alarming in case of a machine error. Warning lights are already installed for each machine in the production hall but are hardly noticeable when one is doing a manual task at another machine. 4

CONCLUSION AND OUTLOOK

In this paper, we briefly introduced our ongoing work on mobile and wearable monitoring services for factory workers. Based on our existing monitoring system and two prototypes with core functionality such as receiving notifications on machine incidents, we presented several promising extensions and research directions for advancing such ubiquitous monitoring services. In our future work, we will prioritize these ideas together with industry partners and start designing and prototyping promising extensions in a worker-centered research approach.

References

[1] Xavier Amatriain and Justin Basilico . 2016 . Past, Present, and Future of Recommender Systems: An Industry Perspective . In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) ( RecSys '16) . Association for Computing Machinery , New York, NY, USA, 211 - 214 . https://doi.org/10.1145/2959100.2959144

[2] Brian P. Bailey and Joseph A. Konstan . 2006 . On the need for attention-aware systems: Measuring efects of interruption on task performance, error rate, and afective state . Computers in Human Behavior 22 , 4 ( 2006 ), 685 - 708 . https://doi.org/10.1016/j.chb. 2005 . 12 .009 Attention aware systems.

[3] Matthias Baldauf , Sebastian Müller , Arne Seeliger, Tobias Küng, Andreas Michel, and Werner Züllig . 2021 . Human Interventions in the Smart Factory - A Case Study on Co-Designing Mobile and Wearable Monitoring Systems with Manufacturing Staf (CHI EA '21). Association for Computing Machinery , New York, NY, USA. Forthcoming.

[4] Nicole Renee Baptiste . 2008 . Tightening the link between employee wellbeing at work and performance . Management Decision 46 , 2 (March 2008 ), 284 - 309 . https://doi.org/10.1108/00251740810854168

[5] Boban Blazevski and Jean D. Hallewell Haslwanter . 2017 . User-Centered Development of a System to Support Assembly Line Worker . In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) ( MobileHCI '17) . Association for Computing Machinery , New York, NY, USA, Article 57 , 7 pages. https://doi.org/10.1145/3098279.3119840

[6] Vanessa Cobus , Stefen Busse, and Wilko Heuten . 2019 . Glass++ Evaluating Multimodal Alarms on Google Glass . In Proceedings of Mensch Und Computer 2019 (Hamburg, Germany) (MuC'19). Association for Computing Machinery , New York, NY, USA, 795 - 799 . https://doi.org/10.1145/ 3340764.3344910

[7] Vanessa Cobus , Hannah Meyer, Swamy Ananthanarayan, Susanne Boll, and Wilko Heuten . 2018 . Towards Reducing Alarm Fatigue: Peripheral Light Pattern Design for Critical Care Alarms . In Proceedings of the 10th Nordic Conference on Human-Computer Interaction (Oslo, Norway) ( NordiCHI '18) . Association for Computing Machinery , New York, NY, USA, 654 - 663 . https://doi.org/10.1145/3240167.3240218

[8] Alberto Vianna Dias da Silva , Lucas Borges, and Vaninha Vieira . 2018 . CDNA: A Context-Aware Notification System for Driver Interruption . In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems (Belém, Brazil) (IHC 2018 ). Association for Computing Machinery , New York, NY, USA, Article 11 , 8 pages. https://doi.org/10.1145/3274192.3274203

[9] OPC Foundation . 2020 . OPC Unified Architecture . https://opcfoundation.org/about/opc-technologies/opc-ua/. Accessed: 2021 -04-06.

[10] Christian Krupitzer , Sebastian Müller , Veronika Lesch, Marwin Züfle, Janick Edinger, Alexander Lemken, Dominik Schäfer, Samuel Kounev, and Christian Becker . 2020 . A Survey on Human Machine Interaction in Industry 4.0 . arXiv: 2002 . 01025 [cs .HC]

[11] Sherry Ruan , Jacob O. Wobbrock , Kenny Liou, Andrew Ng , and James A. Landay . 2018 . Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones . Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1 , 4 , Article 159 ( Jan . 2018 ), 23 pages. https://doi.org/10.1145/3161187

[12] Fangfang Zhou , Xiaoru Lin , Chang Liu, Ying Zhao , Panpan Xu, Liu Ren, Tingmin Xue, and Lei Ren . 2019 . A survey of visualization for smart manufacturing . Journal of Visualization 22 , 2 ( 2019 ), 419 - 435 . https://doi.org/10.1007/s12650-018-0530-2