Digitalization in Smart Factories is allowing virtual, physical asset data, as well as processes data to be connected throughout their lifecycles. Here, digital twins mirror the behaviors of physical assets and can simulate their spatiotemporal statuses. The work systems that employ digital twins have yet to address in-situ information representation to workers and ways to mitigate task information overload. Thus, the key is to only present relevant information when and where it is needed. We propose proxemic interaction patterns, i.e. the distance from the user to the device or between devices, for visualization of this data. Here, we outline how scaling the amount and type of augmented reality visualization could be realized using distance, angle, and orientation of users. We first showcase possible scenarios of how proxemic interaction can support workers in smart factories. We then highlight challenges and opportunities when using proxemic interaction in industrial settings such as manufacturing and warehousing. Finally, we present possible future investigations concerning proxemic interactions in the context of a smart factory. CCS Concepts: • Human-centered computing → Mixed / augmented reality; • Applied computing → Industry and manufacturing.
Data-driven demands of smart factories are creating new opportunities to develop systems and interaction paradigms
based on real-time data access. With this, the concept of a digital twin, i.e., a digital replica of a physical system or
asset, has gained increased emphasis due to the capacity to integrate virtual and physical data of machine processes
and lifecycles [
This challenge can be addressed through innovative means of human-computer interaction (HCI) that carefully consider human capability and functionality in socio-technical spaces. Building on how humans make use of social
Welding Robot #124 E ciency: 256 spots / h Downtime 58min
Show Schematics Start Maintenance
Welding Robot #421 E ciency: 3896 spots / h Downtime 2min
Welding Robot #387 E ciency: 2441 spots / h Downtime 38min
Press #081 Press #079
Welding Robot #588
Welding Robot #583
boundaries for interaction [
In this paper, we propose proxemic interactions in AR for smart factories. In contrast to smart homes or general population settings, smart factories have ever-increasing and complex data flows and operate on a vastly diferent scale reaching in excess of 200, 0002 1. Thus, the scale on which workers need to interact is also diferent. On a macro-scale, workers need to maintain overall production processes; on a micro-scale, each machine and its sub processes need to maintain optimal eficiency. Given this context, we argue that proxemic interactions can help workers maintain a real-time overviews of factory operations by overlaying relevant information as diferent factory spaces are engaged. In a more industry related context, we will explore how proxemic interactions can support workers in their tasks by providing meaningful real-time information using AR.
First, we will map out a possible deployment scenario for proxemic interactions. Then, we will outline the challenges and opportunities in operationalizing such a scenario.
We propose using proxemic interactions to manage the information load presented to production and maintenance
workers. We envision scaling information by displaying more detailed information as the worker maintains close
1https://www.daimler.com/innovation/production/factory-56.html
vicinity and only giving sparse information for processes with greater distance, see Figure 1. We use four proxemic
dimensions that can facilitate such interactions: Movement, Orientation, Distance, and Identity. Movement lets us
understand when a person is walking towards a machine, how quickly (e.g., in the case of an emergency), or when
changing directions. Orientation gives information about the direction a user is facing. Orientation can be inferred by
the positioning of faces and limbs which can in turn suggest diferent postures and gaze direction [
In more typical factories, we see stack lights [
We propose that workers use AR headsets to provide a similar stack light logic and visualization as status indicators. In this context, only the worker or team in charge of a specific machine or process would be shown relevant indicators which can potentially reduce workload as stack lights tend to have salient features that grab attention such as blinking and beeping components when in the red state. We envision these indicators to only be visible when a worker is in view of the machine or process, see Figure 2. As the worker comes close to the error, more specific information will be displayed to assist the worker in correcting the error. Special situations such as emergencies may indeed occur, which would trigger alerts and information displays more exigent in nature and do not depend on the field of view or proximity. 2.2
The real-time virtualization of data in smart factories creates a new space for workers to use data in-situ for task performance. In the past, workers would need to carry or use specialized tools that may be cumbersome or require special tuning, such as a mechanic’s stethoscope or infrared thermometers. In contrast, we imagine a proxemics smart factory where a worker takes advantage of real-time data analysis to visualize detailed information as an AR overlay. Visualized sensor readings from machines that indicate important diagnostic information, such as voltage, heat dissipation, chemical levels, but also aggregated indicators on connected processes, such as the last service date, could be shown in a proxemics-enabled display when in the appropriate area and proper orientation, see Figure 2. 2.3
Getting the most accurate information in-situ is essential given the diversity of data available in the continuous
information flow. As shown in Figure 1, we envision a primary egocentric “Zone 0” that encompasses the user’s identity
as a maintenance worker. This primary zone encapsulates, follows the user, and is also where physical work occurs
in a close-up space where assistive task-specific data is displayed. This zone is instantiated at a consistent and more
user-centric degree of distance that enables the worker to see details of a specific error. Information management in this
stage is critical due to the restricted working space. The Identity dimension is one where information type is pre-filtered
for a specific user’s working tasks [
Summary information alone may often not be enough to complete tasks in a smart factory. Meaningful interaction with proxemics-enabled systems could allow a worker to work more eficiently. Thus, adding interaction capabilities directly into AR visualizations, which scale with distance, adds to the usability of proxemics-enabled systems. This opens a wide range of capabilities, such as displaying additional real-time sensor values, showing reading histories, overlaying schematics. However, more importantly, this can also enable the worker to directly get simulated data after making physical modifications to engage in localized quality assurance processes using real-time data visualizations. 3
Displaying relevant information in-situ into the environment enables the worker to keep an overview of current tasks and to use real-time data as a dynamic feedback assistant to complete tasks much faster than without support. Thus, proxemic interaction will improve eficiency by lowering the worker’s time searching for information and interaction possibilities. Although we present a sketched out design for a proxemics factory in this work, several challenges need to be addressed that we believe will enable this vision. 3.1
Allowing the user to access and work with relevant data in the correct context is key to keeping human workers in-the-loop. Thus, sensor networks to track user position in smart factories are essential to track the worker’s orientation, posture, and gaze to ofer the correct visual assistance. While outdoor tracking has improved massively over the last years, indoor localization is still a highly researched topic [22]. The more stable tracking systems use optical sensors; however, they bring privacy concerns along with them. Regardless of the type, stable tracking is crucial to displaying real-time data in-situ to the worker. 3.2
The current design of the visualization is designed around stack lights. Due to the ubiquity of stack lights, factory
workers are already familiar with their functionality, and adding additional information to them is a logical next
step. However, which information the worker needs and to what detail and functionality should be part of future
investigations. In this next step, we envision visualizations extending beyond one unit in the maintenance process but
then to also assist the user through detailed steps in the repair process [
While Hall [
Proxemic interactions in smart factories require a unique approach due to the data complexity being interfaced with.
The data generated and the diferent user identities accessing and making decisions based on in-situ information will
create an even more complex system of human and digital connectivity. While the fastest way to access data in-situ is
by using an AR display, not all work will require Head-mounted displays with immersive visualization. For some tasks,
tablets or projections can be more efective [
Collaboration is critical to take on complex projects. However, as each worker has their own view and workflow, this
can create conflicts during collaboration. For example, when team members have diferent egocentric “Zone 0” identities,
such as maintenance and production workers, handling conflicts of individual interaction possibilities from disparate
proxemic-aware visualizations will be a unique HCI challenge. As a simple solution to enable collaboration, we propose
that the user with a Zone 0 that has a larger scope may be extended to other users while their individual scopes are
incorporated. Other solutions could be, prioritizing user proximity, creating composite, or merging views [
Smart factories ofer dynamic ways for workers to interact with their surroundings. Displaying relevant information
in-situ for diferent task performance stages enables the worker to keep overviews of task engagement and complete
tasks much faster than in traditional contexts. For proxemic interactions to reach full maturity, HCI investigations will
have to give attention to designing visualizations and interactions for detailed steps in task performance [
Additionally, proper user tracking to enable seamless interactions within proxemic zones will need to be investigated
in ways that mitigate cognitive load and distraction efectively. Rendering discrepancies on diferent devices can also
afect view consistency [
Considerations for diferent cultural groups [
Overall, we see great potential but also specific challenges for proxemic interactions in smart factories. The sheer amount of data generated in smart factories make these potentials particularly valid as information management investigations for specialized human tasks. In this paper, we provided an overview of how proxemics can be used to keep humans in the loop for smart factory processes by showing how meaningful connections can be made between physical infrastructure and complex information spaces that are necessary for operation. This work provides an overview for researchers and industry professionals when considering proxemic interactions in smart factories. [18] Scott S. Snibbe and Hayes S. Rafle. 2009. Social Immersive Media: Pursuing Best Practices for Multi-User Interactive Camera/Projector Exhibits. In
Proceedings of CHI ’09. ACM. https://doi.org/10.1145/1518701.1518920 [19] Miaosen Wang, Sebastian Boring, and Saul Greenberg. 2012. Proxemic Peddler: A Public Advertising Display That Captures and Preserves the
Attention of a Passerby. In Proceedings of PerDis ’12. ACM. https://doi.org/10.1145/2307798.2307801 [20] Robin Welsch, Heiko Hecht, and Christoph von Castell. 2018. Psychopathy and the Regulation of Interpersonal Distance. Clinical Psychological
Science. https://doi.org/10.1177/2167702618788874 [21] Robin Welsch, Christoph von Castell, and Heiko Hecht. 2019. The anisotropy of personal space. PloS one. https://doi.org/10.1371/journal.pone.0217587 [22] Faheem Zafari, Athanasios Gkelias, and Kin K Leung. 2019. A Survey of Indoor Localization Systems and Technologies. IEEE Communications Surveys & Tutorials. https://doi.org/10.1109/COMST.2019.2911558