Time Navigation in a Virtual Environment using Tangible Interactions: application to the domain of History of Science and Technology Pierre Mahieuxa , Sébastien Kubickia , Sylvain Laubéb and Ronan Querreca a Lab-STICC, CNRS, ENIB, Technopôle Brest-Iroise, 29238 Brest Cedex 3, France b Centre François Viète, Université de Bretagne Occidentale, 20 rue Duquesne, 29200 Brest, France Abstract The History of Science and Technology domain studies and describes the evolutions of human technical activities. Historians use new technologies such as Virtual Reality to reconstruct and expose the results of their research. In order to observe the evolutions of the studied systems inside a Virtual Environment, the user must be able to navigate through time. We propose a model of time based on two scales and to use tangible interactions to navigate on these two timescales. Keywords Models, Interaction paradigm, Tangible User Interface, Virtual Reality, istory of Science and Technology, Cultural Heritage 1. Introduction In the History of Science and Technology (HST) domain, Virtual Reality (VR) has notably been used to model and represent in 3D technical and industrial systems within which we find human activities [1]. Let’s consider for example the raising and lowering of a drawbridge inside a harbor, the manufacturing of ship parts inside an arsenal or the routing of resources necessary for the operating of an isolated mine. In the domain of HST, these 3D models are mainly used for cultural mediation and the archiving of these systems. However, little research using these tools is interested in the representation of human activities around these technical devices. To describe the artifacts1 and the activities, historians use ontologies such as CIDOC-CRM 2 and Dolce [2]. These ontologies allow to represent and describe the flow of the activities as procedures3 (e.g., the opening of a bridge) and period4 (e.g., the 16𝑡ℎ century). On the one hand VR, by immersing a user in a Virtual Environment (VE), offers the possibility to visualise and to take part in the realisation of an activity. On the other hand, to allow Proceedings of ETIS 2020, November 16–20, 2020, Siena, Italy Envelope-Open mahieux@enib.fr (P. Mahieux); kubicki@enib.fr (S. Kubicki); sylvain.laube@univ-brest.fr (S. Laubé); querrec@enib.fr (R. Querrec) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings CEUR Workshop Proceedings (CEUR-WS.org) http://ceur-ws.org ISSN 1613-0073 1 Artifact: human product that has a defined function and the ability to perform this function 2 http://www.cidoc-crm.org/ 3 Procedure: sequence of action allowing the accomplishment of a task 4 Period: a coherent set of phenomenons and cultural events in time. Source: CIDOC-CRM navigation between the different periods, some research works have aimed at defining new interactions metaphors (e.g., using time portals [3]). As part of our work, by relying on the theoretical benefits of the Tangible User Inter- faces (TUI) [4, 5], we hypothesise that using a TUI would improve temporal navigation and the comprehension of the temporal evolutions of the activities represented inside a VE. For our first demonstrator we use a Tangible Interactive Tabletop (TIT) allowing, first, to support the interactions with a tangible object and also to show a complementary semantic view of the VE (e.g., maps, diagrams). Our work, therefore, aims to answer these two questions:1) Which metaphors of tangible interactions could we offer so that a user could navigate through time in a virtual environment? 2) Which tangible interactors to use to realise these interactions? 2. Navigation through time In order to formalise human activities asso- ciated with socio-technical systems Laubé et al. [6] described ANY-Artefact, an ontology based on CIDOC-CRM 2 and Dolce [2]. It is used to represent human activities5 , actors6 and knowledge7 . However, ANY-Artefact does not allow the representation nor the re- alisation of these activities inside a VE. To do so Querrec et al. presented MASCARET [7], a meta-model for representing human activities inside a VE. However, there is no notion of temporality in this model. This is why we will rely on these two models to represent the tem- poral aspect of human activities represented in VR. 2.1. A model of time In order to represent time inside an activity, we first propose to align the notions of ac- Figure 1: Model of the representation of tivity between ANY-Artefact and MASCARET. time (notions of P e r i o d , P r o c e d u r e We then add the notions of P e r i o d , P r o c e d u r e and D u r a t i o n ) inside an activity, and D u r a t i o n of action. We present our propo- based on an alignment between sition with the figure 1. ANY-Artefact [6] and MASCARET [7]. In this figure, we represent a P e r i o d (blue frame on the figure 1) which contains the activities defined according to ANY-Artefact (green 5 Activity: aggregation of the concepts of knowledge, actor and artifact to model human cultural and industrial practices. 6 Actor: the person or the virtual character which has a specific function or position. 7 Knowledge: the set of technical knowledge at a given time. oval on the figure 1) and MASCARET (orange activity diagram on the figure 1). According to ANY-Artefact, an activity is composed of knowledge directly associated with the notion of P e r i o d , artifacts and actors (respectively associated with the notions of resources and roles in MASCARET ). These associations are shown in the figure 1 by the dashed lines. We add the notion of D u r a t i o n of action, noted Δ𝑡𝑎𝑐𝑡𝑖𝑜𝑛 , as well as the notion of P r o c e d u r e which we define as a succession of actions. The D u r a t i o n of a P r o c e d u r e , noted Δ𝑡𝑝𝑟𝑜𝑐𝑒𝑑𝑢𝑟𝑒 , is equal to the sum of the D u r a t i o n of every action in the P r o c e d u r e . Finally, in order to navigate through time we propose to create two timescales: the P r o c e d u r e s c a l e associated with the temporality of the P r o c e d u r e s and the P e r i o d s c a l e linked to the evolution of knowledge (red elements on the figure 1). This model is the first step in our research works in order to offer a set of interaction paradigms allowing temporal navigation in a VE. 2.2. Design workshop We organised a design workshop with the aim of proposing tangible interactions and interactors to navigate through time. The workshop took place on an afternoon and lasted for 4 hours. We split the workshop in 2 parts, first a brainstorm session whose objective was to propose ways of representing time in a tangible way, then a prototyping session during which the participants had to propose low-fidelity prototypes. The participants were 17, 11 of them software engineers, 2 are historians, 2 are psychologists, 1 is a designer and 1 is a mechatronics engineer. 2.2.1. Brainstorm During the brainstorm the participants were asked how to represent time in a tangible way. From the 20 distinct answers to this question 13 were inspired from everyday objects (e.g.: hourglass, candle ...), 4 from cultural references (e.g.: flux capacitor from the movie ”Back to the future”, mayan calendars ...) and 3 were metaphorical representations (e.g.: color gradient ...). 2.2.2. Prototypes During the prototyping phase the participants could work in groups or on their own. A total of 11 prototypes were proposed, we classified them in 3 categories according to the interactions they are proposing or their shapes. Multi-part interactors Multi-part interactors are composed of different parts that can be completely in- dependent or attached to one another. The interactions are controlled by the positions and the rotations of each part be it relative or independent to one another. For example the interactor in the figure 2 is composed of 2 rotating parts. The user can navigate between consecutive periods by rotating the 2 discs in the same Figure 2: Picture of a prototype with 2 direction (e.g.: clockwise to go to the next period). If rotating parts. the user turns the discs in opposite directions he will go to the parent period (left part in anti-clockwise rotation and right part in clockwise rotation) or to a child period (left part in clockwise rotation and right part in anti-clockwise rotation). This interactor uses the same movement as the rotation of the clock hands to navigate through time. Multi-function interactors The multi-function interactors all implements multiple interac- tions by using the position of the interactor as well as multiple buttons. Figure 3: Pictures of the polyhedron prototype. Top: view from the top. Bottom: view of one side showing the buttons. The augmented polyhedron prototype in figure 3 uses a screen to display different information to the user. When tilted on its side the interactor changes mode and can then be used to either change period or modify the speed of time. Because the polyhedron has 4 faces on which it can be tilted it is able to provide 4 different modes. The buttons on its sides can then be dynamically modified depending on the selected mode. Affordant interactors Affordant interactors’ shapes are inspired by everyday objects related to time such as hourglass or clock. The interactions are controlled by using the object as if they were the real objects they take inspiration from. In the figure 4 we see an affordant interactor inspired by clock. The user can navigate between periods by moving the bigger clock hand and the smaller clock hand can be manipulated to change the speed of time. This prototype also proposed to add a button in the middle of the clock to switch between different modes. In the future we will draw inspiration from these 11 Figure 4: Picture of a prototype in- eleven low-fidelity prototypes to propose one or more spired by clocks. tangible interactors allowing a user to navigate through time on the two timescales we presented. 3. Implementation In parallel with our work on the model of time and the workshop we implemented a prototype that we will use as a base for our future works. However at the moment this prototype is a proof of concept and does not consider the results of the design workshop. The prototype uses a Tangible Interactive Tabletop (TIT) and a CAVE 8 and is shown in the figure 5. It allows the user to explore the environment in a first-person view and to have a global point of view with the TIT. The TIT is placed outside the CAVE to allow another user to control the temporal naviga- Figure 5: Picture of our prototype in which a tion while the first user is immersed in the user is immersed inside the virtual en- environment. Using multiple displays has vironment representing the Pont Na- been shown to have significant advantages tional in Brest, France between 1861 compared to using a single display setup [8]. and 1944. Inside our virtual environment, we have mod- eled a bridge crossing the Penfeld’s mouth in Brest, France. Because Brest is a military har- bour and the Penfeld is part of the arsenal, the bridge must move to allow the passage of ships. Over time different bridges using different technologies were built. Our prototype currently instantiates two of them: the Pont National (swing bridge, built in 1861 and destroyed in 1944 by Allied bombardment) and the Pont de Recouvrance (vertical-lift bridge, built in 1954) and their opening (respectively lifting) process. Each of these bridges correspond to a P e r i o d . The user can select which bridge (namely P e r i o d ) he wants to observe, trigger the opening P r o c e d u r e and control their course. First, we integrated the 3D models of both bridges in the environment and we then imple- mented their respective opening P r o c e d u r e s . We also developed the interface associated with the TIT (cf. figure 6). The interface is made up of three interaction areas: 1) an aerial view of the environment (in our case the mouth of the Penfeld river), 2) the list of the P e r i o d s generated from our model and 3) the activity diagram associated with the P r o c e d u r e running in the VE (cf., areas 1, 2 and 3 of the figure 6). In order to allow the users to navigate Figure 6: Picture of our prototype’s interface. 8 CAVE: a system using projectors and stereoscopic effects to immerse a user in a virtual environment through time on the timescale we defined ear- lier (P e r i o d s c a l e and P r o c e d u r e s c a l e ) we implemented interaction paradigms based on two metaphors: the time portal and the speedometer. The selection of the observed P e r i o d (which is based on the time portal metaphor) allows the user to control the artifacts he sees (e.g., the Pont de Recouvrance) and the realisable P r o c e d u r e s (e.g., the lifting of the bridge). To do so, the user selects the P e r i o d by positioning the tangible interactor on one of the elements of the P e r i o d list displayed on the area n°2 of the interface. The P r o c e d u r e (e.g., opening the Pont National bridge) can be controlled in two ways: contin- uous and discrete. First, the continuous mode, based on the metaphor of the speedometer, allows to control the speed at which time is passing by. This is controlled by the angle of the interactor beforehand placed on the area n°1 of the TIT’s interface. This navigation mode corresponds by analogy to the fast forward mode in the control of a movie. The discrete mode, based on the metaphor or the time portal, lets the user control the P r o c e d u r e step by step by positioning the interactor on an action of the activity diagram shown on the area n°3 of the interface. This mode is similar to a chapter selection while watching a movie. 4. Conclusion & Future Works The goal of our work is to propose new tangible interactions paradigms to manipulate time inside a Virtual Environment representing human activities. The first step in our work is to model time in technical activities. Our model is based on the ontology ANY-Artefact and the meta-model MASCARET. We add the notions of P e r i o d , P r o c e d u r e and D u r a t i o n . These three notions allow the user to navigate through time on two different scales. We organised a design workshop whose objective was to propose tangible interactions and tangible interactors to implement the different functionalities previously mentioned. During this design workshop 11 low-fidelity interactors prototypes were designed, we plan to draw inspiration from them to design our interactor. In parallel we have implemented a prototype using a Tangible Interactive Tabletop, a generic interactor and, a CAVE. This demonstrator allows the user to navigate according to the two timescales that our model describes. The next step of our work is to implement an interactor taking inspiration in the result of the design workshop. We will then evaluate our proposition in 2 conditions, first in a controlled environment inside our laboratory and in a second time inside a museum. References [1] J.-B. Barreau, F. Nouviale, R. Gaugne, Y. Bernard, S. Llinares, V. Gouranton, An Immersive Virtual Sailing on the 18th-Century Ship Le Boullongne, Presence: Teleoperators and Virtual Environments 24 (2015) 201–219. URL: https://doi.org/10.1162/PRES{_}a{_}00231. doi:1 0 . 1 1 6 2 / P R E S _ a _ 0 0 2 3 1 . [2] G. Kassel, A formal ontology of artefacts, Applied Ontology 5 (2010) 223–246. [3] R. Gaugne, F. Nouviale, O. Rioual, A. Chirat, K. Gohon, V. Goupil, M. Toutirais, B. Bossis, V. Gouranton, Evoluson: Walking through an interactive history of music, Presence: Teleoperators and Virtual Environments 26 (2018) 281–296. [4] G. W. Fitzmaurice, Graspable User Interfaces, Ph.D. thesis, University of Toronto, 1996. [5] E. Hornecker, Beyond affordance: Tangibles’ hybrid nature, in: Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction, TEI ’12, ACM, New York, NY, USA, 2012, pp. 175–182. URL: http://doi.acm.org/10.1145/2148131.2148168. doi:1 0 . 1 1 4 5 / 2 1 4 8 1 3 1 . 2 1 4 8 1 6 8 . [6] S. Laube, S. Garlatti, R. Querrec, Histoire des paysages culturels industriels et humanités numériques : le modèle d’activité humaine ANY-ARTEFACT, in: “Capitalismo en el de- sierto: Materialidades, población y territorios en Atacama (ss. XIX-XXI)”, III Jornada de Antropología e Historia, Instituto de Arqueología y Antropología, Universidad Católica del Norte Centre de recherche et documentation sur les Amériques CREDA UMR7227 Labo- ratorio de Etnografía (Le), Departamento de Antropología, Universidad de Chile Escuela de Historia, Universidad Academia de Humanismo Cristiano, San Pedro d’Atacama, Chile, 2017. URL: https://hal.archives-ouvertes.fr/hal-01950651. [7] R. Querrec, P. Vallejo, C. Buche, MASCARET: creating virtual learning environments from system modelling, in: The Engineering Reality of Virtual Reality 2013, volume 8649, International Society for Optics and Photonics, 2013, p. 864904. [8] A. Kunert, T. Weissker, B. Froehlich, A. Kulik, Multi-Window 3D Interaction for Collabora- tive Virtual Reality, IEEE Transactions on Visualization and Computer Graphics (2019) 1. doi:1 0 . 1 1 0 9 / T V C G . 2 0 1 9 . 2 9 1 4 6 7 7 .