In the History of Science and Technology (HST) domain, Virtual Reality (VR) has notably been used to model and represent in 3D technical and industrial systems within which we find human activities [1]. Let's consider for example the raising and lowering of a drawbridge inside a harbor, the manufacturing of ship parts inside an arsenal or the routing of resources necessary for the operating of an isolated mine. In the domain of HST, these 3D models are mainly used for cultural mediation and the archiving of these systems. However, little research using these tools is interested in the representation of human activities around these technical devices. To describe the artifacts 1 and the activities, historians use ontologies such as CIDOC-CRM 2 and Dolce [2]. These ontologies allow to represent and describe the flow of the activities as procedures 3 (e.g., the opening of a bridge) and period 4 (e.g., the 16 𝑡ℎ century).
On the one hand VR, by immersing a user in a Virtual Environment (VE), offers the possibility to visualise and to take part in the realisation of an activity. On the other hand, to allow navigation between the different periods, some research works have aimed at defining new interactions metaphors (e.g., using time portals [3]).
As part of our work, by relying on the theoretical benefits of the Tangible User Interfaces (TUI) [4,5], we hypothesise that using a TUI would improve temporal navigation and the comprehension of the temporal evolutions of the activities represented inside a VE. For our first demonstrator we use a Tangible Interactive Tabletop (TIT) allowing, first, to support the interactions with a tangible object and also to show a complementary semantic view of the VE (e.g., maps, diagrams).
Our work, therefore, aims to answer these two questions:1) Which metaphors of tangible interactions could we offer so that a user could navigate through time in a virtual environment? 2) Which tangible interactors to use to realise these interactions? ) inside an activity, based on an alignment between ANY-Artefact [6] and MASCARET [7].
In order to formalise human activities associated with socio-technical systems Laubé et al. [6] described ANY-Artefact, an ontology based on CIDOC-CRM 2 and Dolce [2]. It is used to represent human activities 5 , actors6 and knowledge 7 . However, ANY-Artefact does not allow the representation nor the realisation of these activities inside a VE. To do so Querrec et al. presented MASCARET [7], a meta-model for representing human activities inside a VE. However, there is no notion of temporality in this model. This is why we will rely on these two models to represent the temporal aspect of human activities represented in VR.
In order to represent time inside an activity, we first propose to align the notions of activity between ANY-Artefact and MASCARET.
We then add the notions of P e r i o d , P r o c e d u r e and D u r a t i o n of action. We present our proposition with the figure 1.
In this figure, we represent a P e r i o d (blue frame on the figure 1) which contains the activities defined according to ANY-Artefact (green oval on the figure 1) and MASCARET (orange activity diagram on the figure 1). According to ANY-Artefact, an activity is composed of knowledge directly associated with the notion of P e r i o d , artifacts and actors (respectively associated with the notions of resources and roles in MASCARET ). These associations are shown in the figure 1 by the dashed lines. We add the notion of D u r a t i o n of action, noted Δ𝑡 𝑎𝑐𝑡𝑖𝑜𝑛 , as well as the notion of P r o c e d u r e which we define as a succession of actions. The D u r a t i o n of a P r o c e d u r e , noted Δ𝑡 𝑝𝑟𝑜𝑐𝑒𝑑𝑢𝑟𝑒 , is equal to the sum of the D u r a t i o n of every action in the P r o c e d u r e . Finally, in order to navigate through time we propose to create two timescales: the P r o c e d u r e s c a l e associated with the temporality of the P r o c e d u r e s and the P e r i o d s c a l e linked to the evolution of knowledge (red elements on the figure 1).
This model is the first step in our research works in order to offer a set of interaction paradigms allowing temporal navigation in a VE.
We organised a design workshop with the aim of proposing tangible interactions and interactors to navigate through time. The workshop took place on an afternoon and lasted for 4 hours. We split the workshop in 2 parts, first a brainstorm session whose objective was to propose ways of representing time in a tangible way, then a prototyping session during which the participants had to propose low-fidelity prototypes.
The participants were 17, 11 of them software engineers, 2 are historians, 2 are psychologists, 1 is a designer and 1 is a mechatronics engineer.
During the brainstorm the participants were asked how to represent time in a tangible way.
From the 20 distinct answers to this question 13 were inspired from everyday objects (e.g.: hourglass, candle ...), 4 from cultural references (e.g.: flux capacitor from the movie "Back to the future", mayan calendars ...) and 3 were metaphorical representations (e.g.: color gradient ...).
During the prototyping phase the participants could work in groups or on their own. A total of 11 prototypes were proposed, we classified them in 3 categories according to the interactions they are proposing or their shapes.
The multi-function interactors all implements multiple interactions by using the position of the interactor as well as multiple buttons. The augmented polyhedron prototype in figure 3 uses a screen to display different information to the user. When tilted on its side the interactor changes mode and can then be used to either change period or modify the speed of time. Because the polyhedron has 4 faces on which it can be tilted it is able to provide 4 different modes. The buttons on its sides can then be dynamically modified depending on the selected mode. Affordant interactors Affordant interactors' shapes are inspired by everyday objects related to time such as hourglass or clock. The interactions are controlled by using the object as if they were the real objects they take inspiration from.
In the figure 4 we see an affordant interactor inspired by clock. The user can navigate between periods by moving the bigger clock hand and the smaller clock hand can be manipulated to change the speed of time. This prototype also proposed to add a button in the middle of the clock to switch between different modes.
In the future we will draw inspiration from these 11 eleven low-fidelity prototypes to propose one or more tangible interactors allowing a user to navigate through time on the two timescales we presented. In parallel with our work on the model of time and the workshop we implemented a prototype that we will use as a base for our future works. However at the moment this prototype is a proof of concept and does not consider the results of the design workshop.
The prototype uses a Tangible Interactive Tabletop (TIT) and a CAVE8 and is shown in the figure 5. It allows the user to explore the environment in a first-person view and to have a global point of view with the TIT. The TIT is placed outside the CAVE to allow another user to control the temporal navigation while the first user is immersed in the environment. Using multiple displays has been shown to have significant advantages compared to using a single display setup [8]. Inside our virtual environment, we have modeled a bridge crossing the Penfeld's mouth in Brest, France. Because Brest is a military harbour and the Penfeld is part of the arsenal, the bridge must move to allow the passage of ships. Over time different bridges using different technologies were built. First, we integrated the 3D models of both bridges in the environment and we then implemented their respective opening P r o c e d u r e s . We also developed the interface associated with the TIT (cf. figure 6). The interface is made up of three interaction areas: 1) an aerial view of the environment (in our case the mouth of the Penfeld river), 2) the list of the P e r i o d s generated from our model and 3) the activity diagram associated with the P r o c e d u r e running in the VE (cf., areas 1, 2 and 3 of the figure 6).
In order to allow the users to navigate through time on the timescale we defined earlier (Period s c a l e and P r o c e d u r e s c a l e ) we implemented interaction paradigms based on two metaphors: the time portal and the speedometer. The selection of the observed P e r i o d (which is based on the time portal metaphor) allows the user to control the artifacts he sees (e.g., the Pont de Recouvrance) and the realisable P r o c e d u r e s (e.g., the lifting of the bridge). To do so, the user selects the P e r i o d by positioning the tangible interactor on one of the elements of the P e r i o d list displayed on the area n°2 of the interface.
The P r o c e d u r e (e.g., opening the Pont National bridge) can be controlled in two ways: continuous and discrete. First, the continuous mode, based on the metaphor of the speedometer, allows to control the speed at which time is passing by. This is controlled by the angle of the interactor beforehand placed on the area n°1 of the TIT's interface. This navigation mode corresponds by analogy to the fast forward mode in the control of a movie. The discrete mode, based on the metaphor or the time portal, lets the user control the P r o c e d u r e step by step by positioning the interactor on an action of the activity diagram shown on the area n°3 of the interface. This mode is similar to a chapter selection while watching a movie.
The goal of our work is to propose new tangible interactions paradigms to manipulate time inside a Virtual Environment representing human activities. The first step in our work is to model time in technical activities. Our model is based on the ontology ANY-Artefact and the meta-model MASCARET. We add the notions of P e r i o d , P r o c e d u r e and D u r a t i o n . These three notions allow the user to navigate through time on two different scales.
We organised a design workshop whose objective was to propose tangible interactions and tangible interactors to implement the different functionalities previously mentioned. During this design workshop 11 low-fidelity interactors prototypes were designed, we plan to draw inspiration from them to design our interactor.
In parallel we have implemented a prototype using a Tangible Interactive Tabletop, a generic interactor and, a CAVE. This demonstrator allows the user to navigate according to the two timescales that our model describes.
The next step of our work is to implement an interactor taking inspiration in the result of the design workshop. We will then evaluate our proposition in 2 conditions, first in a controlled environment inside our laboratory and in a second time inside a museum.