Every person makes sense of their personal context differently because of their different sets of personal characteristics (intelligence) and behaviour (life choices). However, the machine's understanding of the personal context is radically different from the user's understanding. This limitation is due to the limited definition of the personal context, and the lack of tools to make sense of the personal context. For instance, while the person you are with now can be linked to a name, for people it has more meaning than just a name, e.g., friend and colleague. Additionally, these meanings are not fixed, they may change at any time, and every person can assign additional meaning using different criteria. Thus, effective context recognition requires a complex and dynamic representation of the personal context and the collaboration of the people to fill the cognitive gap of machines.
The addition of human collaboration into the context recognition learning of machines is an important part of supervised machine learning [Vapnik, 2013]. These interactions bring new challenges to the implementation of machine learning algorithms. For instance, humans can be defined as the expert of the supervised algorithms, interacting in an offline fashion by annotating sensor data [Webb, 2003], or the interaction can be directly online, as active learning [Settles, 2009;Hoque and Stankovic, 2012;Hossain et al., 2017]. The human collaboration is important when we are moving into reallife scenarios [Kwapisz et al., 2011].
Other challenges of this collaborative approach are the possibility of overwhelming the humans and the possible differences between the assignment of meaning between people [Chang et al., 2017], thus, making the annotation a personal activity. Then, having humans as the input of the information opens the possibility of human error in the collaboration process [Tourangeau et al., 2000], and this issue is well known in social sciences and psychology, because of response biases in answering self-reports [West and Sinibaldi, 2013], and more importantly, these biases are not well-understood [Freedman et al., 2013].
We propose a novel context model based on the work from [Giunchiglia et al., 2018]. That work focused on ensuring the reliability of annotations, whereas our focus is on improving personal context representations to get closer to work in reallife scenarios. So, we propose to add a more precise representation of personal context that can also work with machine learning algorithms. We formalize a context model based on ontology and use it with the streaming data to have a knowledge representation of context data. This formalization allows moving towards a generic definition of context that can work with existing multi-label machine learning approaches, using a conversion algorithm. Eventually, the last piece of the puzzle will be the design and development of the Streaming System to manage technically the dynamic context data, and it will be organized in the system architecture with modular components for independent development and easy deployment in current cloud environments.
Some examples of our model improvement can be seen compared with our main related work [Giunchiglia et al., 2018;Bontempelli et al., 2020;Zeni et al., 2019]. All of them can take benefit from our novel personal context representation and can use our conversions algorithms to explicitly implement the transformations needed for the machine learning algorithms.
The paper is structured as follows. Section 2 introduces context modeling. Section 3 illustrates our representation of personal context, while we provide the formal representation in Section 4. Then, we show the learning process to transition from our formal representation to machine learning representation in Section 5 and how our formal representation can be converted to a Direct Acyclic Graph (DAG). Finally, Section 6 describes works related to ours, and Section 7 concludes our paper.
When we talk about the context, we concentrate on the context of a person called observer. The observer's context is the representation of a partial view of the world. We describe this context into three main dimensions: the viewpoint, the part-whole relation, and the endurant-perdurant.
Firstly, we have the viewpoint dimension divided as outside viewpoint and observer viewpoint. The outside viewpoint is the view of an ideal observer who can describe everything from a certain point of view. We distinguish this view from the world's static and dynamic properties. In the static property, there are whatever does not change in time, e.g., mountains, buildings, streets. In the dynamic property, there are not only the moving people, but also the moving animals, and facilities in their manifestations, like trains. Then, the observer viewpoint describes how the observer perceives what is around her or him. In this view, we also have the property of being static or dynamic, but it is relative to the movements of the observer.
Secondly, context is a part-whole relation. In our everyday life, when we do things, we are always embedded in the world. From an ontological view, we are part of the whole world. Thus, we call reference context as the element of the outside context with a volume and extension that is large enough to contain all our movements and changes. For instance, the reference context is the city of Trento when the user walks around, or the users' home when they are at home. In turn, our body as a whole has parts (e.g., arms, mind, legs) that are with us all the time, and they define the internal context of the user. The internal context identifies the elements of the user's body at different levels of abstraction. We usually distinguish between physical parts, such as arms, body, fingers, and mental parts, such as mind, memory, emotions. The context as a part-whole relation is divided into reference context and internal context, and both contexts with different dimensions play a role in our life.
Thirdly, context as endurant and perdurant refers to the bigger relation to changes in time or space. Events and actions are perdurants and elements, like me, are endurants.
The context, as viewpoints, defines the reference point from which we construct the context and the context, as partwhole, defines which parts we should consider. So, next, we need to define how we keep track of the context from a quantitative point of view, with a set of quantitative and qualitative measures. Therefore, based on these measures, we introduce the spatio-temporal context.
The spatio-temporal context consists of the temporal and spatial reference context. The former includes dates, times and all the additional notations like weekdays and seasons. The latter contains the world coordinate system. There are various reasons why context should be represented as a spatio-temporal context. First, this is a common representation when we think of the world. Second, any device today can easily retrieve the time and time zones, and the space coordinates (e.g., via GPS). Third, time can be used to measure the changes in all the elements of the world, all evolving at different speeds, thus temporal context allows us to use them together based time. Finally, a lot of data about the spatial reference context and its sub-contexts are available from external sources (e.g., Google Maps, OpenStreetMaps).
The spatio-temporal context, also called objective context, at time t is defined as: The number and type of persons and objects change over time. Hence, we will have a sequence of time-tagged states, namely O = {o 1 , . . . , o n }. We call the sequence O as the streaming context. In the streaming context, within the given reference location, it is easy to compute spatial relations (e.g., near, right, left, in front, far relative to the location) of the different elements among themselves. For instance, the system can compute that the smartphone is in the home building and the smartphone is near the computer.
The spatio-temporal context is also called objective context, since all the relations are computed concerning what is objectively measured, in terms of spatial relations. However, notice that different observers will have different views of the world. For instance, the school building has the function of studyplace from the point of view of a student and is the teacher's workplace. The word function here is used with the precise meaning defined in [Giunchiglia and Fumagalli, 2017;Giunchiglia et al., 2018]. Hence, "the function of an object formalizes the behavior that an object is expected to have" [Giunchiglia and Fumagalli, 2017]. For instance, objects are trains and buildings. The expected behavior may be the purpose of the object (e.g., fridge) or the role of a person (e.g., friend).
The subjective context includes both the objective context elements and the function of persons and objects as seen by the user. Thus, the subjective context at time t is defined as:
where F n (P k ) and F n (O m ) are the functions with respect to a person P k and an object O m , respectively. The number and type of persons and objects, and their functions change over time. The sequence of subjective contexts over time is defined as the subjective streaming context S = {s 1 , . . . , s n }.
In the endurant context [Giunchiglia et al., 2017], its parts are endurants, essentially objects where the spatial extension of their actions is contained by the space defined by the spatial (reference) context. The actions "represents how objects change in time" [Giunchiglia and Fumagalli, 2017]. For instance, running in a park performed by a runner. We also need to represent actions, in particular, the actions that are executed by the endurant me and also by any other elements of the outside dynamic context. Actions can be seen in two ways: (i) actions modeled as processes, namely as sequences of single micro-steps, each of length close to zero; (ii) actions modeled as events, which are often also called perdurants, namely as complete movements which last for a certain duration. Actions as events have key properties, similar to endurants. An event and an action can associate with a set of component sub-events and sub-actions.
Considering the mentioned concepts, the fundamentally different role of space and time should become clear. Whereas the parts of the space context are only used to limit the space where things happen, the parts of time have the main goal to detail how actions get executed. Things get complicated by adding objects, people, and functions as shown in the data representations shown in Table 1.
It is worth noting that each function of a person is associated with a limited set of actions, and the type of action that a person can perform can be considerably limited by knowing his or her function.
The actions apply to me and persons, whereas functions apply to me, other objects and persons. Notice also, that the stored location L is limited by the most specific location that we can compute. This is because the bigger locations are assumed to be static and stored in the system. For events, instead, we store the smallest possible most general event as well as those component actions which are done during a certain period. Thus, for instance, the action/event meeting can have sub-actions such as talking, walking, listening, typing. Table 2 reports the streaming context matrix of Example 1 and shows how the context changes over time.
The streaming context describes the contexts of an observer according to time. To represent each context occurrence, we define a set of notions to build a figure of the current context. We mainly divide the current context into four types of context according to how things compose in space and time. From top to bottom we have the following cases:
• 1L1E (One Location One Event), such as a lecture holds in a classroom;
• 1LME (One Location Multiple Events), such as a sequence of meetings hold in an office, or eat breakfast, lunch and dinner at home;
• 1EMC (One Event Multiple Locations), such as a travel goes from many different places;
• MEMC (Multiple Events Multiple Locations), it is the most complex case, which mixes with the former three cases.
An 1EMC example is shown in Figure 1, which describes the following travel scenario around observer me.
Example 1 In the travel scenario, me is Xiaoyue, she has travel named Travel 1 in the Trentino of Italy from 12:00 to 13:00 of 2th, June 2021. From 12:00 to 12:30 on this day, she takes train 1 from Rovereto to Trento, she sits on seat 1 by herself. From 12:30 to 12:55 on the same day, she walks on Roads 2 from Trento Train Station to Xiaoyue's Home, together with her friend named Haonan. In addition, Xiaoyue talks to Haonan, and Haonan listens to Xiaoyue when they are walking. This scenario involves one event travel and multiple locations.
In general, in the Figure 1 In the rest of this section, we list the attributes of Location, Event, Object, Person and Action, each kind of attributes is represented as a box in the figure. The Location has the following attributes:
• Spatial properties: Coordinates (x i , y i , z i ), Volume (∆x i , ∆y i , ∆y i ) , and InLocation(L i ) shows the super Location for the top level Location;
• Visual properties, namely some properties of Location that can be observed visually;
• Location' functions: F unctionOf (U ) with U ∈ {P 1 , . . . , P k , O 1 , . . . , O M }, which shows location's functions for persons and objects;
• An extra box: including the rest part of the context that happens in the Location.
The Event orders by time and is represented by a box with round corners. Events include actual events and virtual events. An Event can have Sub-Events, the Event and Sub-Event have the following attributes:
• Super Event for the top level event: InEvent(E i ), which shows the super event E i for the current Event;
• Temporal properties: Begin Time -End Time (Date i , T ime i -Date j , T ime j ), which shows the date and time of the start and end of the event;
• An extra box: including the rest part of the context that happens in the Event.
The Object appears in Event and has the following attributes:
• Spatial properties: Coordinates (x i , y i , z i ), In/Far/... (P i /O m /...);
• Visual properties, namely some properties of Object that can be observed visually;
We use the Entity Type Graph (ETG) and the Entity Graph (EG) in ontology to represent the context. ETG is a knowledge graph where nodes are entity types, which are further decorated with data properties. The object properties are presented in the graph representing the relations among the entity types.
In Figure 2, white box nodes are entity types that include data properties with data types, the green box nodes enumerates the values of data property. The object properties are connecting all entity types, represented by diamond nodes with arrows. The inheritance relation in the ETG is represented by an arrow from the super-class to the sub-class, and the subclass inherits all the data properties and object properties of its super-class. The EG populates the entity types and properties defined in the ETG with specific values. It is a data graph where nodes are entities that are connected by object property values representing the relations. Each entity further includes data property values. The streaming context can be viewed as a stream of EGs, in which each EG describes the context at a different time.
We design an EG example as Figure 3 according to the scenario in Example 1. The graph represents the context around "Me", this contains entities shown by nodes, e.g., "Smartphone", "Talking", "Walk". Also, we can see object property values.
AI applications like smart personal assistants provide a service to the users based on their context. The context information is usually not available to the machine, and hence it has to infer the location or the activity of the user from sensor data (e.g., GPS, accelerometer, nearby Bluetooth devices). In our scenario, Xiaoyue is carrying a smartphone that generates a stream of sensor readings, and she annotates the data by answering questions about her context, e.g., "Where are you" and "What are you doing?". The sensor data are aggregated in time windows generating a stream of instances (e.g., the average number of nearby Bluetooth devices in the last 30 minutes). On each incoming instance, the machine decides whether to query the user to acquire the labels. The machine learning technique defines the query strategy, and in the simplest case, the labels are acquired on every instance. belong to i-th concept in H and 0 otherwise. The machine is trained on a stream of examples z t = (x t , y t ) drawn from a ground-truth distribution P (X, Y) that is always consistent with a ground-truth hierarchy H, i.e., if there is an edge from class c i to class c j , then y i = 1 implies y j = 1 and conversely y j = 0 implies y i = 0. The goal of this hierarchical classification tasks is to learn a classifier that recognize well the context on future sensor readings.
The ETG and EG introduced in Section 4 can be used as prior knowledge about the structure of the hierarchy. They encode the available information about the user and the world. Algorithm 1 shows the conversion from ETG and EG to a DAG H. The first step is to convert each entity type (etype) in the ETG as a node in H (lines 3 -5). Second, each entity in EG also becomes a node that is added as a child of the node referring to the etype of the entity (lines 6 -10). The hierarchy encodes the information about the current user, so the Me etype and the corresponding entity (e.g., Xiaoyue entity in Figure 3) are not considered.
The properties of the ETG are grouped in properties that are context depends and properties that are static. The value of the former changes every time the users change their context and, in Figure 2 are Q = {near, use, interact, in, do, happenIn, during, participate}. For instance, if Xiaoyue travel from the city of Trento to Rovereto, the in property will change accordingly. In contrast, the fact that Trento is partOf Italy can be assumed to remain valid even if user's context is changed. This distinction is necessary since the value of context-dependent properties are derived from the output of the context recognition task (e.g., the machine recognizes that the user is in the city of Trento and thus updates the in property accordingly). The object properties that are not contextual are converted as follow. Given a object property p ∈ {isA, partOf, has} that links the etype A to B, then the node referring to etype A becomes a child of the node of etype B. For the other object properties, a new node referring to the property is added as child of the codomain etype node (lines 13 -19). For every object property value i of the prop- erty p linking the entities a and b, a new node c i is added as child of c p (i.e., the node referring to the property P ), and as parent of c b (i.e., the node pointing to the entity b) (lines 20 -25). Finally, all nodes that does not have a parent are connect to the root node and the transitive reduction is applied (lines 27 -30). Figure 4 shows an extract of the DAG resulting from applying Algorithm 1 on EG and ETG presented in Section 4.
Every node in H has a unique identifier that is used to reference back to the ETG and EG. The node name can be translated into a human-readable text that is used to interact with the user. This aspect is left as future work. The concept hierarchy available at the beginning can evolve over time and has to be continually updated. This aspect has been defined as knowledge drift and is addressed in [Bontempelli et al., 2021].
Considering our novel context modelling of the personal context, the context learning, and the architecture presented in this paper, we can show as use cases the papers that individually compartmentalizes examples for these improvements. Current works on context recognition focus on learning the relationship between the input data (sensor readings) and the target concepts (context). The structure of the context is implicitly learned by the implemented machine learning algorithm during the training phase. The new parts are described in Section 3 and compared with related work we can outline the following: 1) the connection between the learned context model; 2) The extension of new dimensions and classification of context, i.e. Internal State, Functions, Actions. The Algorithm 1 bridges the gap between the two modellings allowing context recognition to leverage both machine learning solutions and knowledge graph representation (ETG and EG). For instance, the ETG and EG can generate the questions needed by the machine to interact with the users. For instance, the personal context recognition model shown in [Giunchiglia et al., 2018], showed to be a good approach to increase the accuracy of the context recognition algorithms. Our work described in Section 3 enhance the representation of the personal context with the hope to perform better in real-life scenarios. In practice, existing approaches for context recognition using batch or streaming sensor data [Vaizman et al., 2018;Bontempelli et al., 2020;Zeni et al., 2019] do not leverage on an explicit context modeling. The context representation is implicit in the la-bels used to train the machine learning model. The context formalization introduced in this work can be used to structure the output of these machine learning models according to our representation. Moreover, it can help machine learning approaches to interact with users. For instance, the machine can ask if Haonan is a friend of Xiaoyue since they are walking together. Approaches that use active learning strategies (e.g., [Settles, 2009;Hoque and Stankovic, 2012;Hossain et al., 2017]) can benefit of our representation.
Also, existing frameworks for creating context-aware mobile applications, such as Ferreira et al., do not consider the modelling of the context.
In this paper, we moved forward with a better representation of personal context in real-life environments. We proposed an improved representation of the personal context, adding the internal state, functions, and actions. The learning aspect of our work is the formal definition of an algorithm to transform the streaming input data to ML algorithms. We will put all these components in the system architecture.
In comparison with the work on personal context recognition for human-machine collaboration [Giunchiglia et al., 2018], we have shown an enhancement related to the model representation of the personal context.
Additionally, we have shown how our novel personal context representation can also be leveraged by machine learning algorithms to include prior knowledge about the structure of their output and can be used to drive the interaction with the user. Future work will focus on evaluating the impact of our formalization on an existing approach for fixing mislabeled data when learning the users' contexts [Zeni et al., 2019].
The next step is to propose and implement a modern design of the services related to iLog [Zeni et al., 2014] by a centralised streaming system and linking the personal context data collections with other distributed services of machine learning. This implementation will allow us to test and evaluate our novel context model in near real-life scenarios.
The research conducted by Fausto and Xiaoyue has received funding from the European Union's Horizon 2020 FET Proactive project "WeNet -The Internet of us", grant agreement No 823783.
The research conducted by Marcelo and Andrea has received funding from the "DELPhi -DiscovEring Life Patterns" project funded by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) 2017 -DD n. 1062 del 31.05.2019.