-Nowadays smartphones play a significant role in gathering relevant data about their owners. Micro-devices embedded in Personal Digital Assistants (PDAs) perform a continuous sensing, the phone call lists, PIM (Personal Information Manager), text messages and so on allow to collect and mine data enough for a high-level description of daily activities of a user. This paper proposes an agent able to perform an automated profile annotation by adopting Semantic Web languages. As a proof of concept, the devised agent has been tested in an Ambient Intelligence (AmI) scenario, i.e., a domotic environment where it interacts with its home counterpart to trigger services best matching the user needs. A toy example is presented as case study aiming to better clarify the proposal while an early experimental evaluation is reported to assess its effectiveness.
Keywords—Ambient Intelligence; Agent-based Data Mining; Semantic Web of Things; Home and Building Automation.
Mobile phones are both pervasive and personal –following
the user and having clues about everyday situations– resulting
extremely useful to infer a context. Embedded micro-devices
(accelerometer, digital compass, gyroscope, GPS, microphone
and camera) can be used to extract significant information
about the user: GPS location traces, call and SMS lists,
PIM (Personal Information Management) records including
contacts and calendar, battery charging habits. By leveraging
the smartphone processing capabilities, ever-expanding ways
to investigate behavioral, spatial and temporal dimensions of
the everyday life can be provided. The personal nature of
mobile phones suggest they are well suited for pervasive
computing, but data they are able to collect and process could
be profitably used for a large set of context-aware applications,
like the Ambient Intelligence (AmI) [
This paper presents a smart profiling agent1 which
borrows languages and technologies from the Semantic Web
experience to funnel inarticulate raw individual information
toward a semantically rich glossary. A crawler agent runs
on the user smartphone and performs a multimodal (i.e.,
involving several heterogeneous data sources) and continuous
sensing [
The remainder of the paper is organized as in what follows. Section II contextualizes the overall multi-agent HBA system motivating the proposed approach before presenting both architecture and algorithms of the profiler agent in Section III. The toy example in Section IV acts as a case study while an early experimental evaluation is reported in Section V. Finally, most relevant related work is discussed in Section VI and concluding remarks and future research are in Section VII.
II. SCENARIO: SEMANTIC-BASED HOME AUTOMATION
The user agent proposed in this paper is intended as a
part of a more complex HBA Multi Agent System (MAS) [
The adopted multi-agent system comprised a home mediator agent as well as user and device agents. Each agent adopts the custom service-oriented model sketched in [4, Fig. 4]. Basically, the agent monitors its internal state and inputs; when a significant change occurs, it communicates with the other agents in order to discover suitable services that maximize its utility. The number of both resources/services and agents varied unpredictably (as new users or devices joined or disconnected the system at any time) without redefining the communication paradigm for that.
2RDF (Resource Description Framework) Primer, W3C Recommendation, 10 February 2004, http://www.w3.org/TR/rdf-primer/
3OWL 2 Web Ontology Language, W3C Recommendation, 11 December 2012, http://www.w3.org/TR/owl2-overview/
4See the related project home page
http://sisinflab.poliba.it/swottools/smartbuildingautomation/ for more details.
– The Mediator Agent coordinates the explicit
characterizations of available services, described w.r.t. a reference ontology
modeling the conceptual knowledge for the building
automation problem domain. Furthermore, it acts as a broker in order
to discover the (set of) elementary services that cover (part of)
the request coming from user or device agents.
– The Device Agents are thought to run on advanced devices,
i.e., home appliances with some computational capabilities
and memory availability. Each one can expose one or more
semantic descriptions, i.e., functional profiles to be discovered
by other agents, or alternatively each of them could issue
semantic-based requests to the mediator agent when the device
status changes and then require a home reconfiguration.
– KNX Device Interface Agents support semantic-based
enhancements in case of legacy or elementary appliances, e.g.,
switches, lamps, and so on. In such cases, there is only a static
interaction between agent and device.
– Finally the User Agents, running on mobile clients, send
requests toward the home environment, in order to satisfy user
needs and preferences. W.r.t. the version in [
Figure 1 sketches the general architecture of the profiling
agent. Raw data are extracted from smartphone embedded
micro-devices, communication tools and PIM. The data
mining life cycle consists of the following subsequent stages:
(a) gathering; (b) feature extraction; (c) classification and
interpretation; (d) semantic annotation. High-level information
about user activities, whereabouts, mental and physical status
is inferred and annotated w.r.t. an extension of the HBA
ontology in [
Google Places 1. Points of Interest Recognition. A mining algorithm analyzes the smartphone GPS data in order to: a. identify Stay Points (SPs) through a slightly refined version of the algorithm in [6]; b. for each SP, retrieve the nearest Point Of Interest (POI) via reverse geocoding queries to Google Places5 Web service;
c. associate a “place category” to each POI, so as to further infer the kind of user activity; d. enrich the daily user profile conjoining all detected activities, described w.r.t. a proper HBA ontology.
A SP represents a narrow geographic region where a user stands for a while. In particular, given two subsequent detected GPS locations P1 and P2, a SP satisfies both the following constraints: (i) maximum distance d(P1; P2) < Dmax; (ii) minimum time difference |T1 − T2| > Tmin, where the thresholds were set to Dmax = 200m; Tmin = 350s. An empirical evaluation was executed to assign the thresholds values granting the highest precision of the SP recognition algorithm.
(a) Home POI (b) POI Info (c) Extracted Places (d) Profile mining (e) Food place detail (f) Daily stay period and location visited before
Figure 2 shows the GUI of the profiler prototype on the
GPS-side. The daily GPS trace is drawn on Google Maps
together with detected SPs, depicted as markers on the map
in Figure 2(a). The Home and Workplace POIs are set by
the user in a preliminary configuration step. As said, the
SP classification leverages a Web-based reverse geocoding
service: after comparing Google Places and LinkedGeoData
(LGD) [7] (see Section V for further details) the first one
service has been chosen at the moment, since it provides more
available POIs even if LGD often seems to be more accurate.
In the example reported in Figure 2(c), the agent selected a
SP near to the Politecnico di Bari and all the nearby POIs
were retrieved by means of the Google Places API. The main
category of the nearest POI is used as label of the retrieved
location. Starting from the Google Places classification6, the
6http://developers.google.com/places/documentation/supported types/
reference ontology for domotics in [
Each transportation mode is associated to a semantic-based annotation fragment which includes a given class of the ontology, further extended to include also concepts and properties about user movements. Moreover, the description will include the overall time –in seconds– the user spent during the day for moving, the daily period and possible means of transport used before. Figure 3(c) shows the details about the user profile section related to a transfer by train. 3. User Activity Recognition. Beyond the above components, the profiling agent is completed by a module to detect some user activities. In particular, at the moment the following elementary actions can be discovered: sitting, standing, walking, walking upstairs and dowstairs. Starting from data acquired from the smartphone accelerometer and gyroscope, a supervised Machine Learning (ML) approach is adopted, exploiting the Support Vector Machines (SVM) classifier in [8]. W.r.t. the original approach, the classifier was simplified to improve its efficiency on PDAs and to reduce the training time. The early 568 features used on the dataset9 associated to [8] as input
8http://wiki.openstreetmap.org/wiki/Overpass API
9http://archive.ics.uci.edu/ml/datasets/
Human+Activity+Recognition+Using+Smartphones
(a) Overpass routes
(b) Train Mode
(c) Train Mode details
for the classifier were reduced to 16 (see Table I) by applying
the Recursive Feature Elimination (RFE) algorithm proposed
in [
A training set composed by sensor raw data has been used
to let the classifier learn directly on the mobile device. The
smartphone used for the experimental evaluation is equipped
with an accelerometer and a gyroscope measuring both the
3axial linear acceleration and the angular velocity (tAcc-XYZ
and tGyro-XYZ, respectively) at a fixed sampling rate of 25
ms, which is adequate to identify a human body motion. The
collected data are subsequently processed through two
firstorder low-pass filters. The first one is used to reduce noise,
while the second filter splits the acceleration signal into body
and gravity components (tBody and tGravity). The classifier
has been implemented using Weka-for-Android10, an Android
port of Weka [
The vector containing the extracted features is then used as input of the trained SVM model. Finally, the user profile is enriched with the annotations related to the detected activities. For each of them it will be also considered the overall stay time and the daily period.
IV.
CASE STUDY
In order to clarify the rationale behind the proposed approach and to let emerge the goal of the profiling agent, the following daily scenario is considered as example. The user leaves home early in the morning to go to work. He remains at office until lunch, then reaches a bar for a fast meal. Afterward, he comes back to work, then goes to the gym in the evening and finally returns home late at night. The profiling agent extracts the daily location sequence reported in Table II. Particularly, Home and Office POIs are mapped to the user profile directly as Home and Work activities; Bar is identified as a Food place; Gym is associated to the Sport place category. The agent also recognizes the adopted means of transport and the duration of each trajectory.
Route Home ! Office Office ! Bar Bar ! Office Office ! Gym Gym ! Home
Type car walk walk car car
Duration (min) 30 4 5 11 21
Along the day, the agent also detects the activities of the user: he was seated for about 6 hours (e.g., at work, within the car, during lunch), walked for 35 minutes (e.g., to reach the bar or for short strolls) and was standing for 15 minutes. As a result of the mining and annotation processes, the following profile is extracted (expressed in Description Logic [11] notation w.r.t. the reference ontology)11: User Daily Profile ≡ ∀ wasAtHome:HomeActivity ⊓ ∀ wasAtW ork:W orkActivity ⊓ ∀ wasInF oodP lace:F oodActivity ⊓ ∀ wasInSportP lace:SportActivity ⊓ ∀ movedByCar:CarMode ⊓ ∀ movedByW alk:W alkMode ⊓ ∀ wasSitting:SittingActivity ⊓ ∀ wasW alking:W alkingActivity ⊓ ∀ wasStanding:StandingActivity HomeActivity ≡ Home ⊓ ∀ during:(Morning ⊓ Night) ⊓ ∀ af ter:Gym ⊓ =1945 stayT ime WorkActivity ≡ W ork ⊓ ∀ during:(Morning ⊓ Af ternoon) ⊓ ∀ af ter:(Home ⊓ Bar) ⊓ =32470 stayT ime 11Due to space constraints, some sections have been voluntarily omitted. ⊓ ⊓ ⊓
∀ af ter:W ork ⊓ =474 stayT ime
∀ af ter:W ork ⊓ =5362 stayT ime ∀ ∀ during:Af ternoon during:Evening WalkMode ≡ W alk ⊓ =2115 moveT ime ⊓ ∀ during:Af ternoon ⊓ ∀ af ter:Car SittingActivity ≡ Sitting ⊓ =21436 ∀ during:(Morning ⊓ Af ternoon ⊓ Evening) moveT ime
The above generated profile will be adopted by the user agent to negotiate with the mediator agent at home the environmental situation best fitting needs and mood of the inhabitant via a semantic-based matchmaking. The elementary services and appliances covering the mined user profile as much as possible are automatically activated (or in case deactivated) to increase the overall MAS utility. As an example of this phase, let us consider the following available home services/resources: CookingService ≡ Service ⊓ ∀ wasInSportP lace:( >=1800 stayT ime) ⊓ ∀ wasAtHome:( ∀ af ter:(Sport ⊓ ¬F ood)) ⊓ ∀ suggestedF orF eeling:Hungry SoftLightLevel ≡ LightLevelRegulation ⊓ ∀ wasAtW ork:( >=10800 stayT ime) ⊓ ∀ wasAtHome:( ∀ af ter: ¬Relax) ⊓ ∀ suggestedF orStamina:MentallyT ired ⊓ ∀ suggestedF orDisease:Headache PlayMusic ≡ Service ⊓ ∀ wasAtHome:( ∀ af ter:( ¬W ork ⊓ Relax) ⊓ ∀ during: ¬Night) ⊓ ∀ suggestedF orStamina:Rested ⊓ ∀ suggestedF orDisease: ¬Headache
It should be noticed that service annotations are described in terms of both user features (such as a physical status, mood and health) and daily events which cause the activation. In this way, a service/resource selection can be performed through the matchmaking against the user profile. For example, a cooking service is activated not only if the user explicitly declares he is hungry, but also if the user agent detects he comes back home after a sport activity, performed for more than 30 minutes (expressed in seconds), without eating anything before. In a similar way, a soft lighting setting is selected to improve the comfort at home in case the user is mentally tired and he spent more than 3 hours at work not followed by a restful activity. The extracted user profile can also lead to a deactivation of previously enabled services. For example, the music service is normally activated to welcome the owner at home, but it is unsuitable if the user comes back during the night and in that case it must be turned off.
The above case study is purposely simplified in order to make the presentation of the proposed approach clear and short. In real scenarios, more articulated user profiles and service descriptions can be used.
An overall evaluation of the proposed approach has been
carried out following a reference user for a period of 14
months. Results reported here refer to the first 60 days of
observation. In particular, only the days –24 in the evaluated
dataset excerpt– with at least one Stay Point different from
Home or Workplace have been selected for further
investigation. The profiling agent has been tested on a smartphone
equipped with an ARM Cortex A8 CPU at 1 GHz, 512 MB
RAM, a 8 GB internal storage memory, and Android 2.3.3
as operating system. Done experiments basically aimed to
measure: (i) the amount of data retrieved from services on the
Web; (ii) the turnaround time (for which each test was repeated
four times taking the average of the last three runs); (iii) the
memory usage (for which the final result was the average of
three runs). This experimental analysis only focuses on the user
profiling aspects: [
Figure 4 shows the total number of stay points detected with the mining algorithm compared with the overall GPS coordinates composing a daily trace. It can be noticed that the user agent collects 53 GPS points per day on average, detecting about 3 relevant SPs.
GPS Points
Starting from detected SPs, the results of Google Places and LGD services have been compared in terms of number of retrieved POIs in the neighborhood of each SP. As shown in Figure 4, Google Places usually returns 16 POIs w.r.t. 5 POIs on average retrieved by LGD, so an accurate identification of the locations the user visited is more likely. Nevertheless, as reported in Figure 5, in some cases the LGD replies are longer even though it returns fewer POIs. This is due to the LGD response format including, for each point, information annotated according to Linked Data principles [12]: Google Places uses 830 B per POI on average, whereas LGD uses 1.56 kB.
Google Places
LGD
The time required by the main processing steps for POIs recognition (GPS traces parsing; SPs detection; Google Places/LGD services querying; profile enrichment), transportation mode detection (Overpass service querying; traces comparison; profile enrichment) and activity recognition are reported in Figure 6. Google Places is slightly slower than LGD, but this is due to the greater amount of retrieved POIs. Considering Google Places as reference service, the agent spends about 1.2 s to retrieve the POIs from a detected SP. Activity A Sitting B Standing C Walking D Walking Upstairs E Walking Downstairs Precision % TABLE III.
CONFUSION MATRIX In particular, the last step took about 1.15 s (49% of total time) to parse the ontology and create the semantic-based annotation. The remaining steps require only the 3% of the overall turnaround time, as these procedures use elementary data structures stored in the device main memory. For the transportation mode detection, only 1.7 s were spent to query the Overpass service, while traces comparison is one of the slower operations, needing 3.4 s. The activity recognition process has a very short turnaround time. After a preliminary task (required to train the SVM classifier) taking about 5.6 s and performed when the profiling agent starts, this module needs only 45 ms to extract the 16 reference features for each windows and 6 ms to detect the user activity. Finally, a daily profile was completely composed in about 1.2 seconds. GPS Trace Parsing SPs Detection Google Query LGD Query Overpass Query Traces Comparison SVM Training Features Extraction Activity Recognition Profile Creation 10000 1000 ) s (m100 e m iT 10 1
Processing Task
A further evaluation of the activity recognition module required to measure precision and recall of the classifier. 100 datasets of activities containing a similar number of samples per class have been used. The confusion matrix shown in Table III reports on the weighted precision of the classifier and on single precision and recall values for each activity. It is referred to a single specific dataset with 779 sample vectors. However all confusion matrices for different tests showed similar outputs, varying slightly in the classification results. It is possible to notice that the classifier precision and recall are very high despite the usage of a small set of features.
RAM usage trend was also evaluated and results are shown in Figure 7, where memory peaks are reported. The profiler agent needs very low memory, only 4.2 MB on average, a satisfactory value for current mobile devices.
VI.
The recent popularization of smartphones equipped with
a wide range of embedded sensors and adequate processing
capabilities has attracted increasing research efforts toward
mobile sensing. Lane et al. [
The activity recognition from accelerometer by means of
machine learning is a frequent sensing application. Among
other proposal, noteworthy are [
In the above cited works, however, the knowledge gap between acquired data and the understanding of human behavior is still huge. Stay points and movement patterns require to be interpreted to extract a user profile, implicitly providing knowledge about the user habits. Noteworthy attempts to enrich movement trajectories with semantics are in [19] and [20]. An ontology-based approach for a semantic modeling of trajectories is also proposed in [21]. Trajectories are seen as composed by three main elements: stops, moves and beginends. Each part is described through an annotation referred to a domain ontology and time information are also exploited to annotate activities to enable rule-based queries and to help users validate and discover moving objects.
Although previous solutions add a machine-understandable
meaning to data collected by smartphones, a subsequent
ex12http://www.google.com/mobile/mytracks/
13http://www.endomondo.com
14http://www.google.com/landing/now/
15http://xndme.com/
ploitation in an articulated AmI framework is still missing.
Usually, collected data are only used to indicate detected user
conditions or activities through messages or alerts displayed on
the mobile phone. On the contrary, in the approach proposed
here, the ontology-based characterization of user activities is
used as an input for a context-aware HBA MAS [
The paper presented a lightweight agent able to mine data collected by embedded micro-devices, logs and applications of a smartphone to build a semantic-based daily profile of its user. According to the AmI paradigm, such a description can be exploited to transparently adapt the environment to user preferences, implicitly inferred. In the matter in question, the agent interacts in a multi-agent framework for Home and Building Automation, grounded on knowledge representation theory and reasoning technologies. It has been designed and then implemented as an Android application and experiments in a concrete case study proved its feasibility and effectiveness.
Future work will include a more extensive experimental campaign involving several different users to be profiled and new performance indicators. Particularly, both battery drain and storage peaks will be taken into account to assess the feasibility of a continuous data collection and mining and to compare the provided framework with existing approaches. Also the exploitation of an agent-based framework w.r.t. to classical approaches will be posed under investigation to verify if it results in a more accurate profiling action. Finally, future research will be also devoted to the integration of the current multimodal information. A fusion of information coming from data sources which now are distinct and independent will be pursued in order to reach a more accurate and precise user characterization.
UbiCare.