The volume, velocity and uncontrolled variety of Big Data are changing the way data exploration for data-driven decision making is performed on top of Data Lakes. As data grows, novel methods are needed for data aggregation by means of indicators and multi-dimensional analysis of Data Lakes content, enabling exploration of data according to various dimensions, thus empowering users with diverse roles and competencies to capitalise on the available information. In this paper, we present a computer-aided approach (named PERSEUS, PERSonalised Exploration by User Support) for data exploration on top of a Data Lake. The approach is structured over three phases: (i) the construction of a semantic metadata catalog on top of the Data Lake; (ii) the creation of an Exploration Graph, based on metadata contained in the catalog, containing the semantic representation of indicators and analysis dimensions; (iii) the enrichment of the definition of indicators with personalisation aspects (based on users' profiles and preferences) to identify Exploration Contexts, in turn delimiting portions of the Exploration Graph for a personalised and interactive exploration of indicators. Results of an experimental evaluation in the Smart City domain are presented with the aim of demonstrating the feasibility of the approach.
In dynamic and rapidly evolving environments permeated by the volume, velocity and
uncontrolled variety of Big Data, Data Lakes have been proposed as ground-breaking solutions to
develop applications for data-driven decision making. Data Lakes ensure a suitable degree of
lfexibility for managing diferent types and formats of data sources, since data is loaded “as is”
and transformed only when it becomes necessary [
We model a Data Lake as a set of data sources , each one modelled as ⟨, , ℳ⟩, where: (i) is a set of attributes; (ii) is a collection of data sets, representing the content of the data source regardless its nature (i.e., structured, semi-structured, unstructured); (iii) ℳ is a set of attribute-value pairs containing metadata apt to access the source (e.g., username, password) and other source-specific metadata. Each data set ∈ is defined over a set of attributes ⊆ . An attribute can be either: (i) a simple attribute or (ii) an attribute referencing another data set in the same data source (nesting).
The domain expert is in charge of creating the semantic metadata catalog by means of a
web-based tool supporting basic annotation tasks. The annotation procedure regards only
attributes names and not their values, thus reducing the annotation burden. The steps for the
creation of the catalog are performed incrementally, as soon as new data sources are added.
Lexical enrichment of data source attributes. Each attribute of a data source is associated
with a label referred to as Entity Property, to reduce the gap between the attribute name and
names of concepts used for semantic annotation. To this aim, domain experts are supported
by two external linguistic APIs, conceived to complement each other: (i) an Abbreviations
API [
Semantic annotation of data source attributes. Starting from the Entity Property, the
web-based tool retrieves a suitable concept describing the meaning of the attribute . To this
aim, a set of domain ontologies stored within an open access repository (LOV - Linked Open
Vocabularies [
Semantic metadata catalog population. The semantic metadata catalog constructed over the Data Lake contains: (i) the set of concepts annotating attributes of data sources; (ii) equivalence relationships between pairs of concepts, either associated with the same data source or diferent data sources, which are suggested relying on the metadata set ℳ (e.g., when the concepts annotate attributes belonging to two tables of a relational database) or manually defined by the domain expert (e.g., when involved attributes belong to diferent data sources).
Data sources { "city": "Brescia", coords:[{ "lat": "45.54155", "long": "10.2118" },}] { "city": "Sarezzo", coords:[{ "lat": "45.36186", "long": "10.13109" },}] ...
JSON file "comm"; "city" "Borgo Trento"; "Brescia" CSV file "B..u.ffalora"; "Brescia"
Data sets representation city "Brescia" coords lat "45.54155" long "10.2118" comm "Buffalora" city "Brescia"
City place:City place:City
City city
Semantic metadata catalog city
coords lat Latitude
S1: Latitude osadm: Community Community comm
long Longitude
S1:
Longitude geo:Spatial
Thing
Legend Domain Ont.
Concept Specialised
Concept rdfs:subClassOf Concept specialisation Equivalence relationship
Attributes
Entity Properties Concepts
Example. The left side of Figure 1 illustrates examples of two Smart City data sources and their representation as attributes and data sets. The two sources contain geospatial information of cities and related administrative areas. The right side of Figure 1 shows the semantic representation of the sources in the semantic metadata catalog. The Entity Properties are retrieved from WordNet (e.g., for country attribute) and the Abbreviations API (e.g., lat and long attributes). To find suitable concepts for semantic annotation, the LOV Search Term API is invoked using the Entity Properties as query parameters. Two concepts (Latitude and Longitude) have been obtained as a specialisation of the ones extracted from LOV ontologies (through the rdfs:subClassOf semantic relationship). In the figure, blue arrows denote equivalence relationships between concepts.
Indicators are modelled by data analysts starting from the knowledge retained in the
semantic metadata catalog and through the specialisation of concepts and relationships of a
MultiDimensional Ontology (MDO), containing the conceptual elements that must be taken into
account to model indicators. In the design of the MDO, pivotal concepts from available
foundation ontologies have been exploited to: (i) represent users’ activities (Schema.org ontology), (ii)
characterise indicators and dimensions as analytical data entities (Data Cube ontology) and (iii)
model units of measure for indicators (OM ontology). Further details regarding the conceptual
elements of the MDO can be found in [
Creation of indicator concept. In this step the Indicator concept of the MDO is specialised to extend the indicators hierarchy. Using the takesDataFrom semantic relationship, composite indicators can be defined starting from other fine-grained indicators. For a newly created
Legend Personalisation
Concept Indicator Concept Dimension
Concept rdfs:subClassOf hasPracticable
Activity
Building
Administrator hasPracticable
Activity
Monitoring Pollution Levels
involves Summation
Formula Sum
hasFormula Air Pollution hasAggregationFunction Indicator hasUnitOfMeasure belongsTo ppm
Environment
Exploration context Citizen
Monitoring
Pollution Levels
hasDimension hasDimension
hasDimension Household
takesDataFrom
Spatial Dimension hasLevel hasLevel rollUp
District City rollUp Building rollUp Apartment indicator, a Formula (that, for a composite indicator, reports how to calculate it in terms of its component indicators), the UnitOfMeasure and the AggregationFunction are specified. Link to dimensional hierarchies. Once an indicator has been modelled, it must be bound to one or more dimensional hierarchies. The data analyst may reuse previously created hierarchies or define new ones, relying on the pivotal concepts Dimension and Level from the MDO. Definition of personalisation concepts. The semantic representation of indicators is further enriched by associating them with their target domains (e.g., environment, health) through the belongsTo relationship. Personalisation concepts derived from the MDO are employed to afirm that the awareness of certain indicators impacts particular tasks, requiring end-users to base their decisions on these indicators (e.g., building monitoring, check air pollution). This is achieved by binding the indicator to a UserCategory and an Activity (or one of their sub-concepts) from the MDO. In particular, the hasPracticableActivity relationship binds a UserCategory to an Activity. Finally, the involves semantic relationship links an Activity to one or more Indicators.
Validation of the created indicator. To assist the data analyst in the modelling task, several
constraints are checked through validation rules defined in the MDO: (i) a valid activity involves
at least one indicator; (ii) a valid dimension hierarchy, being associated with an indicator, must
gather at least one dimension level; (iii) a valid indicator belongs to at least one domain, is
explorable according to at least one dimension hierarchy, possibly has a unit of measure and is
involved in at least one activity. The interested reader can find the formulation of the validation
rules in [
Example. In Figure 2, the AirPollutionIndicator is described as a sum of other indicators, has ppm as unit of measure and is linked with the Environment domain. HouseholdCO2 is an example of composite indicator, specialised from CO2Indicator and computed starting from CO2Heaters indicator. All the indicators are associated with SpatialDimension, articulated over the Apartment, Building, District and City levels and connected each other by rollUp relationship. Similarly, indicators are associated with the TimeDimension (not shown here). Lastly, indicators can be explored by both citizens and building administrators (modelled through corresponding concepts) while performing MonitoringPollutionLevels activity.
Once the Exploration Graph has been created, the Data Lake can be explored by relying on: (i) Multi-Dimensional Descriptors, apt to model multi-dimensional basic elements on which exploration is performed; (ii) Exploration Contexts, that identify portions of the Exploration Graph containing indicators compliant with users’ activities; (iii) contextual preferences, to suggest to the user the most promising indicators to start the exploration from within an Exploration Context.
Multi-Dimensional Descriptors. Navigating across the Exploration Graph may be unpractical with a growing number of nodes and edges. Hence, to explore the indicators in , we foster a strategy grounded on the assumption that users inherently explore data according to a multidimensional organisation. In this respect, we defined proper Multi-Dimensional Descriptors over (MDDs), to provide a compact representation of indicators and their dimensional levels. Figure 2 highlights two examples of MDDs for the CO2Heaters indicator.
Exploration contexts. Personalised exploration of MDDs is modelled through a set of soft constraints contained in users’ profiles () for each in the set of users . Soft constraints are modelled as preferences, organised according to Exploration Contexts, that represent the situations in which the user explores the MDDs, influenced by both his/her roles and goals. An Exploration Context is used to delimit a portion of the Exploration Graph , explorable by the user . Available contexts are derived from considering all the distinct pairs of UserCategory and Activity (sub-)concepts. At exploration time, a context can be bound to one or more users’ profiles. Users can manage their profile by selecting/changing the context of interest, choosing it from the ones compliant with their role(s).
Contextual preferences. The portion delimited by an Exploration Context may
contain a high number of indicators, especially when considering a generic activity (such
as “pollution monitoring”). To cope with this issue, contextual preferences help suggesting
the user the indicators which best fit his/her demands. Contextual preferences can be either:
(a) short-term preferences, expressed by the user at exploration time, representing imminent
exploration needs; (b) long-term preferences, stored in user’s profile, which are assumed to be
static or change slowly over time. Contextual preferences are expressed on the set of MDDs
derived from through diferent constructors that rank indicators based on: (i) the distance that
indicators have in the hierarchy induced by rdfs:subClassOf relationships (IND constructor);
(ii) the distance that dimensional levels have in the hierarchy induced by rollUp relationships,
focusing on a specific dimension ( LEV constructor); (iii) the fact that an indicator belongs to
a given domain (DOM constructor). Formalisation details of the constructors are available
in [
(a) ESxmpalortraCtiitoynInEdnigcianteors LAoglioceut Home
Logged in as Citizen. To change role, logout and choose a different role in the login page
Exploration Context Selection ? Role Citizen Activity MonitoringPollutionLevels Forumlate Request for Indicators dDoemsidaeinred (No selection) iDnedsicidaetorerd CO2Indicator Desidered dimension (No Selection) Edit Long-Term Preferences
? Add...
Add...
Add...
Find Indicators
(b) ESxmpalortraCtiitoynInEdnigcianteors LAoglioceut Home / Results Visualisation
Current Exploration Context: Citizen (MonitoringPollutionLevels) Available results ? Eepvxrepaflrueerasetseniodcne I(DNLOEDVM((CS(OEpna2vtIiniardolDincimamteoenrn)ts ) i) o n , District) Indicators returned Displayed results: 1-4
CO2Indicator (District, Year) CO2Indicator (District, Quarter) CO2Indicator (District, Month) CO2Indicator (District, Day)
More Results... 1 2 3
Browse selected indicator
In this section, we present an excerpt of the implementation and the experimental evaluation
conducted in the scope of the Brescia Smart Living project, wherein three diferent typologies
of end-users have been identified as targets for the personalised exploration of indicators:
(i) citizens, willing to explore aggregated data related to their neighbourhood (for example,
average energy consumption, air quality, neighbourhood safety); (ii) property managers,
administering one or more apartment buildings; (iii) technical user plant managers, responsible
for heat distribution in buildings. In particular, we focus here on presenting the procedure
for personalised exploration of indicators, achieved through a prototype web-based GUI
(Figure 3). The usability of the GUI (in terms of facility in finding and exploring indicators)
has been tested by a representative group of 10 users, belonging to the three aforementioned
typologies, completing a standard System Usability Scale (SUS) questionnaire [
GUI for personalised indicators exploration. Personalised indicators exploration is
articulated over four main steps executed through a web-based GUI (Figure 3). To present the four
steps (S1-S4), we consider Alice, a citizen interested in monitoring air pollution levels to decide
whether or not to practise outdoor activities, since pollution has efect on this kind of activities.
(S1) Exploration context selection – The exploration platform proposes Alice to select one of
the Exploration Contexts available for her profile. Figure 3(a) shows the selection of the
AirPollutionMonitoring activity.
(S2) Short-term preferences formulation. – In this step, Alice chooses the desired indicators,
domains and dimensional levels, and the corresponding concepts are mapped by the platform
to DOM, IND and LEV base preference constructors. For instance, in Figure 3(a), when Alice
selects the CO2Indicator, the corresponding IND preference constructor will be automatically
included in the request. The obtained constructors constitute the short-term preferences.
(S3) Short-term and long-term preferences combination. – Short-term preferences in the request
are combined with long-term preferences in the profile () of the user, holding within the
Exploration Context, thus leading to the compound preference P. Long-term preferences are
automatically combined using the Pareto composition operator, since they all assume an equal
importance for Alice. Short-term preferences are combined with long-term ones according to
the prioritization operator (▷), as they address an immediate need. After the request formulation
has been finalised, Alice confirms her choices by clicking the “Find Indicators” button.
(S4) Preference evaluation and indicators exploration. – The compound preference P from the
previous step undergoes an evaluation process to identify the set of best (optimal) MDDs
according to P. Such MDDs are proposed to the user, who can select any of them to explore
indicator values. For example, Alice’s preference evaluation result is displayed in the first page
of the list in Figure 3(b). Finally, Alice selects one of the MDDs (by clicking on the “Browse
selected indicator” button) and the multi-dimensional query apt to retrieve indicator values will
be issued over the underlying Semantic Data Lake. As detailed in [
In this section, we will analyse an excerpt of the literature based on the requirements demanded
by each phase of the PERSEUS approach. Regarding Semantic Data Lake modelling research,
the focus of the latest years has been on the formalisation of models for supporting knowledge
extraction from Data Lakes, building a semantic overlay with diferent techniques (e.g., by
grouping similar attributes for easing querying data sources [
Novel contributions. PERSEUS aims at proposing a combined engineering of diferent
techniques for addressing Semantic Data Lake exploration. With respect to [
In this paper, we presented PERSEUS, a computer-aided approach for data exploration on top of
a Semantic Data Lake. The approach is structured over three phases: (i) the construction of a
semantic metadata catalog on top of the Data Lake; (ii) the creation of an Exploration Graph,
based on metadata catalog, containing the semantic representation of indicators and analysis
dimensions; (iii) the enrichment of the definition of indicators with personalisation aspects (based
on users’ profiles and preferences) to identify Exploration Contexts, in turn delimiting portions
of the Exploration Graph for a personalised and interactive exploration of indicators. Results
of an experimental evaluation in the scope of the Brescia Smart Living project are presented
with the aim of demonstrating the feasibility of the approach. Each phase of the PERSEUS
approach paves the way to further investigation. For instance, regarding preference-based
indicators exploration, we will enhance the preference model by considering the propagation of
preferences across Exploration Contexts, as proposed by [