=Paper=
{{Paper
|id=Vol-2978/casa-paper6
|storemode=property
|title=A Framework for Adaptive Context and User-Related Management of Multimedia Contents (short paper)
|pdfUrl=https://ceur-ws.org/Vol-2978/casa-paper6.pdf
|volume=Vol-2978
|authors=Mariagrazia Fugini,Jacopo Finocchi,Elisa Rossi
|dblpUrl=https://dblp.org/rec/conf/ecsa/FuginiFR21
}}
==A Framework for Adaptive Context and User-Related Management of Multimedia Contents (short paper)==
https://ceur-ws.org/Vol-2978/casa-paper6.pdf
A Framework for Adaptive Context and User-Related
Management of Multimedia Contents
M.G. Fugini 1, J. Finocchi 1, E. Rossi 2
1
DEIB - Politecnico di Milano, Milano, Italy
2
DIG - Politecnico di Milano, Milano, Italy
Abstract
This paper presents the specification of a software platform where knowledge is associated to maps,
through the organization and presentation of geo- and chronological-referenced multimedia
contents, in such as a way that this association is automatically adaptive to different contexts
(application areas) and to different typologies of users.
The platform adaptivity has the purpose of reducing the information overload of the maps and of
offering the user targeted navigation and search functions, tailored to his/her typology (e.g., domain
expert, citizen, public administration operator, decision maker, stakeholder, etc.) and dependent on
the loaded contents. To achieve this, we focus on the problem of selecting and aggregating the
multimedia contents of the map, in a way that depends on the context and on the user. This
reduction is obtained as a dynamic process, using a metamodel in the form of ontology of tags.
Keywords
Georeferenced Data, Data Discovery, Map Enrichment, Context Adaptivity, Multimedia Content,
Multimedia Tagging, Metadata Analysis.
1. Introduction1 The set of tags associated to multimedia data
are modelled in an ontology with:
The typical aim of geographic-based sets of concepts shared among various
multimedia applications is to enrich a map with contexts (e.g., “geo-location” tag, “time”,
georeferenced contents, allowing one to explore “user”, “media_type”, etc.) and
selected areas and run thematic analyses at sets of concepts typical of a context (e.g.
various levels of detail, and with different aims “year_of_discovery” and “host_museum”
and focus, so that is will be possible to perform tags in the Cultural Heritage context,
analytics on multimedia map contents. Such “approval_document” tag in the Urban
analyses usually aim at exploring various levels Planning context, and “level_of_robotization”
of interest on the map contents, and at obtaining and “efficiency_formula” tags in the Industry
zoom levels on the represented realm. context).
In this paper, we propose an adaptive
framework that enriches maps with knowledge Domain experts develop and specialize the
using an ontology of tags, which describe the ontology for each context.
map multimedia contents, the geo references, the The tags guide the dynamic generation of
time references, the context, and the user thematic map layers, as defined in [1], which are
behaviour, so that the contents can adapt to: 1) the basis for content navigation. In fact, the
different contexts; 2) different users. navigation interface is based on a set of layers,
some recommended by the system, others
customized by the user. Each layer contains the
Proceedings Name, Month XX–XX, YYYY, City, Country points of interest belonging to one or more
EMAIL: mariagrazia.fugini@polimi.it (M.G.Fugini); categories, i.e., connected to contents associated
jacopo.finocchi@ mail.polimi.it (J.Finocchi); with those tags. Each tag identifies a category of
elisa5.rossi@mail.polimi.it (E.Rossi)
ORCID: 0000-0002-0692-0153 (M.G.Fugini) contents. The choice of how to split tags among
Copyright 2021 © for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). the various layers is taken dynamically
CEUR Workshop Proceedings (CEUR-WS.org) considering which nodes of the ontology are
actually instantiated by the contents present in teaching activity tasks, to support personalized
the context. flipped learning.
The paper is organized as follows. In Section
2, we review related work. In Section 3, we 3. Associating knowledge to maps:
present the overall approach of associating
knowledge to maps. In Section 4, we discuss issues and our approach
about navigating tags for adaptivity and draw the
conclusions. In our platform, knowledge associated to
geographical maps links geographical elements
with geo- and temporal-referenced multimedia
2. Related work contents. The framework is being developed to
be adaptable to:
Nowadays, geographic-based applications are - various contexts, such as history, cultural
a widespread object of study. Developers are
heritage, urban planning, digital twinning
implementing many different solutions that
of devices and artefacts, and citizen
combine geo-referenced information related to a
map, ending up with maps that can be used either journalism.
as general-purpose exploration tools, or as - different user typologies.
professional tools for specific application The key innovative features of the proposed
contexts. framework are as follows.
Some of these tools combine geographic and 1. Handling of multimedia content: the
multimedia information to improve content framework focuses on the organization and
search, e.g., in [2], where visual features of navigation of georeferenced multimedia, such
pictures are integrated with geographic data, as pictures, video, audio tracks, point clouds,
based on the points of interest concept. unstructured text or documents.
The research in [3] tackles the semantic gap 2. Adaptivity to different contexts and
between semantic needs of users and the users: the framework comprises a meta-
visualization of multimedia content. The structure (semantic tags ontology), which
proposed solution is focused on how to extract allows instantiation onto different contexts,
information from the context of multimedia via self-adaption to loaded contents.
documents and their metadata, rather than from 3. Temporal dimension: chronological
media raw content itself. metadata are treated separately, so that
Also [4] enhances the semantic annotation of specific navigation modes can be provided
multimedia contents by leveraging their web along the time dimension.
context and the user comments. Videos are
indexed via a large set of labels, and are linked Contents are linked to the map through the
with related contents. definition of a number of Points Of Interest
The solution presented in [5] implements a (POI), which are selected and clustered
semantic tag recommendation technique for according to our self-adaptive semantic tag
image tagging, relying on a graph of ontology. Content classification adapts to the
relationships among words. context-specific ontology, by matching the
The development of an adaptive multimedia content metadata with the most relevant ontology
content navigation is a main goal of adaptive node.
hypermedia systems, mainly based on the user In the case of massive contents loaded from
profiles, as discussed in [6]. an existing data source, the classification of
The problem of adaptivity in accessing multimedia under our semantic tags is performed
multimedia contents is often addressed in the by a supervised ML classifier, starting from the
field of e-learning systems. For example in [7], metadata natively associated to each content item
ML techniques are applied to adapt the difficulty in the data sources, along with geo and temporal
level of the presentation to the learner style and tags.
to student assessment.
In [8], the authors adopt an ontology-based
approach to integrate the relevant knowledge in 3.1. Modeling
the content presentation. Their ontology not only
models the domain knowledge but also the
For the representation of geo and temporal the geographic dimension, the temporal
information, we propose a flexible data structure dimension, and the thematic dimension. The first
that joins existing cartographic formats to enable can be navigated with the interactive tools
linking pieces of information one to another and typical of maps, i.e., panning and zooming. The
to map. second is navigable through chronological
The approach consists in adding the following selection filters. The third is navigable through
issues to geographical objects: the definition of content layers, with the support
- unstructured multimedia content, together of an ontology of semantic classes.
with source of information attributes as well The first two dimensions are organized
as domain-specific attributes, and independently of the context, while the thematic
- temporal dimension, to reach the concept of dimension is strongly linked to the context of
4-dimensional map. application.
We started from the definition of a data
The prototype we are developing will be model, aimed at representing and managing the
grounded on an existing open cartographic semantic contents associated to the map. The
system (precisely, OpenStreetMap) with various data model allows integrating the cartographic
added layers of data, representing the context- data with heterogeneous structured data (coming
specific elements. Compared to a classic map, from different sources) and with unstructured
this model allows browsing the data as a 4D data (multimedia), including the chronological
map, enriched with multimedia content. dimension.
Tags representing the temporal dimension The hinge between geographic elements and
allow associating a point of interest to a set of contextual content is the definition of POI. Since
events that occurred in its location or describing knowledge is often related to a geographical
the evolution of the associated contents along element rather than a single point, a POI can also
time. correspond to an area, accompanied by related
The knowledge associated with the map context information.
through the metadata of the multimedia contents
is organized along three information dimensions:
Figure 1: The ontology of tags
Figure 1 shows an outline of the ontology we On the right-hand part of Figure 1, we
are developing. A map element is defined with represent the content associated to the map,
geographic coordinates and can represent a organized into different contexts and including
single point (like a building), a line (like a street) the multimedia content and its related metadata.
or a polygon (an area).
Deriving the content classification from a while transversal modules deal with Data
semantic ontology allows for a language- Quality and Validation.
independent consultation, since each node of the In the topmost area of our architecture, the
ontology describing the contents can be Interface layer allows users to interact with the
expressed with a specific tag for each language analysis modules, to enter new data and to
managed by the system. This is different from visualise the map, dealing with Security and
textual search in metadata, which is instead Privacy issues by filtering accesses according to
dependent on the language used to specify the the different user privileges. Here, some modules
metadata. aim at validating the provenance of the data
sources, and to model the additional data
3.1. Functional architecture according to the target representation, which is
visualised in a personalised way according to the
rights of the different users.
The functional architecture of the proposed
The framework is designed to be used by
framework is shown in Figure 2. different categories of users: Public
The Storage area contains a spatial database
Administrations, professional users such as
for geographical and temporal data connected to architects or urban designers, and individual
the map, together with an additional database citizens. Each category is granted access to
storing information about contexts and different contents and functionalities according
multimedia. to the needs and roles.
In the Analytics area, the Adaptation module
implements the adaptivity of the framework,
Figure 2: Framework overview
testing purposes, we are considering importing a
3.2. Enriching a Map: series of publicly available georeferenced
multimedia contents.
Semantic Annotations In the content acquisition phase, the interface
layer assigns to data a set of attributes derived
After the definition of the general framework, from metadata and from source registration
the following step is to design an interactive tool information, such as privacy, licensing,
aimed at enriching the map layer with content confidence, and so on) and to content, to enable
provided by external and voluntary sources, its thematic classification.
according to crowdsourcing models, since, for
Considering classification of the content, Selection, namely filtering, i.e., showing
either we provide a tree of predefined tags for only some contents and filtering out
homogeneous and language-independent irrelevant contents.
classification, or we leave the contributors free to Grouping, namely clustering, i.e.,
enter user-defined tags in the tree. showing a single object that summarizes
Concerning who can upload content, the various different contents.
framework is open to ordinary users, voluntary Ranking, namely proposing only the
citizens and qualified sources with credentials, most relevant contents according to a
such as public administrations, universities, given order of relevance.
associations, or professionals.
Regarding who can read data, we envisage The framework could adapt to the data
public, private, password-protected information entered into the archive (multimedia data,
or payment-based access. The type of access metadata, and tags). This should occur not
depends on the type of licence the system grants: simply by returning different data (otherwise it
data can be public domain, or can be used under would just be a normal data-driven system) but
creative-common licences, or provided under the by modifying the navigation and presentation
authorisation of the owner. Some types of users methods depending on the type of data. For
can add data, which they can decide to share with example, the modification can be made
other users (a community – selective access, all dependent on: a) whether we are dealing with
users – public access, no users – restricted images or videos or texts etc., or b) on the value
access) and combine them with the existing data. of the tags.
The combination is guided by the ML If applied to multimedia elements, some ML
algorithms, that help classifying data according techniques can be used to generate and better
to what defined in the ontology. structure the tags associated to the different
media contents, in order to run further analyses
4. Discussion about Navigating Tags of clustering or aggregations.
for Adaptivity and Conclusions Our proposed navigation and query mode is
conceived to display information on the map,
considering the different granularity of data,
The presented framework aims at offering a depending on the displayed area. The higher the
query and analysis environment that supports geographical zoom level, the higher the temporal
navigation along tags and enables analyses, e.g., detail and the semantic detail of the returned
for decision making, by intersecting content information. In this way, information on
geo/temporal referenced data coming from the map is presented at different levels of
different sources and with different formats and granularity, automatically linked to each other.
informative content, including multimedia data. The zooming function can operate onto the
The navigation and query interface would three dimensions: spatial, temporal and thematic.
adapt to the context. The way to achieve The first is the classic geographical zoom; the
adaptation is, for example, the selection of second groups events into macro-events or
different query filters or the dynamic generation chronological periods; the third is a semantic
of a layer containing context-dependent zoom, where one can navigate a taxonomy
elements. dynamically derived from the context-specific
As an alternative, we envision the dynamic ontology.
generation of the POI and the clusters of It is also interesting to adapt the user analyses
multimedia content associated with them. The to the context; this is a matter of current study.
idea is that not only the content of the clusters
can be dynamically navigated (which is
somehow obvious), but also their structure 5. Acknowledgements
should adapt to the context. Tags adapt to the
uploaded content or vary based on user queries. This paper is partially supported by the
We plan to have three navigation modes to Projects BigData and Seamless of Regione
reduce information overload in a way that is Lombardia, Italy, and by the EU Horizon 2020
adaptable to the context, where the context is research and innovation programme under grant
defined by the set of entered contents and by the agreement No. 826232, project WorkingAge
user preferences: (Smart Working environments for all Ages).
6. References
[1] H. Karimi, H. Zeinivand, Integrating runoff
map of a spatially distributed model and
thematic layers for identifying potential
rainwater harvesting suitability sites using
GIS techniques, Geocarto International,
(2021), pp. 320-339.
[2] E. Purificato, A. M. Rinaldi, Multimedia
and geographic data integration for cultural
heritage information retrieval, Multimedia
Tools and Applications, 77(20), 2018.
[3] T. Bracamonte, B. Bustos, B. Poblete, T.
Schreck, Extracting semantic knowledge
from web context for multimedia IR: a
taxonomy, survey and challenges,
Multimedia tools and applications, 77(11),
2018.
[4] D. Fernández, D. Varas, J. Espadaler, I.
Masuda, J. Ferreira, A. Woodward, E. Bou,
Vits: video tagging system from massive
web multimedia collections, in: Proceedings
of the IEEE International Conference on
Computer Vision Workshops, 2017, pp.
337-346.
[5] H. K. Hong, G. W. Kim, D. H. Lee,
Semantic tag recommendation based on
associated words exploiting the interwiki
links of Wikipedia, Journal of Information
Science, 44(3), 2018, pp. 298-313.
[6] A. C. T. Klock, I. Gasparini, M. S. Pimenta,
J. P. M. De Oliveira, Adaptive hypermedia
systems, in: Advanced Methodologies and
Technologies in Media and
Communications, IGI Global, 2019, pp.
217-228.
[7] H. Anantharaman, A. Mubarak, B. T.
Shobana, Modelling an adaptive e-Learning
system using LSTM and Random Forest
classification, in: 2018 IEEE Conference on
e-Learning, e-Management and e-Services
(IC3e), 2018, pp. 29-34.
[8] Y. L. Chi, T. Y. Chen, C. Hung, Learning
adaptivity in support of flipped learning: an
ontological problem-solving approach.
Expert Systems, 35(3), 2018.