Information systems used in tourism rely mostly on up-to-date content on aractive places. In addition, these systems increasingly make use of archived photographs, documents, lms, or even ancient paintings and other artwork by integrating such curated content from museums and memory institutions, possibly enriched with user-provided content. Hence the distinction between cultural heritage applications and tourism more and more blurs. Users are not only interested in the current appearance of landscapes, monuments, or buildings, but also in the evolution of these places over time. is requires large multimedia collections which integrate content from several cultural heritage institutions. As a consequence, interactive retrieval systems for historical multimedia are needed that support homogeneous content-based and semantic querying despite the heterogeneity of these collections. In this paper we present City-Stories, a multimedia hybrid content and entity retrieval system. City-Stories is based on a state-of-the-art open source multimedia retrieval system. Multimedia features in City-Stories represent multiple semantic levels: low-level (e.g., color, edge, motion), mid-level (e.g., date, location, objects), and high-level features (e.g., semantic entities, scene category). For the laer, CityStories applies entity recognition and entity linking for identifying semantic concepts and linking objects across media types. Consequently, City-Stories supports various types of cross-modal queries. Moreover, City-Stories uses a map-based visualization layer that facilitates spatial queries and browsing. Finally, City-Stories follows a crowdsourcing approach for content annotation and for enriching curated content with multimedia objects and documents provided by users. e paper shows how the City-Stories system seamlessly combines content-based search with entity-based navigation and leverages the wisdom of the crowd for manual annotations.
•CCS CONCEPTS 1
Multimedia data on places of interest like documents, photos, videos, or user ratings are the most important sources used in information systems for tourists. While systems have so far focused on up-todate content, historical material taken from archives is increasingly gaining importance in order to give tourists more information on how particular touristic sites have developed over time, i.e., how they looked 20, 50, 100, or even more years ago. With the content from museums and memory institutions, the distinction between cultural heritage applications and information systems for tourism increasingly blurs, despite the fact that content may dier signicantly (in terms of media types and formats, age, availability and degree of detail of metadata/annotations, etc.). Moreover, content provided by users out of their private archives is also gaining importance due to the proliferation of social networks and crowdsourcing platforms.
In order to provide integrated access to such heterogeneous content, several important technical challenges need to be addressed:
Multimedia Retrieval. e integrated content should be accessible by a very broad range of dierent query types, such as keyword queries to search in (manual) textual annotations, query-byexample (multimedia search with sample objects), query-by-sketch (multimedia similarity search on the basis of hand-drawn sketches), semantic queries that exploit semantic concepts and links between objects, spatio-temporal queries (i.e., queries on the location and/or time where/when a particular object has been created), and any combination of these modes.
Entity Recognition and Linking. Content coming from dierent sources, in dierent formats, and possibly also with dierent metadata structures has to be integrated to make sure that it can be accessed via a homogeneous interface. is includes standard approaches to schema and data integration, but also more advanced and innovative challenges like entity recognition and entity linking to make sure that links between objects (of the same or even of dierent media types) can be identied, stored as part of the metadata, enhanced with further external sources, and subsequently exploited for query purposes.
Crowdsourcing. In addition to cultural heritage content curated
by archives, user-generated content from private collections is
gaining importance in touristic information systems. In order to
aract the aention to potential content providers, the awareness
of such touristic platforms has to be raised, the technical barrier for
contribution has to be lowered, and users have to be encouraged to
actively participate. is is not only true for the provision of new
content but also for annotations to existing content (e.g., ratings or
experience reports). e rapid adoption of smartphones has made it
possible to also exploit mobile crowdsourcing [
In this paper, we introduce City-Stories, a novel and innovative
system for collecting, managing, and accessing heterogeneous
cultural heritage content for touristic applications. e City-Stories
browser allows to retrieve spatio-temporal knowledge and supports
real-time interactive search in large databases of historical
multimedia collections. From a systems perspective, City-Stories is based
on vitrivr1 [
Low-level features like color, edge, motion, mid-level features like date, location, and high-level features like semantic entities or scene categories.
Spatio-temporal metadata in the collection is directly imported as vector location and timestamp features; the CityStories frontend allows for spatial, temporal, and spatiotemporal queries and the results are displayed in a map and on a timeline, respectively.
Manual annotations and user ratings are provided via crowdsourcing.
Textual metadata (usually in the form of title and description of an item) is subjected to entity extraction, which yields Uniform Resource Identiers (URIs) in a knowledge base. ese entity URIs are then used to pre-compute semantic entity distances between collection items.
Hence, City-Stories combines content-based search with entitybased navigation and leverages the wisdom of the crowd for manual annotations.
e contribution of the paper is threefold. First, we show how multi-feature content-based similarity search providing a plethora of query types can be enriched by entity recognition and entity 1hps://vitrivr.org linking, thereby allowing for cross-media retrieval based on semantic concepts. Second, we show how curated collections and automatically generated metadata can be extended by user-generated content and user-provided metadata, which is particularly relevant in applications for tourists and which complements manually curated cultural heritage collections. ird, we show how all these elements can be seamlessly combined in the City-Stories system.
e remainder of the paper is structured as follows: Section 2 motivates the City-Stories approach with a tourism use case. Section 3 discusses the components needed for the integrated multimedia content and entity retrieval system and Section 4 presents details of the City-Stories system. Section 5 summarizes related work and Section 6 concludes. 2
Consider, as an example for City-Stories, the following use case: Sophia, a tourist from Dublin, is visiting the city center of Berlin. When standing in front of the Brandenburg Gate, one of Berlin’s neoclassical city gates, she has a variety of questions regarding the building and its neighborhood, like ‘What building is this, what was its purpose, when and by whom has it been built?’ or ‘How did the neighborhood of the gate look like around 1900, in the so-called ‘golden’ 1920’s, shortly aer the end of WW2, in the 1970s, before and aer the fall of the iron curtain in 1989 — or at any other point in time in the past?’.
Sophia holds a smartphone on which she accesses the City-Stories query interface. e City-Stories system integrates several cultural heritage multimedia collections, e.g., from the German federal archive or from focused museum collections. Moreover, City-Stories encompasses a large number of photos and associated metadata provided by local citizens. Using City-Stories, Sophia is able to directly browse the content and submit queries of dierent types: Simple location queries: Using the GPS coordinates of her current location, Sophia is able to identify the building she is currently looking at and get access to basic information regarding this building, combined from several data sources on the web.
Temporal queries: On the basis of information from various sources that have been integrated beforehand into the City-Stories system, Sophia is able to query details of the building’s history (photos or historical paintings from dierent stages of the building). Moreover, she will also get information on the building’s neighborhood at dierent points in time, on historical events that took place there, and statistical information (e.g., population of the city at dierent points in time). e laer is based on linked metadata, i.e., metadata enriched with links between objects aer entity recognition has been applied to the content. Combined spatial and multimedia similarity queries: ese queries allow to search for similar buildings (or buildings that take a similar role) in the vicinity of Sophia’s current location.
Multimedia similarity queries: Sophia takes a photo of the Brandenburg Gate with the camera of her smartphone. She will use this photo for a similarity query in order to nd other buildings (in Berlin or in any other European city) that look similar. Here, similarity can be dened either by intrinsic image features, on the base of the object’s metadata or links, or any combination of these.
Combined sketch-image similarity queries: Sophia provides one of the photos of the Brandenburg Gate taken with her smartphone as query input and adds a superimposed sketch (e.g., she draws a typical Berlin double-deck coach in the foreground).
Most importantly, Sophia does not want to use an earmarked smartphone app provided by a local tourist organization with manually curated content specialized for a particular touristic site, as she would have to newly install such an app every time she visits another place. Rather, Sophia is interested in City-Stories, a generic approach that can be used to integrate and access content independent of a concrete location, so she could use this system for her next trips to Singapore, to visually explore the recent growth of the Marina area, or to New York, for instance to allow her to visualize the development of Lower Manhaan over the past 120 years. 3
e multimedia retrieval engine of City-Stories, which is based on the vitrivr system, oers multiple dierent query modes: queryby-example (QbE), query-by-sketch (QbS), temporal, and spatial queries. In a given query we can freely combine these dierent modes, e.g., by providing an exemplary image, drawing a sketch on top of an existing or a provided image, and/or specifying a location. To provide this functionality, we extend the soware stack of vitrivr including Cineast, the extraction and retrieval engine, and ADAMpr o , the database engine.
In Cineast, we dierentiate between an on-line and an o-line phase. e o-line phase includes the feature extraction and the storage of the resulting metadata. In the on-line phase, where the actual retrieval happens, the engine executes a given query and returns a list of documents ordered by similarity.
Cineast uses multiple features in combination to describe a document. In total, there are already 40 content descriptors for videos and images provided by Cineast. ese descriptors extract mainly color, edge, and motion information (where applicable). Building on top of that, we extend Cineast with the following higher level descriptors:
Spatial and temporal similarity features by using geolocalization and timestamp metadata provided by the content (usually via the capturing devices).
A descriptor using semantic entities that are provided by the entity recognition described in what follows in Section 3.2. Semantic concept features provided by a deep neural network. 3.2
In order to integrate the archive data available to us with other data sources such as knowledge bases, we apply named-entity recognition, candidate selection, and entity linking techniques on available text data. Named-entity recognition is the task of identifying mentions of entities, which can take various surface forms, in text. Once an entity mention has been extracted, the corresponding URI in the knowledge base has to be found. Selecting the URI corresponding to a mention of an entity in text is called entity linking.
By linking to a knowledge base, specically to DBpedia2 [
In order to identify entities associated with media items, we use textual media metadata and extract entities from associated titles and descriptions. Consequently, when displaying media items, additional information from the knowledge base can be retrieved and displayed. Knowledge bases tend to oer various aributes for entities, many of which will not be relevant to the viewer. us, 2hp://dbpedia.org TGV candidates
Martigny candidates http://fr.dbpedia.org/ resource/Martigny, 0.9 ________________ ________________ candidate selection
Arrivée du premier TGV {http://fr.dbpedia.org/
resource/TGV} des neiges en gare de Martigny {http://fr.dbpedia.org/resource/Martigny} entity linking when choosing which information to display, we also rank the aributes by their importance to the viewer (for example, when viewing information on a city, the population and the founding year will likely be more relevant to a tourist than the ZIP codes in this city).
Furthermore, we are able to leverage the relationships between media items by extracting the relationships between entities in a graph-structured knowledge base. is also allows to transitively extract related information, e.g., for entities with lile available data. Oentimes, in such graph structures, we can rely on higherlevel categories to provide general information on an entity for which specic information may not be available. 3.3
e crowdsourcing component provides the possibility of enriching content with new data provided by dierent types of users and enhancing the integrated digital collection data.
Collected data is not always complete, i.e., it could be noisy and/or miss particular information. In a tourist application, for instance, information provided by users could miss the location where a point of cultural interest can be found, to whom it belongs, or even the title or a description. Likewise, data might be incorrect or conicting as a result of the integration of collected datasets coming from dierent sources. Hence, these challenges are grouped into two categories: conicting information and missing information, both of which are addressed in City-Stories using crowdsourcing.
Crowdsourcing is able to build an open, connected, and smart
cultural heritage with involved consumers and providers [
In order to optimize the assignment of tasks to the appropriate
crowds, we make use of push crowdsourcing [
In paid micro crowdsourcing scenarios, money as incentive is
the main motivator for workers to contribute and can also be used
for quality control (e.g., constrain a payment on the quality of the
work that has been done). However, it is important to distinguish
malicious users from workers that do not have the intention of
fraud but may aect the overall system due to misunderstandings
or lack of knowledge and experience with the platform [
CITY-STORIES SYSTEM In what follows, we describe details on the implementation of the dierent components of City-Stories and the content that has been integrated in a rst prototype. 4.1
When querying the retrieval engine via the City-Stories UI, for
each given feature, Cineast performs an extraction on the given
query document resulting in a feature vector. Each feature vector is
passed to the ADAMpr o database backend to perform a k-nearest
neighbor (k-NN) similarity search to nd a list of similar documents.
ADAMpr o has been shown to scale to collection sizes of up to 50
million entries and feature vectors with up to 500 dimensions [
In particular, for the spatial and temporal similarity search we use a nearest neighbor query on the two-dimensional geolocation data and the one-dimensional timestamp data, respectively (see Figure 3).
To search for similar documents based on semantic properties, we provide two dierent kinds of features:
Based on entity recognition and linking, each document
is characterized by a list of semantic entities. Using this
list of semantic entities we calculate a pairwise distance to
estimate the similarity between two documents.
e similarity based on semantic concepts utilizes an
AlexNet convolutional network [
Candidates are selected from a database of pairs of surface forms and entities (given as DBpedia URIs). is database is created by processing all Wikipedia articles and extracting hyperlinks where the hyperlink text corresponds to the observed surface form (i.e., the link text) and the link location corresponds to the entity this surface form links to (i.e., the linked article). e frequency of observing a specic (surface form, entity) pair yields a prior probability that a particular surface form corresponds to the linked entity. We rank the candidates using this prior score and select the top candidate to link this mention to an entity in the knowledge base.
Having done the linking, we are able to create a graph of the media items where an edge is present between two items if they contain the same entity and are thus related. Knowing the DBpedia URI of an entity found in the metadata of a media item, we extract relevant aributes for this entity by ranking the aributes with their frequency of appearing in close proximity to this type of entity. 3hp://www.valais-wallis-digital.ch/ 4.3
In parallel to collecting data from institutions such as audio/visual archives, Mediatheque4, and DigitalValais, we emphasize the importance of data sharing from people that have valuable data and information about cultural heritage in private collections. is part of the system enables users to participate and contribute to cultural heritage. Once shared, users’ data is integrated to the data repository. In order to maintain a high level of quality of the crowdsourced data, we apply the following control methods (shown in Figure 5):
Play cards is a test measure used to qualify or disqualify users for solving tasks of a certain category. We use 195 playing cards grouped in 13 dierent categories, where categories represent subject areas of the crowdsourcing tasks. Each card contains both a question and its answer (not visible to the user). At rst users provide information on their topics of interest which intersect with card categories. en they are forwarded to the test phase. To get qualied, they have to correctly answer at least 70% of the questions matched to their interests. Upon two consecutive failures, a user is no longer considered for tasks on these specic topics. Providing the option to choose topics they like or have knowledge on avoids false positives, i.e., eliminating users due to lack of knowledge on randomly assigned questions coming from a pool of predened questions. Moreover, implementing this measure in a game fashion increases the interactivity and the interest of the users. Additionally, users can test their knowledge on topics covered by the cards and at the same time expand their knowledge with additionally provided information by these cards.
A weighted majority voting with reputation system combines
majority voting (MV) with users’ reputation scores. MV [
A screenshot of the crowdsourcing frontend of City-Stories showing how data is shared, the play cards for quality control, and sample annotations is depicted in Figure 6. 5
In general, existing retrieval systems applied to the cultural
heritage domain are either metadata-based or content-based, and few
of them implement both [
EUscreen5 is a project related to multimedia archives that
focuses on the collection, integration, and publication of audio-visual
content. Oomen et al. [
Named-entity recognition is based largely on natural language
processing techniques and is required as a step prior to performing
candidate selection and named-entity disambiguation. e
Stanford NER [
Crowdsourcing has shown to be an eective solution for
problems that are dicult to solve for computers and problems that
require human intelligence [
In this paper, we have presented City-Stories, a novel system that combines content-based similarity search on both historical and contemporary multimedia data with spatio-temporal queries and that exploits semantic analysis of the content for entity recognition and linking. We have deployed City-Stories in a tourist application where cultural heritage content from archives and memory institutions is complemented with content contributed by users via a crowdsourcing approach.
In our future work, we aim to increase the size of the collections available in City-Stories for several selected tourist locations to show the generic applicability of the City-Stories approach. We also intend to perform user studies at these locations to assess the usability of the system and the eectiveness of the integrated content and entity retrieval approach. Moreover, we plan to further exploit the synergies obtained from the combination of all retrieval modes supported in City-Stories, for instance by proactively making recommendations during the retrieval process and by providing additional information from external sources. Finally, we aim to extend the user experience by providing within the user interface an overlay function that allows to superimpose the camera view of a smartphone showing the current view of a place of interest with historical content in order to beer show the development of a particular place. When multiple visual objects of a place are available from dierent periods of time (taken from the same or at least a similar perspective), this will lead to a “history browser” which can be used to steer the overlay with a slider on the timeline, to select the object chosen for the overlay, and to gradually visualize the development.
is work was partly funded by the Hasler Foundation in the context of the project City-Stories. We would like to thank the cantonal archives and the “Mediathe`que” of the canton of Valais and the team of Digital Valais project for delivering a data testbed. tunities and challenges,” in Proceedings of the 5t h International Conference on