=Paper=
{{Paper
|id=Vol-201/paper-18
|storemode=property
|title=Sharing, Discovering and Browsing Photo Collections through RDF geo-metadata
|pdfUrl=https://ceur-ws.org/Vol-201/07.pdf
|volume=Vol-201
|dblpUrl=https://dblp.org/rec/conf/swap/TorniaiBC06
}}
==Sharing, Discovering and Browsing Photo Collections through RDF geo-metadata==
https://ceur-ws.org/Vol-201/07.pdf
Sharing, Discovering and Browsing Photo
Collections through RDF geo-metadata
Carlo Torniai Steve Battle
Multimedia Integration and Communication Center and Steve Cayzer
University of Florence Hewlett Packard Labs
Firenze, Italy Bristol, United Kingdom
Email: torniai@micc.unifi.it Email: steve.battle@hp.com, steve.cayzer@hp.com
Abstract— In recent years the growth in popularity of digital a photo on the web, then jump, for instance, to other photos
photography, together with the development of services and tech- in the field of view or taken nearby. It draws on the network
nologies to annotate and organize data on the Web, have extended effect of the web by including not only the user’s own photos
the possibilities for managing and sharing large numbers of
pictures. Our work explores the kinds of metadata that can be but any photo that can be discovered with suitable metadata.
captured at the time a photo is taken, and ways to share these This includes location (GPS or other mobile location) and
metadata in order to create a browsing experience of distributed heading information to identify the position and direction of
photo collections based on their spatial information and relations. the camera. The photos discovered may have been taken by
We present a prototype system in which an RDF description of different people and are shared on the web. The key to this
pictures, including location and compass heading information,
is used to discover geo-related pictures from other users. A linking is location and heading metadata attached to the photo.
browsing interface that allows users to explore pictures according There are no explicit hyper-links between photos, making it
to the spatial relationships discovered is proposed. easy for people to contribute. Automatic linking is achieved
by the discovery of photos on the semantic web.
I. I NTRODUCTION The main idea is to build RDF descriptions of metadata
With the growing popularity in digital photography, there related to pictures and photo collections and share these
is now a vast resource of publicly available photos on the descriptions in a distributed environment. Spatial relations
web. The availability of cheap GPS devices has made it between nearby pictures are discovered by means of inference
easy to classify, organize and share geotagged pictures on over their RDF descriptions. A web application then uses these
the Web. Geotagging (or geocoding) is the process of adding descriptions to provide a browsable interface. This interface
geographical identification metadata to resources (websites, allows users to explore shared photo collections through their
RSS feed, images or videos). The metadata usually consist of spatial relationships with each other.
latitude and longitude coordinates, but they may also include The paper is organized as follows: the process of choosing
altitude, camera heading direction and place names. metadata and building RDF description of a photo collection is
There has recently been a dramatic increase in the number discussed in Sect. II. Algorithm for building relations between
of people using geo-location information for tagging pictures. pictures is described in Sect. III. In Sect. IV the architecture
The result of a query for pictures with geo:lat tag uploaded of the distributed environment and the process of image
in Flickr1 returns 16,048 results between October 2003 and discovery is presented. Sect. V discusses possible metadata and
October 2004, 89,514 results for the following year and architecture enhancements while in Sect. VI previous works
171,574 results for the period from October 2005 to October based on geotagged images are presented. Finally, in Sect. VII
2006. Following the increasing number of pictures that are we provide conclusions and some future works.
manually geotagged by users, Flickr has recently launched its II. P HOTO COLLECTIONS AND METADATA
own service for adding latitude and longitude information to
a picture. To define the structure and the content of metadata for
In principle, the availability of geotagged pictures, allows a picture description we consider existing RDF schemata that
user to access photos relevant to his or her current location. capture the following information:
However in practice there is a dearth of methods for discov- • Latitude
ering and linking such spatially (and perhaps socially) related • Longitude
photographs. Our work explores the kinds of metadata that • Heading information
can be captured at the time a photo is taken, and ways to link • Author
photos together according to these metadata. The objective of • Date and time
our work is to create an experience where someone can view • Title
• Annotation about location
1 http://www.flickr.com • EXIF metadata
We have used both RDF translation of the EXIF [1] standard Each image is defined according to the Image class de-
and Basic Geo (WGS84 lat/long) vocabulary [2] for latitude scribed in the mindswap ontology2 . The annotation about
and longitude. Heading information and camera related data location is included in the dc:coverage value. Latitude and
(focal length, focal plane resolution and so on) are expressed longitude information in degree-minute-second (d-m-s) nota-
using an RDF version of the EXIF standard. Dublin Core tion are represented by exif:gpsLongitude and exif:gpsLatitude
[3] was selected for defining author, title, date, time and while geo:lat and geo:long contain the decimal degree
annotation about location. To describe the location context (WGS84) notation. North or south latitudes are indicated
we used the Dublin Core dc:coverage tag. The purpose of by exif:gpsLatitudeRef ; while exif:gpsLongitudeRef specifies
dc:coverage is to define the extent or scope of the content whether a longitude is east or west. The exif:gpsImgDirection
of a resource and typically includes spatial location (a place indicates the direction of the image when it was captured. The
name or geographic coordinates), temporal period (a period range of values is from 0.00 (north) to 359.99. A collection of
label, date, or date range) or jurisdiction (such as a named pictures is defined as an RDF list of images with a title and
administrative entity). Additionally, we introduced a hierar- a creator as shown in Listing 2:
chical order into the values of the dc:coverage tags, namely:
Listing 2. Example of RDF pictures collection
Place or area, City, Country. For instance values representing a
picture taken at the Watershed in Bristol would be “Watershed, C a r l o T o r n i a i
c o l l e c t i o n 3
Bristol, UK”. Furthermore, this hierarchical tag could be used h t t p : / / hp . co . uk / semPhoto / p h o t o # C o l l e c t i o n
flexibility in the discovery process. An example of an RDF
Listing 1. Example of an RDF picture description in N3 notation
t e c h n i c a l . owl#> . . HPIM0432 . JPG”/>
@ p r e f i x e x i f : . ....
@ p r e f i x geo : .
a mindswap : Image ;
# Coverage d a t a III. D ISCOVERING PICTURE RELATIONS
dc : c o v e r a g e ” B r i s t o l , UK” ;
The RDF picture descriptions are used to determine the
# Geo I n f o r m a t i o n :
# L a t i t u d e i n d e c i m a l d e g r e e n o t a t i o n (WGS84) spatial relations between pictures. We have chosen to define a
geo : l a t ” 5 1 . 4 4 9 6 8 2 6 ” ; light-weight computation algorithm that provides the following
# L o n g i t u d e i n d e c i m a l d e g r e e n o t a t i o n (WGS84) information:
geo : l o n g ” −2.5976958” ; • Field of view evaluation (moving forward - zoom)
# L a t i t u d e i n d e g r e e−m i n u t e s−s e c o n d s n o t a t i o n • Spatial relations (turning - pan)
e x i f : g p s L a t i t u d e ”51 26 5 8 . 0 ” ;
The field of view relation describes the fact that from a
# Latitude reference picture taken at A (imagea ) one can move towards the picture
e x i f : g p s L a t i t u d e R e f ”N” ;
taken at B (imageb ). The way in which the field of view is
# L o n g i t u d e i n d e g r e e−m i n u t e s−s e c o n d s n o t a t i o n evaluated is shown in Fig. 1. This states that for imageb to
e x i f : g p s L o n g i t u d e ”2 35 5 1 . 0 ” ;
be in the field of view of imagea , one must be able to see
# Longitude r e f e r e n c e point B in imagea , and imageb must have a similar heading
e x i f : g p s L o n g i t u d e R e f ”W” ;
direction to imageb .
# Image D i r e c t i o n The algorithm for field of view evaluation is shown in
e x i f : gpsImgDirection ”320.00” ;
Alg. 1.
dc : c r e a t o r ” C a r l o T o r n i a i ” ; The F OV T HRESHOLD has been set to 150 meters,
dc : d a t e ” 2 0 0 7 : 0 4 : 1 8 T15 : 4 8 : 5 9 ” ;
dc : f o r m a t ” image / j p g ” ;
while the bearing angle threshold Tbear and the heading
dc : t i t l e ” C a b o t Tower from w a t e r f r o n t ” ; direction threshold Thead have been heuristically set to 20
dc : t y p e ” image ” ;
e x i f : brightnessValue ”2389/256” ;
degrees.
e x i f : c o m p o n e n t s C o n f i g u r a t i o n ”48 51 50 49” ; Spatial relations refer to the direction in which you have to
e x i f : c o n t r a s t ”0” ;
e x i f : customRendered ”0” ;
turn, standing in A, in order to see the picture taken at B. If the
exif : dateTimeDigitized ”2007:04:18 15:48:59” ; pictures imagea and imageb have been taken within a given
exif : dateTimeOriginal ”2007:04:18 15:48:59” ;
exif : focalLength ”44.63” ;
range we consider the pictures to be taken in the same location
e x i f : f o c a l P l a n e R e s o l u t i o n U n i t ”3” ; so that their relative spatial position is given by the difference
e x i f : f o c a l P l a n e X R e s o l u t i o n ”20000000/555” ;
e x i f : f o c a l P l a n e Y R e s o l u t i o n ”20000000/555” ;
between their heading information. Referring to Fig. 2 we say
e x i f : g p s V e r s i o n I D ”2 0 0 0” ; that one can turn right from A to B.
e x i f : imageLength ”1952” ;
e x i f : imageWidth ” 2 6 0 8 ” . 2 http://www.mindswap.org/∼glapizco/technical.owl
Algorithm 2 Spatial relations discovering algorithm
for each image pair (imagea , imageb ) in the collection
evaluate distance d(A, B) // distance between A and B
if d(A, B) < DIST AN CE T HRESHOLD
then
dif f angle = HA − HB
case dif f angle
0 to +22.5 OR -337.6 to -360 :
position = F ront
+22.6 to +67.5 OR -292.6 to -337.5 :
position = F ront Right
+67.6 to +112.5 OR -247.6 to -292.5 :
position = Right
+112.6 to +157.5 OR -202.6 to -247.5 :
Fig. 1. Field of view evaluation. If |HA − BA | is less than a given threshold position = Back Right
point B is in the field of view of point A. If |HA − HB | is less than a given +157.6 to +202.5 OR -157.6 to -202.5 :
threshold then the pictures taken in A and B have similar heading direction.
If both these conditions are met then imageb , taken at B is in field of view position = Back
of imagea taken at A. +202.6 to +247.5 OR -112.6 to -157.5 :
position = Back Lef t
Algorithm 1 Field of view evaluation algorithm +247.6 to +292.5 OR -67.6 to -112.5 :
position = Lef t
for each image pair (imagea , imageb ) in the collection
+292.6 to +337.5 OR -22.6 to -67.5 :
evaluate distance d(A, B) // distance between A and B
position = F ront Lef t
if d(A, B) < F OV T HRESHOLD then
+337.6 to +360 OR -0.1 to -22.5 :
evaluate BA // bearing angle between A and B
position = F ront
if (|HA − BA | < Tbear )
set spatial relation(position, imagea , imageb )
// ie point B can be seen in imagea
AND (|HA − HB | < Thead )
// ie imageb and imagea have similar headings
then set f ov relation(imagea , imageb )
We have defined simple properties describing the field of
view (has in f ov) and spatial relations (F ront, Lef t, Right,
Back Lef t, F ront Right, and so on). An example of an
RDF relations model is shown in listing 3.
Listing 3. Example of RDF relations file
2 2 3 . 0 0
< d c : t i t l e>W a t e r s h e d from P e t o B r i d g e
W
N
−2.5976999
5 1 . 4 4 9 6 1 2 5
...
2 1 0 . 0 0
< d c : t i t l e>A r e d B o a t
Fig. 2. Spatial relations evaluation. If d(A, B) is less than a given threshold W
then the spatial relation is given by (HA − HB ). N
−2.5976999
5 1 . 4 4 9 6 1 2 5
...
The algorithm for spatial relations discovering is shown in
Alg. 2. DIST AN CE T HRESHOLD is typically set to 15
meters taking into account the GPS accuracy. IV. D ISTRIBUTED ENVIRONMENT
The output of the algorithm is an RDF model describing A distributed test environment has been implemented in or-
the relations discovered between the pictures. der to evaluate the picture discovery process and the algorithm
for relations evaluation across different photo collections. The client asks other known clients for pictures that have
The distributed environment is composed of a set of the same coverage entries as in its own collection. This is
“clients” (Fig. 3). Each client exposes its photo collection(s) performed by means of SPARQL queries against (similarly
(i.e. RDF metadata) to its peers by means of SPARQL [4] expanded) dc:coverage tags. As a result of this query process
endpoint(s). The clients hold, but do not need to share, the a list of images is returned to the client. Only when potentially
inferred spatial relations between pictures. relevant photos have been discovered and their metadata
The process of discovering related pictures is described retrieved from a remote client do we begin to evaluate the
in Alg. 3. Discovery is performed through queries against specific spatial relationships between them. These images can
remote clients, and does not require the relatively expensive be considered as a “virtual collection” of images; candidates
computation of spatial relations. Instead, photos are selected that may have some relation with the pictures in the client’s
by their coverage, expressed as relatively simple location own photo collection. The client executes the algorithm for
hierarchies. relations evaluation between its collection and the candidate
images. Every relationship discovered is added to the RDF
Algorithm 3 Picture discovery algorithm model. At the end of this process the client will hold all the
expand the coverage tags in the collection relations between their own pictures and pictures of the remote
for each distinct coverage clients.
for each client The distributed environment and the algorithm for relations
query client for matching coverage entries evaluation permit the growth of the RDF relations model.
evaluate relations(client collection, virtual collection) This holds the information required for building the browser
update relation file interface for picture collections. The interface is shown in
Fig. 4.
Fig. 4. Picture browsers interface
The pictures described in RDF can be accessed by a thumb-
nail menu or a Google Maps panel. Moving the mouse over
the markers on the map causes the latitude, longitude, heading
Fig. 3. Distributed Environment
and coverage information for the corresponding picture to be
displayed. The user can browse the pictures by means of
the navigation arrows surrounding the pictures that show the
The first step is the expansion of hierarchical dc:coverage direction in which a user can move from the perspective of
tags (Sect. II) in a client’s own collection. This allows a the current picture. The pictures related by means of the field
SPARQL query to retrieve photos at varying degrees of of view relations can be reached by clicking on the current
granularity. For example, given a picture with the coverage picture.
”Peto Bridge, City Center, Bristol, UK ” the expanded For our experiments we used a set of 100 pictures related to
coverage tags will be the following: 3 different cities. Latitude, longitude and heading information
Peto Bridge,City Center,Bristol,UK were collected on a Suunto G93 watch at the time the pictures
City Center, Bristol, UK were taken and then later injected in the EXIF data for
Bristol, UK
3 http://www.suunto.com
each picture. The RDF collection files were created by a • Topic tags can be mapped to Flickr tags as the URI
batch program reading the EXIF information directly from the for a Flickr tag is simply its URL. The RDF property
pictures. The test distributed environment was composed by 4 used to connect a photograph to a Flickr tag would,
clients. Each client was implemented using Joseki4 SPARQL however, need to be a custom property. The tag hierarchy
server running as a web application under Apache Tomcat. can be represented within RDF using rdfs:subClassOf or
The browsing interface was developed as a web application skos:broader13 .
using Jena5 and Velocity6 . Our ontology reuses some of these existing ontologies
V. D ISCUSSION : ALTERNATIVE REPRESENTATION , for picture metadata definition. RDF translation of the EXIF
ADDITIONAL METADATA , SCALABLE ARCHITECTURE
standard and Basic Geo (WGS84 lat/long) vocabulary are used
for latitude and longitude. Heading information and camera re-
In our approach we used RDF as format to describe photo lated data (focal length, focal plane resolution and so on) are
collections and metadata related to the pictures they contain. expressed using an RDF format for the EXIF standard. Dublin
This offers the following advantages: Core describes author, title, date, time and annotation about
• RDF is expressly designed to provide a standard, ex- coverage. We have introduced our own vocabulary for defining
tensible format for machine readable metadata. RDF is field of view and spatial relations as described in Sect. II.
an open standard, allowing widespread deployment and Our approach for hierarchically structured locations uses
consumption. Using RDF means that the pictures can be the dc:coverage property and the values it may contain. This
shared and reused more easily. approach is very lightweight compared to relations defined
• RDF is “syntax neutral”; different RDF vocabularies more formally but has the following advantages:
share the same syntax. This allows us to mix different • simple expression of the ‘Place or area, City, Country’
vocabularies, and load any vocabulary into any tool. order
• Many ontologies related to pictures metadata are already
• tag-like format that users may easily create
available in RDF format. • more accessible than a series of properties values
The following ontologies are examples of those that can be The advantages of letting users define their own vocabulary
used in order to define pictures metadata: for classifying information has already been demonstrated by
• W3C [5] suggests three simple schemata - Dublin Core the growth of tagging community, while the effectiveness
(for title and description), a technical schema (for camera of folksonomies in information classification and retrieval is
type, lens) and a content schema (oft-used tags like Baby, becoming more and more relevant. One could extend our
Architecture and so on). approach using constraints on tag-like format of property
• Time can be dealt with as a Dublin Core tag or by treating values, or indeed link photographs using controlled vocabu-
events as first class entities [6]. laries. Other metadata can be added to the proposed picture
• Space can be described using precise geographical de- description. In particular, it would be interested to add social
scriptors, like latitude and longitude7 . To represent hier- metadata related to pictures so that social relations, other than
archical relations such as “England contains London” we spatial, can be discovered and presented to the users providing
could use formal approaches like the ‘space’ ontology8 . a social exploration of shared picture collections.
A more ambitious, though incomplete, schema based Our prototype has been a useful proof of concept but is not
on ISA standards has also been proposed9 . Differing yet suitable for real deployment. A P2P architecture would
degrees of accuracy can be catered for by taking a provide an optimization of query caching and routing between
‘layered’approach10 (‘within 10m’, ‘within 100m’, ‘within the different clients at the expense of complexity in the client
10km’). An alternative approach is to consult a controlled implementation. However, a centralized server, which would
vocabulary with concrete place names. act as the repository of the pictures’ metadata and evaluate
• Device metadata is provided within a photo in EXIF the spatial relationships between users’ pictures with batch
format, for which an RDF version exists. Other terms processes, allows the development of a simple web based
relevant to cameras such as focal length are represented service without the need of a client-side application. This
in Morten Frederickson’s Photography Vocabulary11 and is a lighter-weight solution for users who wouldn’t have to
in Roger Costello’s Camera ontology12 . download and install a full software application.
4 http://www.joseki.org/
Compared to other approaches and applications, our system
5 http://jena.sourceforge.net/ has the benefit of standard metadata descriptions that can
6 http://jakarta.apache.org/velocity/ easily be shared and reused in many different applications and
7 http://www.mindswap.org/2004/geo/geoOntologies.shtml (accessed Octo- services. The browser application built on top of these descrip-
ber 2006) tions is an example of what can be done using our approach.
8 http://space.frot.org/ontology.html (accessed October 2006)
9 http://loki.cae.drexel.edu/ wbs/ontology/iso-19115.htm (accessed October RDF provides flexibility in how spatial information is encoded,
2006) processed and computed. One can imagine for example a
10 http://esw.w3.org/topic/GeoOnion (accessed October 2006)
browser based on social networks or an algorithm combining
11 http://www.wasab.dk/morten/2003/11/photo (accessed October 2006)
12 http://www.xfront.com/camera/camera.owl (accessed October 2006) 13 http://www.w3.org/TR/swbp-skos-core-spec/ (accessed October 2006)
latitude, longitude, coverage and geographic thesauri for more VII. C ONCLUSION
accurate spatial labeling. The lightweight approach proposed In our work we have presented a prototype system providing
for computing picture relations, and indeed the choice to rely ways to:
purely on metadata rather than on information gathered from
• represent geographical metadata related to pictures
heavyweight image processing, makes our solution suitable for
• discover pictures relations according to the metadata
real time and web based applications.
• present the geotagged pictures and their relations
An algorithm for inferring spatial relations between differ-
VI. R ELATED W ORK
ent pictures using location and compass heading information
There has been much interest recently in using geo-location embedded in the RDF description of the pictures has been
information to relate different picture and to create an en- presented. A testing environment for metadata sharing and
hanced photo browsing experience. relations discovering has been implemented so that users’
In Sharing Places14 , multimedia annotation (photo, video photo collections are enhanced by relations with other users’
and audio) can be associated with physical locations to create pictures.
a ’mediascape’. These trails, based on GPS information and We have shown how, based on geographical metadata ex-
enriched with annotation, can be accessed over the web or pressed in RDF, it is possible to build a service for discovering,
downloaded to a suitable device (e.g. PDA) and experienced linking and browsing geographical related photo in a novel
in the real world. The trails can be tagged, published for others way. Our future work will deal with experiments on large test
to find, remixed and shared. This approach relies on a central beds in order to obtain meaningful performance evaluation,
server and doesn’t provide annotation in a standard sharable improve scalability, and improve the user interface.
metadatada format. R EFERENCES
Images are arranged according to their location in the World- [1] W. W. W. Consortium, “Exif vocabulary workspace - rdf schema,” W3C,
Wide Media Exchange [7] while time and location are used http://www.w3.org/2003/12/exif/, Tech. Rep., accessed 1 October 2006.
to cluster images in PhotoCompas [8]. Realityflythrough [9] [2] ——, “Basic geo (wgs84 lat/long) vocabulary,” W3C,
http://www.w3.org/2003/01/geo/, Tech. Rep., accessed 4 October
presents a very friendly user interface for browsing video from 2006.
camcorders equipped with GPS and tilt sensors, and a method [3] D. U. Board, “Dcmi metadata terms,” Dublin Core Metadata Initiative,
for retrieving images using proximity to a virtual camera is http://dublincore.org/documents/dcmi-terms/, Tech. Rep., accessed 1 Oc-
tober 2006.
presented in [10]. [4] W. W. W. Consortium, “Sparql protocol and rdf query language,” W3C,
In Photo Tourism [11] a system for interactively browsing http://www.w3.org/TR/rdf-sparql-query/, Tech. Rep., accessed 4 October
and exploring large unstructured collections of photographs is 2006.
[5] ——, “Describing and retrieving photos using rdf and http,” W3C,
presented. Using a computer vision-based modelling system, http://www.w3.org/TR/photo-rdf/, Tech. Rep., accessed 1 October 2006.
photographers’ location and orientation are computed along [6] ——, “Rdf calendar workspace,” W3C, http://www.w3.org/2002/12/cal/,
with a sparse 3D geometric representation of the scene. Full Tech. Rep., accessed 1 October 2006.
[7] R. Toyama, K. Logan and A. Roseway, “Geographic location tags on
3D navigation and exploration of the set of images and world digital images,” in MULTIMEDIA ’03: Proceedings of the eleventh ACM
geometry, along with auxiliary information such overhead international conference on Multimedia. New York, NY, USA: ACM
maps is provided by the photo explorer interface. In contrast Press, 2003, pp. 156–166.
[8] A. Naaman, M. Paepcke and H. Garcia-Molina, “Metadata sharing for
to our system (based on a distributed environment in which digital photographs with geographic coordinates,” in Proccedings of 10th
metadata related to photo collections are exchanged in real International Conference on Cooperative Information Systems, 2003.
time between users in order to discover relationships between [9] N. J. McCurdy and W. G. Griswold, “A systems architecture for
ubiquitous video,” in MobiSys ’05: Proceedings of the 3rd international
pictures) a complex computer-vision based algorithm is used conference on Mobile systems, applications, and services. New York,
to provide spatial relationship between images. NY, USA: ACM Press, 2005, pp. 1–14.
These approaches provide a user experience enhanced [10] R. Kadobayashi and K. Tanaka, “3d viewpoint-based photo search and
information browsing,” in SIGIR ’05: Proceedings of the 28th annual
by geo-information but don’t rely on standard format international ACM SIGIR conference on Research and development in
for metadata nor provide a distributed environment for information retrieval. New York, NY, USA: ACM Press, 2005, pp.
exchanging metadata. As already pointed out [12] we believe 621–622.
[11] N. Snavely, S. M. Seitz, and R. Szeliski, “Photo tourism: exploring photo
that metadata related to pictures and their locations should collections in 3d,” ACM Trans. Graph., vol. 25, no. 3, pp. 835–846, 2006.
be expressed in a common and sharable standard so that they [12] S. Cayzer and M. H. Butler, “Semantic photos,” Hewlett Packard Labs,
may be used by other applications. Sharing picture metadata http://www.hpl.hp.com/techreports/2004/HPL-2004-234.html, Tech.
Rep., accessed 4 October 2006.
across a distributed environment using an open standard such
as RDF can lead to interesting evolutions in the way in which
pictures and other multimedia geotagged content are shared,
discovered and browsed.
14 http://www.sharing-places.com