- In recent years the growth in popularity of digital photography, together with the development of services and technologies to annotate and organize data on the Web, have extended the possibilities for managing and sharing large numbers of pictures. Our work explores the kinds of metadata that can be captured at the time a photo is taken, and ways to share these metadata in order to create a browsing experience of distributed photo collections based on their spatial information and relations. We present a prototype system in which an RDF description of pictures, including location and compass heading information, is used to discover geo-related pictures from other users. A browsing interface that allows users to explore pictures according to the spatial relationships discovered is proposed.
With the growing popularity in digital photography, there is now a vast resource of publicly available photos on the web. The availability of cheap GPS devices has made it easy to classify, organize and share geotagged pictures on the Web. Geotagging (or geocoding) is the process of adding geographical identification metadata to resources (websites, RSS feed, images or videos). The metadata usually consist of latitude and longitude coordinates, but they may also include altitude, camera heading direction and place names.
There has recently been a dramatic increase in the number of people using geo-location information for tagging pictures. The result of a query for pictures with geo:lat tag uploaded in Flickr1 returns 16,048 results between October 2003 and October 2004, 89,514 results for the following year and 171,574 results for the period from October 2005 to October 2006. Following the increasing number of pictures that are manually geotagged by users, Flickr has recently launched its own service for adding latitude and longitude information to a picture.
In principle, the availability of geotagged pictures, allows a user to access photos relevant to his or her current location. However in practice there is a dearth of methods for discovering and linking such spatially (and perhaps socially) related photographs. Our work explores the kinds of metadata that can be captured at the time a photo is taken, and ways to link photos together according to these metadata. The objective of our work is to create an experience where someone can view a photo on the web, then jump, for instance, to other photos in the field of view or taken nearby. It draws on the network effect of the web by including not only the user’s own photos but any photo that can be discovered with suitable metadata. This includes location (GPS or other mobile location) and heading information to identify the position and direction of the camera. The photos discovered may have been taken by different people and are shared on the web. The key to this linking is location and heading metadata attached to the photo. There are no explicit hyper-links between photos, making it easy for people to contribute. Automatic linking is achieved by the discovery of photos on the semantic web.
The main idea is to build RDF descriptions of metadata related to pictures and photo collections and share these descriptions in a distributed environment. Spatial relations between nearby pictures are discovered by means of inference over their RDF descriptions. A web application then uses these descriptions to provide a browsable interface. This interface allows users to explore shared photo collections through their spatial relationships with each other.
The paper is organized as follows: the process of choosing metadata and building RDF description of a photo collection is discussed in Sect. II. Algorithm for building relations between pictures is described in Sect. III. In Sect. IV the architecture of the distributed environment and the process of image discovery is presented. Sect. V discusses possible metadata and architecture enhancements while in Sect. VI previous works based on geotagged images are presented. Finally, in Sect. VII we provide conclusions and some future works.
To define the structure and the content of metadata for
picture description we consider existing RDF schemata that
capture the following information:
• Latitude
• Longitude
• Heading information
• Author
• Date and time
• Title
• Annotation about location
• EXIF metadata
We have used both RDF translation of the EXIF [
Listing 1. Example of an RDF picture description in N3 notation @prefix mindswap : <h t t p : / / www. mindswap . org / ˜ g l a p i z c o / t e c h n i c a l . owl#> . @prefix dc : <h t t p : / / p u r l . org / dc / e l e m e n t s /1.1/ > . @prefix e x i f : <h t t p : / / www. w3 . org / 2 0 0 3 / 1 2 / e x i f / ns#> . @prefix geo : <h t t p : / / www. w3 . org / 2 0 0 3 / 0 1 / geo / wgs84 pos#> . <h t t p : / / b i g p i c t u r e / p i c t u r e s / HPIM0459 . JPG> a mindswap : Image ; # Coverage d a t a dc : c o v e r a g e ” B r i s t o l , UK” ; # Geo I n f o r m a t i o n : # L a t i t u d e i n d e c i m a l d e g r e e n o t a t i o n (WGS84) geo : l a t ”51.4496826” ; # L o n g i t u d e i n d e c i m a l d e g r e e n o t a t i o n (WGS84) geo : l o n g ” −2.5976958” ; # L a t i t u d e i n degree−minutes−s e c o n d s n o t a t i o n e x i f : g p s L a t i t u d e ”51 26 5 8 . 0 ” ; # L a t i t u d e r e f e r e n c e e x i f : g p s L a t i t u d e R e f ”N” ; # L o n g i t u d e i n degree−minutes−s e c o n d s n o t a t i o n e x i f : g p s L o n g i t u d e ”2 35 5 1 . 0 ” ; # L o n g i t u d e r e f e r e n c e e x i f : g p s L o n g i t u d e R e f ”W” ; # Image D i r e c t i o n e x i f : g p s I m g D i r e c t i o n ” 3 2 0 . 0 0 ” ; dc : c r e a t o r ” C a r l o T o r n i a i ” ; dc : d a t e ” 2 0 0 7 : 0 4 : 1 8 T15 : 4 8 : 5 9 ” ; dc : f o r m a t ” image / j p g ” ; dc : t i t l e ” Cabot Tower from w a t e r f r o n t ” ; dc : t y p e ” image ” ; e x i f : b r i g h t n e s s V a l u e ”2389/256” ; e x i f : c o m p o n e n t s C o n f i g u r a t i o n ”48 51 50 49” ; e x i f : c o n t r a s t ”0” ; e x i f : customRendered ”0” ; e x i f : d a t e T i m e D i g i t i z e d ” 2 0 0 7 : 0 4 : 1 8 1 5 : 4 8 : 5 9 ” ; e x i f : d a t e T i m e O r i g i n a l ” 2 0 0 7 : 0 4 : 1 8 1 5 : 4 8 : 5 9 ” ; e x i f : f o c a l L e n g t h ” 4 4 . 6 3 ” ; e x i f : f o c a l P l a n e R e s o l u t i o n U n i t ”3” ; e x i f : f o c a l P l a n e X R e s o l u t i o n ”20000000/555” ; e x i f : f o c a l P l a n e Y R e s o l u t i o n ”20000000/555” ; e x i f : g psVe rsi onI D ”2 0 0 0” ; e x i f : imageLength ”1952” ; e x i f : imageWidth ”2608” .
Each image is defined according to the Image class described in the mindswap ontology2. The annotation about location is included in the dc:coverage value. Latitude and longitude information in degree-minute-second (d-m-s) notation are represented by exif:gpsLongitude and exif:gpsLatitude while geo:lat and geo:long contain the decimal degree (WGS84) notation. North or south latitudes are indicated by exif:gpsLatitudeRef ; while exif:gpsLongitudeRef specifies whether a longitude is east or west. The exif:gpsImgDirection indicates the direction of the image when it was captured. The range of values is from 0.00 (north) to 359.99. A collection of pictures is defined as an RDF list of images with a title and a creator as shown in Listing 2:
Listing 2. Example of RDF pictures collection <r d f : D e s c r i p t i o n > <dc : c r e a t o r >C a r l o T o r n i a i </dc : c r e a t o r > <dc : t i t l e >c o l l e c t i o n 3 </dc : t i t l e > <r d f : type>h t t p : / / hp . co . uk / semPhoto / photo # C o l l e c t i o n </ r d f : type> <r d f : f i r s t > <mindswap : Image r d f : a b o u t =” h t t p : / / b i g p i c t u r e / p i c t u r e s /
HPIM0428 . JPG”/> </ r d f : f i r s t > <r d f : r e s t r d f : p a r s e T y p e =” C o l l e c t i o n ”> <mindswap : Image r d f : a b o u t =” h t t p : / / b i g p i c t u r e / p i c t u r e s /
HPIM0429 . JPG”/> <mindswap : Image r d f : a b o u t =” h t t p : / / b i g p i c t u r e / p i c t u r e s /
HPIM0432 . JPG”/> . . . .
</ r d f : r e s t > </ r d f : D e s c r i p t i o n >
The RDF picture descriptions are used to determine the spatial relations between pictures. We have chosen to define a light-weight computation algorithm that provides the following information: • Field of view evaluation (moving forward - zoom) • Spatial relations (turning - pan)
The field of view relation describes the fact that from a picture taken at A (imagea) one can move towards the picture taken at B (imageb). The way in which the field of view is evaluated is shown in Fig. 1. This states that for imageb to be in the field of view of imagea, one must be able to see point B in imagea, and imageb must have a similar heading direction to imageb.
The algorithm for field of view evaluation is shown in Alg. 1.
The F OV T HRESHOLD has been set to 150 meters, while the bearing angle threshold Tbear and the heading direction threshold Thead have been heuristically set to 20 degrees.
Spatial relations refer to the direction in which you have to turn, standing in A, in order to see the picture taken at B. If the pictures imagea and imageb have been taken within a given range we consider the pictures to be taken in the same location so that their relative spatial position is given by the difference between their heading information. Referring to Fig. 2 we say that one can turn right from A to B.
2http://www.mindswap.org/∼glapizco/technical.owl
Algorithm 1 Field of view evaluation algorithm for each image pair (imagea, imageb) in the collection evaluate distance d(A, B) // distance between A and B if d(A, B) < F OV T HRESHOLD then evaluate BA // bearing angle between A and B if (|HA − BA| < Tbear) // ie point B can be seen in imagea AND (|HA − HB| < Thead) // ie imageb and imagea have similar headings then set f ov relation(imagea, imageb)
The algorithm for spatial relations discovering is shown in Alg. 2. DIST AN CE T HRESHOLD is typically set to 15 meters taking into account the GPS accuracy.
The output of the algorithm is an RDF model describing the relations discovered between the pictures.
Algorithm 2 Spatial relations discovering algorithm for each image pair (imagea, imageb) in the collection evaluate distance d(A, B) // distance between A and B if d(A, B) < DIST AN CE T HRESHOLD then dif f angle = HA − HB case dif f angle 0 to +22.5 OR -337.6 to -360 : position = F ront +22.6 to +67.5 OR -292.6 to -337.5 : position = F ront Right +67.6 to +112.5 OR -247.6 to -292.5 : position = Right +112.6 to +157.5 OR -202.6 to -247.5 : position = Back Right +157.6 to +202.5 OR -157.6 to -202.5 : position = Back +202.6 to +247.5 OR -112.6 to -157.5 : position = Back Lef t +247.6 to +292.5 OR -67.6 to -112.5 : position = Lef t +292.6 to +337.5 OR -22.6 to -67.5 : position = F ront Lef t +337.6 to +360 OR -0.1 to -22.5 : position = F ront set spatial relation(position, imagea, imageb)
We have defined simple properties describing the field of view (has in f ov) and spatial relations (F ront, Lef t, Right, Back Lef t, F ront Right, and so on). An example of an RDF relations model is shown in listing 3.
Listing 3. Example of RDF relations file <r d f : D e s c r i p t i o n r d f : a b o u t =” h t t p : / / b i g p i c t u r e / p i c t u r e s /
HPIM1375 . JPG”> <b i g p i c t u r e : h a s i n f o v r d f : r e s o u r c e =” h t t p : / / b i g p i c t u r e / p i c t u r e s / HPIM1351 . JPG” /> <e x i f : g p s I m g D i r e c t i o n>22 3. 00</ j . 0 : g p s I m g D i r e c t i o n> < d c : t i t l e>Watershed from P e t o B r i d g e</ d c : t i t l e> <e x i f : g p s L o n g i t u d e R e f>W</ e x i f : g p s L o n g i t u d e R e f> <e x i f : g p s L a t i t u d e R e f>N</ e x i f : g p s L a t i t u d e R e f> <g e o : l o n g>−2.5976999</ g e o : l o n g> <g e o : l a t>51.4496125</ g e o : l a t> </ r d f : D e s c r i p t i o n>
. . . <r d f : D e s c r i p t i o n r d f : a b o u t =” h t t p : / / b i g p i c t u r e / p i c t u r e s /
HPIM1351 . JPG”> <b i g p i c t u r e : B a c k R i g h t r d f : r e s o u r c e =” h t t p : / / b i g p i c t u r e / p i c t u r e s / HPIM1350 . JPG” /> <e x i f : g p s I m g D i r e c t i o n>21 0. 00</ j . 0 : g p s I m g D i r e c t i o n> < d c : t i t l e>A r e d Boat</ d c : t i t l e> <e x i f : g p s L o n g i t u d e R e f>W</ e x i f : g p s L o n g i t u d e R e f> <e x i f : g p s L a t i t u d e R e f>N</ e x i f : g p s L a t i t u d e R e f> <g e o : l o n g>−2.5976999</ g e o : l o n g> <g e o : l a t>51.4496125</ g e o : l a t> </ r d f : D e s c r i p t i o n> . . .
A distributed test environment has been implemented in order to evaluate the picture discovery process and the algorithm for relations evaluation across different photo collections.
The distributed environment is composed of a set of
“clients” (Fig. 3). Each client exposes its photo collection(s)
(i.e. RDF metadata) to its peers by means of SPARQL [
The process of discovering related pictures is described in Alg. 3. Discovery is performed through queries against remote clients, and does not require the relatively expensive computation of spatial relations. Instead, photos are selected by their coverage, expressed as relatively simple location hierarchies.
Algorithm 3 Picture discovery algorithm expand the coverage tags in the collection for each distinct coverage for each client
query client for matching coverage entries evaluate relations(client collection, virtual collection) update relation file
The client asks other known clients for pictures that have the same coverage entries as in its own collection. This is performed by means of SPARQL queries against (similarly expanded) dc:coverage tags. As a result of this query process a list of images is returned to the client. Only when potentially relevant photos have been discovered and their metadata retrieved from a remote client do we begin to evaluate the specific spatial relationships between them. These images can be considered as a “virtual collection” of images; candidates that may have some relation with the pictures in the client’s own photo collection. The client executes the algorithm for relations evaluation between its collection and the candidate images. Every relationship discovered is added to the RDF model. At the end of this process the client will hold all the relations between their own pictures and pictures of the remote clients.
The distributed environment and the algorithm for relations evaluation permit the growth of the RDF relations model. This holds the information required for building the browser interface for picture collections. The interface is shown in Fig. 4.
The first step is the expansion of hierarchical dc:coverage tags (Sect. II) in a client’s own collection. This allows a SPARQL query to retrieve photos at varying degrees of granularity. For example, given a picture with the coverage ”Peto Bridge, City Center, Bristol, UK ” the expanded coverage tags will be the following: <dc:coverage>Peto Bridge,City Center,Bristol,UK</dc:coverage> <dc:coverage>City Center, Bristol, UK</dc:coverage> <dc:coverage>Bristol, UK</dc:coverage>
The pictures described in RDF can be accessed by a thumbnail menu or a Google Maps panel. Moving the mouse over the markers on the map causes the latitude, longitude, heading and coverage information for the corresponding picture to be displayed. The user can browse the pictures by means of the navigation arrows surrounding the pictures that show the direction in which a user can move from the perspective of the current picture. The pictures related by means of the field of view relations can be reached by clicking on the current picture.
For our experiments we used a set of 100 pictures related to 3 different cities. Latitude, longitude and heading information were collected on a Suunto G93 watch at the time the pictures were taken and then later injected in the EXIF data for each picture. The RDF collection files were created by a batch program reading the EXIF information directly from the pictures. The test distributed environment was composed by 4 clients. Each client was implemented using Joseki4 SPARQL server running as a web application under Apache Tomcat. The browsing interface was developed as a web application using Jena5 and Velocity6.
V. DISCUSSION: ALTERNATIVE REPRESENTATION, ADDITIONAL METADATA, SCALABLE ARCHITECTURE In our approach we used RDF as format to describe photo collections and metadata related to the pictures they contain. This offers the following advantages: • RDF is expressly designed to provide a standard, extensible format for machine readable metadata. RDF is an open standard, allowing widespread deployment and consumption. Using RDF means that the pictures can be shared and reused more easily. • RDF is “syntax neutral”; different RDF vocabularies share the same syntax. This allows us to mix different vocabularies, and load any vocabulary into any tool. • Many ontologies related to pictures metadata are already available in RDF format.
The following ontologies are examples of those that can be
used in order to define pictures metadata:
• W3C [
Our ontology reuses some of these existing ontologies for picture metadata definition. RDF translation of the EXIF standard and Basic Geo (WGS84 lat/long) vocabulary are used for latitude and longitude. Heading information and camera related data (focal length, focal plane resolution and so on) are expressed using an RDF format for the EXIF standard. Dublin Core describes author, title, date, time and annotation about coverage. We have introduced our own vocabulary for defining field of view and spatial relations as described in Sect. II.
Our approach for hierarchically structured locations uses the dc:coverage property and the values it may contain. This approach is very lightweight compared to relations defined more formally but has the following advantages: • simple expression of the ‘Place or area, City, Country’ order • tag-like format that users may easily create • more accessible than a series of properties values The advantages of letting users define their own vocabulary for classifying information has already been demonstrated by the growth of tagging community, while the effectiveness of folksonomies in information classification and retrieval is becoming more and more relevant. One could extend our approach using constraints on tag-like format of property values, or indeed link photographs using controlled vocabularies. Other metadata can be added to the proposed picture description. In particular, it would be interested to add social metadata related to pictures so that social relations, other than spatial, can be discovered and presented to the users providing a social exploration of shared picture collections.
Our prototype has been a useful proof of concept but is not yet suitable for real deployment. A P2P architecture would provide an optimization of query caching and routing between the different clients at the expense of complexity in the client implementation. However, a centralized server, which would act as the repository of the pictures’ metadata and evaluate the spatial relationships between users’ pictures with batch processes, allows the development of a simple web based service without the need of a client-side application. This is a lighter-weight solution for users who wouldn’t have to download and install a full software application.
Compared to other approaches and applications, our system has the benefit of standard metadata descriptions that can easily be shared and reused in many different applications and services. The browser application built on top of these descriptions is an example of what can be done using our approach. RDF provides flexibility in how spatial information is encoded, processed and computed. One can imagine for example a browser based on social networks or an algorithm combining 13http://www.w3.org/TR/swbp-skos-core-spec/ (accessed October 2006) latitude, longitude, coverage and geographic thesauri for more accurate spatial labeling. The lightweight approach proposed for computing picture relations, and indeed the choice to rely purely on metadata rather than on information gathered from heavyweight image processing, makes our solution suitable for real time and web based applications.
There has been much interest recently in using geo-location information to relate different picture and to create an enhanced photo browsing experience.
In Sharing Places14, multimedia annotation (photo, video and audio) can be associated with physical locations to create a ’mediascape’. These trails, based on GPS information and enriched with annotation, can be accessed over the web or downloaded to a suitable device (e.g. PDA) and experienced in the real world. The trails can be tagged, published for others to find, remixed and shared. This approach relies on a central server and doesn’t provide annotation in a standard sharable metadatada format.
Images are arranged according to their location in the
WorldWide Media Exchange [
In Photo Tourism [
These approaches provide a user experience enhanced
by geo-information but don’t rely on standard format
for metadata nor provide a distributed environment for
exchanging metadata. As already pointed out [
In our work we have presented a prototype system providing ways to: • represent geographical metadata related to pictures • discover pictures relations according to the metadata • present the geotagged pictures and their relations An algorithm for inferring spatial relations between different pictures using location and compass heading information embedded in the RDF description of the pictures has been presented. A testing environment for metadata sharing and relations discovering has been implemented so that users’ photo collections are enhanced by relations with other users’ pictures.
We have shown how, based on geographical metadata expressed in RDF, it is possible to build a service for discovering, linking and browsing geographical related photo in a novel way. Our future work will deal with experiments on large test beds in order to obtain meaningful performance evaluation, improve scalability, and improve the user interface.