This paper discusses the adaptation of the Scene of Crime Information System developed within an EPSRC-funded project, to the collection of data within the ImageCLEF track of the Cross Language Evaluation Forum 2003. The adaptations necessary to participate in this activity are detailed, and initial results are briefly presented. ImageCLEF is concerned with the retrieval of images from a specific collection by the captions associated to those images, and is running in relation to an EPSRC-funded project at Sheffield University (Eurovision, GR/R56778/01). The image collection consists of around 28,133 images from the photographic collection provided by St Andrews University Library (Clough et al. 2003). The 28133 images are each referred to and annotated by a single text file, and the full set of annotations are contained within one SGML-based document1. Each annotation comprises identifiers to the text file and the image files (DOCNO, SMALL_IMG, LARGE_IMG), the caption of the image (HEADLINE), a set of categories that have been assigned to this image (CATEGORIES), a database record identifier (RECORD_ID) and an unlabelled chunk of text describing the image, denoted below in italics.
From the above example, it is apparent that some of the categories assigned to the images may not be wholly reliable. While some of the associations are clear: jetty Fife rowing (boat) fishing (rods) piers and landing stages Fife all views rowing boats, rowing, fishing vessels angling, fresh water fishing, fishing equipment others could be associated to information that appears, but is not in the correct context – the combination of “Open Championship” and “St Andrews” being candidates for explaining the golfing categories – while the assignment of a “battlefields” category is less easily obvious.
The automatic ad hoc retrieval task aims at the ranked-retrieval of upto 1000 images from the Eurovision collection. The images are to be retrieved in response to a set of pre-formulated queries. The queries themselves comprise of 50 topics. Each topic has an English query, plus narrative description of the expected result of the query, and the English query has been translated into 5 other languages, French, German, Spanish, Italian and Dutch. Some queries have more than one translation for a given language.
The retrieval results are to be assessed by personnel from the University of Sheffield such that they can be evaluated using the trec_eval program with recall and precision metrics. Similar to TREC, the results will subsequently be published.
An example topic encoded in XML2 is shown below: <top> <num>Number: 25</num> <EN-title n="1">Golf course bunkers</EN-title> <EN-narr>A relevant image will show a picture of a golf course in which a bunker can be clearly identified. The picture must be a photograph or a postcard, but not a drawing, e.g. a plan of the golf course. A bunker is a sandy hollow formed by wearing away of the turf, or nowadays an artificial sand-hole with a built-up face. An example relevant document is [stand03_1714/stand03_7020].</EN-narr> </top> <top> <num>Number: 25</num> <DE-title n="1">Golfplatz Bunker</DE-title> </top> <top> <num>Number: 25</num> <FR-title n="1">Bunkers de terrain de golfe</FR-title> </top> <top> <num>Number: 25</num> <IT-title n="1">Un bunker in un percorso di golf</IT-title> <IT-title n="2">bunkers in un campo di golf</IT-title> </top> <top> <num>Number: 25</num> <ES-title n="1">Búnkers en un campo de golf</ES-title> <ES-title n="2">Pista de golf</ES-title> </top> <top> <num>Number: 25</num> <NL-title n="1">Bunkers op een golfbaan</NL-title> </top>
The example shown is for Topic 25, for which Golf course bunkers has been translated once into each of German, French and Dutch, and twice each for Spanish and Italian. With multiple translations for some languages for the 50 topics, we have the following number of queries for the various languages: 2 Similar character issues as reported previously were also fixed for this collection.
These 421 queries are to be made against the 28,133 annotations to retrieve images from the collection.
The EPSRC-funded Scene of Crime Information System (SoCIS) project was run from October 1999 to March 2003. The aim of the project was to study the link between images and texts within a specialist domain context. A method has been outlined for developing an intelligent content-based image retrieval (CBIR) system, which can store and retrieve images based on the linguistic descriptions of the images. The corpus-based method uses the lexical and semantic properties of specialist texts for extracting key terms and for discovering the ontological organisation of the terms.
A prototype CBIR system was developed in the Java programming language for demonstrating the efficacy of the
corpus-based method. The system, which is based on a 3-tier architecture of client, server, and database, can be accessed
via a local intranet. SoCIS is an intelligent CBIR system that automatically: (a) labels (and indexes) images by keywords
as well as relational facts extracted from the descriptions provided by domain experts; (b) extracts physical features of an
image; (c) populates a database comprising domain-specific terminology, together with the semantic relationships
between terms, starting from a random selection of collateral texts of the domain; and (d) learns to link image and text by
using neural networks
The SoCIS prototype3 was evaluated using images normally used for the training of Scene of Crime Officers (SoCOs)
together with a description provided by the SoCOs as well as other collateral texts like crime scene reports and forensic
science research papers and manuals. The question of (inter) indexer-variability, the variances in the output of different
indexers for the same image, has been explored in the project
SoCIS was specifically targeted at the use of specialist languages – or Languages for Special Purposes (LSP)
The first step necessary was the translation of the various queries to English. Without in-house software, we relied upon translation engines as found on the Internet. Some work was done in an attempt to exploit Google’s translation tools for this purpose, however there were difficulties encountered in this. Eventually, Altavista’s Babelfish was selected as the principal translation engine (http://babelfish.altavista.com/), however since this system does not translate Dutch, FreeTranslation.com (http://www.freetranslation.com/) was also used.
To translate the queries, Java code was used to wrap definitions of the query syntax used by these sites (with the
HTTP POST command being used in both cases). Each query was posted to the site with its requested translation
language pair, and the HTML result was retrieved. Using the Java JTidy utility, the resulting HTML was converted to
XML
The results of translating the various languages for topic number 25 (Golf course bunkers) are shown in the table below:
German Golf course shelter French Bunkers of ground of gulf Italian (1) A bunker in a distance of golf Italian (2) bunkers in a golf course Spanish (1) B??nkers in a golf course Spanish (2) Track of golf
Dutch Bunkers on a wave job
Immediately, certain of these translations will cause problems with the retrieval. The topic identifies the image stand03_1714/stand03_7020 as being relevant. In the run, this was located only for English, Italian (2), and Dutch at ranks 798, 798 and 45 respectively. The quality of returned translation will therefore have a significant impact on the results being returned. 4.2.
The thesaurus construction module of SoCIS was developed to provide a query expansion facility for the system. There
are general-purpose thesauri or lexicons available such as Wordnet4, which could be used but are inadequate in specialist
domains due to a deficiency in specialized terminology. For example, the two key compound terms ‘forensic science’
and ‘crime scene’ are not present in Wordnet. The method we developed was based on the analysis of a representative
domain-specific text corpus to automatically extract key terms and relationships, which were then used to build the
thesaurus
A program was written to query a Wordnet database to provide a set of synonyms and hyponyms for each of the query terms. In Wordnet, English nouns, verbs, adjectives and adverbs are ordered into synonym sets (synsets). Each synset can be said to contain the words that represent a specific concept. The synsets are then linked to each other based on semantic relations such as antonymy, hyponymy and meronymy. Given a query term, the program returns all the words in the synset that the particular term is an element of, as well as all the hyponyms of each synset element to a specified level in the hierarchy. Initially we planned to go down 2 levels in the hierarchy but ended up using just the synonyms due to system performance issues related to the large number of expanded terms returned, which is discussed in section 5. Taking the query “Boats on Loch Lomond” as an example, the term ‘boat’ returned 53 expanded words going down one level in the hierarchy. Some synonyms returned were: travel on water, sauceboat, gravy boat; some hyponyms returned included motorboat, mail boat, mailboat gondola, propel by oars, propel by paddles, yacht, and so on. ‘Loch’ returned one synonym lough while ‘Lomond’ was not present since it is a proper noun. The very common term ‘man’ had 131 expanded words going down one level and 344 expanded words going down two levels with words such as private, make swollen, belly out, candy striper, Homo erectus, clothes horse, ridicule with a satire, and gentleman.
Some basic morphological analysis was also carried out for each query term to account for the use of variants such as singular or plural terms as well as the verb or adjective forms. The morphology module uses standard rules (for example if a word ends with ‘ss’ or ‘h’ then the plural form is usually derived by adding an ‘es’) as well as some common exceptions (for example the plural of ~man will be ~men). This was also important for the query expansion part since Wordnet only has singular forms of words as part of the synsets so a plural word used as the query term will return no results. 4.3.
Each keyword carried a proportion of its frequency in an annotation divided by the total number of terms allocated to this annotation. The original keyword was then multiplied with weight 1, each expanded term (synonyms) returned by WordNet with weight 0.9, and words containing substrings of the original keywords with weight 0.1. The total ranking was then given by:
Rank = ∑ td × w f
t N d
Where ftd is the term frequency of term t in document d, wt is the weight of a term t as described previously, and Nd is the total number of words in document d.
The main factor to have an effect on the performance of SoCIS was that the system has been designed for the analysis of free text in specialist domains whereas with the ImageCLEF collection we were dealing with structured texts in a general domain. This resulted in difficulties for SoCIS when indexing the images – the indices produced were relatively unreliable due to the different syntactic structure of the ImageCLEF text when compared to free text, which also affected the ranking. One example here is that the system considered all the category terms given by the ImageCLEF description in the XML document (since they where enclosed in square brackets) as a single compound term. Also due to the fact that we used Wordnet for query expansion, we encountered problems associated with polysemous words as well as different word forms (see the example of boat and man in section 4.2). Due to the amount of time it was taking to process the expanded queries (some times reaching up to 300 words, see section 4.2) we had to limit the expansion to just synonyms of the original query terms. Even so we had six computers running in parallel to finish the processing, which was taking approximately 8 hours per language.
Although the combination of features outlined above would require significant efforts to develop as a usable real-world system (parallelisation and optimisation issues at least), the combination of technologies and techniques presented did enable participation in the ImageCLEF track. A system that in principle would allow a user to query a collection of images that have been annotated in English, using a query in one of six languages has been prototyped from this combination. According to the abstract from the Eurovision project, such a system had not been implemented or researched. Though far from perfect, the evaluation of the results obtained at this stage is important.
Across all languages, the following sets of results were obtained (missing topics and quantities for that topic are given in the third column):
For this selection of 6 topics, the exemplar image is only found for English for topic 21. This is an initially disappointing result. We consider, first, the top image being retrieved for each of these topics. 7 (En) stand03_1749/stan
d03_22144 14 (En) stand03_1502/stan
d03_16737 21 (En) stand03_1675/stan
d03_22740 28 (En) stand03_1714/stan
d03_7540 35 (En) stand03_1851/stan
d03_7899 42 (En) stand03_1590/stan
d03_28349 7 (Fr) 14 (Fr) 21 (Fr) 28 (Fr) 35 (Fr) 42 (Fr)
No results stand03_1853/stan d03_12134 stand03_1675/stan d03_22740 stand03_2046/stan d03_13818 stand03_1851/stan d03_7899 stand03_1590/stan d03_28349 7 (De)
No results 14 (De) stand03_1857/stan
d03_9586 21 (De) stand03_1675/stan
d03_22740 28 (De) stand03_2046/stan
d03_13818 35 (De) stand03_1851/stan
d03_7899 42 (De) stand03_2054/stan
Littlehampton. The
The Castle, Loch an Eilein Engraving of a painting of a Biblical scene, [Noah, family and the Ark at Mount Ararat].
Old Tom Morris, golfer, St Andrews. (ca 1900) Trossachs. Loch Achray, Trossachs Church and Ben An or Binnein (Ben A 'an).
Samuel Messieux, refugee from Paris and teacher of French at Madras College [South Street], St Andrews.
Boat of Garten.
Engraving of a painting of a Biblical scene, [Noah, family and the Ark at Mount Ararat].
Kingsbarns. Old Grave Stone, Kingsbarns Churchyard.
Trossachs. Loch Achray, Trossachs Church and Ben An or Binnein (Ben A 'an).
Samuel Messieux, refugee from Paris and teacher of French at Madras College [South Street], St Andrews.
View of ship at sea.
Engraving of a painting of a Biblical scene, [Noah, family and the Ark at Mount Ararat].
Kingsbarns. Old Grave Stone, Kingsbarns Churchyard.
Trossachs. Loch Achray, Trossachs Church and Ben An or Binnein (Ben A 'an).
Motherwell. Town Hall.
7 (It) 14 (It) 21 (It) 28 (It) 35 (It) 42 (It) d03_18895 stand03_1587/stan d03_28525 stand03_1502/stan d03_16737 stand03_1675/stan d03_22740 stand03_2046/stan d03_13818 stand03_1778/stan d03_4502 stand03_2054/stan d03_18895 7 (Es) stand03_1587/stan
d03_7524 14 (Es) stand03_1853/stan
d03_12134 21 (Es) stand03_1675/stan
d03_22740 28 (Es) stand03_2046/stan
d03_13818 35 (Es) stand03_2092/stan
d03_14170 42 (Es) stand03_1590/stan
d03_28349 7 (Nl) stand03_1587/stan
d03_7524 14 (Nl) stand03_1502/stan
d03_16737 21 (Nl) stand03_1974/stan
d03_11773 28 (Nl) stand03_2046/stan
d03_13818 35 (Nl) stand03_1853/stan
d03_21295 42 (Nl) stand03_2054/stan d03_18895 [Walker family?] Untitled portrait of a man.
The Castle, Loch an Eilein Engraving of a painting of a Biblical scene, [Noah, family and the Ark at Mount Ararat].
Kingsbarns. Old Grave Stone, Kingsbarns Churchyard.
Launch X.
Motherwell. Town Hall.
Man in theatrical costume. [St Andrews ?].
Boat of Garten.
Engraving of a painting of a Biblical scene, [Noah, family and the Ark at Mount Ararat].
Kingsbarns. Old Grave Stone, Kingsbarns Churchyard.
Lochgilphead. Crinan Canal at Samuel Messieux, refugee from Paris and teacher of French at Madras College [South Street], St Andrews.
No results The Castle, Loch an Eilein Brompton Oratory. Altar of Our Lady of Good Counsel.
Kingsbarns. Old Grave Stone, Kingsbarns Churchyard.
Fettercairn. Cairn o' Mount and Clatterin' Brig Motherwell. Town Hall.
These tables of results show some interesting features. For Topic 7, 3 of the queries returned no results, while those that did have a different first result. For topic 14, 5 of the 6 results refer to just 2 images. For topic 21, all but then Dutch result refer to the same image. For topic 28, all but the English result refer to the same image, however judging by the caption, the English result is the best. For topic 35, one image is referred to in 3 results. For topic 42, 2 images are equally referred to.
For Topic 14, the top 5 results have been taken once for each language, and the similarity matrix between these results is as follows:
The top 5 results show degrees of similarity between the English, Italian and Dutch results, with German and Spanish showing similarities, and French showing the most marked behavioural difference. This top 5 have captions as follows: The Castle, Loch an Eilein Boat of Garten.
Dunkeld. Loch of Craiglush and Creag nam Mial (Creagnam Hill).
Linlithgow Palace and Loch, from the air.
Bearsden. St Germain's Loch.
It would appear that a number of Lochs, apart from Loch Lomond with any boats on have been discovered in response to this query! Indeed, none of the 16 results above make mention of Lomond.
From this, it is apparent that although similar behaviour is achieved for certain language translations, the end result of retrieval is not correctly weighted. The initial concern that translation would have a significant bearing on retrieval is perhaps now not so relevant as the retrieval itself.
Taking a list of the exemplar images for retrieval, the ranking (where it exists) of that image within the 1000 results for each language was considered. For each language, if the exemplar image was retrieved within the first 1000, this was counted. If it was retrieved within the top 100 results, this was also noted. The following table presents the results obtained.
Nl De En Fr It Es 20 21 28 33 42 51 50 50 50 51 103 117 17 11 8 11 1 7 971 973 798 995 967 884 319.55 309.05 257.96 337.52 353.14 314.52
In top 100 7 11 12 11 14 19
For two queries in Italian, both for Topic 19, the exemplar image was retrieved in first place. This is certainly a result of interest given the analysis of other results in this paper. In the above table, the first column represents the language code, the second the amount of exemplar images retrieved in the 1000 results, the third is the amount of queries, the fourth and fifth show the highest and lowest ranking of the exemplars, with the sixth column showing the average ranking. Column 7 shows the quantity of exemplars occurring in the first 100 retrieved results. This set of results tends to indicate that there is some value to the approach taken here, but how that compares to other approaches remains to be seen.
Increasingly, images are being indexed and retrieved by both their visual content and by related texts such as captions
that describe the image
Numerous improvements suggest themselves, for example if the system could be grid-enabled then the different processing modules, as well as instances of the same module, could be run as a service, in parallel, which would significantly improve the processing time. The ranking mechanism needs to be further refined and tuned by carrying out more trial runs. To improve the query expansion one suggestion could be to use part-of-speech information from the query sentence to filter out some of the irrelevant expanded terms returned from Wordnet – for example in the query “Boats on loch Lomond”, the term boat is being used in the noun and not the verb form so the synonyms (propel by oars, propel by paddles) related to the verb form of boat would not be retrieved. Otherwise, an attempt could be made to analyze the British National Corpus5, which might perhaps yield only the more frequently used term associations.
Since the system deals with image retrieval, we are investigating methods of effectively combining text-based with
image-based retrieval techniques. The physical features of an image such as colour, texture, and shape can be extracted
and used in combination with the text features. This technique when incorporated into a system that learns how to index,
would result in a significant improvement in performance
This work was partially funded by the European Union through the Generic Information-based Decision Assistant (GIDA: IST-2000-31123) Projects and the EPSRC through the Scene of Crime Information System (SOCIS: GR/M89041/01) project.