1 Personal photo retrieval is different from many search tasks in that all the targets are either known to the user, or they are about something the owner has seen. For this reason, generating queries for searching personal photos will more likely rely on an individual's memory. In a pilot study of Personal file re-finding, the results suggested a change of retrieval efficiency over time for different types of queries due to recall reliability. While auto-annotating of the photos with contextual information, we are seeking to develop a query weighting strategy, which takes the recall reliability into account, and give feedback to the user based on the searching queries to improve the searching efficiency. This work is supervised by Dr. Gareth J.F. Jones and funded by grant CMS023 under the Science Foundation Ireland Research Frontiers Programme 2006.
The prevalence of digital cameras (including camera mobile
phones) and advances in storage devices, while enabling the
recording of every memorable moment in our lives, is
leading to huge amount of personal digital image data,
which can be difficult for searching. Although techniques of
content based retrieval for static images are becoming
increasingly sophisticated, they may not suffice the needs of
efficient retrieval in such large collections, due to the varied
image quality and severe redundancy of content, and most
importantly, that people do not always remember what the
exact contents are, even of photos taken by themselves [1].
One possible solution is to do information retrieval (IR)
basing on the photos’ metadata (e.g. annotations) which the
user can remember. The reason is that searching for personal
photos can rather be viewed as information re-finding, as
opposed to general information seeking in an unknown
collection, like the World Wide Web. It largely depends on
the individual’s memory about the photos. For example,
while we are looking for certain photos, we usually have
some recollection of the occasion in which they were taken.
In cases where no assistant retrieval tool is provided, we
must reply on browsing the corresponding folders to find the
required photos if the folders are labelled with time, location
or event name, etc. This suggests that people may often have
better memory about contextual information related to
events than the details of the photos. Thus, we assume that if
the photos are also annotated with contextual information
that people remember well, the IR may be more efficient. In
fact, some standard forms of context data are already been
integrated into Personal Retrieval Systems, such as
MediAssist [
In our previous pilot studies [
We assume that if the document can be annotated with what the user remembers at the time of searching, the retrieval effectiveness can be maintained. We aim to develop an algorithm, which can continuously update the status of the metadata, so that when the recalled metadata is entered as query, it can be weighted dynamically according to the estimated recall reliability as well as other traditional IR methods. Above all, annotations are required for all the photos to apply the algorithm.
Manual annotation of large volumes of photos is unrealistic.
Thus, we need to do this automatically by capturing context
information which can be synchronized with the photos. For
example:
1) Time and Date can automatically be embedded into the
photos when they were created.
2) Location can be recorded by GPS devices.
3) Weather and light status can be determined by combining
time and location information at the time of creation [
We propose to develop a dynamic weighting system, structuring the data partly based on a memory model, so that the recall reliability can be evaluated, with feedback to the user to generate more efficient queries.
Information processing theories have argued that the human memory exists in Associated Networks, that nodes of remembered information are linked to each other so that they can be retrieved tracing from the links [7]. In our model, we propose to create links from attributes to items and links between attributes or same level items. We assume that the same level links are usually created based on time proximity (belonging to the same events), but which types of attributes tends to link with each other is yet to be explored.
Instead of weighting independent nodes like the Page rank algorithm does, this model weights the links’ strength, which estimate the likelihood of information on one end being retrieved if cued with the other end. Based on memory and learning theories, we propose to integrated several factors into an algorithm, such as time lapse from the last time the two nodes were associated, frequency of occurrence of the link, and encoding quality calculated from various factors [7]. The links’ weight automatically updates (e.g. when triggered by encoding of new items).
The retrieval system is based on the above structure, inheriting traditional IR strategies, and implemented with a query evaluation and feedback mechanism. Simple semantic processing will be applied to the entered queries, mainly expanding query words to arrays of synonyms from the database. The Query evaluation step will estimate the reliability of recalled query features based on the memory model. In the first stage of our system developing, the searching interface will allow the users to judge the reliability of their recall for each query (very sure, guessed, etc.). This will also combined with traditional IR methods to provide feedbacks to the user about the queries’ efficiency as well as potential possible queries or combination of queries based on the links, and leave the final decision to the user to refine their queries.
The above proposed system still needs a series of user studies which will be based on our on going data collection.
[1] Sellen, A. J., Fogg, A “Do life-logging technologies support memory for the past?: an experimental study using sensecam.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '07, ACM, New York, 2007, pp. 81-90.