A decade ago, Nakazato proposed 3D MARS, an immersive virtual reality environment for content-based image retrieval. Even so, the idea of taking advantage of post-WIMP interfaces for multimedia retrieval was no further explored for content-based retrieval. Considering the latest low-cost, o -the-shelf hardware for visualization and interaction, we believe that is time to explore immersive virtual environments for multimedia retrieval. In this paper we highlight the advantages of such approach, identifying possibilities and challenges. Focusing on a speci c eld, we introduce a preliminary immersive virtual reality prototype for 3D object retrieval. However, the concepts behind this prototype can be easily extended to the other media.
Copyright c 2011 for the individual papers by the papers’ authors.
Copying permitted only for private and academic purposes. This volume is
published and copyrighted by the editors of euroHCIR2011.
on visual queries. However, most existing solutions still face
major drawbacks and challenges to be tackled. Among
others, extensively identi ed in Datta's survey [
Notably, a decade ago, a new visualization system for
contentbased image retrieval(CBIR) was proposed by Nakazato and
Huang from the University of Illinois. The 3DMARS [
In this paper we bring up the work of Nakazato and Huang as a starting point to the exploration of new possibilities for result visualization in multimedia information retrieval. With the spreading of stereoscopic viewing and last generation interaction devices outside lab environment and into our everyday lives, we believe that in a short time users will expect richer results from multimedia search engines than just a list of thumbnails. Following this rationale, and despite it could be applied to any type of media, we will focus our approach on 3D object retrieval (3DOR).
The rst and most noticeable 3D search engine, at least
within researchers working on this area, is the Princeton
3D Model Search Engine[
Additionally to queries by example and sketch-based queries,
the FOX-MIIRE search engine[
Outside the research eld, Google 3D Warehouse3 offers a text-based search engine for the common user. This online repository contains a very large number of di erent models, from monuments to cars and furniture, humans and spaceships. However, searching for models in this collection is limited by textual queries or, when models represent real objects, by its georeference. On the other hand, the results are displayed by model images in a list, with the opportunity to manipulate a 3D view of a selected model.
Generally, the query speci cation and visualization of results in commercial tools for 3D object retrieval, usually associated with 3D model online selling sites, did not di er much from those presented above. The query is speci ed through keywords or by example and results are presented as a list of model thumbnails.
These traditional approaches to query speci cation and result visualization do not take advantage of latest advances of neither computer graphics or interaction paradigms. Current hardware and software are capable of handling millions of triangles per frame and generating complex e ects in real-time. Additionally, the growingly common use of new human-computer interaction (HCI) paradigms and devices brought new possibilities for multi-modal systems.
The recent dissemination among common users of new HCI
paradigms and devices (e.g. Nintendo Wiimote4 or
Microsoft Kinect5 ) brought new possibilities for multi-modal
systems. For decades, the \windows, icons, menus, pointing
device" (WIMP) interaction style prevailed outside the
research eld, while post-WIMP interfaces were being devised
and explored [
Particularly, the use of gestures to interact with system has
been part of the interface scene since the very early days. A
pioneering multimodal application was \Put-that-there" [
Recent technological advances allowed development of
lowcost, lightweight, easy to use systems. With limited
resources, novel and more natural HCI can be developed and
explored. For instance, Lee [
Generally, post-WIMP approaches abandoned the traditional
mouse and keyboard combination, favouring devices with six
degrees of freedom (DoF). Unlike traditional WIMP
interaction style, where it is necessary to map the inputs from a 2D
interaction space to a 3D visualization space, six DoF
devices allow straightforward direct mapping between device
movements and rotations and corresponding e ects on the
three-dimensional space. This represents an huge leap to the
concept of direct manipulation, which, according to
Shneiderman [
Combining six DoF devices with stereoscopy, it is possible
to make a multi-modal immersive interaction with direct
and natural manipulation of objects shapes within virtual
environments. This may be experienced using immersive
displays (e.g., HMDs, CAVEs) [
Nakazato focused his work on query result visualization. Thus 3D MARS supports only query-by-example mechanism to specify the search. The user select one image from a list and the system retrieves and displays the most similar images from the image database in a 3D virtual space. The image location on this space is determined by its distance to the query image, where more similar images are closer to the origin of the space. The distance in each coordinate axis depend on a pre-de ned set of features. The X-axis, Y-axis and Z-axis represent color, texture and structure of images respectively.
The interaction with the query results is done through a wand that the user holds while freely walking around the CAVE, as depicted in Figure 1. By wearing shutter glasses, the user can see a stereoscopic view of the world, which provides a full immersive experience. In such solution, visualizing query results goes far beyond scrolling on a list of thumbnails. The user navigates among the results in a three-dimensional space.
The 3D MARS was a catalyst for the incitement proposed in this paper: explore immersive visualization systems for multimedia information retrieval. Following that idea, we devised an immersive 3D virtual reality system for the display of query results of queries for 3D object Retrieval.
Taking advantage of the new paradigms in HCI, we
propose an immersive VR system for 3D object retrieval
(ImO-Ret). The version of the system presented in this
paper relies on a large-screen display, the LEMe Wall [
Regardless of the hardware details, the Im-ORet allows the
user to browse the results of a query to collection of 3D
objects in an immersive virtual environment. The objects are
distributed in the virtual 3D space according to their
similarity. This is measured by the distance of each result to
the query, which stands in the origin of the coordinates. To
each of the three axis is assigned a di erent shape matching
algorithm. The similarity to the query returned by the
corresponding algorithm determines the coordinate. Current
version of Im-O-Ret uses the Light eld Descriptors [
Similar to the 3D MARS, this work opens a myriad of new possibilities. By assigning di erent shape matching algorithms to each axis, one can adapt the query mechanism to speci c domains, producing more precise results. Applying transparency to results, it is possible to overlay results of distinct queries. Adding e ects to results, such as glow or special colors, it order to convey additional information. Since query results are not images or thumbnails, but threedimensional models, it is possible to navigate around them in the virtual environment and even manipulate them. Moreover, instead of a static view of the result, displaying it as a 3D object that can be rotating over one axis, o ers a better perception of the model. Adding stereoscopy will improve even more the visualization since the user gains depth perception over the environment.
The combined use of VE and devices with six DoF, provides a more complete visualization and makes interaction more natural, comprehensible and predictable. Their use, will also add some challenges to the implementation of such system.
While in traditional 3DOR systems the query results are
represented and ordered as a list of thumbnails ordered by
a given similarity measure, when we move to a virtual
environment, the distribution of results in a 3D space becomes a
challenge. How query results should arranged in 3D space to
be meaningful to the user remains an open question. In our
approach we select three shape descriptors and assigned each
one to a coordinate axis, but this is a preliminary approach.
We believe that a nal solution is more complex that this.
Further investigation on this topic is clearly required.
On the other hand, the way users navigate and interact with
objects in an immersive environment and interact with it
still an open issue. Norman[
Above all, an important challenge remains open. No easy query speci cation mechanism has been presented, neither in traditional search engines, nor with new HCI paradigms. Although sketch-based queries apparently provide good results, they greatly depend on the ability of the user to draw a 3D model, which hinders the goal of a widely used, contentbased, 3D search engine.
We believe that recent advances in low-cost, post-WIMP enabler technology, can be seen as an opportunity to overcome some drawbacks of current multimedia information retrieval solutions. Combined with the dissemination of stereoscopic visualization as a commodity, these interaction paradigms will acquaint common users with immersive virtual reality environments.
In this paper we highlight that such scenario is a fertile ground to be explored by search engines for multimedia information retrieval. In that context, we identi ed two major research topics: query result visualization and query specication. While the latest requires further study, we already started tackling the rst one.
We developed a novel visualization approach for 3D object retrieval. The Im-O-Ret o ers the users an immersive virtual environment for browsing results of a query to a collection of 3D objects. The query results are displayed as 3D models in a 3D space, instead of the traditional list of thumbnails. The user can explore the results, navigating in that space and directly manipulating the objects. Looking back to 3D MARS, the initial work proposed by Nakazaro, we realize it was a valid idea that fell almost into obliviousness. We expect that our preliminary work, which lies over concepts introduced by 3D MARS, could prove the goodness of our incitement to explore the possibilities offered by immersive virtual environments to the multimedia information retrieval.
The work described in this paper was partially supported by the Portuguese Foundation for Science and Technology (FCT) through the project 3DORuS, reference PTDC/EIAEIA/102930/2008 and by the INESC-ID multiannual funding, through the PIDDAC Program funds.