Currently, almost all information retrieval systems utilize a textual interface for users to express their information needs. While this works perfectly well for searching text document collections, there are domains in which textualizing a query or context is much harder. One of these domains is the preservation of cultural heritage practices in the form of artifact usage, which involves the capture and representation of intangible assets. We have started work on a project in which we want to focus on the domain of (hand-held) tools and their use, going beyond a description in the form of text or 2D images. Our aim is to record the gestures necessary to operate the tools and to make a collection of tools accessible with this information, providing a much richer context.
The importance of preserving cultural heritage has been steadily on the rise over the past
few decades. Computer science is contributing to this effort by providing platforms for
digitizing, modeling, and visualizing artifacts or archaeological sites and making them
accessible to a wider audience [
We plan to concentrate our efforts on hand-operated tools, which are so common
in our daily lives that we rarely pay attention to them. Nevertheless, they are intriguing
for they perfectly combine appearance with dense, sometimes complex inherent
organization. Some of these objects present a basic form that developed over decades or even
centuries with little variations. The grasping and handling of these tools for proper use
– to make them work – add dimension: for example, Andre´ Leroi-Gourhan talks of the
intermeshing of tools and motive gestures [
Similar to Rosner et al. [
We want to make collections of objects accessible by describing the way they are
handled. A user describes with gestures how an item is used or moved around, which
can be accomplished with the help of data gloves, other haptic input devices, kinetic
user interfaces, or hand tracking technology such as a Leap Motion controller [
While computer science can provide the tools to enable the digitization of cultural assets and practices, it cannot give answers to questions concerning the overall philosophy of pursuing an approach like this. Involving the field of cultural and art history, as well as the field of design history and applied design, we focus on two main questions. First, how can digitized collections of objects be enhanced by visual and tactile, even interactive features for augmented information conveyance on cultural heritage? Second, how can historic product development be transposed in a modern discourse of knowledge transfer and thus be of interest even for contemporary product design?
Current digital collections of cultural objects (either operating offline or online) generally follow similar principles. The objects are categorized by basic physical features, such as material, form, and size, and by descriptions, such as texts and images, and the system offers structured search functionality, e.g. with keywords or simple image matching. Although this kind of interface works well with, e.g. books or two-dimensional artwork, it is less suitable when dealing with objects having specific modes of operation in addition to their three-dimensional appearance. For instance, (hand-operated) tools and their use are an interesting example of cultural assets whose functional aspects are either completely missing or at least underrepresented in current digital collections.
While there is interesting research work in the area of computer science trying to go
beyond keyword or feature search, we are not aware of any involving gestures to describe
the handling of artifacts that go further than moving or placing them in a virtual space.
In the following we give a brief overview of existing techniques. Ichida et al. present a
method for interactive retrieval of virtual 3D shapes using physical objects [
In an art-historic and design context, research and analysis of functional objects do exist,
but the results of this analysis are usually embedded in a discourse defined by text.
Therefore, many aspects of how an object actually functions or why it was designed in a
specific way depend on text comprehension and context awareness, whereas visual (or even
experienced) conveyance of such information is mostly missing. In particular, the efforts
that go into trying to establish design research as a scientific discipline in its own right
rely on (textual) descriptions, semantics, aesthetics, and cognition theories [
We are about to embark on a project about building a framework for the retrieval of information from a digitized collection of hand-operated tools via gestures with the plan of implementing a prototype of the framework, demonstrating the technical feasibility. We have identified several important steps on the way to the finished prototype. First, we need to lay the theoretical foundations by analyzing a specific collection of handoperated tools and identifying their fundamental properties in terms of usage. This means categorizing and describing the objects, not only in terms of their appearance, but also in terms of their mode of operation, going beyond mere technical descriptions, adding cultural and historical aspects. For example, scissors are used in a very typical fashion (similar to shears, clippers, or pliers) by repeatedly moving the thumb and one or several fingers towards and away from each other.
Second, we need to investigate models for representing hand-operated tools
according to their modes of operation. This not only involves the movement of a hand through
space, but also relative positions of fingers to each other and the positions of joints and
other parts of the hand. A gesture would be described as a spatio-temporal sequence of
basic events or movement primitives. For our scissor example this means identifying the
repeated opening and closing movements of the digits of a hand. We also need to
investigate which motion-tracking technology is suitable for this task. Currently, we are
looking into data gloves, Leap Motion controllers [
The insights gained during the first two steps will be applied when it comes to actually digitizing the artifacts. This involves creating virtual representations of the objects (e.g. via 3D scanning) and digitizing gestures and movements. Finally, for querying the digitized collection a measure determining the similarity of gestures has to be defined, as this will be crucial for the retrieval, comparison, and ranking of the returned objects. The importance of different features considered for the similarity measure depend heavily on the cultural aspects of the results obtained earlier. So, when querying the system with the gestures for using scissors, the answer would contain different types of scissors, but also include similar tools such as the aforementioned shears, clippers, and pliers (although these tools would be ranked lower).
From the point of view of computer science, our goal is to extend the boundaries of today’s information retrieval systems by developing a novel, gesture-based query interface that is much better suited for describing the information need of a user searching for manually operated artifacts. Particular hand movements used for operating tools are usually hard to translate into text, but easy to demonstrate. Consequently, it makes much more sense to capture these movements directly, transferring them into an internal representation that can be utilized for retrieval without taking a detour via text. For the moment, the main aim of our approach is to support professionals and researchers, such as designers, archaeologists, anthropologists, and historians, conducting searches in artifact collections. Once the technology becomes more inexpensive, it could also be used by museums to develop interactive displays.
However, our work goes beyond the technological dimension, we also want to
generate new impulses in a cultural setting: with the help of our project, the investigation into
the interface between man and object can be simulated, manipulated, improved, and
visualized. We want to contribute to the current philosophical discourse about the so-called
material culture by introducing the new dimension of direct experience of a forgotten
or lost object, thereby bridging the gap between oral history and the reconstruction of
knowledge by heuristic narrative methods. This may even spark a new debate about the
actor network theory by Bruno Latour, stating that objects are actors on an equal
footing with other participants in social systems, contributing substantially to civilization by
being able to create and preserve human communication [