=Paper=
{{Paper
|id=Vol-175/paper-26
|storemode=property
|title=Pen-based Acquisition of Real World Annotations for Document Information Spaces
|pdfUrl=https://ceur-ws.org/Vol-175/20_staeudel_penbased_final.pdf
|volume=Vol-175
|dblpUrl=https://dblp.org/rec/conf/semweb/StaudelKA05
}}
==Pen-based Acquisition of Real World Annotations for Document Information Spaces==
https://ceur-ws.org/Vol-175/20_staeudel_penbased_final.pdf
Pen-based Acquisition of Real World Annotations for
Semantic Desktops
Markus Stäudel1 , Bertin Klein2 , and Stefan Agne2
1
Fachhochschule Kaiserslautern, Standort Zweibrücken
Amerikastr. 1, 66482 Zweibrücken, Germany
mast0005@student-zw.fh-kl.de
2
German Research Center for Artificial Intelligence (DFKI GmbH)
Postfach 20 80, 67608 Kaiserslautern, Germany
{bertin.klein,stefan.agne}@dfki.de
Abstract. We explain our strategy towards a Semantic Desktop and our research
and prototype. In this paper we complement these with the paradigm of paper —
scribbling, annotating, revising—, an information interface for knowledge work,
which has been well-tried and elaborated over the centuries. Everybody is highly
trained for it. The combination can instantly improve the conditions of knowl-
edge work. By removing one more felt barrier, it has the potential to enhance a
phenomenon called flow, providing users with efficiency and satisfaction at work.
The implementation is built around a pen transmitting its movements and a state-
of-the-art handwriting recognition.
1 Introduction
The regular (wooden) desktop seems to us the authoritative archetype for any semantic
desktop. Semantics on the regular desktop is found in structures that are meaningful to
users, and in their creation, change, and maintenance ([15], compare also [2, 4]). Our
approach to new breeds of semantic desktops starts at the concepts and procedures on
regular desktops. [6, 16] Further, our foremost goal is to reduce the users cognitive load
and his mental distraction, because a mind relieved from distractions, draws the best
results from the sensed input and his prior knowledge. [19] Thus we try to sensitively
elicit the real tasks and distractions3 of users and quietly assist these (cf. [4, 9]).
Notwithstanding major achievements in personal computing, reading and writing
demand for paper. In a study aimed at knowledge capturing needs and uses at desktops
Adler et al [1] conclude among other interesting things that: “Reading occurs more fre-
quently in conjunction with writing than it does in isolation. Thus, it appears that writ-
ing (in a variety of forms) is an integral part of workrelated reading. Designers need
to seriously consider the importance of developing reading devices which also support
the marking or writing of documents during the reading process.” [1] Other work em-
phasizes the impact of the reading-writing combination on creativity. [13, 14] Notwith-
standing, that creativity is a well-understood issue lately, there is also a much more
3
Any minimal, intermediate, even unconscious goal or indirection which requires a microsec-
ond of conscious attention.
profane view on the topic: even on such simple tasks as dragging and moving, users
perform better with a stylus (on some canvas) than with a trackball or a mouse. [12]
We conclude that users should be enabled to print and work with paper. TabletPCs
are an approach, however, with the same OCR software and only 40 Euros of hardware,
we additionally get the high resolution of paper and the freedom to skim over and
spread pages. While scribbling and annotating, the strokes are captured and sent to the
electronic desktop. Plug-in modules can edit the electronic version of the document.
The plug-in module actually first completed, is a Microsoft handwriting recognition,
the output of which is used to automatically place comments at any position into the
original electronic document.
2 The Whole Approach: Semantics, Users, Desktops
Writing desks exist for two reasons. First, they are needed as a space for documents and
reminder notes, while forging ahead in constructing their semantics, as long as their
meaning has not been fully incorporated into the mind of the user. Second, computer
desktops lack important, useful elements. — Supporting users in the first by improving
the latter, we have the goal to deliver flow: It happens that one starts a day in the office
at ones writing desk and suddenly one kind of wakes up, and it is two hours later. One
has achieved a lot, but cannot easily tell what in detail one has done. This phenomenon
is called ”flow”. It enhances individual work, especially where creativity is involved,
and yields to the efficiency and satisfaction of individuals in their work. [3]
Fig. 1. A regular writing desk it the archetype of a Semantic Desktop. The right side conveys an
idea of the visualization elements (comprising 3D) with which we are experimenting.
2.1 In Support of the Semantics of a Document Collection
With books, folders, reports, notes, a calendar, todo-lists —i.e. a collection of documents—
people organize their knowledge or information, in order to fulfill their daily work. Se-
mantics here is that what such collection of documents mean to their owner, comprising
the topics, statements, conclusions, interrelations, and implications discovered by the
owner in these documents. We like the iterative flavor of “Keeping found things found”
from the University of Washington, which we consider an important aspect of the so-
called hermeneutic circle, i.e. the laborious and time consuming integration of every
new document into the other knowledge known by a person. The expert might notice,
that we conceive of semantics for a user as something like RDF or equally Topic Maps,
in the brain of the user. [5]
Our implementation and research is based on the idea that thinking involves the
senses, e.g. the visual. Figure 1 on the left shows a design sketch of our semantic desk-
top. On the right it shows a real screen and a mockup of the (really running) 3D screen.
The left image comprises not yet completely implemented visual elements like docu-
ment viewers, document stacks, Post-Its, meta-information (links), and a calendar tool
with enhanced fisheye view, and active links to documents. The right side shows our
visualization equipment, trying to tap the 3rd dimension. The monitor pair is actually
used like that, as it was found to be more efficient to complement 3D viewing with a
regular 2D view. We also experiment with a powerwall.
2.2 Virtual Desktops are Lightweight and Semantic
With all my paper documents from my writing desk as PDF documents on a USB stick I
can carry them with me at all times. A near future smart USB stick will be pluggable to
displays — perhaps a futuristic projector or my stylish glasses — and allow me to spend
a few seconds or longer at my writing desk. Note, that all my todo-lists, my calendar,
my notes, my documents, are there, my entire writing desk.
The instant advantage of a virtual, semantic desktop can be sketched with electronic
“Post-Its”: Electronic “Post-Its” can be retrieved, e.g., by its creation time, “this morn-
ing”, and with a mouse click one finds the document to which it refers. “Post-Its” can
also refer to several documents and thus establish commented links. They can also in-
clude links to appointments, todo-lists, etc. It is also possible to let popup a required
document at a specific time and at a desired text position depending on automatically
extracted information about the document. Paper “Post-Its”, on the other hand, lack
these features.
3 Gaining from Paper Editing
Imagine I am asked to do a review of an article, enclosed as a PDF. I print it, put the
paper into a clamp and start commenting. The pen transmits its movements to the clamp.
On my screen I have opened the PDF version of the document and all the comments
I write on the paper appear synchronously in the PDF, nicely in Times Font. When I
physically turn a page I tip into the lower right corner of the page, and the editor turns
page too. When I finish, I just save the document and mail it back.
Paper is an input catalyst. Many projects to overcome the boundedness of paper
failed miserably. [15] Field research has elicited that a couple of tasks are done ex-
clusively or with high tendency with paper, like revising another’s person text [15],
collaborative authoring, telephone activities or at planning and thinking. [16] It is pos-
sible to navigate quickly, apply annotations, underline important words or passages,
read through again and the like. This is not comparably possible on today computer
desktops. [13] Sure, only as an exception it is easier to actually edit with paper.
Fig. 2. Annotating a printed dokument and the result on the screen.
A stylus, a clamp, and our Gesture Port can do the job. The 40 Euro of hardware
needed comprises a stylus and a clamp. The stylus is equipped with a transmitter, which
sends coordinates (plus some events like PENUP, PENDOWN, PENMOVE, ...) wire-
lessly to a receiver embedded in the clamp and further to the desktop software (usually
vie USB). The Gesture Port interfaces input devices to our Desktop framework. An in-
termediate unit converts the movements received into meaningful gestures. Here, we
started with a (rather good) pre-release Microsoft handwriting recognition. Further, we
committed (without loss of generality of the approach) to a PDF editor, as the (fixed)
target application.
A special set of stylus gestures changes the interpretation mode, so that the interpre-
tation unit then sends commands to the target or changes the target. This allows to serve
different stylus based activities, as drawing (movements cause lines to appear), text
editing (characters are recognized and inserted, striking through deletes text), text for-
matting (underlining, italicizing, ...), layout editing (paragraphs are cut, merged, moved
...), annotating (textual annotations and links are edited/inserted, the visible document
text remains unaltered). Currently, only positioning and editing of comments are wholly
implemented. In general, the Gesture Port can also interface the data glove visible in
Figure 1. However, for writing we found the data glove rather inconvenient.
4 Related Work
Recently, there was the assortment of related work with respect to Semantic Desk-
tops made meet under the label of “Personal Information Management”. [7] Certainly,
www.gnowsis.org remains to be added. A couple of approaches are specificly rele-
vant to our paper philosophy: DigitalDesk [20], VideoMosaic [10], Ariel [11], Inter-
activeDesk [17], EnhancedDesk [8], MetaDesk [18]. Most of these Desk Systems try
to or actually use input devices like cameras and scanners. Some articulate goals very
similar to ours, but the approaches differ astoundingly. E.g. instead of a working space
on the desk to read and write, displays are embedded into the surface. Our specific task
of annotating documents is not addressed at all.
References
1. A. Adler, A. Gujar, B. Harrison, K. OHara, and A. Sellen. A diary study of work-related
reading: Design implications for digital reading devices. In Proceedings of CHI 98, 1998.
2. T. Berners-Lee. Design issues – architectural and philosophical points, October 1998.
3. M. Csikszentmihalyi and I. Csikszentmihalyi. Optimal Experience : Psychological Studies
of Flow in Consciousness. Cambridge University Press, 1988.
4. M. Eldridge, A. Sellen, and D. Bekerian. Memory problems at work: Their range frequency,
and severity. Technical Report EPC-1992-129, RXRC Cambridge Technical Report, 1992.
5. V. Gallese and T. Metzinger. Motor ontology: The representational reality of goals, actions,
and selves. Philosophical Psychology, 13(3), 2003.
6. R. Harper, A. Munro, H. Hook, and D. Benyon. Information that counts: A sociological view
of information navigation. In Social Navigation of Information Space, 1999.
7. W. Jones, H. Bruce, N. Belkin, V. Bellotti, S. Dumais, J. Grudin, J. Gwizdka, A. Halevy,
D. Karger, D. Levy, M. Perez-Quinones, and J. Raskin. The pim workshop — an nsf-
sponsored invitational workshop on personal information management, January 2005.
8. H. Koike, Y. Sato, and Y. Kobayashi. Integrating paper and digital information on enhaced-
desk: a method for real-time finger tracking on augmented desk system. ACM Trans. on
Computer-Human Interaction, 8, 2001.
9. M. Lamming, P. Brown, K. Carter, M. Eldridge, M. Flynn, G. Louie, P. Robinson, and
A. Sellen. The design of a human memory prosthesis. Computer Journal, Vol. 37(3), 1994.
10. W. Mackay. Augmented reality: linking real and virtual worlds. In In Proceedings of ACM
Conference on Advanced Visual Interfaces, AVI ’98, 1998.
11. W. Mackay, D. S. Pagani, L. Faber, B. Inwood, P. Launiainen, L. Brenta, and V. Pouzol. Ariel:
augmenting paper engineering drawings. In CHI ’95: Conference companion on Human
factors in computing systems, 1995.
12. I. S. MacKenzie, A. J. Sellen, and W. Buxton. A comparison of input devices in elemental
pointing and dragging tasks. In Proceedings of SIGCHI 91, 1991.
13. K. O’Hara and A. Sellen. A comparison of reading on-line and paper documents. In Pro-
ceedings of CHI 97, 1997.
14. K. O’Hara, A. Taylor, W. Newman, and A. Sellen. Understanding the materiality of writing
from multiple sources. International Journal of Human-Computer Studies, Vol. 56(3), 2002.
15. A. Sellen and R. . Harper. The Myth of the Paperless Office. MIT Press, 2003.
16. A. Sellen and R. Harper. Paper as an analytic resource for the design of new technologies.
In Proceedings of CHI 97, 1997.
17. A. Toshifumi, M. Kimiyoshi, K. Soshiro, and H. Shojima. Interactivedesk: a computer-
augmented desk which responds to operations on real objects. In Conference companion on
Human factors in computing systems, 1995.
18. B. Ullmer and H. Ishii. The metadesk: models and prototypes for tangible user interfaces. In
Proceedings of UIST ’97, 1997.
19. J. Waterworth and E. Waterworth. Affective creative spaces: the interactive tent and the
illusion of being. In Proc. of The Int. Conf. on Affective Human Factors Design, 2001.
20. P. Wellner. Interacting with paper on the digitaldesk. Communications of the ACM, 36(7),
1993.