Pen-based Acquisition of Real World Annotations for Semantic Desktops Markus Stäudel1 , Bertin Klein2 , and Stefan Agne2 1 Fachhochschule Kaiserslautern, Standort Zweibrücken Amerikastr. 1, 66482 Zweibrücken, Germany mast0005@student-zw.fh-kl.de 2 German Research Center for Artificial Intelligence (DFKI GmbH) Postfach 20 80, 67608 Kaiserslautern, Germany {bertin.klein,stefan.agne}@dfki.de Abstract. We explain our strategy towards a Semantic Desktop and our research and prototype. In this paper we complement these with the paradigm of paper — scribbling, annotating, revising—, an information interface for knowledge work, which has been well-tried and elaborated over the centuries. Everybody is highly trained for it. The combination can instantly improve the conditions of knowl- edge work. By removing one more felt barrier, it has the potential to enhance a phenomenon called flow, providing users with efficiency and satisfaction at work. The implementation is built around a pen transmitting its movements and a state- of-the-art handwriting recognition. 1 Introduction The regular (wooden) desktop seems to us the authoritative archetype for any semantic desktop. Semantics on the regular desktop is found in structures that are meaningful to users, and in their creation, change, and maintenance ([15], compare also [2, 4]). Our approach to new breeds of semantic desktops starts at the concepts and procedures on regular desktops. [6, 16] Further, our foremost goal is to reduce the users cognitive load and his mental distraction, because a mind relieved from distractions, draws the best results from the sensed input and his prior knowledge. [19] Thus we try to sensitively elicit the real tasks and distractions3 of users and quietly assist these (cf. [4, 9]). Notwithstanding major achievements in personal computing, reading and writing demand for paper. In a study aimed at knowledge capturing needs and uses at desktops Adler et al [1] conclude among other interesting things that: “Reading occurs more fre- quently in conjunction with writing than it does in isolation. Thus, it appears that writ- ing (in a variety of forms) is an integral part of workrelated reading. Designers need to seriously consider the importance of developing reading devices which also support the marking or writing of documents during the reading process.” [1] Other work em- phasizes the impact of the reading-writing combination on creativity. [13, 14] Notwith- standing, that creativity is a well-understood issue lately, there is also a much more 3 Any minimal, intermediate, even unconscious goal or indirection which requires a microsec- ond of conscious attention. profane view on the topic: even on such simple tasks as dragging and moving, users perform better with a stylus (on some canvas) than with a trackball or a mouse. [12] We conclude that users should be enabled to print and work with paper. TabletPCs are an approach, however, with the same OCR software and only 40 Euros of hardware, we additionally get the high resolution of paper and the freedom to skim over and spread pages. While scribbling and annotating, the strokes are captured and sent to the electronic desktop. Plug-in modules can edit the electronic version of the document. The plug-in module actually first completed, is a Microsoft handwriting recognition, the output of which is used to automatically place comments at any position into the original electronic document. 2 The Whole Approach: Semantics, Users, Desktops Writing desks exist for two reasons. First, they are needed as a space for documents and reminder notes, while forging ahead in constructing their semantics, as long as their meaning has not been fully incorporated into the mind of the user. Second, computer desktops lack important, useful elements. — Supporting users in the first by improving the latter, we have the goal to deliver flow: It happens that one starts a day in the office at ones writing desk and suddenly one kind of wakes up, and it is two hours later. One has achieved a lot, but cannot easily tell what in detail one has done. This phenomenon is called ”flow”. It enhances individual work, especially where creativity is involved, and yields to the efficiency and satisfaction of individuals in their work. [3] Fig. 1. A regular writing desk it the archetype of a Semantic Desktop. The right side conveys an idea of the visualization elements (comprising 3D) with which we are experimenting. 2.1 In Support of the Semantics of a Document Collection With books, folders, reports, notes, a calendar, todo-lists —i.e. a collection of documents— people organize their knowledge or information, in order to fulfill their daily work. Se- mantics here is that what such collection of documents mean to their owner, comprising the topics, statements, conclusions, interrelations, and implications discovered by the owner in these documents. We like the iterative flavor of “Keeping found things found” from the University of Washington, which we consider an important aspect of the so- called hermeneutic circle, i.e. the laborious and time consuming integration of every new document into the other knowledge known by a person. The expert might notice, that we conceive of semantics for a user as something like RDF or equally Topic Maps, in the brain of the user. [5] Our implementation and research is based on the idea that thinking involves the senses, e.g. the visual. Figure 1 on the left shows a design sketch of our semantic desk- top. On the right it shows a real screen and a mockup of the (really running) 3D screen. The left image comprises not yet completely implemented visual elements like docu- ment viewers, document stacks, Post-Its, meta-information (links), and a calendar tool with enhanced fisheye view, and active links to documents. The right side shows our visualization equipment, trying to tap the 3rd dimension. The monitor pair is actually used like that, as it was found to be more efficient to complement 3D viewing with a regular 2D view. We also experiment with a powerwall. 2.2 Virtual Desktops are Lightweight and Semantic With all my paper documents from my writing desk as PDF documents on a USB stick I can carry them with me at all times. A near future smart USB stick will be pluggable to displays — perhaps a futuristic projector or my stylish glasses — and allow me to spend a few seconds or longer at my writing desk. Note, that all my todo-lists, my calendar, my notes, my documents, are there, my entire writing desk. The instant advantage of a virtual, semantic desktop can be sketched with electronic “Post-Its”: Electronic “Post-Its” can be retrieved, e.g., by its creation time, “this morn- ing”, and with a mouse click one finds the document to which it refers. “Post-Its” can also refer to several documents and thus establish commented links. They can also in- clude links to appointments, todo-lists, etc. It is also possible to let popup a required document at a specific time and at a desired text position depending on automatically extracted information about the document. Paper “Post-Its”, on the other hand, lack these features. 3 Gaining from Paper Editing Imagine I am asked to do a review of an article, enclosed as a PDF. I print it, put the paper into a clamp and start commenting. The pen transmits its movements to the clamp. On my screen I have opened the PDF version of the document and all the comments I write on the paper appear synchronously in the PDF, nicely in Times Font. When I physically turn a page I tip into the lower right corner of the page, and the editor turns page too. When I finish, I just save the document and mail it back. Paper is an input catalyst. Many projects to overcome the boundedness of paper failed miserably. [15] Field research has elicited that a couple of tasks are done ex- clusively or with high tendency with paper, like revising another’s person text [15], collaborative authoring, telephone activities or at planning and thinking. [16] It is pos- sible to navigate quickly, apply annotations, underline important words or passages, read through again and the like. This is not comparably possible on today computer desktops. [13] Sure, only as an exception it is easier to actually edit with paper. Fig. 2. Annotating a printed dokument and the result on the screen. A stylus, a clamp, and our Gesture Port can do the job. The 40 Euro of hardware needed comprises a stylus and a clamp. The stylus is equipped with a transmitter, which sends coordinates (plus some events like PENUP, PENDOWN, PENMOVE, ...) wire- lessly to a receiver embedded in the clamp and further to the desktop software (usually vie USB). The Gesture Port interfaces input devices to our Desktop framework. An in- termediate unit converts the movements received into meaningful gestures. Here, we started with a (rather good) pre-release Microsoft handwriting recognition. Further, we committed (without loss of generality of the approach) to a PDF editor, as the (fixed) target application. A special set of stylus gestures changes the interpretation mode, so that the interpre- tation unit then sends commands to the target or changes the target. This allows to serve different stylus based activities, as drawing (movements cause lines to appear), text editing (characters are recognized and inserted, striking through deletes text), text for- matting (underlining, italicizing, ...), layout editing (paragraphs are cut, merged, moved ...), annotating (textual annotations and links are edited/inserted, the visible document text remains unaltered). Currently, only positioning and editing of comments are wholly implemented. In general, the Gesture Port can also interface the data glove visible in Figure 1. However, for writing we found the data glove rather inconvenient. 4 Related Work Recently, there was the assortment of related work with respect to Semantic Desk- tops made meet under the label of “Personal Information Management”. [7] Certainly, www.gnowsis.org remains to be added. A couple of approaches are specificly rele- vant to our paper philosophy: DigitalDesk [20], VideoMosaic [10], Ariel [11], Inter- activeDesk [17], EnhancedDesk [8], MetaDesk [18]. Most of these Desk Systems try to or actually use input devices like cameras and scanners. Some articulate goals very similar to ours, but the approaches differ astoundingly. E.g. instead of a working space on the desk to read and write, displays are embedded into the surface. Our specific task of annotating documents is not addressed at all. References 1. A. Adler, A. Gujar, B. Harrison, K. OHara, and A. Sellen. A diary study of work-related reading: Design implications for digital reading devices. In Proceedings of CHI 98, 1998. 2. T. Berners-Lee. Design issues – architectural and philosophical points, October 1998. 3. M. Csikszentmihalyi and I. Csikszentmihalyi. Optimal Experience : Psychological Studies of Flow in Consciousness. Cambridge University Press, 1988. 4. M. Eldridge, A. Sellen, and D. Bekerian. Memory problems at work: Their range frequency, and severity. Technical Report EPC-1992-129, RXRC Cambridge Technical Report, 1992. 5. V. Gallese and T. Metzinger. Motor ontology: The representational reality of goals, actions, and selves. Philosophical Psychology, 13(3), 2003. 6. R. Harper, A. Munro, H. Hook, and D. Benyon. Information that counts: A sociological view of information navigation. In Social Navigation of Information Space, 1999. 7. W. Jones, H. Bruce, N. Belkin, V. Bellotti, S. Dumais, J. Grudin, J. Gwizdka, A. Halevy, D. Karger, D. Levy, M. Perez-Quinones, and J. Raskin. The pim workshop — an nsf- sponsored invitational workshop on personal information management, January 2005. 8. H. Koike, Y. Sato, and Y. Kobayashi. Integrating paper and digital information on enhaced- desk: a method for real-time finger tracking on augmented desk system. ACM Trans. on Computer-Human Interaction, 8, 2001. 9. M. Lamming, P. Brown, K. Carter, M. Eldridge, M. Flynn, G. Louie, P. Robinson, and A. Sellen. The design of a human memory prosthesis. Computer Journal, Vol. 37(3), 1994. 10. W. Mackay. Augmented reality: linking real and virtual worlds. In In Proceedings of ACM Conference on Advanced Visual Interfaces, AVI ’98, 1998. 11. W. Mackay, D. S. Pagani, L. Faber, B. Inwood, P. Launiainen, L. Brenta, and V. Pouzol. Ariel: augmenting paper engineering drawings. In CHI ’95: Conference companion on Human factors in computing systems, 1995. 12. I. S. MacKenzie, A. J. Sellen, and W. Buxton. A comparison of input devices in elemental pointing and dragging tasks. In Proceedings of SIGCHI 91, 1991. 13. K. O’Hara and A. Sellen. A comparison of reading on-line and paper documents. In Pro- ceedings of CHI 97, 1997. 14. K. O’Hara, A. Taylor, W. Newman, and A. Sellen. Understanding the materiality of writing from multiple sources. International Journal of Human-Computer Studies, Vol. 56(3), 2002. 15. A. Sellen and R. . Harper. The Myth of the Paperless Office. MIT Press, 2003. 16. A. Sellen and R. Harper. Paper as an analytic resource for the design of new technologies. In Proceedings of CHI 97, 1997. 17. A. Toshifumi, M. Kimiyoshi, K. Soshiro, and H. Shojima. Interactivedesk: a computer- augmented desk which responds to operations on real objects. In Conference companion on Human factors in computing systems, 1995. 18. B. Ullmer and H. Ishii. The metadesk: models and prototypes for tangible user interfaces. In Proceedings of UIST ’97, 1997. 19. J. Waterworth and E. Waterworth. Affective creative spaces: the interactive tent and the illusion of being. In Proc. of The Int. Conf. on Affective Human Factors Design, 2001. 20. P. Wellner. Interacting with paper on the digitaldesk. Communications of the ACM, 36(7), 1993.