We explain our strategy towards a Semantic Desktop and our research and prototype. In this paper we complement these with the paradigm of paper scribbling, annotating, revising-, an information interface for knowledge work, which has been well-tried and elaborated over the centuries. Everybody is highly trained for it. The combination can instantly improve the conditions of knowledge work. By removing one more felt barrier, it has the potential to enhance a phenomenon called flow, providing users with efficiency and satisfaction at work. The implementation is built around a pen transmitting its movements and a stateof-the-art handwriting recognition.
The regular (wooden) desktop seems to us the authoritative archetype for any semantic
desktop. Semantics on the regular desktop is found in structures that are meaningful to
users, and in their creation, change, and maintenance ([
Notwithstanding major achievements in personal computing, reading and writing
demand for paper. In a study aimed at knowledge capturing needs and uses at desktops
Adler et al [
We conclude that users should be enabled to print and work with paper. TabletPCs are an approach, however, with the same OCR software and only 40 Euros of hardware, we additionally get the high resolution of paper and the freedom to skim over and spread pages. While scribbling and annotating, the strokes are captured and sent to the electronic desktop. Plug-in modules can edit the electronic version of the document. The plug-in module actually first completed, is a Microsoft handwriting recognition, the output of which is used to automatically place comments at any position into the original electronic document. 2
Writing desks exist for two reasons. First, they are needed as a space for documents and
reminder notes, while forging ahead in constructing their semantics, as long as their
meaning has not been fully incorporated into the mind of the user. Second, computer
desktops lack important, useful elements. — Supporting users in the first by improving
the latter, we have the goal to deliver flow: It happens that one starts a day in the office
at ones writing desk and suddenly one kind of wakes up, and it is two hours later. One
has achieved a lot, but cannot easily tell what in detail one has done. This phenomenon
is called ”flow”. It enhances individual work, especially where creativity is involved,
and yields to the efficiency and satisfaction of individuals in their work. [
With books, folders, reports, notes, a calendar, todo-lists —i.e. a collection of documents—
people organize their knowledge or information, in order to fulfill their daily work.
Semantics here is that what such collection of documents mean to their owner, comprising
the topics, statements, conclusions, interrelations, and implications discovered by the
owner in these documents. We like the iterative flavor of “Keeping found things found”
from the University of Washington, which we consider an important aspect of the
socalled hermeneutic circle, i.e. the laborious and time consuming integration of every
new document into the other knowledge known by a person. The expert might notice,
that we conceive of semantics for a user as something like RDF or equally Topic Maps,
in the brain of the user. [
Our implementation and research is based on the idea that thinking involves the senses, e.g. the visual. Figure 1 on the left shows a design sketch of our semantic desktop. On the right it shows a real screen and a mockup of the (really running) 3D screen. The left image comprises not yet completely implemented visual elements like document viewers, document stacks, Post-Its, meta-information (links), and a calendar tool with enhanced fisheye view, and active links to documents. The right side shows our visualization equipment, trying to tap the 3rd dimension. The monitor pair is actually used like that, as it was found to be more efficient to complement 3D viewing with a regular 2D view. We also experiment with a powerwall. 2.2
With all my paper documents from my writing desk as PDF documents on a USB stick I can carry them with me at all times. A near future smart USB stick will be pluggable to displays — perhaps a futuristic projector or my stylish glasses — and allow me to spend a few seconds or longer at my writing desk. Note, that all my todo-lists, my calendar, my notes, my documents, are there, my entire writing desk.
The instant advantage of a virtual, semantic desktop can be sketched with electronic “Post-Its”: Electronic “Post-Its” can be retrieved, e.g., by its creation time, “this morning”, and with a mouse click one finds the document to which it refers. “Post-Its” can also refer to several documents and thus establish commented links. They can also include links to appointments, todo-lists, etc. It is also possible to let popup a required document at a specific time and at a desired text position depending on automatically extracted information about the document. Paper “Post-Its”, on the other hand, lack these features. 3
Imagine I am asked to do a review of an article, enclosed as a PDF. I print it, put the paper into a clamp and start commenting. The pen transmits its movements to the clamp. On my screen I have opened the PDF version of the document and all the comments I write on the paper appear synchronously in the PDF, nicely in Times Font. When I physically turn a page I tip into the lower right corner of the page, and the editor turns page too. When I finish, I just save the document and mail it back.
Paper is an input catalyst. Many projects to overcome the boundedness of paper
failed miserably. [
A stylus, a clamp, and our Gesture Port can do the job. The 40 Euro of hardware needed comprises a stylus and a clamp. The stylus is equipped with a transmitter, which sends coordinates (plus some events like PENUP, PENDOWN, PENMOVE, ...) wirelessly to a receiver embedded in the clamp and further to the desktop software (usually vie USB). The Gesture Port interfaces input devices to our Desktop framework. An intermediate unit converts the movements received into meaningful gestures. Here, we started with a (rather good) pre-release Microsoft handwriting recognition. Further, we committed (without loss of generality of the approach) to a PDF editor, as the (fixed) target application.
A special set of stylus gestures changes the interpretation mode, so that the interpretation unit then sends commands to the target or changes the target. This allows to serve different stylus based activities, as drawing (movements cause lines to appear), text editing (characters are recognized and inserted, striking through deletes text), text formatting (underlining, italicizing, ...), layout editing (paragraphs are cut, merged, moved ...), annotating (textual annotations and links are edited/inserted, the visible document text remains unaltered). Currently, only positioning and editing of comments are wholly implemented. In general, the Gesture Port can also interface the data glove visible in Figure 1. However, for writing we found the data glove rather inconvenient. 4
Recently, there was the assortment of related work with respect to Semantic
Desktops made meet under the label of “Personal Information Management”. [