Introduction

Charles Bourne and Douglas Engelbart open their 1958 paper, "Facets of the Technical Information Problem," [1] with:

RECENT world events have catapulted the problem of the presently unmanageable mass of technical information from one that should be solved to one that must be solved. The question is receiving serious and thoughtful consideration in many places in government, industry, and in the scientific and technical community.

If networked computers will be the "printing presses of the twenty-first century" and beyond, then networked semantic desktop applications will be the workstations of many of those knowledge workers mentioned by Stefan Decker and Martin Frank in their 2004 paper, "The Social Semantic Desktop" [6]. Today, knowledge workers are accustomed to the use of applications such as email, calendar, word processing, spreadsheets, and more. Each of those applications can be viewed as stand-alone entities, each facilitating the accomplishment of some particular task, but in no particular sense integrated in ways we shall call semantic with each other. An appropriate interpretation of John Stuart Mill's 1873 [14] call for fundamental changes in our modes of thought would suggest that we look at semantic integration of the tools with which we perform knowledge work.

In this paper, we will introduce a new semantic desktop system called IRIS (for Integrate-Relate-Infer-Share) and explain the context in which it was built, as part of the CALO Cognitive Assistant project. As we describe our quest to construct the kinds of tools that will significantly enhance the desktop user's experience and augment the user's ability to perform knowledge work, we will explain our design decisions, progress, and shortcomings.

Background and Requirements

I believe that at the end of the century the use of words and general educated opinion will have altered so much that one will be able to speak of machines thinking without expecting to be contradicted.

-Alan Turing, Computing Machinery and Intelligence 1950

Before discussing the IRIS project, we will briefly describe CALO 1 , an artificial intelligence application for which IRIS serves as the semantic desktop user interface. Requirements from the CALO program have greatly influenced our design for IRIS.

IRIS has been developed as part of SRI's CALO2 project, one of two projects funded under DARPA's "Perceptive Assistant that Learns" (PAL) program 3 . The goal of the PAL program is to develop an enduring personal assistant that "learns in the wild," evolving its abilities more and more through automated machine learning techniques rather than through code changes. DARPA expects the program to generate innovative ideas that result in new science, new and fundamental approaches to current problems, and new algorithms and tools, and to yield new technology of significant value to the military and commercial sectors. Led by SRI International, 250 researchers and developers from 25 universities and companies are working on CALO.

CALO is a cognitive software system that can reason, learn from experience, be told what to do, explain what it is doing, reflect on its experience, and respond robustly to surprise. CALO's mission is to serve its user as a personal assistant, collaborating in all aspects of work life: organizing information; preparing information artifacts; mediating person-person interactions; organizing and scheduling in time; monitoring and managing tasks; and acquiring and allocating resources.

To understand and reason about the dynamics of the user's work life, CALO requires a semantically coherent view into the user's life and a mechanism for interacting with the user in a natural work setting. Our solution was to outfit CALO researchers with a semantic desktop, called IRIS, that enables them to outline the key ele-ments in their environment, specifically: the projects the user works on; the key participants in various roles for these projects; the way in which accessed information relate to people, projects, and tasks in the user's life; and the priorities of tasks, messages, documents, and meetings. CALO plays the role of a collaborative teammate participating in this exercise, learning how to populate much of the semantic content and relationships on behalf of the user and the rest of his team.

In approaching the design and development of IRIS, we took much inspiration from the work of Douglas Engelbart. While Ted Nelson's Xanadu4 [15] was arguably the first project to set the stage for modern hyperdocument processors, Engelbart's Augment5 was the first system to actually find engagement in group document processing and sharing. In 1968, at the Fall Joint Computer Conference in San Francisco, Engelbart demonstrated Augment before a live audience. 6 Augment displayed many of the capabilities we now want to build into modern semantic desktop applications. Augment, the program, saw commercial application, and is still used today by Dr. Engelbart in his day-to-day activities. Efforts are under way to create open source variants of the Augment system [19]. At the same time, work continues on the development of an Open Hyperdocument System [9] guided by Dr. Engelbart. CALO, Engelbart's work, the paper by Decker and Frank [6], and an earlier paper by Gradman [10] combine to provoke background thoughts that drive the evolution of our requirements. Here are several that animate the IRIS project:

1. "Real" enough to do daily work. As dictated by Engelbart's notion of bootstrapping, 7 we should develop using that which we are developing. To convince people to give up their current mail program, web browser, or calendar in favor of IRIS, we will need to offer a full-featured experience that supports all of their specific needs: mail encryption, spam filters, calendar servers, synchronization with PIMs 8 , embedded flash, etc. Rather than implement these features ourselves, we opt to find and integrate the most mature third-party applications available into our developments. 2. Implemented in, or able to easily integrate with, Java. This requirement comes from the fact that many of the machine learning components we will include from CALO researchers are implemented in Java. 3. Ontology-based knowledge store. We require the ability to model rich semantic structures that can capture every aspect of a user's work environment. 4. Capable of supporting organization of personal knowledge assets. This implies providing for the ability of users to organize their information resources in ways that suit individual needs ("just for me" [4]) while maintaining semantic interoperability with other CALO installations. 5. Cross-platform. IRIS should be able to run on Windows, Macintosh, and Linux platforms, to support as widely as possible the diverse CALO community.

Related Work

With these requirements in mind, we set about looking at candidate solutions that could meet our needs. We started by looking at Mitch Kapor's Chandler9 project, which certainly belongs in the semantic desktop category. The Chandler web site says this:

With Chandler, users will be able to organize diverse kinds of information for their own convenience --not the computer's convenience. Chandler will have a rich ability not only to associate and interconnect items, but also to gather and collect related items in a single place creating a context sensitive "view" of many types of data, mixing-and-matching email, mailing lists, instant messages, appointments, contacts, tasks, free-form notes, blogs, web pages, documents, spreadsheets, slide shows, bookmarks, photos, MP3's, and so on. While Chandler's vision resonated with what we wanted for IRIS, its early stage of development and long product roadmap made Chandler an unsuitable starting point.

We next explored Haystack10 from MIT. When we discovered this project [12], we were amazed how well it fit our initial designs for IRIS, in terms of both architecture and user interface design, with the added benefit of being Java-based and open source. We invited Dennis Quan to visit SRI to discuss the internals of Haystack in relation to our perceived needs. We learned much from the visit and did, indeed, begin the task of adapting Haystack's significant code base to our framework. Haystack's approach to ontology-driven architectures was to create a language, Adenine [3]. With Adenine, all user interface objects, the overall system architecture, and information assets are defined in an ontology. IRIS took a slightly different approach; an OWL ontology defines information assets. Instead of an API based on a language like Adenine, IRIS implements specialized APIs for each OWL class. This provides programmers with a convenient, object oriented access to the knowledgebase. The user interface layout for Haystack and IRIS are greatly similar; both rely on a threecolumn view structure, where the three concerns of navigation, focus of attention, and context are each presented in their own view. For a variety of reasons, we ended up moving in a different direction, but Haystack and Dr. Quan's deep knowledge of the subject gave us a solid start.

The next system we evaluated was the Radar Networks 11 Personal Radar, a very impressive semantic desktop that turned out to share many of the goals and requirements for IRIS: Java-based, ontology-driven, user centric. We negotiated, and CEO Nova Spivack agreed to join the CALO project to help combine elements of Personal Radar into the IRIS code-base. In particular, we adopted their Semantic Object framework, a very fast triple-store implementation, and certain elements from their SWING-based user interface. These are described in more detail in Section 4.

Well down the path of implementing IRIS, we came across Gnowsis 12 . Gnowsis [18] appears to offer integration with many of the same third-party applications as IRIS, and to share many similar philosophies regarding application and data integration. Where IRIS and Gnowsis currently diverge may lie in the way in which those applications are integrated. Whereas Gnowsis appears to have fairly loose integration with standalone applications, using adapters to copy references from applications into a local server and a separate browser for navigating and searching the data, IRIS has chosen to be more tightly integrated at the user interface level, providing an "embedded suite" of applications. Each plug-in application is instrumented such that IRIS captures semantic events as they occur. For example, IRIS "knows" which web page is being browsed, or which email has been opened for reading. Tight integration of applications is particularly useful to IRIS's learning framework and components, which can offer real-time suggestions as the user works with information. Having said that, it is worth noting that, as work on IRIS progresses, we are beginning to relax the tight integration between the presentation layer and the backside. Indeed, notions of external presentation elements are now under consideration.

Most recently, we discovered MindRaider 13 , a project arguably close to IRIS, Haystack, and Gnowsis in spirit and intent. MindRaider's open source license precludes us from looking closely at the source code, but we observe, while running the program, that there are profound similarities between its ontology-driven architecture and that of IRIS. Without examining implementation, we suspect that a central ontology is at work determining classes of information assets and constraining relationships between those classes. We also note the same interesting parallels in user interface design as mentioned with Haystack. We suspect that HCI (user interface design) will eventually rise to be at least as important to the success of semantic desktops as is semantic interoperability among platforms.

Topic Maps [17] is another research area we are tracking, as we consider IRIS a kind of topic map for personal information assets. A topic map 14 provides a container for proxies for subjects, called topics. Each subject, which is anything that can be the focus of thought or discussion, is represented by one topic. Each topic is a kind of container for links to all known information about the subject. Topics have properties that provide subject identity and other properties of the subject. Topics can play roles in association with other topics. For instance, a topic, which is a proxy for the IRIS user, can play roles such as member in a meeting, speaker at a conference, parent or spouse, and more. Topics are associated with occurrences. For instance, the topic for a particular personal computer can be linked with occurrences such as web pages where that computer can be purchased, or where an online help system is found. Topics are also repositories for all possible ways to name the topic. With this, individuals can assign names for things in their personal space; that personal space thus gains the "just for me" [4] flavor. Since IRIS is an "ontology-driven" platform, the addition of a topic map structure to the IRIS knowledgebase facilitates this "just for me" characteristic. User assigned names and relationships can be added without affecting the IRIS ontology.

The IRIS Semantic Desktop

IRIS is an application framework for enabling users to create a "personal map" across their office-related information objects. IRIS is an acronym for "Integrate. Relate. Infer. Share." In the following text, we will adopt these four terms as organizing subsections, as we describe IRIS's design, architecture, implementation, and usecases.

IRIS -Integrate

Hypertext is a form of storage, a new form of literature, and a network that just might revitalize human life.

-Ted Nelson 1965 IRIS is first and foremost an integration framework. Whereas in today's packaged applications suites, where only loose data integration exists 15 (usually limited to the clipboard and common look-and-feel for menus and dialog boxes), IRIS strives to integrate data from disparate applications using reified semantic classes and typed relations. For instance, it should be possible to express that "File F was presented at Meeting M by Tom Jones, who is the Project Manager of Project X," even if the file manager, calendar program, contact database, and project management software are separately-developed third-party applications. In a Topic Maps fashion, there should be a single instance that represents each concept, and all that is knowable about that concept should be directly accessible from that instance [17].

The IRIS framework offers integration services at three levels (Figure 1):

1. Information resources (e.g., an email message, a calendar appointment) and the applications that create and manipulate them must be made accessible to IRIS for instrumentation, automation, and query. IRIS offers a plug-in framework in the style of the Eclipse 16 architecture or the JPF framework 17 , where "applications" (components with a graphical user interface) and "services" (processing components with no GUI of their own) can be defined and integrated within IRIS. Apart from a very small, lightweight kernel, all functionality within IRIS is defined using the plug-in framework, including user interface, applications, back-end persistence store, learning modules, harvesters, and so forth. 3. The IRIS user interface framework allows plug-in applications to embed their own interfaces within IRIS, and to interoperate with global UI services such as the notification pane, menu and toolbar management, query interfaces, the link manager, and suggestion pane. IRIS comes "out of the box" with several integrated office applications:

• Email: After initially integrating Java-based Columba, 18 we moved to Mozilla 19 for email, as it is one of the most popular, full-featured, cross-platform applications available. To integrate Mozilla with Java, we adopted and made significant extensions to the JREX 20 package, and then ran Email as an embedded XUL 21 application.

• Web browser: Mozilla provides an much better web browsing experience than our initial integration effort, Java-based CALPA.22 • Calendar: We selected OpenOffice GLOW23 because it is Java-based and iCAL compliant, interoperates with the Sun/Netscape calendar server used by SRI, and has a very nice user interface. We believe there remains room for improvement in our calendar application. • Chat: We implemented our own interface to the Jabber24 instant messaging backend. • File explorer: We wrote our own in Java.

• Data editor/viewer: To view and edit data records such as people, projects, tasks, and any other ontology-based object in the KB, we used a forms package from Radar Networks' Personal Radar software. The IRIS user interface provides the "shell" for hosting each of these embedded applications (figure 2). Two collapsible side panels frame the main application window, one for selecting among available applications, the other for displaying and editing semantic links for the selected application object and presenting contextual suggestions from the learning framework. Applications can add toolbars to the IRIS frame, and when selected, an application's menu items are "merged" with IRIS menu functionality present for all applications. IRIS provides an extensible context-sensitive online help system and several methods for querying information resources within and across applications. An example of a natural language query supported by iAnywhere's Answers Anywhere25 IRIS plugin is "find email from Vinay last week related to the CALO project."

IRIS -Relate

Information is both more and less real than the material universe. It's more real because it will survive any physical change; it will outlast any physical manifestation of itself. It's less real because it's ineffable. For example, you can touch a shoe, but you can't touch the notion of "shoe-ness" (that is, what it means to be a shoe). The notion of shoe-ness is probably eternal, but every shoe is ephemeral.

-Steven R. Newcomb [17, page 32] IRIS is used to semantically integrate the tools of knowledge work. What do we mean by this? We use the term "semantic" in the sense used by the Semantic Web community, where markup technologies are being wedded to the tools of semantic representation (e.g., ontologies, OWL, RDF). This facilitates putting data on the web in such a way that machines can access it, make meaningful references to it, and perform manipulations on it, including reasoning and inference. In that sense, IRIS provides a knowledge representation by which the artifacts of a user's experience such as email messages, calendar events, files on the disk or found on the web, can be stored and related to each other across applications and across users.

When defining the ontology to be used for IRIS, a design choice had to be made: Do we use a small, simple ontology or a complex, more-expressive ontology? We first implemented a fairly large, yet straightforward, ontology. However, the requirement that IRIS interoperate with CALO's reasoning and learning capabilities drove us to adopt CALO's pre-existing ontology, which supports roles, events, and complex data structures. CALO's ontology is called CLib,26 the Component Library Specification, which consists of definitions for everyday objects and events, as well as axioms to support the beginnings of common-sense reasoning. For IRIS, we translate CLib, implemented in a knowledge language called KM, to OWL. We chose OWL27 as the data representation in IRIS because it is a W3C-approved standard that allows for a flexible data schema and query that supports inheritance. Currently IRIS supports the OWL Lite subset, with future plans to support OWL DL.

IRIS provides a framework for harvesting application data and instrumenting user actions in IRIS applications. The harvesting of data refers to importing external data into semantic (ontology-based) structures. For example, if given the specification of an email instance, harvesting APIs exist to create ontology structures for the email, addresses, and people associated with those addresses. These ontology structures are available in the instrumentation API for application events. This data is then translated once again to an external event publish/subscribe model that allows other IRIS plug-ins or external applications to access the data.

IRIS -Infer

If you invent a breakthrough in artificial intelligence, so machines can learn, that is worth 10 Microsofts.

-Bill Gates, quoted by NY Times 28

One of the key differentiators of IRIS, compared to many semantic desktop systems, is the emphasis on machine learning and the implementation of a plug-and-play learning framework. We see machine learning as one of the solutions around a key issue limiting the semantic web's growth and mass adoption: Who is going to enter all of the required links and knowledge?

Here we present a typical use case of how learning components integrated within the IRIS framework combine to progressively construct a semantic representation of the user's work life.

Step 1: Email Harvesting: As the user receives email in Mozilla, IRIS automatically harvests messages, adding them as semantic instances in the knowledge base. As part of this process, names in the address fields are normalized (e.g., "Rich Giuli" will match "Richard Giuli"), links are created to existing contact records in the KB, and new records are added for people not in the KB. Events indicating new email messages and people records are published to the Instrumentation Bus for other learning components to consume.

Step 2: Contact/Expertise Discovery: When contact records containing a name and email address are added to the KB, the DEX service (from UMass), a CALO component, wakes up and tries to discover additional information for that person. Contact information is discovered, as well as a "gist" representing a person's expertise, composed of keywords and noun phrases that are significant for the person.

Step 3: Learn from Files: In a similar fashion to email, IRIS harvests information from files on the user's desk. Currently, SEMEX [7] (from UWashington) opens LaTeX, BiB, and Microsoft Office files (Word, Excel, PowerPoint) to add content (e.g., publication references) to people in the contact KB.

Step 4: Project Creation: Clustering algorithms in IRIS are applied to the user's email to propose new projects to be added to the KB. For each project instance, a label for the project is proposed using the most salient phrase in the email cluster, keywords are added that provide a "gist" of the project, and links are added to project participants using the people in the from/to fields for the email cluster. IRIS provides a user interface where the differences between multiple clustering algorithms can be explored. Currently, three algorithms have been integrated into the framework: Carrot2/Lingo, based on singular value decomposition [16]; an algorithm based on agglomerative clustering and social network analysis [11]; and an algorithm based on linear optimization with user-specified centroids.

Step 5: Classification According to Project: Leveraging the textual content and relations extracted for projects, people, and files, a Bayesian classifier is applied to hypothesize relationships between projects and objects such as emails, files, web pages, and calendar appointments. IRIS's suggestions are displayed to the user, who can optionally provide feedback to the algorithm by indicating the correct values (Figure 3). In the "CALO Suggests" pane, learned hypotheses about an email are presented, including reply urgency, meeting detection, project association, and others. The user can provide feedback about the system's choices, and the system will adapt accordingly.

Step 6: Higher-level Reasoning: A number of specialized reasoners within CALO continually examine events in the user's activity stream and attempt to make useful inferences. For instance, when the user clicks on an email, IRIS attempts to predict whether the user will/should reply to the message (Figure 3). Another plug-in applies text summarization techniques to produce a gist that will be faster for the user to read. In several use cases, multiple reasoners are combined to produce a single prediction. If the user provides negative feedback to a resulting hypothesis, each individual reasoner will adapt itself accordingly, and a meta-learner will use the intermediate results from each predictor to improve its own logic about how their results are combined. Implemented examples of this approach include the "Meeting PrepPack" reasoner and the "Meeting Request" detector. This use case gives readers a flavor of the types of learning components that have been integrated within IRIS to help construct and then leverage a semantic model representing the user's work life. We feel that we have just scratched the surface of the types of useful learning-based functionality that can be integrated into the IRIS semantic desktop, and we are eagerly anticipating continued development, working with members of the CALO and open source communities.

IRIS -Share

Prior to the Internet, the last technology that had any real effect on the way people sat down and talked together was the table.

-Clay Shirky, at Emerging Technology Conference 200329 Sharing information is one of the four key concepts that make up the IRIS vision. We feel that the ability to learn and leverage semantic structure in organizing one's work life will be greatly enhanced in a collaborative setting. Shared structures are essential for both end-user applications, such as team decision making and project management, and for infrastructural components such as machine learning algorithms, which improve when given larger data sets to work on.

In the first version of IRIS, we experimented with a simple collaborative functionality using a Jabber-based transport mechanism. Changes to shared data were written to a "chat room" space representing an ACL group; each IRIS client would remember what changes it had seen previously, and upon startup, initialize its KB by applying all recently recorded change actions. This approach had the benefit that it supported real-time collaborative work between online participants as well as enabling "latecomers" to join and be initialized to the common state. However, with no locking mechanisms, users were plagued by inconsistencies, and we temporarily removed the collaboration feature.

In the coming year, the CALO project has planned functionality and applications that will require infrastructure to support collaborative team decision making, as well as reasoning over a shared document space. We are therefore currently revisiting what approaches to take regarding a collaboration infrastructure for IRIS.

Evaluations & Conclusions

IRIS is now in daily operation as the primary office environment used by the authors and several other members of the CALO community. In addition, as part of the formal evaluations of the CALO project, IRIS and its learning components were used extensively by 15 users during a few weeks of testing, giving CALO an opportunity to learn "in the wild" through observation and interaction with its user. After this experimentation period, we interviewed the users to understand what they liked, what features were missing, and how IRIS generally should be improved.

We were encouraged that most of the feedback was quite positive, with many of the users stating that they were generally pleased with the robustness of the system and impressed with the capabilities of IRIS to learn and provide useful data to them. In particular, the capability to automatically discover contact and "gist" information for people from whom they receive email was much appreciated. Several reports of events where IRIS made a significant positive contribution were noted. For instance, one user, after skimming a long email from his boss, wondered why IRIS was flagging the message as a meeting request. Upon closer reading, he discovered that towards the end of the message, his boss was actually requesting a meeting with him later that afternoon, with an expected deliverable.

Despite the positive feedback, a number of issues still need to be addressed before most users will be willing to adopt IRIS as their primary work environment:

1. Performance was the Number 1 issue. Many noted that the startup time for IRIS was quite slow, and actual use was sluggish, particularly during email use, where many research-quality components would process each selected message. Subsequent analysis revealed that 68% of the startup time and 72% of memory use could be attributed to three learning components; these will be candidates for optimization in the future.

2. Many user interface issues were noted, in particular regarding real-estate management for smaller screens, several inconsistencies in UI design, and the desire to use drag-and-drop. Also, several minor user interface bugs were mentioned, the most annoying being a proclivity for IRIS to pop up or become the selected window in an unsolicited way when activity (such as the arrival of an important email message) occurs. Improvements to the user interface experience will become a significant priority for the near-term roadmap.

3. Finally, many users, excited by the glimmer of intelligence that IRIS (and the cognitive assistant CALO) at times seemed to exhibit, made numerous suggestions about use cases they thought the pair would be able to help them with in the future to aid their productivity.

In the coming months, we will aggressively pursue these and other roadmap items, with a particular emphasis on making IRIS usable and useful for collaborative teams.

We remain enthusiastic about the potential for coupling semantic representations, machine learning, user interface design, and real-world office systems.