The increasingly huge volume of personal information stored in a desktop computer is characterized by disparate models, unstructured contents, and implicit knowledge. Aiming at a semantic rich environment, a number of Semantic Desktop frameworks have been proposed, concentrating on different aspects, including organization, manipulation, and visualization of the data. In this paper, we propose a layered and semantic ontology-based framework for personal information management, and we discuss its annotations, associations, and navigation. We also discuss query processing in two cases: query rewriting in a single personal information application, PIA, and that between two PIAs.
In 1945, Vannevar Bush put forward the first vision of personal information
management (PIM) system, Memex, by pointing out that the human mind “operates by
associations”, and we should “learn from it” in building Memex [
PI Space (C:\) papers
WISE03-1.pdf WISE03-submission.pdf WISE03-camera.pdf JoDS05.pdf photos
WISE myself.jpeg talk.jpeg with sam.jpeg talks
WISE03.ppt IDEAS04.ppt AP2PC04.ppt Super-invited.ppt emails
Final submission of WISE.eml Meeting on Monday.eml WISE photos.eml
Register for WISE.eml
including the data model, query language, and user interface, should provide multiple
ways to manipulate data in a powerful and flexible manner. Furthermore, a PIM system
should possess the capability for seamless communication (or interoperability) with
external sources (possibly in another PIM system), e.g., in a peer-to-peer (P2P) way [
Example 1. Figure 1 presents a fragment of PI space, which consists of four directories of files in the hard drive C:\. The papers directory contains four papers of the format pdf, photos\WISE contains three pictures taken at the WISE ’03 conference, talks contains four Powerpoint files that are respectively the slides of four talks, and emails contains four saved email messages. Even if the concrete contents of all these files are unknown, we can tell from their names (or the names of their respective directories) that several of them appear to be related to one another. Unfortunately, their storage in different and possibly unrelated directories does not show such inter-relationships, thus resulting in possible difficulties in locating the wanted information. Some keywordbased searching techniques, e.g., offered by the Google Desktop Search,1 can retrieve all files that are relevant to WISE. However, without further inspection of the contents of each file, the user may not be able to discover certain associations between them, e.g., that file JoDS05.pdf is an extended journal paper of WISE03-camera.pdf.
From this example, we can see that the lack of semantic associations among the stored data could be a handicap for data and knowledge discovery. In this paper, we focus on issues of semantic data organization and management in PIM, by taking the following approach:
1) We propose a layered framework for PIM, in which multiple ontologies playing a
variety of roles are employed. Specifically, the resource layer stores all the PI resources
(using URIs), metadata of the PI, and all kinds of associations using RDF. The domain
layer contains the ontologies specific to various domains that are used to structure the
data and categorize the resources. The application layer, built on top of the domain
layer, is where the user constructs different application ontologies for different purposes
of data usage. This layered architecture enables: i) a semantics-rich environment for
1 http://desktop.google.com
personal information management; ii) a flexible and reusable system, by decoupling the
domain and application ontologies, so that the construction of application ontologies
for different applications can reuse the underlying domain ontologies. We argue that
this provides certain advantages over the use of a single domain model for all the PI
(e.g., [
2) We discuss in detail how to utilize superimposed information for semantic
organization, focusing on the construction of resource-file and resource-resource
associations. We also present the idea of 3D navigation, which is a combination of the vertical,
horizontal and temporal navigation in the PI space. The idea is inspired by some
existing PIM systems including MyLifeBits [
3) In our framework, the basic unit for the user to manage the Semantic Desktop is the personal information application (PIA). Each PIA aims to accomplish or assist a specific task (e.g., bibliography management, paper composition, and trip planning). The PIAs can be standalone, with their own application ontology, user interface, and workflows. Meanwhile, they can communicate with each other as if in a P2P network, by means of the connections (mappings) established between their application ontologies. In this sense, different PIAs interoperate at a semantic level. We describe query processing in our framework in two cases: within a single PIA or between two PIAs, in a P2P query processing mode.
Among the existing approaches to PIM, the Gnowsis project from DFKI2 aims at
a Semantic Desktop environment, which supports P2P (or distributed) data
management based on Desktop services [
MyLifeBits [
Other approaches to PIM include Chandler,3 Lifestreams [
3 http://www.osafoundation.org/ PIM 1
Application Layer
Application
Ontology 1 Domain Layer
Domain
Ontology 1 Resource Layer
Resource repository (RDF)
Association
Application Ontology 2 Domain Ontology 2 Resource -file index
PIM 2
Application Layer
AOpnptloicloagtiyoni . . .
Application Ontology n Domain
Ontology m File Metadata
PIM 3
Application Layer
AOpnptloicloagtiyonj . . .
PI Space
Contacts Simple-content Bibtex
...
Textual
Papers
Reports Complex-content Emails
Slides ...
Nontextual: (Video, Audio, Pictures, ...)
Relational database, XML
The rest of the paper is structured as follows. In Section 2, we describe the layered framework and its main components. The semantic organization of the PI (including the concepts of annotation, association, and representation) is discussed in Section 3. Section 4 and Section 5 focus on two main ways of data manipulation, namely, navigation and query processing. Finally, we conclude in Section 6. 2
Our framework follows the principle of superimposed information, i.e., data or metadata
“placed over” existing information sources [
Dimensions: 3072 £ 2048 pixels Device make: Canon Color space: RGB Focal Length: 75 ......
Domain ontologies. A number of ontologies are published on the Web. Examples of such ontology libraries include DAML Ontology Library,4 the Semantic Web Ontologies,5 and the Prote´ge´ OWL ontologies.6 The ontologies in these libraries are typically designed and organized for different domains such as Conference, Person, Photo, and Email. In our framework, the domain ontology layer is designed to be loosely-coupled with the other layers, to enable the insertion and removal of ontologies as “plug-ins”. Resource-file index and RDF repository. One of the roles of domain ontologies is to provide the basis for data classification. In order to establish the connections between the files and the concepts in the domain ontologies, we treat each file as a resource, which is then classified as an instance of one or more concepts. The resource-file index is a local database storing these connections between resources and files. Furthermore, the various types of associations among resources (as instances of association of concepts in the domain ontologies) are stored in an RDF repository. The resource-file index and the RDF repository are both in the resource layer, providing resource instances for the domain ontologies in the domain layer above.
Application ontology. Above the domain layer is the application layer, which contains the ontologies for different applications. The domain ontologies, as an intermediate layer between the applications and the data, are meant to enhance the reusability and flexibility of the framework. More specifically, the application ontologies are defined as views of the domain ontologies, which can be reused for the construction of different application ontologies. In our framework, each personal information application (PIA), is associated with an application ontology, has access to relevant data, and is functionally independent of other applications. It may be infeasible to have a single ontology to cover various applications, e.g., for trip planning and paper writing. Instead, as many PIAs as needed can be designed in one or more PIM systems, where the PIAs can interoperate (e.g., through P2P query processing) for the purpose of integrating relevant information. This issue is elaborated on in Section 5.
Besides the data components described above, a PIM system also needs some functional components to perform all kinds of data and metadata processing, to make the framework work as a whole. Such components include an indexer (for establishing and managing the indexes of the files), a wrapper (for identifying and extracting resources 4 http://www.daml.org/ontologies/ 5 http://www.schemaweb.info 6 http://protege.stanford.edu/plugins/owl/owl-library/ from the files), and an ontology designer (for importing and editing an ontology). Because of space limitations, we do not elaborate further on these components. 3
The layered architecture of our PIM framework described previously enables the reusability and the organization of semantically rich data for PIM. In this section, we discuss in detail the mechanisms that our framework uses to support the semantic organization of the PI space, including those for semantic annotation, association, and representation. 3.1
Given that the data in the PI space is the base information, all the other data components in our framework are actually superimposed information over this base. The most fundamental function of the superimposed information is to provide semantic annotations of the base information to enable powerful and accurate data access. We discuss the following two aspects: File description. It is especially important to provide the searcher with a detailed description of the nontextual files. When performing a keyword-based searching, the searcher matches the submitted keywords (e.g., “Canon”) or key-value pairs (e.g., “Maker:Canon”) with the property-value pairs of the file description, to find the right files requested by the user. Even for textual files, taking into account such metadata will improve the effectiveness of full-text searching.
Domain ontologies. Given that a file is identified as a resource, we are able to annotate the file using a domain ontology, by associating the resource with a concept of an ontology. The domain ontology provides not only a context for understanding the data, but also semantic clues for the precise data retrieval. For example, the user can query the PI using a query language for RDF instead of using keywords. We note that a file can be an instance of more than one concept, according to different classification criteria. 3.2
In our framework, semantic associations are used to relate all the data (base information) and metadata (superimposed information). There are two classes of associations: the resource-file associations that are actually the resource-file indexes and the resource-resource associations that are instances of the domain ontologies and are stored in the RDF repository.
Resource-file associations. In addition to the ontological resources that are used to identify (through data classification) the files, a (textual) file may contain and refer to a number of resources. Therefore, the resource-file associations can be one of the following: identification, containment, and reference.
Example 2. Suppose that the user has saved an email message, which is an announcement of a seminar, as shown in Figure 3. First, the email message can be classified as an instance of the concept Email, provided that the concept exists in some domain ontology. Then, the system can generate for the concept SeminarAnnouncement and its properties a new instance (i.e., resource), which is associated with the saved email by the relationship containment. Finally, a reference association can be established between the resource http://www.tliap.nus.edu.sg/ (e.g., of the concept WebsiteAddress) and the email message.
The process of setting up the resource-file associations is the one of recognizing
resources from the file description and/or the file content and then mapping them to the
ontological concepts. The user may determine the degree to which the resources should
be extracted from a file and its description. For instance, in the previous example, the
user can further create resources for the title and abstract of the seminar, and for the
biography of the presenter. It is expected that this process (as well as the process of
discovering resource-resource associations, as discussed later) can be maximally
automated, to reduce the user’s burden. For this purpose, we may utilize the following
methods:
– Keyword extraction. From the text of a file, keywords can be extracted based on a
thesaurus or be highlighted manually by the user. Each keyword can be considered
a resource contained by the file. The matching of the resources with the concepts
in the domain ontologies can be guided by a thesaurus such as WordNet.7
– Hyperlink analysis. For the textual files that include hyperlinks to classified
resources (e.g., a citation of a paper or a link to a webpage), we create for each
hyperlink a reference-type resource-file association, as well as a resource-resource
association between the referring resource and the referred one.
– Natural language processing. We can utilize known techniques (e.g., [
Resource-resource associations. We borrow from the Object Oriented Design (OOD) techniques the following four types of relationships between objects: instantiation (i.e., membership), property, aggregation (i.e., whole/part), and generalization (i.e., inheritance). These four relationships, which are used in object models, are adopted to describe the associations among concepts as well as resources in our framework. Note that “property” refers to a pattern as identified, for example, using natural language processing techniques, which corresponds to a user-defined property. For example, writes can be a property of the concept Author, connecting Author to the concept Book. Table 1 summarizes the resource-resource associations in our framework.
By using the previously described techniques, we can discover the resources and their associations implied in the PI, and classify them into the domain ontologies, thus populating the ontologies. In the example of Figure 3, it is possible to extract a pattern hSingapore; implements; ITSi, which could then be classified as an instance of an ontological pattern such as hOrganization; implements; Systemi, where Organization and System are two concepts, and implements is a property. Note that the user is allowed to choose the granularity of this knowledge (resource and associations) discovery process, ranging from only taking the whole file as a single resource to analyzing the detailed contents of the file.
In addition, ontology mappings may be established between correspondences that
connect concepts in different domain and application ontologies. Currently, we consider
equivalence as the only semantics for the mapping between two concepts, although
richer semantics of the mappings could be considered [
9 http://www.w3.org/TR/rdf-schema the RDF data, such as rdfs:Class, rdf:Property, and rdf:type. Table 2 summarizes the RDFS vocabularies that are used to represent different types of associations.
The use of RDF as the data model and RDFS as the ontology language in our
framework is motivated by the nature of the RDF as a Web resources description mechanism
and the fact that the PI is represented as a set of interrelated resources. In contrast, XML
is not chosen because it cannot represent semantic associations [
The extension to RDFS is as follows: we define in a namespace (abbreviated using the prefix rdfx) a new RDF property, contains, which is used to represent the aggregation relationship. For the representation of the instantiation and generalization relationships, we use rdf:type and rdfs:subClassOf, respectively. The property relationship is represented naturally by an RDF property defined in the user-defined namespace. Figure 4 gives a concrete example of an RDF representation. 4
It is critical for a Semantic Desktop to provide the user with the capability to access the stored data in a variety of ways. The user may want to browse the information by means of the flexible and intelligent navigation in the information space, including the base and superimposed information. The user may also desire that certain query facilities (e.g., keyword-based searching or certain query languages) be provided by the framework. In this section, we discuss the navigation in the data space of a Semantic Desktop. Query processing is discussed in the next section.
The semantic data organization in our framework enables the navigation in the PI space, making use of useful hints (e.g., the context of a concept being browsed) so as to facilitate the user’s understanding of data. More specifically, by taking into account the layered architecture, the semantic navigation in our framework can be performed in three directions: (1) In vertical navigation, the user follows a path across layers. Two cases are possible for this way of navigation: top-down from the application ontologies to the stored files and bottom-up from the stored files to the application ontologies. (2) Ontology for attending a conference
Person writtenBy Paper extendedVersion Journal attends publishedAt Conference presentedAt Talk
presentedBy r e y a L Email n o it a c li p p A receivedBy sentBy
Person
In horizontal navigation, the user follows links of concepts (or resources) within one layer. Typically, there are three cases of horizontal navigation, corresponding to each layer: application-to-application navigation, domain-to-domain navigation, and file-tofile navigation. (3) In temporal navigation, the user can navigate by following references in chronological order, each being a resource for the same real world object with a time stamp associated with it. For example, the user may want to look at different versions of a research paper.
All the base and superimposed information in the framework forms a directed graph, where the vertices are the resources in the ontologies and the files stored in the PI space, and the edges are the associations between the resources and files. We say that the three directions of navigation together provide the capacity of a 3-dimension (3D) navigation mechanism, which can facilitate the construction of a browser. For instance, suppose the user is browsing a specific application ontology in a visualized browser. When the user clicks on the node of a concept in the ontology, the browser can then choose to Application Ontology Email display the instances of the concept thus selected (by vertical navigation), the context of the concept in the domain (also by vertical navigation), and the associated concepts in other application ontologies (by horizontal navigation). Compared to the traditional navigation approach that is based on hierarchical directories, 3D navigation is based on semantic associations, similarly to those that humans establish between concepts. Example 3. Consider the scenario shown in Figure 4. The spirit of 3D navigation is demonstrated in the browser of Figure 5. The current resource (file) that the user is browsing is an email message (i.e., wisephotomsg), which has some photos attached, which were taken at WISE ’03. The concepts that this resource belong to are highlighted (in white) so as to show the contexts to which they belong. All associated resources are categorized and shown on the right tabbed pane, which provides a guidance for the user in navigating the PI space. The bottom-right pane shows the timeline of different versions of the current resource (if they exist) or all the resources belonging to the same concept as the current resource. 5
Unlike navigation, which is an interactive process, query processing is performed without further intervention from the user. To retrieve relevant data from the PI space, the user’s request may be posed as a sequence of keywords or as a query formulated in a certain query language.
The keyword-based search matches the input keywords and the vector of words in
the candidate documents, calculates the similarity for each of the matches, and returns
to the user the results after ranking them [
Since the two approaches complement each other, it is desirable to provide both of
them. In this section, however, we mainly focus on query processing in our framework.
We choose to express the queries in RDQL [
In our framework, the user query is formulated in RDQL (RDF Data Query Language),
which uses an SQL-like syntax [
i
In our framework, an application ontology is constructed over one or more domain
ontologies, and the files in the PI space are formalized as instances of the concepts in
the domain ontologies. If we consider the application ontology as the global ontology
(since the user query is posed on it), the whole system can be seen as a GaV data
integration system [
There are a number of algorithms for query rewriting in relational or XML data
integration systems [
By assuming that there are no integrity constraints over the application ontologies and the user queries are formulated in c-RDQL, we give the formal description of our query rewriting algorithm in a single PIA, which we call ADREWRITING (for rewriting from Application ontologies to Domain ontologies), as follows. We note that we do not consider the namespaces of ontologies for simplicity of the description. Algorithm ADREWRITING Input:1. q1 over the application ontology G: ans(X) :- p1(X1); :::; pm(Xm); 2. M: the mapping table between G and domain ontologies S1; :::; Sn.
Output: q2: A c-RDQL query over S1; :::; Sn. 1. headq2 = ans(X); bodyq2 = null; 2. For i = 1 to m do 3. (c1; c2) = name of the classes referred to by (x1; x2), for Xj = (x1; x2); 4. Search M to find (d1; d2) such that f(c1; d1); (c2; d2)g are two class correspondences in M; 5. Traverse S1; :::; and Sn by following all kinds of associations, to find the vertices, v1; :::; vk, connecting from d1 to d2; 6. If k = 0 then add p(x1; x2) (or p(x2; x1)) to bodyq2 , if there exists p connecting d1 to d2 (or d2 to d1); 7. Else for j = 1 to k ¡ 1 do 8. Add p(x^j ; x^j+1) (or p(x^j + 1; x^j )) to bodyq2 , if p is not a mapping and connects vj to vj+1 (or vj+1 to vj ); 9. Add p(x1; x^1) (or p(x^1; x1)) to bodyq2 , if p is not a mapping and connects d1 to v1 (or v1 to d1); 10. Add p(x^k; x2) (or p(x2; x^k)) to bodyq2 , if p is not a mapping and connects vk to d2 (or d2 to vk); 11. q2 = headq2 :- bodyq2 ; Example 4. Suppose the user wants to list all conference papers with their authors and journal version, using the query q1 : ans(x; y; z) :- writtenBy(x; y), extendedV ersion(x; z), which is posed on the application ontology of publication management. For the variables (x; y; z), we get the classes that they refer to as (Paper, Person, Journal), as indicated by Line 3. By looking into M, we find the corresponding class sequence as (Publication:InProceedings, Publication:Person, Publication:Article), where the names before the colons are domain ontology names. From Lines 5 to 10, we compute the predicates in the body of q2 as follows.
q2: ans(x; y; z) :- editor(x; y), extends(z; x)
By executing q2 over the RDF repository as shown in Figure 4, we get the answer f(#wise03papercamera, #xiao, #jods05), (#wise03papercamera, #cruz, #jods05)g. 5.2
Application to application (A2A) query processing occurs when an application is
attempting to retrieve relevant data from another semantically related application, to
answer a query. If the PIAs are considered as connected peers (i.e., service providers for
certain data access), the A2A query processing is similar to that in peer-to-peer (P2P)
systems [
A2A query processing consists of two steps of query rewriting. First, we rewrite the original query q, which is posed on the application ontology G1, to a query q0 on the other application ontology G2, according to the mappings between G1 and G2. Then, q0 is rewritten to a query q00 on the domain ontologies, to which G2 is mapped. Answers are obtained by executing q00 on the RDF repository. The second query rewriting is exactly the one described by the algorithm ADREWRITING, whereas the first rewriting is slightly different from ADREWRITING. In particular, unlike the total mapping from an application ontology to the domain ontologies, some of the concepts in G1 may not be mapped to those in G2. Therefore, the answers returned by q00 may contain null values or Skolem functions for the unmapped concepts or properties.
The A2A mappings can be derived by composing the mappings between G1 and the
domain ontologies, inter-domain mappings, and those between G2 and the domain
ontologies. To evaluate both query rewriting processes, we need to check the equivalence
(or containment) between a query and its rewriting. A correct query rewriting is the one
that is equivalent to (or maximally contained in) the query. These two issues (reasoning
on mappings [
In this paper, we present our design of a PIM system. We propose a layered ontologybased framework, which aims to provide a semantics-rich environment for personal information organization and manipulation. The multiple ontologies existing in different layers of the architecture explicitly support the data semantics. Furthermore, the decoupling of the domain layer and the application layer enhances the flexibility and reusability of the framework. Specifically, we discuss in detail the semantic-enriched data organization, including the use of file descriptions and domain ontologies as annotations, and the construction of resource-file and resource-resource associations. We also introduce the idea of 3D navigation, which is used in a desktop browser. We discuss query processing in our framework in two cases: within a single personal information application, PIA, and between two PIAs, using application to application (A2A) communication. A formal query rewriting algorithm is presented for the single PIA case.
In the future, we will continue the study and implementation of our framework. It is clear that a lot of the success of PIM systems lies on the successful automation of the different mechanisms that are needed. In particular, we will look further into the automation of the conceptualization of full-text files and that of matching resources to ontological concepts. Also, we will elaborate on the idea of 3D navigation both by studying a model for temporal navigation and by carrying out user studies. The study of A2A communication, including data exchange, collaboration, and query processing will also be continued. While RDQL queries are expressive, they may not be suitable for most users. We are therefore exploring visual queries that can express a class of RDQL queries “appropriate” for the semantic desktop.