Mailing list archives (i.e., the compilation of the messages posted up-to-now) are often published on the web and indexed by conventional search engines. They store a vast knowledge capital. However, the ability to automatically recognize and process the information is mostly lost at publishing time. As a result, the current mailing list archives are difficult to query and have a limited use. This paper describes an usage of the Semantic Web technologies in order to avoid the information loss and to allow new applications to exploit the information in a more powerful way.
Electronic mail (e-mail) remains one of the most popular applications of the Internet. Besides direct messaging between individuals, mailing lists exist as private or public forums for information exchange in communities with shared interests. Mailing list archives are compilations of the previously posted messages that are often converted into static HTML pages for their publication on the web. They represent a noteworthy portion of the contents that are indexed by web search engines, and they capture an impressive body of knowledge that, however, is difficult to locate and browse.
The root of these problems can be traced back to the translation procedure that is run to transform the e-mail messages into static HTML pages. This task is fulfilled by scripts that create an static HTML page for each message in the archive. In addition, some indexes (by date, by author, by thread) are generated and usually splitted by date ranges to avoid excessive growth.
On the one hand, this fixed structure reduces the flexibility when users browse the mailing list archives using their web browsers. On the other hand, some of the meta-data that were associated to each e-mail message are lost when the message is rendered as HTML for presentational purposes.
We propose to use an ontology and RDF (Resource Description Framework
The rest of the paper is organized as follows: Section 2 introduces the SIOC ontology and our extensions to it, and then some software applications are described in Section 3. We close the paper with the conclusions and a discussion on future plans in Section 4.
An ontology to capture the meta-data of a discussion forum, such as a mailing list, was clearly
recognized as the first milestone to fulfill the purpose of the project. Fortunately, DERI
Galway has developed SIOC (Semantically-Interlinked Online Communities,
http://siocproject.org/), an ontology that provides a vocabulary to interconnect different discussion
methods such as blogs, web-based forums and mailing lists
The SIOC ontology was designed to express the information contained both explicitly and implicitly in Internet discussion methods. Several software applications, usually deployed as plug-ins, are already available to export SIOC data from some popular blogging platforms and content management systems. The effort, however, is focused on web-based communities (weblogs, webforums), while little has been done so far to extend the coverage to legacy nonweb communities, such as mailing lists and Usenet groups.
SIOC is specified in OWL, and their instances can be expressed in RDF. Therefore, they
can be easily linked to other ontologies. The obvious choice here is FOAF
SIOC is an almost perfect match for our purpose. Each mailing list becomes an instance of
sioc:Forum, messages sent to the list become instances of sioc:Post (as well as their replies),
and the people subscribed to the list are sioc:Users. The Dublin Core
However, additional object properties were required in order to retain the sequence of messages published in a mailing list. Thus, we extended the SIOC ontology with two properties defined in a separate namespace: swaml:previousByDate and swaml:nextByDate. Both properties are defined with sioc:Post as their domain and range. An RDF representation of a sample message is shown in Figure 2.
The ontology itself provides no service to end users. Software tools are required, and we built two of them as part of this project1: ● SWAML is a non-interactive, command-line application whose main purpose is to translate mailboxes into sioc:Forum instances in RDF.
● Buxon is a graphical browser for sioc:Forum instances. 1 Our applications are available at http://swaml.berlios.de/
Each tool has a precisely defined role, fulfilling the need to generate RDF data and to consume data, respectively, as depicted in Figure 3. The following paragraphs provide further detail on SWAML and Buxon.
SWAML covers the data-generation phase, and it is intended to be used by mailing list
administrators, who usually have access to the archives in raw format. The most popular
format for mailing list archives is the “mailbox” (or “mbox”), as defined in RFC 4155
Parsing the mailbox and rebuilding the discussion threads may be sometimes tricky.
Although each mail message has a supposedly unique identifier in its header (Message-ID,
defined by RFC 2822
Actually, SWAML goes further than just a format-translation tool. A dedicated subroutine that runs as part of the batch execution, but may be also separately invoked on any sioc:Forum, tries to find a FOAF description for each sioc:User. To the best of our knowledge, there is not any web service to fetch FOAF descriptions from a given e-mail address, so we mocked it. Some of the authors of this paper are also currently working on a functional implementation of such a service as part of a different project.
The last step of the SWAML processing chain generates a KML
Buxon is a multi-platform desktop application written in PyGTK. It allows end users to browse the archives of mailing lists as if they were using their desktop mail application. Buxon takes the URI of a sioc:Forum instance (for example, a mailing list exported by SWAML, although any sioc:Forum instance is valid) and fetches the data, retrieving additional files if necessary. Then, it rebuilds the conversation structure and displays the familiar message thread list (see Figure 5).
Buxon also gives users the ability to query the messages, searching for terms or filtering
the messages in a date range. All these queries are internally translated to SPARQL
There is a lot of ongoing effort to translate data already reachable on the web into formats which are Semantic Web-friendly. Most of that work focuses on relational databases, microformats and web services. However, at the time of this writing and to the best of our knowledge, e-mail was almost excluded from the Semantic Web. This project, in combination with the generic SIOC framework, fills this gap, conveniently providing an ontology and a parser to publish machine-readable versions of the archives of the countless mailing lists that exist on the Internet.
The SWAML project fulfills a much-needed requirement for the Semantic Web: to be able to refer to semantic versions of e-mail messages and their properties using resource URIs. By re-using the SIOC vocabulary for describing online discussions, SWAML allows any semantic web document (in particular, SIOC documents) to refer to e-mail messages from other discussions taking place on forums, blogs, etc., so that distributed conversations can occur across these discussion media. Also, by providing e-mail messages in SIOC format, SWAML is providing a rich source of data, namely mailing lists, for use in SIOC applications.
Some benefits arouse from the availability of these data. In the first place, data can be fetched by user applications to provide handy browsing through the archives of the mailing lists, providing features that exceed what is now offered by static HTML versions of the archives on the web.
Secondly, the crawlers of the web search engines can use the enhanced expressivity of the RDF data to refine search results. For instance, it becomes possible to filter out repeated messages, advance in the fight against spam, or introduce additional filter criteria in the search forms.
Another consequence of no lesser importance is that each e-mail message is assigned a URI that can be resolved to a machine-readable description of the message. This actually makes possible to link a message like any other web resource, and therefore enriches the expressivity of the web.
We are exploring some directions for future work. Some of them are:
● Integration of the SWAML process with popular HTML-based mailing list archivers, such
as Hypermail or Pipermail, would be a giant push to speed up the adoption of SWAML. It
is well known that one of the most awkward problems of any new technology is to gain a
critical mass of users. The semantic web is not an exception. A good recipe to tackle this
problem is to integrate the new technology into old tools, making a smooth transition
without requiring any extra effort from users. Merging the SWAML process into the batch
flow of tools such as Hypermail would allow to generate both HTML and RDF versions of
the archives. Those could reside side-by-side on the web server, even sharing the same
URI by means of content-negotiation
Obviously, such information can not be automatically derived from a RFC 4155-compliant
mailbox. However, it is conceivable that it can be added by other means, such as social
tagging using folksonomies, or parsing the RDFa that may exist in the e-mail messages
that are sent in XHTML format. The inherent community-based nature of mailing lists can
be exploited to build recommendation systems
The authors would like to express their gratitude to Dr. John Breslin and Uldis Bojars from DERI Galway, whose support and contributions have been of great help to this project. Also to Ignacio Barrientos by his contribution packaging the project for Debian GNU/Linux.