The architecture of JobOlize, depicted in Figure 1, is divided into two core components, a knowledge base providing a domain ontology as well as an extraction rule base and a pipeline consisting of different extraction components. Parts of the system are realized on basis of the existing information extraction infrastructure GATE (General Architecture for Text Engineering) [ 3 ] which has been favored mainly because it is a mature open source implementation providing a highly extensible pluggable architecture.

e-Recruitment Domain Ontology. For representing the annotation vocabulary used by our extraction system, a light-weight domain ontology has been developed containing 15 core concepts of job offers which should be extracted (e.g., job title, IT-skill, language skill, operation area and graduation). A design goal in this respect was to heavily build on reasonable concepts of existing ontologies in the area of e-recruitment and human resource (HR) management, backed up by our experience in developing ontologies [ 20 ], [ 22 ], [ 23 ]. Most notably, from the ontology of Mochol et al. [ 17 ] which is based on the comprehensive HR-XML standard [ 10 ], we have adopted the concept of skill levels, whereas the work of Garcia et al. [ 7 ] was influential for our notion of offer characteristics. Our ontology also considers specialisation hierarchies of IT skills (e.g., “Oracle“ and “MySQL” are specializations of “DBS”) and equivalence relationships (e.g., “DB” is semantically equivalent to “DBS”), which are of particular interest for the extraction process. Finally, to consider also job offers in different languages, each concept contains a language property.

Extraction Rule Base. The second part of our knowledge base contains about 50 extraction rules ranging from very simple ones, responsible, e.g., for matching input tokens with ontology concepts to rather complex ones for, e.g., job title detection. The rules are in fact conditionaction tuples formulated using the regular-expressionbased language JAPE (Java Annotation Pattern Engine) provided by GATE and callouts to Java code for realizing more complex tasks.

Pipeline of Extraction Components. On basis of the second core part of our architecture, the pipeline, annotations of web pages are incrementally built up by streaming input pages through the different components of the pipeline, each of them being responsible for a certain annotation task, thereby adding new or manipulating already existing annotations. Parts of this pipeline are realized on basis of GATE which provides not only predefined, pluggable components for realizing basic extraction tasks but also allows to plug-in self-developed, customized ones. To be more specific, JobOlize reuses existing NLP-based components for tokenizing and stemming and furthermore realizes customized components on basis of Java for preprocessing the input Web pages (e.g., eliminating JavaScript code), for post-processing (e.g., exporting annotated Web pages as XML documents) and for the core task of identifying those tokens of the input Web page, which are relevant for our purposes, e.g., IT-skill with an associated skill-level like “basic knowledge in Java programming”. The identification of relevant tokens is performed on basis of four different customized components, as described in the following, which are making use of the above mentioned knowledge base.

Initial Annotation. The first component is responsible for an initial annotation of the tokens of the input Web page with appropriate concepts defined by our domain ontology. For most of the tokens, this task is straightforward, but considering the domain concepts IT-skill and language skill, context-driven processing is required in order to determine their corresponding levels. In particular, not only the position of the skill level with respect to the skill type itself is taken into account by means of appropriate JAPE rules, but also, e.g., if it is located within the same sentence or not.

Page Segmentation. The remaining three components promote the basic idea of context-driven extraction even further, with the ultimate goal to improve the quality of the extraction results. For this, the Web page is first segmented into three parts, a top part, a content part and a bottom part, simply taking the content itself into account. This is done by identifying common text fragments between two job offers of the same Web site on basis of the well-known Longest-Common-Subsequence algorithm (LCS) [ 11 ]. These common text fragments represent the top and bottom parts of a page and contain tokens which are most probably irrelevant for further processing. For example, the occurrence of “powered by Typo3” within the bottom page would lead to an IT-skill annotation of “Typo3” during the initial annotation phase, being refined by page segmentation, i.e., classified as irrelevant.

Block Identification. The content part of the Web page identified before, is further divided into so-called blocks, representing a couple of tokens which “visually” belong together and are normally grouped under a header title. For the identification of such blocks, first, context in terms of layout information (e.g., a <b>-tag) and structural information (e.g., a <li>-tag) is considered by appropriate JAPE rules. In a second step, context in form of content information is used to categorize the identified blocks. In particular, on basis of their header title and the corresponding domain concepts defined by our ontology, blocks are categorized and annotated as requirements, responsibilities, offer characteristics and contact details, i.e., those chunks of information most commonly found in job offers.

Relevance Assignment. The final component of our extraction process assigns pre-defined relevance values, between 0 and 1, to the initial annotations, depending on the block category, the annotated token is contained in. E.g., in case that a token annotated as IT-skill is part of the requirements block it gets a relevance of 1, whereas if it is part of the bottom part, it gets a relevance of 0,25, only. During post-processing, annotated IT-skills with a relevance lower than a certain threshold, are eliminated from the result.

For ensuring an acceptable quality of the extraction results, information extraction out of arbitrarily structured and heterogeneous job offers is reasonable in a semiautomatic way only. Thus, the results generated automatically by the extraction system, which are in our case, annotations of the target Web pages containing the job offers, have to be assessed by a user and potentially corrected accordingly. For this task, we provide a rich client interface which allows the user not only to initiate the extraction process by entering the URL’s to the desired target Web pages, but especially to visualize the results of the extraction process in terms of the annotated Web pages and the possibility to manipulate existing annotations or to add additional ones directly in a Web browser2. The screenshot in Figure 2 depicts on the left hand side a menu sidebar and on the right hand side the original Web page.

Annotation Highlighting. Within the sidebar, the user can choose from an annotation list (cf. c ), representing the concepts of our domain ontology, which of the automatically generated annotations should be indicated within the Web page on the right hand side. To preserve the look and feel of the original Web page, annotations are indicated by marking the concerned parts of the Web page 2 It has to be noted that the GUI provided by GATE is realized as a heavy-weight desktop application only, which among others, lack a proper visualization of the extraction results. with corresponding colors and highlighting those, being currently focused on, by means of a flash effect.

Annotation Details. The details of annotations (e.g., the relevance values assigned) are shown for each annotation, both, within the sidebar (cf. d ) and directly within the Web page, activated by a mouse over effect. This functionality also serves as a simple form of explanation component, which allows tracing back the genesis of a certain annotation.

Annotation Manipulation. Finally, the details of each of the automatically generated annotations can be manipulated in accordance with the underlying job ontology, the annotation can be completely deleted or new ones can be added. Thus, e.g., a language skill can be modified from German to English (cf. e ). Manipulations are immediately propagated to the Web page at the right hand side.

Implementation Aspects. Concerning implementation aspects, the rich client interface is realized as a Firefox extension using XUL (XML User Interface Language [ 26 ]), allowing for portability of the application, JavaScript and CSS. The primary reason to favor the Firefox extension mechanisms instead of alternatives like Apache’s MyFaces [ 19 ], was the rather easy realization of a rich client interface experience as in native desktop applications, considerably reducing programming effort. Annotation highlighting as well as manipulation is based on DOM. As a pre-requisite for asynchronously initiating the extraction process (executed on the server) directly from within the rich client interface, the underlying extraction system had to be ported to the used Tomcat Web server.

The goal of this case study is to evaluate the amount of improvement in extraction quality when employing our context-driven extraction process, compared to a simple ontology- and rule-based annotation (called “initial annotation” in Section 2.1).

Set-up of the Case Study. The test set of our case study consisted of 32 randomly chosen online IT job offers from 16 different Austrian recruitment Web sites, partly differing considerably in content, structure, and layout. The extraction quality has been tested for three different concepts of our job ontology, namely IT skill, operation area, and language skill, since each of them poses unique challenges to the extraction process. Finally, we employed the common metric used for evaluating the quality of information extraction, comprising three different measures [ 5 ]: First, precision, which reflects the share of extracted content which is relevant among all extracted content. Second, recall, which depicts the share of extracted content which is relevant among all content. Third, F-measure, which is a balanced measure, combining recall and precision. These measures were computed using the AnnotationDiff tool, provided by GATE [ 8 ].

Interpretation of Results. Figure 3 shows the results of the case study. Overall, it can be seen that our contextdriven extraction process outperforms a non-contextdriven extraction, as expected. The values for recall were equal in all three cases, simply due to the fact that the initial annotation determines all potentially relevant token candidates, while the subsequent components of our extraction pipeline do not further contribute to the recall values.

The most significant gain in extraction quality could be observed regarding the ontology concept operation area. Precision and F-measure showed an increase of 64% and 38%, respectively. The reason for this improvement is that, in several Web pages of the test set, operation areas also describe the business focus of the company, not only the profile required from a job seeker. This completely different intention of using operation areas can be detected by our context-driven extraction process, since operation areas of a company are naturally positioned in the top part of a Web page, whereas operation areas required from a job seeker occur within a dedicate block of the content part.

1,2000 1,0000 0,8000 0,6000 0,4000 0,2000 0,0000 iirscenoP llceaR .-seaFM iircseonP llceaR .-seaFM iirsceonP llceaR .-seaFM Operation IT-Skill Language

Area Skill Context-driven Extraction Non Context-driven Extraction

Precision and F-measure for IT-skills increased by 30% and 20%, respectively. The reason for this improvement is similar to that given for operation areas – there are job offers where pretended IT-skills show up at the bottom part of the Web page as already argued in Section 2.1. Our context-driven extraction process allows again eliminating these false positives, although there seems to be not that many false positive IT-skills than operations areas out there, as indicated by the lower improvement in extraction quality.

Finally, the least improvement could be observed for language skills, being nearly the same for precision (8%) and F-measure (6%). This rather marginal advance is due to the fact that language skills are more or less unambiguously used in job offers, seldom leading to a false positive content extraction.

There are already numerous information extraction techniques available [ 14 ]. Considering our rich client interface first, most existing systems build, as already mentioned, on desktop-based interfaces. The remaining ones, e.g., KIM [ 21 ] provide restricted functionality only, in that they visualize the annotated Web page only, but do not allow arbitrary manipulations.

Second, regarding the extraction itself, most existing techniques represent the targeted Web page in form of a DOM tree. On this basis, some of them just separate content from non-content parts (cf. e.g., [ 4 ]), often focusing on certain kinds of structure only, such as tables (cf., e.g.,[ 16 ]). Other approaches add additional context to improve extraction quality, either in terms of content information (cf. e.g., the “Pagelet” approach of [ 1 ]) or using structure (cf. the WISDOM approach of [ 13 ] and the template detection approach of [ 2 ]), or layout information (cf. e.g., the “Styletree” approach of [ 27 ] and the visual feature approach of [ 9 ]).

Compared to these systems, we approach the extraction process from a different angle, since we do not represent the input Web page as a DOM tree but let our extraction system directly work with the original Web page, thus preventing to be dependent on the decreased fidelity of DOM representations. Another major difference is that JobOlize does not focus on certain structures of the Web pages but rather deals with arbitrary structures, which is made possible by the combination of NLP techniques of the underlying GATE infrastructure and our contextdriven extraction process. And finally, we do not consider a certain kind of context in isolation, but rather consecutively apply all kinds of context available, i.e., content, structure and presentation information to produce the resulting annotations and calculate corresponding relevance values.

Combining different kinds of context counteracts intricacies of HTML encodings. As one would expect, the liberty of users to ignore any accessibility or wellformedness criteria when designing HTML pages, using e.g., proprietary enumerations instead of explicit <li>tags, different font sizes instead of explicit <H>-tags or HTML tables instead of CSS, considerably challenges the extraction process. It has been shown that the combination of different kinds of context as done in JobOlize eases the task to deal also with these intricacies of HTML encoding in a proper way.

Eliminating irrelevant parts of a web page considered harmful. One has to be aware that eliminating parts of a Web page which are presumably considered as irrelevant as done in JobOlize on basis of the LCS algorithm can also be counterproductive, e.g., in case that the eliminated part indeed contains links to further skills, self refreshing date items, or RSS-based news-tickers. In addition, the accurateness of the used algorithm completely depends on a proper choice of the two Web pages used for initial comparison. When choosing two job offers with, e.g., similar skill demands, these blocks could, because of their overlap, be falsely classified as irrelevant.

Firefox extensions and XUL proved to be adequate for rich client interfaces. Our experience of using Firefox extension mechanisms and XUL confirms that the development of a rich client interface is possible without much training and programming effort, being additionally facilitated by a comprehensive documentation and a steadily enlarging developer community. Nevertheless, as common in the Web Engineering area, one must expect permanent changes of the underlying SW-components which required, in our case, due to fundamental changes to the Firefox API from version 2 to 3, considerable re-coding.

Annotations partly conflicting with original HTMLtags. To visualize also the annotated Web page within the rich client interface, JobOlize currently merges the HTML-tags of the original page with the tags representing the annotations. There are, however, cases where this leads to irresolvable HTML validity conflicts, for example, if a title element overlaps an end-of-list HTML-tag. Currently, our rich client interface is not able to visualize annotations causing such conflicts, requiring more advanced visualization techniques such as an overlay concept.

Guidelines how to apply information extraction systems required. Not least due to the high degree of extensibility provided by GATE, guidelines how to apply an extraction system depending on certain characteristics of the domain are urgently needed. Especially the specification of extraction rules for JobOlize turned out as a trial and error process, getting completely lost in several rule dependencies which might be alleviated by a visual (pattern-based) ruleeditor or by some kind of decision support system, applying further AI expertise to the area of information extraction.

Measures for evaluating the quality of information extraction needed. During the case study, it emerged that, e.g., for evaluating the skill levels, a simple Boolean measure in terms of true positives and false positives is not adequate. For example, instead of “perfect” for a skill level the slightly different value “very good” might have been extracted or instead of “Oracle” for a skill the more generalize concept “DBS” might have been extracted. These examples argue for a measure, which takes a similarity aspect into account, e.g., derived from relations defined within the ontology. These explorations confirm the argumentation of Lavelli et al. [ 15 ], that the field still lacks standard datasets, evaluation procedures and appropriate measures.

Acknowledgements: We kindly thank our colleague Simon Thalbauer for valuable contributions to the realization of Jobolize.