The Web contains an enormous quantity of information which is usually formatted for human users. This makes it difficult to extract relevant content from various sources. In the last few years some authors have addressed the problem to convert Web documents from unstructured or semi-structured format into structured and therefore machine-understandable format such as, for example, XML. In this paper we briefly survey some of the most promising and recently developed extraction tools.
Although XML can be regarded as a lingua franca of the Web, nowadays
almost all information available in Web sites is coded in form of HTML
documents. This situation in unlikely to change in short or even medium term
for at least two reasons: the simplicity and power of HTML authoring tools,
together with a valuable inertia to change markup language. From the point
of view of anyone interested in extracting information from Web sites, on
the opposite, the difference between HTML and XML is evident. Although
they are both derived from SGML, HTML was designed as a
presentationoriented language. On the contrary, XML has among its points of strength the
separation between data and its human-oriented presentation, which allows
data-centered applications to better handle large amounts of data. Another
fundamental advantage of XML is the availability of powerful instruments
for querying XML documents, namely XQuery/XPath[
Information extraction from Web sites is often performed using wrappers.
A wrapper is a procedure that is designed to access HTML documents and
export the relevant text to a structured format, normally XML. Wrappers
consist of a series of rules and some code to apply those rules and,
generally speaking, are specific to a source. According to [
Besides the HTML page structure, effective wrappers consider also the structure of hyperlink as it may reveal relevant information. Depending on the type of Web search engine the following kinds of results can be obtained: • one-level one-page result: one page contains all the item descriptions; • one-level multi-pages: a series of pages linked one to another, all containing the item description; • two-level pages: a chain of pages, each containing a shortened description of items, each linking to a detailed page.
In this section a brief overview of some information extraction tools will be
given. The idea is to illustrate the main features of tools belonging to the
family of the so-called HTML aware tools (see [
The LiXto project was started by Gottlob et al. at TUWIEN and is now
developed and sold by the LiXto GMbh software house. LiXto [
Fetch Agent Platform [21] is another example of commercial information extraction tool.It is based on two major components, the AgentBuilder which provides a visual environment that allows a user to construct web agents, and the AgentRunner which automatically performs the tasks specified by the agent, and produces structured data. The framework also provides a tool able to monitor Web target pages, specifying which data fields are to be checked. The extraction rules are based on landmarks (groups of consecutive tokens) that enable a software agent to locate the start and end of fields within a page. The extraction algorithm that learns these landmarks based on examples labeled by the user and uses the hierarchical structure of the page to constrain the learning problem.
RoadRunner [
The Dynamo Project [
HTML documents contain a mixture of information to be published, i.e., meaningful to humans, and of directives, in the form of tags, that are meaningful to the browsers and determine the appearance on the screen. Moreover, since the HTML format is designed for visualization purposes only, its tags do not allow sophisticated machine processing of the information contained therein.
Among other things, one factor that may prevent the spread of the Semantic Web is the complexity of extracting, from existing, heterogeneous HTML documents machine-readable information. Although the Dynamo project addresses only a fraction of the Semantic Web vision, management of HTML documents needs some technique to locate and extract some valuable and meaningful content. Therefore, a set of annotations, in form of meta-tags, were defined; they are inserted inside HTML in order to highlight informational content that is essential for the creation of a RSS feed. In this application, meta-tags are used as annotations, to describe and mark all interesting information, in order to help in the extraction and so-called XML-ization phases. Notice that with pages that are dynamically generated out of some template (which is the case with practically all on-line fora) Dynamo annotation is done, manually but only once and for all, over the page template.
Once HTML documents are processed by Dynamo, annotated semantic structures are extracted and organized into a simple XML format to be stored and used as a starting point for document querying and transformation. The structure of the XML output resembles the structure of meta-tags previously defined and the RSS XML structure, in order to facilitate transformations from the former to the latter. At the moment, a version of Dynamo is undergoing a phase of testing in several forum of the Milan Community Network (Rete Civica Milanese). In the past few years, many approaches to the problem of Information Extraction (IE) by means of Wrapper Induction (WI) systems have been tackled. Previously proposed taxonomies will be briefly examined in this section. Hsu and Dung [22] classified wrappers into 4 categories: • • • • hand-made wrappers using general-purpose programming languages; designed programming languages; heuristic-based wrappers; WI approaches.
A complete categorization was made by Laender et al. [
They also compared among the tools using these features: degree of automation, support for complex objects, page contents, availability of a GUI, XML output, support for non-HTML sources, resilience and adaptiveness.
Sarawagi [24] classified Web sites wrappers according to the amplitude of the tasks they are able to face. So he distinguishes record-level wrappers, capable to extract elements of a single list from a Web page, page-level wrappers which extract elements of multiple records and, finally, site-level wrappers which can extract and convert into structured format an entire Web site.
More recently, Chang et al. [
In this paper we presented a short survey of most recent tools for the extraction of information from Web sites. All the tools presented here automatically generate wrappers in order to accomplish their task and all of them provide output data in XML format, thus focusing on the meaning of data rather than on their graphical representation.
There are a series of current and future applications where information extraction tools can fully exploit their power. One of the most promising seem to be the comparison of items, for example in commercial aggregators. The possibility for a user to compare different offerings of the same object is a feature currently not supported by online auction sites.
Even in the area of communication, the possibility of aggregating and querying information automatically extracted from different Web news sites seems really promising, specially in conjunction with the features offered by XML-based query engines. This, together with more flexible and powerful extraction tools will certainly help paving the road to the semantic web. 20. De Cindio F., Fiumara G., Marchi M., Provetti A., Ripamonti L.A. and Sonnante L. (2006) Aggregating information and enforcing awareness across communities with the Dynamo RSS feeds creation engine: preliminary report. OTM Workshops (1) 2006: 227-236 21. Fetch Technologies, available from http://www.fetch.com 22. Hsu C.-N. and Dung M. (1998) Generating finite-state transducers for semistructured data extraction from the web. Journal of Information Systems 23(8): 521-538 (1998) 23. Chang C-H., Hsu C.-N. and Lui, S.-C. (2003) Automatic information extraction from semi-structured web pages by pattern discovery. Decision Support Systems Journal, 35(1): 129-147 (2003) 24. Sarawagi S. (2002) Automation in information extraction and integration, Tutorial of VLDB (2002)