Automatic extraction of structured data from web pages is one of the key challenges for the Web search engines to advance into the more expressive semantic level. Here we propose a novel data extraction method, called ClustVX. It exploits visual as well as structural features of web page elements to group them into semantically similar clusters. Resulting clusters reflect the page structure and are used to derive data extraction rules. The preliminary evaluation results of ClustVX system on three public benchmark datasets demonstrate a high efficiency and indicate a need for a much bigger up-to-date benchmark data set that reflects contemporary WEB 2.0 web pages.
Tomas GRIGALIS a;1
For the Web search engines to advance into a more expressive semantic level, we need
tools that could extract the information from the Web and represent it in a machine
readable format such as RDF [
In this work we address the problem of automatic information extraction from structured Web data, such as lists of products in online stores. We propose a novel approach, called ClustVX, which is fully automatic, scalable, and domain independent.
ClustVX is based on two fundamental observations. First, vast amount of information on the Web is presented using fixed templates and filled with data from underlying databases. For example, Fig. 1(a) shows three Data Records (DRs) representing information about three digital cameras in an online store. The three DRs are listed according to some unknown to us style template and the information comes from a database. This also means, that each DR has almost the same Xpath (tag path from root node in HTML tree to particular web page element), where only a few node numbers differs.
Second, although the templates and underlying data differ from site to site, humans understand it easily by analyzing repeating visual patterns on a given Web page. We hypothesize, that the data which has the same semantic meaning is visualized using the same style. Therefore humans, viewing such a web page, are able to comprehend its unique structure quickly and effortlessly and distinguish items photos, titles, prices and etc. For example in Fig. 1(a) prices are brown red and bold, title is green and bold, text ”Online Price” is grey.
ClustVX exploits both of these two observations by representing each web page element with a combination of its Xpath and visual features such as font, color and etc. For each visible web page element we encode this combination into the string called Xstring. Clustering Xstrings allows us to identify visually similar elements, which are located in the same region of a web page and in turn have same semantic meaning. See Fig. 1(b) where price elements are clustered together according to their Xstring. Subsequent data extraction leads to a machine readable structured data. The result of this extraction is shown in Fig. 1(c). Our preliminary evaluation on three public datasets demonstrate that the new method is able to consistently achieve high recall and precision in extracting structured data from given web pages.
(a) An example of three digital cameras (Data Records) in a web page
(b) A cluster with visually similar price elements
Image 2 Image 3
Samsung ES80 $84.95 Fujifilm FinePix T300 $174.95 Vivitar ViviCam F529 $84.95 (c) Desired extraction result
In the following we present a brief review of the current related research work and then, in Sec. 2 we outline the ClustVX system. We present experimental results in Sec. 3 and, finally, outline the necessary future research directions and further aspects of experimental evaluation in Sec. 4.
Data extraction systems can be broadly divided into supervised and unsupervised categories. Supervised learning approaches require some manual human effort to derive the extraction rules, while automated data extraction systems work automatically and need no manual intervention to extract data.
In this work we focus on the latter as we believe that only fully automatic systems can be applied for web-scale data extraction. Our proposed ClustVX system belongs to this category.
One widely adopted technique to automatically detect and extract DRs is to search
for repetitive patterns in HTML source code by calculating the similarity of HTML tree
nodes. Variations of simple tree matching algorithm [
Contrary to the above recent system VIDE [
Combining previous two approaches ViNTs [
Systems like TextRunner [
In summary, none of the systems can properly handle visual and structural features of web page to effectively extract structured web data. The ClustVX system proposed in this work fully exploits visual and structural information and achieves promising results.
The ClustVX processes a given Web page in the following steps: 1. Preprocessing the page. In this stage the web page is cleaned from all HTML text formatting tags, such as < b >, < em >, which appear in the middle of a text and may hinder the clustering process. Visual features of web page elements acquired from browser’s API are embedded into HTML tags for processing in the next step. 2. Generating Xstring representation of each HTML element. Each visible web page elements is represented by a Xstring, by which the elements are later clustered. As we see in Fig. 1(b) Xstring consists of a) tag names from Xpath b) visual features of that element (font style, color, weight, etc.). Structural features (string of tag names) identifies position in HTML document. Visual features enhance understanding of semantic similarity between web page elements. 3. Clustering of web page elements. All visible web page elements are clustered according to their Xstring. Resulting clusters contain only semantically similar web page elements. In Fig. 1(b) at (#1) we see a cluster of price elements. 4. Extraction of structured data. By analyzing Xpath of clustered visually similar web page elements, extraction rules are induced and data is extracted. In Fig. 1(b) at (#2) is a Xpath of the region in a page where Data Records are located. Each Data Record is enclosed in a DIV tag (#3). The final path of price elements inside a Data Record is (#4).
To evaluate ClustVX approach we use the following three publicly available benchmark
datasets containing in total of 7098 data records: 1) TBDW Ver. 1.02 [
The positive preliminary results showing that ClustVX can achieve higher than 0.98
F-Score encourage further development and evaluation of ClustVX on real-world data.
We see a must to have a data set containing thousands of pages from different web sites.
To create such a huge data set we are planning to exploit the power of crowdsourcing
by the help of Amazon Mechanical Turk service [
In this paper we presented ClustVX system, which, by exploiting visual and structural features of web page elements, extracts structured data. The preliminary evaluation of ClustVX on three publicly available benchmark data sets demonstrated, that our method can achieve very high quality in terms of precision and recall. Our future work will be concentrated on creating a new huge benchmark data set, dealing with extremely malformed HTML source code and comparing ClustVX system to competing approaches.