Automatic navigating and gathering information from Deep Web sites requires the use of Web Wrappers in order to simulate human interaction with Web sites. Web Wrappers have some drawbacks: their implementations are specific to the accessed site and also their source code needs a constant maintenance in order to support new changes on Web site. In this work we propose an annotation model for Deep Web sites that could be used for data extraction from the point of view of a Web client. Using these annotations will enable Web Wrappers to be more adaptable to Web site changes.
user-friendly tools that allow natural language references like Chickenfoot 7, and also graphical IDE solutions like OpenKapow RoboMaker 8.
The main common characteristic among all current Wrapper development tools is that, although their operation is simple, they still have strong dependencies with Web site structure, originating a continuous effort of maintenance with the created Wrappers. In our work we will define a formalization of Web structure that can be used for semantic annotation of Deep Web sites in order to represent their navigation operation. These annotations combined with current Wrapper development tools will allow implementation of Wrappers focused on the Web site model, avoiding overlapping with site structure. The maintenance required by new Web site changes will be isolated inside this model layer. Any future modifications on the structure of the Web site will need only to change the semantic annotation for the Web site, and all the Wrappers that make use of these annotations will not need to be changed.
One of the main problems raised with the navigation model generation is
the election of a point of view. In [
In this document, section 2 will introduce previous works related with Deep Web navigation modeling and data extraction. The annotation model for expressing navigation graphs that we present is described in section 3 by defining its main classes and properties. Also we present an example of annotation for a shop Web site that uses our model. Section 4, describes how can be used the annotations for the extraction of information by using an example that performs a query over different sources; and finally, future works and conclussions are described in section 5. 2
The task of generating a navigation model from a Web Site has been previously
faced, but previous works do not provide a Web model annotations formalization.
7 http://groups.csail.mit.edu/uid/chickenfoot/
8 http://openkapow.com/
There were attempts to automatically extract the model using real navigation
examples like [
Deep Web navigation model generation is treated by [
One of the main problems of Deep Web remains in that a Deep Web page
cannot be always represented uniquely with an URL9. The problem of
referencing Deep Web pages is addressed in [
Extracting semantic data from Web sites is a problem that have been faced
in Marmite[
In our demo system we use Chickenfoot as Wrapper development tool. The
main characteristics of this tool and its natural language references to HTML
elements are presented on [
The objective of a Web client is to reach a specific state through interacting with the Web. Our model represent the states and transitions that composes the navigation as a graph that describes all possible situations that could occur in a Web site from client’s point of view. In the graph, vertexes represents all possible logical states that require an action produced by the user, while edges represent the actions produced that allow the transitions between logical states.
Every state could be divided in elements called fragments, and also these fragments could be divided into new fragments. Every fragment represents a type of semantic content that can be extracted by selecting part of the element that it belongs. Figure 1 shows an example of state division in different fragments. Fragments of logical states will allow us to extract semantic information from Deep Web pages if we know the location of the state inside the navigation graph of the Web site. 9 Uniform Resource Locator, defined on http://tools.ietf.org/html/rfc1738
A transition represents the actions available for the user. These actions will allow changing among different states in the Web site. Transitions are composed of, first, an ordered sequence of interactions that originates the change of state, and second, a list of all possibe destinations that could be feasible by using the transition. There could be different possible destinations because the Web site could act in different ways as a consequence of dynamic factors like the state of the service framework, the interactions historical, date and time, etc.
Our model of states and transitions for the navigation graph is represented by the following types of elements: 1. PageState: a uniquely identifying state that represents a Deep Web page. A state contains fragments. 2. Fragment: represents an element inside a PageState or another Fragment. It is domain of the following properties: (a) fragmentOf: Defines this fragment as part of another fragment or part of a PageState as a hierarchy. (b) locatedBy: a XPath expression that identifies the position of the fragment inside the XHTML representation of the PageState. (c) numResults: an integer that defines how many times this fragment appears inside his father Fragment or PageState in hierarchy. (d) optional: a boolean that indicates if the fragment appears optionally inside his father Fragment or PageState. It may happen when a page has variable elements. (e) semantic: a set of RDF triples that represents the knowledge refered by this fragment. This property is also defined in Input and Action. 3. transitions: represents all possible transitions to other states that could be followed from this state. It has the following properties: (a) actions: any transition is originated by an ordered sequence of actions (instances of Action class) (b) sources and destinations: all possible states that could be source or destination of this transition respectively. (c) precondition and postcondition: preconditions and postconditions required in order to use this transition for travelling from sources to destinations. Postconditions can be used for distinguishing between different possible destinations. 4. Action: an interaction that could be performed over a fragment. It contains the following properties: (a) hasType: type of interaction (clicking element, selecting element, entering text. . . ). It is represented by a Chickenfoot command in our demo annotations. (b) inputs: all the form values needed for performing the interaction are refered in this property as a list that contains elements of type Input.
For each input it must be provided a name and a description.
We have represented this formalization in an OWL10 ontology that can be
accessed in [
In figure 2 we have represented an example of the annotated navigation model for searching products inside a typical Shop of Books site. From initial page at this site represented by node state0, there are two fragments: element0 (a searching textbox) and element1 (a button that initiates the searching task). From state0 it is possible to go to state1 through transition0. This transition requires to perform two actions (represented by the orderedActions list of actions): the first action is performed by inserting a search string inside element0 textbox and the second action is performed by clicking element1 button. At state1, there are three fragments that can be accessed: element2, element3 and element4. Because each of them is a part of other fragment, the XPath expression referred by the locatedBy property is relative to its father’s locatedBy property. For example, element4 is constructed with element3 and element2 locations, and so its location should be //div[@id=’atfResults’][$INDEX]//div[@class=’productPrice’].
In this figure we have used an hypothetical ontology for defining concepts of Shops, but the annotation model presented can be combined with any ontology for defining semantic knowledge inside the Fragments, Actions or Inputs of the model. This knowledge can be provided as RDF data inside the property semantic of these elements. 10 http://www.w3.org/TR/owl-features/
FRAGMENT locatedBy semantic
element4 //div[@class=’productTitle’] ADD({this log:content ?text .
this :fragmentOf ?father.}
log:implies {?father shop:productTitle ?text.})
Shop of Books Web Site FRAGMENT locatedBy semantic
element1 //input[@id=’navGoButtonPanel’]
FRAGMENT locatedBy semantic
element0 //input[@id=’twotabsearchtextbox’] fragmentOf FRAGMENT locatedBy semantic element3 [$INDEX]
ADD(this a shop:Product.)
ADD(this shop:searchedBy :searchText.) FRAGMENT locatedBy semantic
element2 //div[@id=’atfResults’] ADD(this a shop:ListOfProducts.) fragmentOf
precond = ’’ fragmentOf
postcond = ’’ orderedActions state1
ACTION enter(element0, $INPUT); semantic ACTION click(element1); semantic inputs INPUT type description semantic $SEARCHED_PRODUCT
rdfs:String String representing product ADD({this log:content ?text}
log:implies {:searchText a :Variable. :searchText log:content ?text.})
Representing the semantic of a specific element inside a state for a Deep Web
site requires not only the knowledge inferred from the specific state, but also the
knowledge of all the transitions that have been followed for reaching the state.
This knowledge may be transformed during the transitions, and new knowledge
may be added or removed from the client’s working memory of assertions,
depending on the client interactions with the Deep Web site, as in a Production
System or Rule-Based System [
As can be seen in figure 2, Fragment, Input and Action have defined the semantic property that indicates which knowledge (represented as RDF triples or inference rules) is added (ADD) or deleted (DEL) from the working memory. Adding a RDF triple simply adds itself to the working memory, while adding a RDF rule executes it inside working memory and every time new RDF triples are added, the rule must be reapplied. Deleting a triple or a rule eliminates its effect from the working memory. The effect of adding or removing RDF data of client’s working memory follows these rules: 1. If the client is located at a State that contains fragments, the semantic property of the fragments is processed inside working memory from outer fragments to inner fragments following the fragmentOf property. Fragments at the same level of indexation can be processed in any order. 2. If the client has travelled through a Transition that contains a ordered list of actions, the semantic property of the actions is processed following the same order of the ordered actions list. 3. If the client uses an input defined in a fragment of a State or in an action of a Transition, the semantic property of the input is processed after all the other semantic properties of the State or Transition.
Semantic information is associated with every instance of Fragment, Action or Input by using the semantic property. With this property, the user that is annotating a Web site indicates which kind of information does the fragment, action or input represents. Its content is expressed in RDF.
For defining semantic information, in figure 2 we make use of the following properties and concepts (based on the built-in CWM reasoner functions11): – log:implies → Property that relates antecedents and consequents of a RDF expressed rule. Antecedents and consequents are defined between brackets { and }. – log:content → This property relates the logical representation of an element with its string representation. It is used for defining the text content of an input, or for defining the specific values of an information. – this → When it is used inside the semantic property of an element, it represents the element itself. It is used for adding knowledge to an element.
In order to facilitate sharing and reusing annotations, it is needed the use of ontologies for modeling this semantic content. 11 http://www.w3.org/2000/10/swap/doc/CwmBuiltins.html 4
The content of the semantic property allows to locate a particular information
inside the navigation map by performing a query that specifies the conditions in
which the information can be found in the working memory. As our model deals
with knowledge expressed as RDF triples, we have selected SPARQL12 as query
language for this purpose. SPARQL allows to indicate the Named Graph[
A set of conditions expressed for a specific navigation model in the query can be achieved by a list of transitions. The actions of these transitions could need to provide client inputs. Also these inputs could be needed for accessing specific fragments in a state. In the example at figure 3, RSS map requires an input called INDEX for accessing a fragment inside rss state0, and Shop of Books map requires the input SEARCHED PRODUCT for performing the actions of transition0. Inputs are the points where the SPARQL query can relate data among different maps. Because inputs require that an information must be supplied, the SPARQL query must face with the possible situations with these rules: – If the conditions states clearly the specific value of the input, use this value. – If the conditions indicate that the information required by the input must be obtained from another graph, then the query must first perform its actions in that other graph. This happens in the example with Shop of Books map at condition ?product2 shop:searchedBy ?title because title is refered by the RSS map. – If the information required can not be infered, then the SPARQL query must use all possible informations that could be used for the input. In the example, this happens with the input INDEX, that is not provided, so the query must iterate among all the elements.
We have developed a demo system that may be accessed through [
orderedActions ACTION enter(element0, $SEARCHED_PRODUCT); semantic
RSS Interesting Books FRloAcaGtMedEBNyT ADD({thiselleo/mgti:tecleonnt2tent ?text . semantic this :fraloggm:eimntpOlife?sfather.} {?father shop:productTitle ?text.})
fragmentOf FRloAcaGtMedEBNyT [e$lIeNmDeEnXt1] semantic ADD(this a shop:Product.)
inputs fragmentOf deIsNctyrPipUpetTion APDosDit(i${oBtnhrOidosfOflsoK:bIgon_:ocIteNokgnDietnerEsniXtd?etelixstt} semantic {:positionlSoegl:eicmtepdlieas:Variable.
:positionSelected log:content ?text.})
FRloAcaGtMedEBNyT ADD(this a sehl/eo/mpit:eeLmnist0tOfProducts.) semantic ADD(this shop:searchedBy :searchText.)
fragmentOf
rss_state0
performing Web tasks like clicking a link, entering data in a HTML form, etc.
Chickenfoot scripts are written in a superset of Javascript. One of its main
characteristic is its support of natural language naming for HTML elements which
eases development of Web automatic tasks to end users [
The scripts are generated from SPARQL queries that can use annotations
expressed in the formalization that we have presented in this article, also accesible
as OWL at URL [
Accessing and integrating data from non-structured heterogeneous sources is a problem that currently is solved with Web Wrappers despite its drawbacks. Web Wrappers are also tools that may be used to give structure to that nonstructured information and made it available to the Web of Data. We think that wrapper development must be adapted to the new environment of linked data using semantic-oriented wrapper definitions, and not site-specific implementations. In order to achieve this, we are interested in researching more powerful languages and development tools for this new semantic approach of semantic wrapper development.
We think our model will improve development of Wrappers, because all the Web site structure changes will require only to modify the annotation model, and all Wrappers that make use of the annotations will be corrected. However, it would be adequate an exhaustive evaluation of user effort with our annotation model, that will be accomplished in future work.
We are interested in continuing working on ontologies integration with semantic wrapper implementations in order to exploit the common concepts among different Web sites. The same semantic wrapper implementation may be used in any Web site that is annotated using the same ontology. 6
This work has been partially funded by the spanish Ministry of Education and Science, project ITACA No. TSI2007-65393-C02-01