This position paper as part of a PhD thesis is a contribution to an automatic retrieval of information from the Deep Web. Addressing current limitations of the Deep Web Information Retrieval leads to the prevailing lack of semantics regarding the retrieval process. Focusing this problem from the information providing services perspective, indicates the signi cant potential of additional semantic annotations provided by websites. Web query interfaces, the interfaces to the majority of available information on the Deep Web, are interpreted as Semantic Deep Web Services (SDWS). The introduction of a SDWS annotation leads to great potential for Information Retrieval services based on the large variety of information available on the Deep Web.
A continuously increasing amount of content on the web is not directly accessible
and indexable by search engines. The content might, for example, be hidden in
non-public, inaccessible areas or might be stored in background databases and
therefore only accessible through web query interfaces. This part of the web is
known as the Deep Web (or Hidden Web) in contrast to the Surface Web which
can be easily accessed and indexed by common search engines [
The Surface Web consists of mostly static content, which is directly
interlinked with static hyperlinks. "Search engines rely on hyperlinks to discover new
webpages [...]" [
reach this part of the web. This is caused by the fact, that search engines "[...]
typically lack the ability to perform form submissions" [
Considering current arising services on the web like the Google Knowledge Graph, "we can use [...] [these services] to answer questions you never thought to ask and help you discover more"1. These services are related to Knowledge Discovery, but in general the bene t from the automatic discovery of new knowledge from existing information on the web is depending on an excellent Information Retrieval. As the retrieval of information from the Deep Web is still limited, Knowledge Discovery services are also still limited in their potential. Therefore, more e cient and targeted retrieval mechanisms for the Deep Web are needed to achieve full potential of Knowledge Discovery services.
The usage of semantic annotations for information on the web play a
crucial role "to assimilate information from multiple knowledge sources" [
The retrieval and indexing of Deep Web content have been addressed from di erent perspectives in the past. The e ort has mostly focused speci c applications to discover, retrieve and index structured data from the Deep Web. This includes special emphasis on the automatic web query interface interpretation.
Common approaches focusing on exposing Deep Web content can be
classi ed to surfacing and virtual integration approaches. The surfacing approach
focuses a search engine initiated process to index the search result pages for
precomputed (randomized) queries to discover Deep Web content on large scale [
Regarding the discovery and cataloging of Deep Web sources Hicks et al. [
Furche et al. [
Until today, there still exists no general domain independent solution for
the Deep Web Information Retrieval problem. Just a fraction of total available
data in background databases may be covered by common state of the art
approaches. This is particularly due to the fact, that for large data sets there
exist nearly endless possible permutations of search results. This especially
applies to the retrieval of dynamic content. Therefore, it seems to be improbable
to improve retrieval and indexing mechanisms towards reaching a 100%
coverage of all available Deep Web content. Consequently, this is not the focus of
our current research. Currently still limited mechanisms have already "[...]
succeeded largely by targeting narrow domains where a search application can be
ne-tuned to query a relatively small number of databases and return highly
targeted results" [
To step forward towards a Semantic Deep Web, which is the superordinated long-term objective, it is necessary to focus on additional research questions resulting from previously identi ed limitations. For the targeted retrieval especially of dynamic Deep Web content, the need of an e cient and in an ideal case fully automatic approach is essential. Therefore, the focus needs to be set to these challenges: content providing service Discovery, Invocation & Execution and Composition. Addressing these challenges will ensure the discovery of appropriate web query interfaces providing access to relevant content (! Discovery ), the appropriate query mapping and query submission (! Invocation & Execution) and the service interoperability (! Composition). Common approaches for Deep Web Information Retrieval focus these challenges from the information 4
retrieving services perspective. The conceptual idea being introduced in this section focuses these challenges from the information providing services perspective.
Common semantic annotation standards like RDFa and schema.org microdata address particularly the annotation of web content and do not have means for the prevailing lack of semantics at the crucial point of the Deep Web Information Retrieval process. This crucial point is regarding the web query interfaces. To improve common crawling, indexing and content retrieval mechanisms and to ensure new mechanisms, a semantic annotation for query interfaces is suggested. This will reuse the query interfaces originally intended for human users in a combined computer and human readable format. The abstract concept, to describe query interfaces in a computer readable format, is derived from the semantic annotation of Web Services. Standards like Semantic Annotations for WSDL and XML Schema (SAWSDL) provide a machine readable Web Service annotation describing the functionality and retrievable data. A semantic annotation for query interfaces will provide machine readable information for henceforth called Semantic Deep Web Services (SDWS). 3.2
An implementation of the SDWS annotation should meet the following fundamental criteria: SDWS interface semantics, providing a generalization of SDWS interfaces (! abstract ) with the ability to include own vocabularies as for example thesauri (! extendable). Going more into details, a SDWS annotation prototype should provide information about general properties regarding the content that is provided by the SDWS (! content properties) and concrete interface eld properties to describe the semantic structure and internal structural dependencies of the SDWS interfaces (! eld properties).
The prototype SDWS content properties describe the content domain of the retrievable information, the content language, as well as the content type. The content type attribute may be described based on schema.org microdata and the supplementary usage of other vocabularies. An additional content property might provide information about the amount of available data (property: count ). These content properties are just the extendable basis for this prototype providing general information about the retrievable content. Further ideas for the extension of SDWS content properties will be discussed in the following section.
A simple example for the SDWS content properties is provided in Fig. 1, describing the basic SDWS eld properties for the interface of the subject portal EconBiz 2. EconBiz provides highly relevant content for the domain of economics and business studies and access to speci c content types (various types of CreativeWork and information about Event s).
The prototype SDWS eld properties describe the eld type (e.g. selectField, inputField ), as well as the input domain and output range of each particular SDWS interface eld. The input domain attribute describes valid input values of a speci ed SDWS eld. Furthermore, it is a trigger for the output range attribute, 2 http://www.econbiz.de/en/ Semanti cation of Query Interfaces to Improve Access to Deep Web Content 5 highly relevant content for a specific domain contentDomain: #economics [AND]
#businessStudies contentType: http://schema.org/CreativeWork [AND]
http://schema.org/Event contentLanuage: #multilingual contentCount: 8.913.444 as its input value de nes the restriction set for the retrieval process at time of form submission (examples, Fig. 2-4). Additionally, a vocabulary attribute may reference for instance a thesaurus that can be used as suggest-value vocabulary for the particular domain to ensure a targeted retrieval.
The basic SDWS eld properties example in Fig. 2 refers to a standardized vocabulary, the STW Thesaurus for Economics. The STW provides "vocabulary on any economic subject" containing "[...] more than 6,000 standardized subject headings and about 19,000 entry terms to support individual keywords"3. This thesaurus is the basis for the annotation on metadata level in EconBiz and will therefore ensure a targeted retrieval. The bene t of the vocabulary property will especially apply to digital libraries but also to other domains. Therefore, regarding simple SDWS interfaces, this might be one of the most appropriate use cases for SDWS eld properties as there is no complex interface structure. fieldType: inputField fieldDomain: xsd:String fieldRange: http://schema.org/CreativeWork#all [AND]
http://schema.org/Event#all fieldVocabulary: #STWthesaurus
Focusing on more complex SDWS interfaces, the example in Fig. 3 contains chunks of related elds that a ect each other. The rst selectField as part of the marked chunk de nes the relation to the other chunks. The second selectField as part of this chunk de nes the input eld domain and restricts the input eld range of the inputField that is part of the focused chunk. Furthermore, Fig. 4 considers some exemplary e ects triggered by the selection of di erent select values of the second selectField within the focused chunk in Fig. 3.
More complex examples as illustrated in Fig. 3 and 4 demonstrate that the semantic meaning behind a SDWS interface might be quite complex and automated form understanding approaches will quickly reach their limits. Especially 3 http://zbw.eu/stw/versions/latest/about.en.html 6
Arne Martin Klemenz, Klaus Tochtermann chunk of related fields that affect each other fieldType: selectField fieldDomain: #predefined (xsd:boolean) fieldRange: define #chunkRelation fieldType: selectField fieldDomain: #predefined (xsd:string) fieldRange: define #inputFieldDomain
restrict #inputFieldRange the automated detection of related elds and the detection of complex relations within chunks might be the most di cult part, where common approaches fail.
In addition to the SDWS interface annotation, it is advisable to link every SDWS interface from the websites root index. This will ensure a targeted SDWS discovery and can be realized by using XML Sitemaps to de ne a SDWS retrieval index. The SDWS annotation itself is suggested to be embedded directly to each particular SDWS interfaces. 4
The introduced SDWS annotation will lead to great potential for new
information retrieval mechanisms and plays a signi cant role for the improvement of
current mechanisms. Queries might for example be automatically mapped to
various SDWS at the same time based on the SDWS annotation (! abstract
semantic querying ). In accordance with Heath et al. the vision of "users [...]
interacting with the Web as a data space" [
vide licensing information exactly at that point where the information itself is being provided e.g. based on the Creative Commons licensing model.
Overall, this approach will make webmasters aware of their responsibility to add SDWS annotations to SDWS interfaces in addition to current semantic content annotations. This process requires additional e ort on the one hand, but on the other hand it also enables the webmasters to control the information content that may be retrieved by various retrieving services like search engines. For now webmasters may only use common HTML attributes like nofollow or noindex and the Robots Exclusion Standard to control the crawling behavior on their websites. The SDWS annotation ensures the targeted in uence of the webmaster. Furthermore, only the webmaster knows the exact semantic statement intended by the implemented SDWS interface. Regarding web content, search engines rely on semantic content markups as it is more reliable than current automatic content interpretation approaches. Therefore, it is obvious, that this will also apply to the annotation of SDWS interfaces. 5
This paper addressed the lack of semantic information regarding web query interfaces in the process of Information Retrieval from the Deep Web. Transferring the concepts of semantic web content annotations on the one hand and Semantic Web Service Descriptions on the other hand, leads to the great potential of semantic annotations for SDWS interfaces. Equivalent to semantic web content annotations, the SDWS annotation provides an unambiguous semantic interpretation of the SDWS interface. A variety of current information retrieval mechanisms and form understanding systems try to analyze SDWS interfaces automatically by focusing the Deep Web Information Retrieval challenge from the retrieving services perspective. Instead of relying on these, the introduced SDWS interface annotation is focusing this challenge from the information providing services perspective.
In general, this approach follows the open knowledge sharing paradigm as
part of the Semantic Web vision from Berners-Lee et al. [
This approach will contribute to domain independent and automatic Information Retrieval mechanisms based on the introduced SDWS annotation. Manual e ort for currently still limited Deep Web Information Retrieval mechanisms will be reduced or even eliminated. Furthermore, these retrieval mechanisms will bene t regarding their e ciency and can be adapted targeting broader domains. 6
Future work will especially focus on the critical evaluation based on further research studies. The de nition of a concrete SDWS annotation syntax based
Arne Martin Klemenz, Klaus Tochtermann on the usage of existing annotation standards will concern the challenge how retrieving services will learn to understand the SDWS annotation. Reduction of manual e ort for the annotation process also requires further e ort. A semiautomatic generation process may provide support for the de nition of SDWS annotations. This process may be based on sampling and probing the background database utilizing promising automated form understanding approaches. This may lead to semi-automatic generation approaches for the SDWS annotation.