The convergence of semantic web techniques with web service technologies has enabled the emergence of so-called semantic web services. This new kind of services enacts the automatic manipulation of services by software programs, to perform tasks such as automatic service location, composition, and invocation. In this paper, we propose methods and techniques that enable the semi-automatic generation, deployment, semantic annotation and classification of web services.
Since the emergence of the semantic web (Berners-Lee et al., 2001), many research efforts have been aiming to use semantics to endow web services with a much higher potential for automation. These efforts have resulted in a new research trend called semantic web services (Terziyan et al., 2003). The basis of this trend is to attach some semantic information to current WSDL-based web service descriptions (Christensen et al., 2001) in order to enable their analysis and manipulation by software programs. This manipulation would be useful to enact powerful capabilities such as automatic location, selection, invocation or composition of web services.
Nevertheless, semantic web services are facing today some problems and limitations that restrain their advancement and evolution. From our point of view there are two main problems to be solved:
• There is no consensus on which language has to be used when describing the semantic information about web services, although there are two main proposals that are gaining momentum at the present time, OWL-S (Martin et al., 2004) and WSMO (Roman et al., 2005).
• There is a lack of automatisms to help web service developers in the creation of services, the annotation of their semantic information or any other task related to the web service life cycle.
The research presented here is set forth to help overcome these problems. More specifically, our work proposes:
• A realistic approach to the semantic information that can be obtained and used from a web service.
• A set of automatisms that will allow web service developers to perform some tasks related to web services. More precisely: creation, deployment, testing, semantic annotation, and classification.
The solutions developed in this work have been implemented and integrated in a web platform called Federica. Besides describing this platform in some detail, the focus of this paper will be to explain our research ideas and discuss the results achieved so far.
The rest of the paper is structured as follows. Section 2 shows all the details of our work towards the semi-automatic generation of web services. Next we explain our approach to the semi-automatic classification of web services. Finally, on Section 4, we provide some conclusions, as well as the next foreseen steps.
The critical mass of available web services, let alone semantic ones, is still quite limited today. This is an important practical barrier for the advancement of research and innovation in this field, which is difficult to achieve without a sufficient testbed to try and evaluate the innovations. Artificial examples (i.e. built by the innovators themselves) hardly provide an objective basis for measuring the usefulness and performance of new proposals, not to mention the considerable cost implied in building the testbed, just for experimentation purposes.
The semi-automatic generation of web services, from such a widespread commodity as are web applications, can help with this necessity, and is an interesting research problem by itself. Of course, the expected quality of automatically generated services should not be the same as that of manually defined ones, but we aim at achieving a sufficient quality for the services to be useful for a variety of purposes, where of course, if needed, the generated web services can be completed or refined by a programmer. Moreover, such a facility as we are proposing here can be helpful in the transition from the current World Wide Web of applications to a (Semantic) Web of services.
The idea of the automatic generation of web services from web applications has already been addressed in former research works. Because of its relevance for our research, it is worth citing the work developed by Pham (2004), which already proposes the creation of web services from web applications. In Pham's proposal, web services are used only as gateways to web applications, so that any program can invoke programmatically the functionalities provided by web applications. When the generated web services are called, they translate their input parameters to HTTP parameters, send them to the application server, and wait for the response. Our work takes a step further from this, by recognizing or generating data types, finding associations of types with ontology concepts, if any, and automatically classifying the generated services. Overall, our research aims to push forward the goals undertaken by Pham et al towards the generation of semantically-enhanced web service descriptions.
It is also interesting to cite the early work by Sahuguet et al (1999) in a similar direction. Although this work does not use an explicit notion of web service, since web service standards had not yet seen the light by that time, their approach is very similar to the one proposed by Pham. The main differences can be attributed to the status of the technology at the time of publication -while Pham uses web service languages and tools to build a gateway to web applications, Sahuguet et al. use non-standard descriptions (manually generated) and generate Java applications as a gateway to the functionality.
The process flow of our approach to the generation of web services is shown in Figure 1. The input to the automatic generation system is the entry web page of an application. The page is parsed ("HTML parser" box in the figure) into a easier to process in-memory data structure, which is analyzed in order to produce a WSDL description (WSDL Generator module) for the service to be generated, plus some additional semantic descriptions (Semantics Generator module). An implementation of the WSDL service is automatically generated (Service Implementor module) and deployed into a web service support platform (Axis and Tomcat have been used). Once the service is deployed, it can be invoked from any web service client that adheres to the generated WSDL description. One such client is automatically generated for testing purposes. Calls to the generated web service (SOAP requests) are automatically deferred to the original web application (HTTP request) by the generated service implementation. The result (a web page source code) is returned to the service client (SOAP response).
The steps enumerated above will be explained in more detail in each of the following subsections. More precisely, in section 2.1, we explain how WSDL descriptions are extracted from the web interface of applications. In section 2.2, the linkage of web services with web application functionality, and the deployment of the service, are described. Then, in section 2.3, we show how the execution of the generated web services is managed.
For the automatic analysis and description of the functionality provided by a web application, the system we propose takes as input a) the URL of the web page which gives access to the application, and b) a name to identify the service to be generated. Our approach for the generation of WSDL descriptions is based on the inspection of the source code of the entry web page to the application, more specifically the HTML forms, as the basic UI construct for web applications. Of course web application UIs are being implemented today using other client-side technologies (JavaScript, Flash, etc.) beyond plain HTML, but as a first approach to the problem, we have circumscribed it to HTML forms.
In addition to the UI elements to interact with the application, the source code of a web page through which a web application is accessed includes all kinds of additional content, just as any other web page does: purely informative contents (titles, instructions, logos, advertisements, etc.), navigation elements (hyperlinks), plus page style and layout details. Although all these elements surrounding UI controls could be exploited to find cues which help in the analysis of forms, in our current approach we only inspect the UI elements themselves.
From the analysis of the web UI, our system determines the operations of the web service to be generated, their inputs, and the type of these, and this information is retained in a WSDL description. The generation procedure works through the following steps:
1. A service with a unique wsdl:portType is defined for the whole application. 2. The application page source is read from its URL, and the HTML forms that the page contains are extracted using an HTML parser. 3. A wsdl:operation is created for each of the forms found in the page.
4. An output message and an input message are defined for each wsdl:operation. 5. The "input", "select" and "textarea" HTLM components are identified as inputs for the service (that is to say, as wsdl:part elements in the "in" wsdl:message) for each of the forms. 6. Depending on the HTML control type and attributes, a different data type is assigned to each service input, a new XML schema type being defined in some cases. Table 1 shows the correspondence between UI controls and service message part data type. 7. The wsdl:output message will contain only one wsdl:part of xsd:string type which, once the service is invoked, will return the source code for the web page that is returned from the web application form. Figure 2 shows an example of the generated WSDL description for the Google home page.
<?xml version="1.0" encoding="ISO-8859-1"?> … <wsdl:types> … <simpleType name="hlEnumeration"> <restriction base="xsd:string"> <enumeration value="es" /> </restriction> </simpleType> … <simpleType name="btnGEnumeration"> <restriction base="xsd:string"> <enumeration value="Búsqueda en Google" /> <enumeration value="" /> </restriction> </simpleType> … <simpleType name="metaEnumeration"> <restriction base="xsd:string"> <enumeration value="" /> <enumeration value="cr=countryES" /> <enumeration value="lr=lang_es" /> </restriction> </simpleType> </schema> </wsdl:types> <wsdl:message name="fRequest"> <wsdl:part name="hl" type="tns1:hlEnumeration" /> <wsdl:part name="ie" type="tns1:ieEnumeration" /> <wsdl:part name="q" type="xsd:string" /> <wsdl:part name="btnG" type="tns1:btnGEnumeration" /> <wsdl:part name="btnI" type="tns1:btnIEnumeration" /> <wsdl:part name="meta" type="tns1:metaEnumeration" /> <wsdl:portType name="GoogleSearchIF"> <wsdl:operation name="f" parameterOrder="hl ie q btnG btnI meta "> <wsdl:input name="fRequest" message="impl:fRequest" /> <wsdl:output name="fResponse" message="impl:fResponse" /> </wsdl:operation> </wsdl:portType> <?xml version="1.0" encoding="ISO-8859-1"?> … <wsdl:types> … <simpleType name="hlEnumeration"> <restriction base="xsd:string"> <enumeration value="es" /> </restriction> </simpleType> … <simpleType name="btnGEnumeration"> <restriction base="xsd:string"> <enumeration value="Búsqueda en Google" /> <enumeration value="" /> </restriction> </simpleType> … <simpleType name="metaEnumeration"> <restriction base="xsd:string"> <enumeration value="" /> <enumeration value="cr=countryES" /> <enumeration value="lr=lang_es" /> </restriction> </simpleType> </schema> </wsdl:types> <wsdl:message name="fRequest"> <wsdl:part name="hl" type="tns1:hlEnumeration" /> <wsdl:part name="ie" type="tns1:ieEnumeration" /> <wsdl:part name="q" type="xsd:string" /> <wsdl:part name="btnG" type="tns1:btnGEnumeration" /> <wsdl:part name="btnI" type="tns1:btnIEnumeration" /> <wsdl:part name="meta" type="tns1:metaEnumeration" /> <wsdl:portType name="GoogleSearchIF"> <wsdl:operation name="f" parameterOrder="hl ie q btnG btnI meta "> <wsdl:input name="fRequest" message="impl:fRequest" /> <wsdl:output name="fResponse" message="impl:fResponse" /> </wsdl:operation> </wsdl:portType> … </definitions>
Once an WSDL is generated, our system provides an automatic implementation of the service, consisting of an application wrapper from which calls to the service are transferred as requests to the initial web application. To achieve this, a link engine has been implemented using Axis for sending and receiving SOAP messages (Gudgin et al., 2003) in the communication with web services.
Using the Axis WSDL2Java tool, a set of empty Java classes is generated from a WSDL description, which once implemented will define the desired web service functionality. Service deployment is also done with Axis, the engine of which processes the SOAP messages received by the service.
The result of this process is a fully functional web service, whose operations carry out the same tasks as those the web application provides.
In order to try the generated services, our environment provides the automatic generation of a client and a web interface to invoke the generated services. This facility is an invaluable tool for development and testing. Furthermore, we plan to integrate this capability with a sophisticated editor with which developers can refine the service response description, and make it more precise, e.g. as an XML Schema data type, rather than a plain web page fragment source. Furthermore, this option opens further possibilities to augment descriptions with richer semantics, in the spirit of the Semantic Web Services vision. These would also be useful, in particular, for the automatic classification of the service, which is addressed in the next section. We have started the development of this edition tools, but, at the present time they are not yet integrated with the rest of the platform.
With the techniques described in previous sections, we have the capacity to generate functional web services almost automatically and have them deployed (and stored) in a repository. Nevertheless, those services are stored in a chaotic way, without any type of structure. This lack of structure in the service storage can be seen as a problem because tasks such as, for instance, searching for a concrete service, gain much complexity. In addition, if we analyze the current situation of web services we see that classification is already an important concern by itself.
Nowadays, UDDI (OASIS UDDI, 2004) is the most accepted and used protocol for publishing, searching, and finding services over the web. These actions are usually performed using UDDI registries, which can be seen as service repositories easily accessed through a URL. In these registries, the published services are classified using some kind of taxonomy (i.e. UNSPSC, NAICS, etc.). Nevertheless, this classification is performed manually by the human publisher of the service. Due to the huge quantity of service classes in taxonomies as the ones mentioned above, the classification process could be very complex and tedious.
What we are proposing here is to use a software agent to help service publishers in the classification task. In order to meet this goal, in the context and spirit of the generation process described in previous sections, our basic starting point for any automated processing of the generated services are WSDL descriptions. Nevertheless, the information contained in a WSDL description is not sufficient to perform any type of classification. So, we need more information about services, their parameters, their operations, etc.
In conclusion, we need a higher level description of the service including some semantic information that enables a software agent to help publishers in the classification task.
In addition, as we are trying to classify web services in a taxonomy, we will need, of course, to define that taxonomy. For this purpose, our system can use any available taxonomy, such as the standards currently recommended by UDDI registries, e.g. UNSPSC or NAICS. Now starting from a taxonomy and a means to represent the semantic information of a web service, the classification process is based on comparing the new generated web services with some services previously classified in the taxonomy. From this comparison, we obtain a similarity measure representing the probability that a service should be placed under a certain classification category in the taxonomy. The definition of similarity measures between taxonomy concepts has been already addressed, for example, by Bernstein et al. (2005).
All the details about each issue taking place in the classification process can be found in the following subsections. More precisely, in section 3.1, all the information about the representation of web service semantics can be found. Next, in section 3.2, we will explain some details related to the taxonomy we are going to use and the previously classified services we will use for comparisons. Finally, in section 3.3, we will show the complete process we will apply to obtain the similarity measure between services and taxonomy classes.
Our first step towards the achievement of the classification mechanism is to obtain, define and represent all the extra needed semantic information for a service. As it has been already said, there are mainly, at the present time, two initiatives on semantic description languages for web services: OWL-S and WSMO. Both of these languages provide a highly generic way to describe semantic information about a web service. Because of this, they are considerably complex and difficult to use, even for very simple services. Therefore, we have decided to define a simpler ontology that fits better with our purposes. We do not discard using one of these initiatives in the future, as we gain more generality in our own service descriptions.
We have chosen OWL (McGuiness et al., 2004), widely accepted in the semantic web community, as the ontology-definition language for our service ontology. With this language, we have developed a simple ontology containing all the concepts (classes) and relations (properties) that are described in WSDL service descriptions. The ontology is shown in Figure 3. In the current status of our research, the ontology does not include much semantics about the different service concepts, but we expect to extend it with new information as soon as it can be automatically or semiautomatically retrieved.
Using the aforementioned ontology we have an OWL representation of the service in terms of the information described in WSDL. The "extra semantics" in this description with respect to plain WSDL is held:
• In the "serviceType" property of the "WebService" class. This allows linking the description of the service to a type of service defined in other ontologies (or taxonomies), i.e. the one explained on the section 3.2. • In the "parameterType" property of the "Parameter" class. This allows linking the description of a data type to a type defined in other data type ontologies. With this ontology we add to WSDL the possibility of representing the classification of a web service and a higher-level description of data types. Regarding the latter, and in order to enrich service parameter descriptions, we have defined a data type ontology to which parameter data types will be mapped. This mapping means that the range of the "parameterType" property of the web service ontology described above is conformed by the classes of the data type ontology, the main elements of which are shown in Figure 4.
As can be seen, the ontology defines three kinds of data types:
• Simple data types: This is an ontological representation of XML Schema data types. The subclasses represent the different kind of data types that can be used in XML Schema (although the classes represented each data type have not been shown in the figure).
• Complex data types: This is an ontological representation of XML Schema complex types, i.e. structures, which could appear in a WSDL. Obviously, as we do not know, a priori, which structures will be in the analyzed WSDL descriptions, this class is initially empty.
• Enumerated data types: Despite these data types could be seen as restricted simple types, we have decided to place them under a separate class. This decision is based on the fact that, due to the basis of the generated web services (this is, web applications), the appearance of enumerated data types is very common (i.e. every combo box of an HTML form). Thus, with this ontology we have an OWL representation of the different data types used by a service. These representations are very useful when trying to classify a service, as they can be used as source for a data type similarity comparison between two services.
Then, with the union of both ontologies we have all the service information needed represented in an ontological way and the possibility of using software agents (i.e. based on OWL reasoners) to perform the classification of the service.
Given a set of classified WSDL descriptions under some classification taxonomy, and a newly generated web service description, based on the service ontology and techniques described in previous sections, in order to deploy the new service, and make it available in a service registry, an appropriate classification category should be assigned to the new service, e.g. for easy retrieval.
As it has been already said, we use the classification standards used by UDDI registries (such as UNSPSC or NAICS) as the taxonomy where our web services will be classified, but our techniques are independent from the standard used. To facilitate our development, and in anticipation of future semantic-web-oriented extensions of our work, we are using an OWL representation of the taxonomy. This has the advantage that the same tools, e.g. OWL reasoners, can be used to process service descriptions and classification hierarchies in a uniform way (e.g. for the computation of similarity measures between services and categories, reasoning about class hierarchies, etc.).
We have developed a heuristic for automated service classification, based on the comparison of an unclassified service with available classified services, whereby a measure of the likelihood that the service can be correctly classified under some category is computed. This problem is related but different from the ones addressed by other service comparison techniques found in the literature (Bernstein et al., 2005;Sirin et al., 2004;Bansal and Vidal, 2003;Field and Hoffner, 2003). In particular, in our approach, the fact that two services are found similar does not mean they are interchangeable for e.g. automatic service selection, composition, or invocation. Instead, the similarity measures provide a ranking of candidate classifications for a new service, which can be of great help for a human service administrator that is facing classification taxonomies with thousands of categories.
Our heuristic works as follows. Let S be the set of all web services, and let C be a classification taxonomy. Since C is used to classify the services in S, we may define C as a subset of the parts of S, P(S), i.e. each c∈C is a labelled set of services c ⊂ S (note that it is possible that c = ∅). Given a new service s∈S, we want to find the category c∈C that maximizes its similarity with s. We measure the similarity between s and c is by the following formula: ∑ ∏ s which can be created or completed with little effort. Some current gaps, that are requiring an effort from our part in compensation, include the construction of a large-enough repository of manually classified WSDL service descriptions (say, by the hundreds), to serve as testbed for our techniques, and some current limitations of available semantic web tools, such as the OWL reasoners. It is to be expected that our workarounds can be removed as these tools keep reaching maturity.
Rather than aiming at, or starting from, a maximum semantic web service representation expressiveness, as in OWL-S and WSMO, we have approached our research objectives in a bottom-up approach, starting from consolidated, ready-to-use technology, such as WSDL and OWL, already being adopted by industry as of today. From this standpoint, our long-term goals are the same as those of OWL-S and WSMO, namely to bridge the gap between current web service technology, and the vision of Semantic Web Services.
Our next steps in this direction include: growing the collection of services generated by our platform, stored in our repository; extend the semantic information represented in our service ontology, that can be obtained in an automated way; complete the development of the edition/administration tools to customize and improve the quality of the generated services; further testing, evaluation, and tuning of the automatic web service classification techniques.
We would like to thank Ruben Lara for his thorough revision, comments, and suggestions on the early drafts of this paper, which greatly helped to improve our work.
whereby the similarity between a service and a category is computed in terms of the similarity between the service and the services classified under that category. We have chosen the above formula because it meets the following desired properties:
• sim(s,c) ∈ [0,1] provided that sim(s,s') ∈ [0,1] ∀s'∈c • sim(s,c) ≥ sim(s,s') ∀s'∈ c (in particular this means that sim(s,c) = 1 if sim(s,s') = 1 for some s').
• sim(s,c) increases monotonically with respect to sim(s,s') ∀s'∈c, • Since ∀s'∈c, sim(s,c) = sim(s,s') + (1 -sim(s,s')) sim(s, c -{s}), sim(s,c) can be computed efficiently (i.e. with linear Θ(|c|) complexity). Now, the similarity between two services is measured in terms of the similarity of their operations and parameters. If we denote by P s the set of the parameters of service s, and by OP s the set of its operations, the similarity of two services is defined as: Developing a measure for comparing service operations is still work in progress in our research as of this writing, so in the meantime we are working with ( , )
The similarity between two parameter sets P and P' is computed as the average of the best possible pairwise similarities obtained by an optimal pairing of the elements from the two sets. This can be formalized as follows. We define top(P,P') as the pair (p,p') ∈ P × P' that maximizes sim(p,p'). Then let P 1 = P, P 1 ' = P', P k = P k-1 -{p k-1 } and P k ' = P' k-1 -{p' k-1 }, where (p k-1 ,p' k-1 ) = top(P k-1 ,P' k-1 ). With these definitions, the similarity between two parameter sets is given by: (
In our current model, the similarity between two parameters is defined as the similarity between their respective types. Let T denote the set of all types (i.e. the set of domain ontology classes). We define the similarity between two types t∈T, t'∈T as:
where h(t, t') denotes the distance between t and t' in the type hierarchy (e.g. the distance between a type and its immediate supertype is 1).
With this process, we can have finally a vector of similarity measures between a new analyzed service and all the taxonomy classes. Then it is time for service publishers to decide on which class (from a ranked list) the service best fits.
Our work shows that automatic web service generation is feasible. In fact, we have implemented all the ideas shown in this document and have developed a first version of the Federica platform (accessible at http://rhadamanthis.ii.uam.es:8080/federica). In that platform it can be seen that tasks such as generation and deployment of web services are done automatically. It also includes the tools needed for web service execution and testing. Nevertheless, the edition tools that have been discussed in section 2 are not yet integrated with the rest of the platform.
Our work on semi-automatic classification has some commonalities with the research on web services matchmaking (Sirin et al., 2004;Bansal and Vidal, 2003;Field and Hoffner, 2003). The approaches in those proposals are based on the matching between IOPEs (input, output, precondition and effects). As available datasets providing such descriptions are scarce today, at present we restrict our matching methods to service inputs and outputs. Of course, as both semantic web services technology and our research evolve, we could extend and improve our classification techniques by exploiting these new semantic features. We expect this to be an important direction of progress for our work, since the richer the description of services, the more advanced automation techniques can be devised.
Meanwhile, the goal of this paper is to show it is possible to develop automated assistance capabilities for service generation and classification, using only syntactic descriptions (WSDL), service taxonomies, and data type ontologies (in OWL), which are already in use in the current state of semantic web and web service technology development and adoption, or