A typical software engineer spends a signi cant amount of time and e ort reading technical manuals to nd answers to questions especially those related to features, versions, compatibilities and dependencies of software and hardware components, languages, standards, modules, libraries and products. It is currently not possible to provide a semantic solution to their problem primarily due to the non-availability of comprehensive semantic datasets in the domains of information technology. In this work, we have extracted, integrated and curated a linked open dataset (LOD) called LOaD-IT exclusively on this domain from a variety of sources including other LODs such as Freebase and DBPedia, technical documentation such as JavaDocs and others. Further, we have built a technical helpdesk system using a semantic query engine that derives answers from LOaD-IT. Our system demonstrates how productivity of the software engineer can be improved by eliminating the need to read through lengthy technical manuals. We expect LOaD-IT to become more comprehensive in the future and to nd other related practical applications.
Software developers, test engineers, researchers and students face technical queries in the course of their daily work. The solution to their problems, more often than not, is to go through some technical documentation, search for information, look for a similar question on on-line forums, or to ask somebody who is considered an expert in that domain. Unfortunately, information from these sources is rather unstructured, distributed and di cult to obtain. The process of going through the technical documentation is tedious and time consuming. What the person ideally wants is a quick and precise answer to the question perhaps augmented by a suitable visualization to aid comprehension. The objective of our work is to enhance workplace productivity of information technology workers by reducing the e ort and time spent in searching for technical information.
There is currently no readily available LOD that covers the factual
knowledge sought by a typical information technology (IT) worker. Thus, our solution
begins by semanticizing the relevant documentation by creating a linked open
data store, which we call LOaD-IT (available as an RDF dump in N-Quads
format at http://datahub.io/dataset/load-it). The potential of this highly
specialized linked data is tapped by building Kappa, a technical assistant application
prototype (Kappa is hosted at http://kanoe.org/lod/loadit-kappa.html). Kappa
takes keyword queries as input and provides speci c factual answer(s) instead
of large number of HTML pages that may or may not contain the correct and
complete answer. It also saves the user from scrolling through long manuals and
documents to manually extract the answer. In order to provide results in the
form of facts, the application rst performs query interpretation as described in
[
Existing semantic datasets [
Another concurrent objective is to generate RDF quads from technical manuals, user guides and other documentation that are widely available but demand considerable amount of time and e ort to sift through. Semanticization of this kind of documentation will contribute to raising the quality of answers (i.e., recall and precision) that the dataset is capable of providing at a ne grain of detail. Extraction of RDF facts from free owing paragraphs of explanatory texts amounts to solving core problems of natural language processing and semantic information retrieval. Therefore, we take the practical approach of selecting `low-hanging fruit? by focusing on relatively well-structured technical documentation. Java Platform Standard Edition 7 Documentation API was selected for this purpose because of its de nite structure in HTML tables and consistent layout, thereby facilitating extraction of clean and accurate facts. The widespread use of Java in industry and academia is another factor why this documentation API is a good choice for our purposes.
The methodology adopted for constructing the LOD has two parts: 2.1 Filtering Existing Datasets A set of carefully chosen seed words from the domain of IT and software engineering is used to select matching resources from Freebase and DBPedia. For each of these resources, all triples matching the resource in their subject elds are selected. All other triples are ltered out as irrelevant to our domain of interest. These operations were carried out using Unix Shell scripts. The numbers of triples obtained by this method from Freebase and DBPedia are shown in Table 1 below. 2.2 Extracting triples from technical documentation 1. A crawler is built to cover the entire documentation collection. 2. A screen scraping package is designed and built to parse the pages and extract facts after a thorough study of the layout of each type of page. 3. Rules are written for logical mapping into quads: (a) Check if any reuse of predicates is possible from existing vocabularies like DC, FOAF, SKOS, and SIOC. Reuse wherever possible. (b) If compatible predicates are not found, create new predicates that can appropriately describe the relationships. For instance, to link a Java method to its return type or modi er, a new predicate is required. (c) Create and publish the required explanatory pages for predicates in RDF and HTML formats. URLs are used to generate quads: (Subject, Predicate, Object, Context) according to the conventions of Linked Open Data. 4. A suitable software library is used to load the quads into a database or data store.
The documentation of Java 7 SE is obtained from the o cial web site and parsed
using a Python Library called BeautifulSoup4 [
Two example scenarios and results are presented below to illustrate the usage of Kappa and LOaD-IT.
Scenario 1 The user wants to know what to write in the import statement in a Java program if he has a need to use a method from the Driver Class in Java. In fact, the user may not even know whether Driver is the name of a Class or a Package.
Keywords entered: Package Driver Facts retrieved: java.sql type package java.sql member Driver Driver type class
A graphical rendering of these three triples in shown in Fig. 1(b) Scenario 2 The user wants to know the details of the Integer Class in Java. Keywords entered: Integer Facts retrieved:
Integer comment Integer class wraps a value of the primitive type int in an object Integer member highestOneBit,hashCode,valueOf,getInteger..
Integer type class (a) Schematic diagram of the architecture of (b) Visualization of
Our primary aim was to create a linked open dataset for the IT domain. After ltering existing datasets, we created a crawler and used screen scraping to extract RDF quads from semi-structured technical documentation. We integrated and normalized the dataset to ensure its compatibility with other LODs and our application Kappa. A reasonably large technical dataset was obtained and its potential was illustrated by a simple yet meaningful and useful application, a fact- nding engine that serves as a technical helpdesk for practicing software engineers.
The dataset can be further enlarged and enriched to include RDF quads extracted from other technical documentation and on-line discussion forums. Further, it can be re ned and grown over time by adopting a crowd-sourcing approach. Additionally, a vocabulary can be created speci cally for predicates that are useful in a software engineering context and can be standardized to bring in global acceptance. The facts that are rendered can be enriched by natural language processing techniques to bring them closer to proper English.
We expect LOaD-IT to nd several other useful applications in the near future. An immediate possibility is to use it as a controlled vocabulary plus ontology for corporate knowledge management. It can also be used to further automate the generation and management of program comments and documentation in software engineering by integrating it with IDE for various programming languages and platforms. None of these is readily possible without LOaD-IT wherein a federated retrieval engine would have to query various general-purpose LODs as well as document repositories and the Web. LOaD-IT together with Kappa promises to deliver the levels of precision and recall to make them useful in real-world software engineering situations.
A demonstration of our application Kappa is available at http://kanoe. org/lod/loadit-kappa.html.