The paper reports on an experience of designing and implementing an enterprise search application. One of the motivations for the work is the lack of previous studies that deal with the problem of evaluating an enterprise search application. The reason for this lack depends on the building a signi cant test collection, because each company may be interested in dealing with di erent types of topic and data format possibly with atypical content formats.
Enterprise search is the term commonly used to de ne information retrieval in a business setting, o ering users a way to search for informative content generated within their companies. By comparing it with web search and desktop search, we can identify some distinctive features: the content is stored in di erent information systems (e.g. data sources like customer relationship management software (CRM), enterprise resource planning software (ERP), intranets), di erent le formats are used (e.g. pdf, docx, xls), including both structured (e.g. database) and unstructured content (e.g. scanned documents), and di erent access levels give users di erent rights.
The purpose of enterprise search is to enable users to e ectively nd the
information they need to perform their tasks, while at the same time requiring
a minimal e ort for the users and low costs to be sustained by the company in
terms of ine ciencies [
One of the motivations for the work reported here is the lack of previous works that deal with the problem of evaluating an enterprise search application. The reason for this lack may depend on the di culties in building a signi cant test collection: each company can be interested in dealing with very di erent types of topics and data format possibly with atypical content formats (e.g. in the case of the study reported in this paper CRM events were also dealt with), and the company may only be able to allocate limited resources to this type of activity.
In the present work we present a real case study of the design, development and evaluation of an application of enterprise search within a company, and draw attention to the issues that had to be overcome to complete the task. The paper also details the technological stack employed to build and test the enterprise search application.
In Figure 1 we sketch the di erent steps to design and implement the enterprise application together with its evaluation. The gure can also be used as a sort of outline of the paper.
The bibliographic research we conducted showed a lack of work on assessment and evaluation methodologies of enterprise search. This may be due to the fact that the importance of conducting an assessment to contribute to the improvement of the quality o ered by information retrieval services has not yet been perceived, especially in the industrial sector. We must also consider that most of the research in this eld is being carried out within companies, so we cannot exclude the possibility that many companies simply prefer not to disclose results and information considered to be strategic.
The results we present are based on [
The objective of this work was to produce a prototype able to perform the effective retrieval of documents that constitute the di erent collections present within the SIAV internal installation of the document management platform Archi ow. The system is designed to integrate the collections of documents with other content from sources other than Archi ow itself, in order to provide the end user with a spectrum of documents related to a speci c subject of interest. An example which clari es this aim is a search that requires the documents related to a particular customer: in addition to providing these documents, while maintaining e cacy as a metric, the system will provide additional related contents, like the latest tickets opened by the customer (available in the helpdesk management system), recently opened business opportunities with the customer derived from CRM or news on the web concerning the customer. This allows us to anticipate possible information needs, thus obtaining an advantage in terms of e ciency, in accordance with a business intelligence perspective.
The reference users are the professionals within SIAV. The company made available a set of data and documents from di erent information sources, including those managed by the di erent systems available within the company; in this way we were able to use these information sources to build a collection to evaluate the system.
The platform adopted is Apache Solr [
3.1
The di erent data sources that have been considered are: { Documents: The documents are a subset of the document base of SIAV, that is, a part of the documents that are managed by the local installation of Archi ow at SIAV headquarters. The documents are of di erent types (such as invoices, emails, project sheets, etc.) and of di erent formats (including DOC, DOCX, TXT, PDF, TIF, MSG, EML, RTF, XLS, and XML). The metadata of each document are stored in an associated card from which the metadata can be exported and extracted in CSV format. For this category of data sources each document, together with the metadata related to it, is considered a basic unit of information. { Events: SIAV is equipped with an issue tracking system that collects the tickets regarding the technical problems reported by their customers. The company manages the ticket elements through a CRM, which allows the coordination of the activities of the sta working at the helpdesk and the technical department. In this way all the actions performed by the sta involved are recorded in the system. Those actions are named events. It is possible to export the events from the CRM, and each event is enriched with a set of descriptive attributes. A total of 70,475 events were considered and studied; those events were collected over a period of about ve years. { Web Pages: During the work only a focused portion of the Web was examined; since the study on this portion of the Web is preliminary, the related results are not reported here.
It is important to note that objects within di erent sources may actually contain related information; e.g. a document can be related to a particular project and also a ticket item can be linked to it. One of the requirements of the system was to allow the user to exploit this information. For all considered data sources, the reference text language is Italian. 3.2
Since the documents were in di erent formats, an ingestion utility had to be developed and used to generate a new version of each document, in a format usable by Solr, for the necessary preprocessing and indexing procedures.
The rst part of the ingestion work was the extraction of the textual contents from the documents: for each document le the text was extracted and saved in a simple text le, so as to allow the subsequent loading and veri cation through Solr of the quality of the extraction procedures. This processing step has been supported by di erent technical tools.
The documents of the used collection were: { communications (11,974) { bids and tenders (59,763) { project reports (4,962) for a total of 76,699 indexed documents.
The permanent memory space necessary for the storage of the preprocessed documents is 2.13 GB while the permanent memory used to store the original documents was more than 60 GB. For each original document, the nal document with a valid format to be imported into Solr was created by adding the metadata of the original document to its extracted textual content. To manage and index all the documents (textual and metadata documents) through the search engine the library package SolrNet5 has been used.
Information on document number and permanent memory size needed to store the indexed documents are reported in Table 1 and 2.
Document Sources Total Archi ow Documents 76,699
Total Events 70,475
Total Document Number 147,174
Evaluation in IR is a well-established practice that has been studied for many
years [
The test collection to be used for the evaluation has to be built before starting the evaluation experiments and it is composed of the triple: C = D; T; RJ .
In this case the possibility of using an existing test collection was rejected because there is no test collection that is suitable to the type of tasks of interest. Therefore the test collection needed to be built. To build the test collection di erent alternatives were studied and the following operational strategy, exploiting a pooling methodology, was eventually decided. Of the three categories of documents { documents from Archi ow, events and web pages { it was decided to use documents and events that are peculiar to the company; web pages were not considered strategic for the design purpose. With the support of a thorough analysis of the tasks of interest conducted with the company management, ve topics of interest for each document category were de ned, and each topic was translated by a domain expert into a query. The pooling was done using Solr,
Terrier6 and two Desktop Search named Copernic7 and Windows Search8. For each of these search engines, the following actions were conducted: indexing of the document base; execution of queries to produce the run; calculation of the number of documents to be judged on the basis of di erent depths of cut.
The achieved results [
The de nition of the document base to be used and the subsequent evaluation of the results raised some issues: the formulation of the topics was really challenging; the process of de ning the queries required the systematic cooperation between domain user experts and IR experts; the calculation and veri cation of all the calculated metrics was supported by MATTERS9, a software toolkit written in Matlab for information retrieval evaluation; many activities requiring great time and e ort were needed to obtain the pool. 5
Before this project SIAV used to rely on third party products to provide its customers with advanced search features, because the base search module in SIAV Archi ow provided only metadata based search.
The purpose of this work for SIAV was twofold: (1) develop a more exible retrieval system and (2) acquire the necessary knowledge to put the enterprise search application into production, maintain it and in the future be able to enrich it with new desired features, in order to cope with new search needs of its customers. These goals were positively met, but there is still the need of an evaluation platform to continue improving the developed system. While enhancing the search system, the developer will in fact need a tool to check the e ectiveness of new versions. The methodology presented in this work can be a valuable starting point for the development of an automated procedure.
We thank the anonymous referees for their fruitful comments and careful reading, which improved this article.