One of the reasons that the existing IR systems do not Collaborative information retrieval (CIR) and search-driven adequately support collaboration is that there are not good software development (SDD) are both new emerging research models and methods that describe users' behavior during fields; the first one was born in response to the problem of collaborative tasks. To address this issue, the community satisfying shared information needs of groups of users that has adopted CIR as an emerging research field in charge to collaborate explicitly, and the second to explore source-code establish techniques to satisfy the shared information needs retrieval concept as an essential activity during software de- of group members, starting from the extension of the IR velopment process. Taking advantages of the recent con- process with the knowledge about the queries, the context, tributions in CIR and SDD, in this paper we introduce a and the explicit collaboration habits among group members. plug-in that can be added to the NetBeans IDE in order CIR community identifies four fundamental features in this to enable remote teams of developers to use collaborative multidisciplinary field that can enhance the value of collasource-code retrieval tools. We also include in this work borative search tools: user intent transition, awareness, diexperimental results to confirm that CIR&SDD techniques vision of labor, and sharing of knowledge [2]. give out better search results than individual strategies. In addition, SDD is a new research area motivated by the observation that software developers spend most of their Categories and Subject Descriptors time searching pertinent information that they need in order to solve their tasks at hand. We identified that SDD context H.5.3 [Information Interfaces and presentation (e.g., was a very interesting field where collaborative IR features HCI)]: Group and Organization Interfaces; H.3.3 [Information could be greatly exploited. For this reason we use the phrase Storage and Retrieval]: Search Process. collaborative SDD to refer to the application of di↵ erent collaborative IR techniques in the SDD process [3]. General Terms It's known than some IDE incorporate tools with support Design, Human Factors. for developer's collaboration practices, but without making emphasis in source-code retrieval. In this sense, the objec-
In the last few years, Information Retrieval (IR) Systems
have become critical tools for software developers. Today
we can use vertical IR systems focused in integrated
development environment (IDE) extensions for source-code
retrieval as such Strathcona [
Copyright c 2011 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by the editors of EuroHCIR2011. 2.
There is a small body of work that investigates methods
to join collaborative information retrieval and search-driven
software development. On the one hand, some researchers
have identified di↵ erent search scenarios where it is
necessary to extend IR systems with collaborative capabilities. For
example, in the Web context, SearchTogether [
CIR systems can be applied in several domains, such as
travel planning, organizing social events, working on a
homework assignment or medical environments, among many
others. We identified software development as another
possible application field where much evidence of collaboration
among programmers on a development task can be found.
For example, concurrent edition of models and processes
require synchronous collaboration between architects and
developers who can not be physically present at a common
location [
However, current SDD systems do not have support for explicit collaboration among developers with shared technical information needs, which frequently look for additional documentation on the API (Application Programming Interface), read posts for people having the same problem, search the company’s site for help with the API, or looking for source code examples where other people successfully used the API. Fortunately, in the last few years, some researchers have realized that collaboration is an important feature, which should be analyzed in detail in order to be integrated with operational IR systems, upgrading them to CIR systems.
As an approach to these situations, we propose in this
work the COSME plug-in [
In order to support such CIR techniques, COSME provides some collaborative services in the context of SDD: • The embedded chat tool enables direct communication among di↵ erent developers. • Relevant search results can be shared with the explicit recommender mechanisms. • Another important feature is the automatic division of labor. By implementing an e↵ ective division of labor policy the search task can be split across team developers, thereby avoiding considerable duplication of e↵ ort. • Through awareness mechanisms all developers are always informed about the team activities to save e↵ ort. Awareness is a valuable learning mechanism that help the less experienced developers to view the syntax used by their teammates, being an inspiration to reformulate their queries. • All search results can be annotated, either for personal use, like a summary, or in the team context, for discussion threads and ratings. 3.
To improve software developers with shared technical
information needs we implemented the COSME front-end as
a NetBeans IDE plug-in. The principal technologies that
we used to implement it include the CIRLab framework [
In this section we are going to show how collaborative features applied to SDD improves the traditional operation without them. Then if we consider the null hypothesis (H0) that AT SDD ACSDD, our alternative hypothesis (H1) is that the collaborative work should help to improve the retrieval performance in a SDD task: AT SDD < ACSDD, where TSDD stands for Traditional SDD and CSDD for Collaborative SDD. To evaluate our proposal we compare 10 group interactions in two di↵ erent kinds of search scenarios (SS) on SDD, SS2k+1 and SS2(k+1); k 2 0, . . . , 9. SS2k+1 represents a team of developers that use a conventional IR system, this means that developers do not have access to techniques of division of labor, sharing of knowledge, or awareness (traditional SDD – TSDD), while S2(k+1) represents a team of developers that uses a CIR system. Then, 5 teams worked in a TSDD context (those with odd subindexes) and the other 5 with CSDD (even subindexes). In both search scenarios, we used COSME with the appropriate configurations for both settings. The search scenario was a common task proposed to a group of developers without Java background: select the most relevant classes to manage GUI (Graphical User Interface) components using di↵ erent Java API with a total of 2420 files. Specifically, Jidesoft (634), OpenSwing (434)), SwingX (732)) and Swing (620). We have focussed on these API because they are directly related to the context of the experiment although they are not complete: we have only considered their most relevant API packages for the experiment.
For evaluation purposes, we created our own test collection: a group of 10 experts proposed a set of 100 topics strongly related to the objective of the experimentation, then their corresponding queries were submitted to each of the following search engines: Lucene, Minion, Indri and Terrier. A document pool was obtained by ranking fusion and later the experts, grouped in pairs, determined the relevant documents for each topic.
In collaborative SDD, it is very important to analyze the interaction among group members, therefore, unlike the evaluation of a traditional SDD system, we can not fix the queries. Then each participating group could freely formulate their queries to the search engine. In order to compare team results, the search engine identified the most similar queries formulated by the members of the teams with respect to those formulated by experts. If the system found enough similarity and if they occur in all the groups, then these queries are considered that deals with the same topic and selected for group comparison purposes. The similarity measure between queries is calculated by Equation 1. A user query (qu) and an expert query (qe) are considered to be the same if they are within a given similarity threshold. A new query qu0 is obtained applying the Porter stemmer algorithm to qu’s terms, and analogously, we would obtain qe0.
sim(qu, qe) = | qu0 T qe0 | = | qu0 S qe0 | (1)
In Equation 1, is a value between 0 and 1. For this experiment we assumed that there exists an expert’s relevance judgement to qu only if 9 N2N+1 , where N =| qu0 S qe0 |, selecting the relevance judgements that correspond to max for each qe.
In order to measure the e↵ ectiveness of the described SST SDD
and SSCSDD scenarios, we considered as evaluation
measures the metrics proposed by Pickens et al. in [
According to the documents that each team selected for each common topic, F1s measure was computed. In order to accomplish the statistical analysis of the results, we use the non parametric test of Wilcoxon (all against all). The Monte Carlo method was used and adjusted with the 99% trust intervals and 10000 signs. It was considered the existences of significance (Sig.) as appear in Table 1.
We could notice significative di↵ erences between TSDD and CSDD groups, considered two by two. As F1s values for CSDD groups are better than those computed from TSDD groups for those cases, then we could conclude that when teams works supported by collaborative tools, they obtain better results. From Table 1, we could realize that apart from SS5, each SST SDD has got at least one SSCSDD with significant di↵ erence values of F1s. With this results we accept H1, because AT SDD < ACSDD. 0, 054 0, 290 0,090 0, 190 0, 029† SS10 0, 290 †: significant di↵ erence (0, 01 Sig < 0, 05) ‡: highly significant di↵ erence (Sig < 0, 01)
Collaboration in SDD is just being recognized as an important research area. While in some cases collaborative SDD can be handled by conventional search engines, we need to understand how the collaborative nature of sourcecode retrieval a↵ ects the requirements on search algorithms. Research in this direction needs to adopt the theories and methodologies of SDD and CIR, and supplement them with new approach constructs as appropriate. In this work we present COSME as a collaborative SDD tool that helps team developers to find better sources than searching with traditional SDD strategies, as well as an experimental approach that confirms our hypotheses.
Our ongoing work focuses on the COSME back-end which poses fundamental research challenges as well as provides new opportunities to let group members collaborate in new ways:
(i) Profile Analysis. We aim to analyze the user-generated data using various techniques from the study of di↵ erent collaborative virtual environments and recommender systems. With the results, our goal is to provide better personalized search results, support the users while searching and recommend users to relevant trustworthy collaborators.
(ii) P2P/hybrid-network Retrieval. Due to scalability and privacy issues we favor a distributed environment by means of a P2P (peer-to-peer) retrieval feature based on hybrid architecture to store the user-generated data and collections (CASPER – CollAborative Search in PEer-to-peer netwoRks). The main challenges in this respect are to ensure a reliable and e cient data analysis.
This work has been partially supported by the Spanish research programme Consolider Ingenio 2010: MIPRCV (CSD200700018), the Spanish MICIN project TIN2008-06566-C04-01 and the Andalusian Consejer´ıa de Innovacio´n, Ciencia y Empresa project TIC-04526. We also would like to thank Carmen Torres for support and discussions and for all of our experiment participants.