This paper focuses on developing lightweight tools for knowledge sharing and collaboration by communities of practice operating in the field of information retrieval. The paper contributes a motivating scenario, a characterization of these communities, a list of requirements for collaboration, and then a system design proposed as a proof-of-concept implementation that is being evaluated.
This paper focuses on the problem of supporting knowledge sharing and collaboration in communities of practice that operate in the field of information retrieval (IR). These communities include developers, researchers, and stakeholders who periodically collect and use scientific data produced by the experimental evaluation of IR systems. Specifically, the communities considered include those involved in three specific IR domains: Patent, Cultural Heritage, and Radiology.
The research context of the work reported in this paper is the PROMISE NoE. This project aims at advancing the current tools for IR communities to perform experimental evaluation of complex multimedia and multilingual information systems. The ultimate goal of the project is to develop a unified infrastructure for the community to efficiently collect and reuse data, knowledge, tools, methodologies, and communities of end users. In this context, providing adequate support for collaboration is crucial. Herefrom the specific goal of the work reported in this paper: designing and evaluating lightweight support for knowledge sharing and collaboration. Currently, the following problems result from lack of suitable collaboration tools: 1) Greater effort is required by individual members, who contribute as volunteers, for sharing knowledge and collaborating. In the long term, this discourages broader participation. 2) Poor reuse of content and process information across the multiple instantiations of similar experimental evaluation processes. Over time, this leads to inefficient processes: e.g., content is alPresented at EuroHCIR2012. Copyright c 2012 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. ways recreated from scratch, successful processes (best practices) cannot be reused, novices cannot be easily trained based on shared experience. 3) The overall community cannot easily reflect on (and thus reengineer) its own workflow around specific TRECs. 2.
The starting point of our analysis is a typical IR evaluation campaign (lab). In a typical scenario, Adam (lab organizer) is preparing an IR experiment and evaluation task and spends time and resources for coordinating, communicating and assembling people and resources in order to proceed with the overall evaluation task, e.g. recruiting people that will be responsible for different evaluation task(s). Communication and sharing of information may be different within different across sub-tasks. Furthermore, they may be different between labs without any awareness among actors of the similarities/differences in the evaluation task processes. Thus, it is important to identify the stages in the evaluation task process as well as how collaborative and information sharing activities are manifested. 3.
The CLEF experimental platform involves a series of CLEF Labs
and one or more tracks within each Lab. Each Lab as well as each
track involves a certain set of tasks that could be considered as a
task process or workflow. In order to define and describe these
tasks we have investigated the lab and track organizers of a CLEF
experiment, how they performed their work and what steps they
went through during their work. Furthermore, we have extracted
requirements for collaborative information handling and
information sharing activities specifically [
Some of the most common tasks that we observed as part of a typical evaluation campaign include the followings: submission, preparation, track definition, topic creation, data set, relevance assessment, summarizing, and finalizing. 3.1
Within an evaluation campaign many people are involved in different tasks, such as organizing, creating topics, managing collections, handling participants and submission, choosing measures, and running the final evaluations.
The set of actors involved in PROMISE activities is not homogeneous and depends on the domain which is taken into account. Looking at three domains (patent, medical, and cultural heritage), we defined the following actors: - organizers: people who are in charge of preparing a campaign; it is possible to distinguish domain organizers and track organizers; - participants: people who run their algorithm(s) according to the actual tasks; - relevance assessors: people who make the relevance assessment; - topic creators: people who define topics for a given task; - site administrators: (e.g. system administrators); - other researchers - annotators: people who annotate resources to highlight some hidden information.
Each of these actors is described along with a set of activities or tasks. Moreover, a user can have more than one role. 3.2
The collaborative work may be performed in a non-structured manner using basic tools for collaboration in an ad hoc fashion. The following are the most commonly used tools for collaboration: - E-mail: the most common way to organize the work and spread information; - Face-to-face meetings: useful to discuss more effectively about problems and solve them; it is complex if people don’t work in the same building; - Video conference tools (e.g. Skype): used instead of face-to-face communication; - Shared workspaces (e.g. shared document editor, desktop sharing): useful to share documents. However an issue may arise where many people work on the same document.
As mentioned in the introduction, a more suitable collaborative tool is needed to help researchers to accomplish their tasks. To realize it we have to overcome some limitations.
The first one is the impossibility to define a common detailed workflow due to the presence of different domains, each of them with specific needs. This makes it difficult to realize a collaborative environment completely specified. Despite this limit, it is possible to individuate some common needs such as: communication with other actors, access to data of previous campaign, sharing of task flow of actual evaluation campaign, and sharing workspace with actors involved in the same tasks.
Another aspect that characterizes the work of people involved in a lab is that there is an alternation between individual and collaborative work, which is in contrast to a too rigid environment. The basic idea of our system is to improve the actual tools used in IR community without defining the collaborative environment in a too rigid way. Our purposes in this paper are: 1) Collecting the tools actually used (e.g. Skype, Googledocs, etc.) in a structured environment. 2) Making available to users other collaborative tools (polls, news). 3) T@GZ: a social software system for organizational information sharing. 4.1
In order to get information on the actual tasks users performed, we made requirements elicitation through a number of questionnaires to track organizers within the three predefined domains of Patent, Cultural Heritage, and Radiology in the CLEF platform. Working through these questions we identified: a) different roles of actors involved in the CLEF experiments, b) requirements for collaborative information handling activities and information sharing and c) links between roles and collaborative events. Furthermore we describe each task stage regarding subtask involved (fig.1): - Submission task: preparing a lab proposition including sub tracks, acceptance of the lab or not. - Preparation task: preparation of a CLEF lab flyer including details of each lab, obtaining databases, preparing a copyright agreement, preparation of the web page. - Track definition task: definition of broad tasks, start of registration for the participants. - Topic creation task: preparation of detailed topics for each task, release of topics, checks of the copyright forms. - Data set: data access for all registered participants with signed copyright forms for their tracks. - Relevance judgment task: preparation of the judgement system; finding qualified judges; submission of all runs by participants; pooling of the runs to create documents to judge; judgement of the documents for relevance; evaluation of all submitted runs. - Summarizing task: release of ground truth; submission of participants’ papers with results and technical descriptions; analysis of the results and submission of overview paper. - Finalizing task: CLEF workshop and labs with discussion; feedbacks on CLEF; preparation of next year of CLEF; distribution of responsibilities. 4.2
Support the community in managing the process:
- Tasks and roles. Various roles are involved in communities work
on experimental evaluation of IR systems. Multiple tracks and tasks
are part of a process occurring in an evaluation experiment such as
CLEF. Within a track, some of the tasks are interdepend. Specific
roles perform specific tasks. One member can play multiple roles.
The set of the role and tasks changes across different IR domains.
- Assume both individual and collaborative works. Many tasks
are conducted individually and the individual work is interleaved
with collaborative work. Support both individual and collaborative
works and faster transitions.
- Only some of the steps in the work-flow are fully specified before
or at the outset of the process, others are defined during the process.
Needs related to existing work tools:
- As for other communities of knowledge workers [
Needs related to new collaborative functions that might support collaboration: - Groups. Group creation is tied to the task. - Polls. The polls component should be integrated with the other components (this is an example of function not available in email). - Collaborative workspace. It is desirable for the members of the community to have easy access to the shared resources and typical steps, which, ideally, should be made available all in one place. - Process visualization. We observed visible differences in the different instantiations of the work-flows elicited from different subcommunities. It allows people to become aware of differences and similarities in the way sub-communities go about performing the same process.
Recent research has pointed to the importance of investigating
and supporting collaboration in the field of Information Retrieval
(IR). For example, [
A key distinctive property of the IR communities is that their
workflows cannot be fully specified a priori. That is, if we
consider a continuum from highly specified to highly unspecified
processes, then we could classify the instances of workflows of the IR
communities as intermediate cases along this continuum. [
Articulating further the design requirements for supporting
semistructured workflows, [
To fulfill the requirements described on Section 4, we devised
the architecture shown on Figure 3. It refers to the classical CLEF
experiments organization, that is arranged in terms of different
domains (e.g., Medical, Patents, etc.). For each domain one or more
tracks are available. As described on Section 3, the organization
of a track is a complex collaborative process and encompasses
several tasks that exhibit some precedence relationships. The task flow
of a track is formalized using an Extended Light livE Gantt
(EleGantt) and is shown within the CUI to the user, acting as the main
entry point for collaborative activities (Figure 4). A suitable
administrative interface allows for adapting the task flow of a track
to procedural changes. According to [
T@GZ is a social software system for organizational information sharing. In T@GZ the user can share by simply sending an email message with the content to be shared, and addressing the message to one or more topic specific keywords. For example, one might use the address, bizdev@share.X.com, for referring to information related to "business development" topic (see Figure 4, top left). Thus, the content of that email is ’tagged’ by the keyword ’bizdev’. Any mail may have multiple tags attached in this manner, in the ’To’ or ’CC’ fields, using any client. While enabling easy publishing and re-finding of this information, the system does not induce people to send additional emails other than those that they are already sharing. Focusing now on the implementation of T@GZ in the whole system, using a set of predefined tags (i.e., the tags associated to the elegantt’ s tasks), T@GZ provides a means for indexing the emails that are exchanged among the organizers of the tracks, including links to smart attachments. The work-flow engine is aware of the elegantts and using time information and inspecting the KB sends through email different kinds of notifications (e.g., a deadline is approaching, it is time to move to the next step, etc.) to people involved in the tracks organization.
The Collaborative User Interface (CUI) is the Web based access point to all the collaborative activities (see figure 4). It is basically split in two subcomponents. The first one allows for managing personal user collaborative information (left part of the picture), e.g., messages, polls, etc. The second one refers to the whole process and allows for both exploring it using EleGantt, discovering people, roles, and tags and browsing the whole set of tagged emails. Moreover, the CUI contains a set of tabs that allows for accessing the collaborative tools that have been specified in the EleGantt. In order to provide the user with an unique access point to the process resources, the CUI sends tagged emails to T@GZ, containing a link to the collaborative resources (e.g., link to dropbox folder or to Google Docs document) 7.
As result of our investigation we identified some general challenges or open issues for this domain: 1) find a good balance between the need for flexibility to fit various, partially-defined processes and the need for enough specification in order to allow automation 2) identify the set of predefined tags (some common and some domain specific) 3) semi-automate the tagging process (e.g., an intelligent assistant).
The work in this paper has been supported by the PROMISE NoE (contract n. 258191) project as a part of the 7th Framework Program of the European commission (FP7/2007-2013). 8.