This paper introduces the NTCIR Workshops, a series of evaluation workshops that are designed to enhance research in information access technologies, such as information retrieval, text summarization, question answering, text mining, etc., by providing infrastructure of large-scale evaluation. A brief history, test collections, and recent progress after the previous CLEF Workshop are described with highlighting the difference from CLEF in this paper. To conclude, some thoughts on future directions are suggested. The NTCIR Workshops [1]1 are a series of evaluation workshops designed to enhance research in information access (IA) technologies including information retrieval (IR), cross-lingual information retrieval (CLIR), automatic text summarization,
question answering, text mining, etc.
The aims of the NTCIR project are: experiments, informal atmosphere, and to encourage research in information access technologies by providing large-scale test collections reusable for to provide a forum for research groups interested in cross-system comparisons and exchanging research ideas in an to investigate methodologies and metrics for evaluation of information access technologies and methods for constructing large-scale reusable test collections.
That is to say, the main goal of the NTCIR project is to provide infrastructure of large-scale evaluation. The importance of large-scale evaluation infrastructure in IA research has been widely recognized. Fundamental text processing procedures for IA such as stemming and indexing include language-dependent procedures. In particular, processing texts written in Japanese or other East Asian languages such as Chinese is quite different from processing English, French or other European languages, because there are no explicit boundaries (i.e., no spaces) between words in a sentence. The NTCIR project therefore started in late 1997 with emphasis on, but not limited to, Japanese or other East Asian languages, and its series of workshops has attracted international participation.
The term information access (IA) includes a whole process to make information in the documents usable for the user who has problems or information needs. A traditional IR system returns a ranked list of retrieved documents that are likely to contain information relevant to the users needs. This is one of the most fundamental and core processes of IA. It is however not the end of the story for the users. After obtaining a ranked list of retrieved documents, the user skims the documents, performs relevance judgments, locates the relevant information, reads, analyses, compares the contents with other documents, integrates, summarizes and performs information-based work such as decision making, problem solving, writing, etc., based on the information obtained from the retrieved documents. We have looked at IA technologies to help users utilize the information in large-scale document collections. IR, summarization, question answering, etc are a family, in which the same target is aimed while each of the 2 technologies has been investigated by different communities with least interaction .
As shown in Figure 1, we have looked at both traditional laboratory-type IR system testing and the evaluation of challenging technologies. For the laboratory-type testing, we placed emphasis on IR and CLIR with Japanese or other Asian languages and testing on various document genres. For the challenging issues, the targets are the shift from document retrieval to technologies that utilize information in documents, and investigation of methodologies and metrics for more realistic and reliable evaluation. For the latter, we have paid attention to users information seeking task in the experiment design. These two directions have been supported by a forum of researchers and discussion among them.
From the beginning, CLIR has been one of the central interests of the NTCIR, because CLIR between English and own-languages is critical for international information transfer in Asian countries, and it was challenging to perform CLIR between languages with completely different structures and origins such as English and Chinese or English and Japanese.
Lab-typed IR Test New Challenges Asian Languages/cross-language Intersection of IR + NLP Variety of Genre Parallel/comparable Corpus To make information in the documents more usable for users!
Realistic eval/user task Forum of Researchers Idea Exchange Discussion/Investigation on Evaluation methods/metrics
In the NTCIR, a workshop is held once per about one and half years. Since we respect the interaction between participants, we call a whole the process from document release to the final meeting as workshop. Each workshop selects several research areas called Task, or "Challenge" for more challenging task. Each task has been organized by the researchers of the domain and a task may consist of more than one subtasks. Figure 2 shows the evolution of the tasks in the NTCIR Workshops and Table 1 is a list of subtasks and test collections used in the tasks [5-7].
th
As shown in Table 1, the 4 NTCIR Workshop hosts 5 tasks, CLIR, Patent Retrieval Task (PATENT), Question Answering Challenge (QAC), Text Summarization Challenge (TSC), and WEB Task (WEB) and their sub-tasks. 2 In addition to the above, how to define the question of the user before the retrieval is also included in the scope of the IA although it has not been explicitly investigated in NTCIR.
Japanese IR t Cross-lingual IR a s Patent Retrieval k s Web Retrieval
Term Extraction/ Role Analysis Question Answering Text Summarization
Tasks (Research Areas) of NTCIR Workshops
*The4ntuhmbweorrokfsthhoep
Registration 3rd workshop 2st workshop 1st workshop As shown in Figures 3 and 4, the number of participants has been gradually increasing. Different tasks attracted different research groups although many are overlapped, or changed the participating tasks over workshops. Many international participants were enrolled to CLIR. Patent Retrieval task attracted many participants from company research laboratories and veteran NTCIR participants. WEB task has participants from various research communities like machine learning, DBMS, and so on. The number of collaborating teams across different organizations is increasing in recent NTCIRs. 120 s 100 p u o rgG 80 n i t icpa 60 i t r Pa 40 f o # 20 0
chinese
Chinese .Korean J!E EJ!!CE,E!J、 x!CJEK 1st(1998-9) 2nd(2000-1) 3rd (2001-2) 4th(2003-4) QA Summarization Term Extraction Web Retrieval Patent Retrieval NonJapanese IR CLIR Japanese IR Fig 3 Number of Participating Groups Fig 4 Participating Groups per Task
The test collections constructed for the NTCIR Workshops are listed in Table 2. In the NTCIR project the term test collection is used for any kind of data set usable for system testing and experiments although it often means IR test collections used in search experiments. One of our interests is to prepare realistic evaluation infrastructure, and those efforts include scaling up the document collection, document genres, languages, topic structure and relevance judgments.
Documents were collected from various domains or genres. Format of the documents are basically the same as TREC or CLEF and are plain text with SGML-like tags. Each of the specialized document genre collections contained characteristic fields for the genre Web collection contains html tags, hyperlinks, URL of the document, etc., and patent collection has tags indicating document structure of patent, and both patent and scientific document collections have parallel corpora of English and Japanese abstracts. The task (experiment) design and relevance judgment criteria were set according to the nature of the document collection and user community who use the type of documents in their everyday tasks.
A sample topic record is shown in Figure. 5. Topics are defined as statements of users requests rather than queries, which are the strings actually submitted to the system, because we wish to allow both manual and automatic query construction from the topics. Emphasis has been shifted towards the topic structure capable more realistic experiments as well as to see the effect of background information of the topic. The characteristics are summarized as followings;
Topic Structure: Topic Structure has slightly changed in each NTCIR. A topic basically consists of a <TITLE>, a description <DESC>, and a detailed narrative <NARR> of the search request as similar to those used in CLEF and TREC. It may contain additional fields as shown in Table 3. Most of NTCIR collections contain a list of concepts <CONC>, but they are not heavily used by participants.
NTCIR Test Collections; IR and QA collection
task NTCIR-4 QA news J 1998-1999 J*
QA NTWCEIRB-4 IR (htmWle/tbext) NW100G-01 mu*lt4iple cra2w0l0e1d in 11,038,720 100GB J* J:Japanese, E:English, C:Chinese (Ct:Traditional Chinese, Cs: Simplified Chinese), K:Korean; "+" indicates the document collection newly added for NTCIR-4 * English translation is available ** gakkai subfiles: 1997-1999, kaken subfiles: 1986-1997 *3: kkh : Publication of unexamined patent application, jsh: Japanese abstract, paj: English translation of jsh *4: almost Japanese or English (some in other languages) documents
summaries genre filename lang # of doc types analysts total# NTCIR Text Summarization collection
task NTCIR-2
single doc news SUMM NTCIR-2
single doc news
TAO NTCIR-3 single doc
news SUMM multi doc Sample Topic
written statement of users needs <TOPIC> <NUM>0010</NUM> purpose/background <TITLE CASE="b">Aurora, conditions, observation</TITLE> <DESC> I want to know the conditions that give rise to an aurora for
observation purposes </DESC> <NARR><BACK>I want to observe an aurora so I want to know the Rjcuerdleigtvmearenincate tcbhoeenhdpiniltadiocinte.s<a/nnBedAcCteimsKse>a<srRyoEfooLnrEliyt>sdAoouccrcuoumrrareeonnbtcsseethravanatdtpiotrhonevriedmceeoacrhddsad,niteiisotmnc.allist information such as the weather and temperature at the time of occurrence are relevant. </RELE></NARR> <CONC>Aurora, occurrence, conditions, observation,
mechanism</CONC> <RDOC>NW003201843, NW001129327, NW002699585</RDOC> <USER>1st year Masters student, female, 2.5 years search </TOPexICpe>rience</USER> user attribute given rel docs WEB3, changed to be a "query", a string put into a search engine by users and defined as a comma-separated term lists up to three terms.
Structured <NARR>: Originally a narrative <NARR> was defined and instructed to the topic authors that it may contain background knowledge, purpose of the search, detailed explanation of the topic, criteria for relevance judgment, term definitions, etc. Since NTCIR-3 WEB, such information categories in <NARR> explicitly marked by tags like <BACK>, <RELE>, etc. as
Mandatory runs: Any combination of topic fields is allowed to use in experiments for research purpose. In the Workshop, the Mandatory Runs are defined in each task, and every participant must submit at least one mandatory run using the specified topic field only. The purpose of this is to enhance the cross-system comparison based on the common condition and see the effectiveness of the additional information over it. Mandatory runs are originally <DESC> only, then gradually shift to <TITLE> only as well as <DESC> only.
Relevance judgments are done by pooling, and the format and methods are basically the same as other evaluation projects including CLEF and TREC. The differences shall be summarized as follows;
Pooling strategies are slightly different according to each of the task
・ Additional interactive recall-oriented searches are done to improve the exhaustivity (NTCIR-1,-2) [8]
・ Additional interactive recall-oriented search are done by professional patent intermediaries (PATENT) [
Relevance judgment files may be prepared to each of the target language document sub-collections in CLIR For 4, it helps assessors to judge consistently over a long list of pooled documents to be judged (typically 2000 - 3000 documents). Relevance judgments may change over assessors and over time. If relevant documents are appeared intensively in the first part of the list, it is easier for the non-professional assessors to set and confirm their criteria for relevance judgments, and then they can always refer those documents to re-confirm their own relevance judgment criteria when they go down to the lower ranked document. We understand they may be suffered by order effect of the ranked list of pooled documents in judgments, but we intentionally have used this strategy as practical and most effective one in our environment based on the comparative tests and interviews with assessors.
For 5, in multilingual CLIR, a topic can not always obtain sufficient number of relevant documents on every language document sub-collection, and this is the natural situation in multi-lingual CLIR. As a result, some topics can not be usable experiments on specific language documents. We can not find the way to manage this issue and only strategy we could take in NTCIR-4 CLIR is to increase the number of topics, so that larger number of topics can be used common across the document sub-collections and then improve the stableness of the evaluation.
Assessors are users of the document genre, judgments are done by the topic author except CLIR in NTCIR-3 and -4 since topics are created in cooperation of multiple countries, and then translated into each language and tested usability on each 3 Topic authors are instructed to sort the terms in <TITLE> in descending order of importance to express the search request resembling the way language document sub-collection. Judging other users topics is sometimes hard for users and take longer time.
First two NTCIRs used two assessors per topic then tested inter-assessors consistency and found that the inconsistency among multiple assessors on a topic does not affect the stableness of the evaluation when tested on sufficient number of topics. Based on this, single assessor per topic is used in and after NTCIR-3.
For the evaluation, trec-eval program [
For Question Answering, MRR is used for subtask-1, return 5 possible answers and no penalty for wrong answers, and F-measure for subtask-2, return one set of all the answers and penalty will be given for wrong answers, and subtask-3, series of question. For Text Summarization, content based and readability based intrinsic evaluation was done in NTCIR-3 for both single document and multi-document summarization, and proposed new evaluation methodology based on revision (edit distance) on system summaries by professional analysts who created the model summaries.
After our previous reports at CLEFs [2-4] and the overview papers in the Proceedings of the NTCIR-3 [7], several additional analyses were done on the NTCIR-3 results and collection.
For PATENT retrieval task, though a new strategy for cross-genre retrieval called term distillation was proposed by Ricoh group and worked well on the collection, many research questions regarding patent retrieval were remained unsolved in NTCIR-3. The questions are, for example; 1. Is there any particular IR model (or weighting scheme) specifically effective on Patent? 2. Influence of the wide variation of document length (from 100 words to 30,000 word tokens in a document!) 3. Indexing (Character bi-gram vs. Word-based) 4. Target document collections: Fulltext vs. abstract (many commercialized systems used abstracts only) For 1., it has been reported that tf is not effective on Patent at the SIGIR 2000Workshop on Patent Retrieval, but we could not find the concrete answers to the question through the NTCIR-3.
To answer these question, the NTCIR-3 Patent Task organizers conducted additional experiments on the patent collection
and newspaper collection, and tested 8 different weighting schemes including both vector space as well as probabilistic models,
on 6 different document collections, using 4 different indexing strategies, character bi-gram, word, compound terms, hybrid of
character bi-gram and word; and 3 different topic length on a system. The results will be reported in [
For WEB, one participating group was consisted as a collaboration of research groups with strong background of
content-based text retrieval and of web-link analysis, worked well at NTCIR-3 WEB. Further analysis on the effect of link on
WEB collection, link-based approaches are generally worked well especially on the short queries like using TITLE only, or
more specifically the first term of the TITLE, i.e. the most important terms for the users (topic authors) [
As shown in Table 1, the 4th NTCIR Workshop hosts 5 tasks, CLIR, PATENT, QAC, TSC, and WEB and their sub-tasks. Evaluation schedule varies according to each task.
April 2003: Document Release
20 February 2004: Evaluation Results Release 2-5 June 2004: Workshop Meeting at NII, Tokyo Japan For the further information including late registration of the task participation, please consult NTCIR web sites at; http://research.nii.ac.jp/ntcir and http://research.nii.ac.jp/ntcir/ntc-ws4, or contact the author.
Since this is the second multilingual CLIR at NTCIR, the same task design will be continued from the previous one. Minor revision was made only to solve the major problems raised in the assessment on the NTCIR-3 as follows; ・ ・ ・ ・ ・
Enlarge the English and Korea document collections comparable to Chinese and Japanese. 2.7GB in total. New sub-task of Pivot Language Bilingual CLIR Restrict the pair of topic and document languages, so that comparison will be done in fruitfully Set T-only run as mandatory as well as D-only run Question type topics were categorized according to the nature and types of the answers in order to take a good balance of the topic set.
The new sub-task, pivot CLIR uses English as a Pivot language, then test the effectiveness of the transitive CLIR. It is one of the practical approaches of Multilingual CLIR in the environment with less availability of the direct translation resources but rich in those between each of the languages and English.
QAC plans three subtasks as previous one at NTCIR-3. Among the three, subtask-1 and -2 will be done without major change. Only exceptions are; use different question sets for each of subtask-1 and -2, and increase the number of topics containing multiple answers. It was decided to avoid overestimate of the groups ignoring the possibility of multiple answers and returning the first priority answer only to the every question in subtask 2.
QAC subtask-3, answering to the series of question, is one of the major focus of the NTCIR-4 QAC. We plan to increase the number of sequence as well as task design aiming to tackle the problems resembling the real-world Report Writing task based on a set of relevant documents. The task design also related to the TSC, content-based evaluation of multi-document summarization will be done by set of questions. This is, more fundamentally, what kind of aspects of an event or topic that users want to know. Some of the questions may be more appropriate for the current factoid oriented QA and others may covered by summarization. IR covered both and those focus of QAC and TSC has many intersection of the focus of the CLIR to see the categorization of question types.
Both PATENT and WEB plan (1) Main task(s) and (2) Feasibility or Pilot studies for more challenging tasks as follows; PATENT- Main: Invalidity Task:
To search patents to invalidate the query patents. Claims of the query patents are used as query and they are segmented into components of the invention or technologies consisting of the investigation, then search related patents. A patent may be invalidated by one patent or by combination of multiple patents. Return document IDs as well as relevant passages.
PATENT - Feasibility: Long term research plan over NTCIR4-5. Automatic Patent Map Creation A kind of Text mining -- Detect sets of technologies used in a set of patents, extract them, and make a table to show the relationship between technologies and patents, and evolution or trends among them.
WEB - Main: Informational Search and Navigation Oriented Search, in which find most informative and reliable page WEB Pilot: Geographical oriented and Topical Classification of the Search results For the details, please visit the website of each task, which are linked from the NTCIRs main web site.
Brief history of NTCIR and recent progress after NTCIR-3 are reported in this paper. One of the characteristic features of the NTCIR is targeting Information Access technologies, in which a whole process for users to obtain and utilize the information in the documents are interested in and see the intersection between all the related technologies including IR, Summarization, QA, Text mining, etc., and treat them as like a family. Other aspects are, for see the users information task behind the laboratory-typed testing. We are in the process of the fourth-iteration in a series. Evaluation must be changed according to the technologies evolution and change of the social needs. We have been and are struggling for this. Collaboration and any leads and advices are always more than welcome.