Overview of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019) Toine Bogers Samuel Dodson Luanne Freund Aalborg University Copenhagen University of British Columbia University of British Columbia Denmark Canada Canada toine@hum.aau.dk dodsons@mail.ubc.ca luanne.freund@ubc.ca Maria Gäde Mark Hall Marijn Koolen Humboldt-Universität zu Berlin Martin-Luther-Universität Royal Netherlands Academy of Arts Germany Halle-Wittenberg and Sciences maria.gaede@ibi.hu-berlin.de Germany Netherlands mark.hall@informatik.uni-halle.de marijn.koolen@di.huc.knaw.nl Vivien Petras Nils Pharo Mette Skov Humboldt-Universität zu Berlin Oslo Metropolitan University Aalborg University Germany Norway Denmark vivien.petras@ibi.hu-berlin.de nils.pharo@oslomet.no skov@hum.aau.dk ABSTRACT significant benefits for the IR community, in particular enabling This paper presents an overview of the BIIRRR 2019 workshop the re-use of aspects of the test collections or shared tasks in other at CHIIR 2019, which had the explicit aim of understanding and IR research. Comparable efforts have been undertaken to bring this promoting re-use of resources for interactive IR experimentation. paradigm to IIR research, such as the TREC Interactive [22] and Session [6] tracks, the INEX Interactive track [21], and the Inter- CCS CONCEPTS active Social Book Search track [24]. The high degree of variation between IIR studies, however, has meant that none of these have • Information systems → Users and interactive retrieval; • achieved similar degrees of standardization and re-use. It seems General and reference → Empirical studies; Evaluation; that the traditional shared task structure and test collections (i.e. KEYWORDS resources to search on) are not successful in inducing re-use in IIR. Instead, an equivalent type and level of re-use is more likely IIR, re-use, secondary use, research design to be achieved through increased sharing of research design (as- pects), enabling better comparability, more transparent reporting, 1 DESCRIPTION and greater methodological standards and rigour. What would be the equivalent of a re-usable TREC test collection IIR research exhibits a large variety of research designs and for the interactive information retrieval (IIR) community? The goal methods [20]. This methodological diversity and richness needs of the BIIRRR 2019 workshop was to answer this question by con- to be investigated to identify potentials and ways of representing tinuing existing community-driven efforts to develop approaches these research materials suitable for re-use. Therefore we need to for the collection, organization, maintenance, and sharing of re- develop an understanding of how and when researchers currently sources for IIR experimentation. These efforts developed out of re-use or would like to find and re-use materials. This is, in fact, an discussions at the CHIIR 2017 workshop on Supporting Complex IIR research problem itself! Search Tasks (SCST 2017) [1] and were expanded upon during the The interactive workshop format is used to gather information BIIRRR workshop at CHIIR 2018 [3, 4]. and input from the IIR community regarding the kind of methods The information retrieval (IR) community has a strong tradition used, their experiences with both re-using materials and making of making research data of system-based experimentation available materials available for re-use, and ideas on how to move towards for re-use, as exemplified by the development of test collections, increased sharing and re-use. The organizers already represent a shared tasks and relevance assessments in large-scale initiatives wide range of IIR research perspectives, but to achieve the goal of such as TREC1 , CLEF2 , NTCIR3 , and FIRE4 . These efforts have had encouraging re-use a high degree of community input and commit- 1 https://trec.nist.gov/ ment is needed in order to identify feasible re-use aspects as well 2 http://www.clef-initiative.eu/ as limitations. 3 http://research.nii.ac.jp/ntcir/ 4 http://fire.irsi.res.in At the BIIRRR 2018 workshop, a number of high-level focus areas were identified [4]. To make further progress on these issues, the Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on focus of the 2019 edition centered on the following five topics: Human Information Interaction and Retrieval (CHIIR 2019), 14 March 2019, Glasgow, UK 2019. Copyright for the individual papers remains with the authors. Copying permitted • Terminology The terms used in IIR studies, their defini- for private and academic purposes. This volume is published and copyrighted by its editors. tions, and origins. Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2019), 14 March 2019, Glasgow, UK Bogers, T. et al. • Methodology General and specific methodologies employed discussions at the popular SCST 2017 workshop (co-located with in IIR studies, their origins, and their re-use. CHIIR 2017) identified a strong desire within the IIR community • Research design The overall research design structures and to address the issues around re-use. This led to the BIIRRR 2018 patterns employed and their potential for re-use. workshop at CHIIR 2018, which focused exclusively on the re-use • Re-use Existing resources re-used as part of IIR studies, how issue and resulted in the publication of a summary paper [4] as well to find them, and issues with re-using them. as a grant proposal, and spawned several informal follow-up meet- • Documentation What and how to document aspects of IIR ings, which planned the BIIRRR 2019 workshop. It also served as a studies to maximize the potential for re-use. starting point for a concrete, community-driven effort focused on BIIRRR 2019 addressed these topics by combining short paper the challenges and opportunities for designing and implementing a presentations with breakout groups. This workshop summary paper platform for the collection, organization, maintenance, and sharing first describes earlier and related efforts preceding the BIIRRR 2019 of resources in IIR experimentation. workshop (section 2), before delving into the focus areas (section Further afield, related efforts and platforms can be identified, 3). Section 4 describes the workshop program, the short papers particularly in the social sciences6 . There, the discussion on re-use presented and contains a brief outline of the breakout sessions. of quantitative versus qualitative research data appears to grapple Section 5 discusses the outcomes and continuing activities planned with similar challenges as the comparison between system-based after the workshop. IR evaluation and IIR experimentation [2, 9]. 2 EARLIER AND RELATED EFFORTS 3 BIIRRR 2019 FOCUS AREAS There have been several successful gatherings directed toward ad- In order to incorporate the community’s experience, the workshop dressing the need for considering how to collect, organize, maintain, invited experience papers that detail methodological and re-use and share research resources for conducting IIR experiments. IIR aspects of previously published or in-press IIR studies. Rather than campaigns on this topic include the TREC Interactive Track (1997– focus on research questions and results, experience papers were to 2002) [22], the INEX Interactive Track (2004–2010) [21, 26, 29], the focus on the following aspects of IIR studies, which are generally Cultural Heritage in CLEF (CHiC) Interactive Task (2013) [24, 30], under-reported in scientific publications: and the interactive Social Book Search (iSBS) task (2014-2016) [12, 13, 15] which provided great insight into the challenges and • Terminology What terminology did you use to describe opportunities for long-term, re-usable IIR research materials. the different components of the study? Why did you choose While these demonstrate the ongoing interest in standardising this terminology? How did you develop this terminology? the evaluation of IIR studies [20], they also show that establish- • Methodology What overarching and specific methodolo- ing and maintaining a collaborative platform for the re-use of IIR gies did you employ in the study? How did you decide which research instruments is still an open issue. This is due to the com- methodologies to employ? Examples of overarching method- plexity of IIR studies, which require a combination of system- and ologies include qualitative/quantitative/mixed methods, the- user-centered evaluation approaches [19] as well as some flexibility ory/practice/design, distant/close reading, big data/small and individuality. In addition to the tasks and document collections data. Specific methods include, for instance, log studies, eye that are needed and provided in most shared tasks, participants, tracking, A/B testing, and simulated work tasks. search contexts, tasks, processes, systems, datasets and evaluation • Research designs What research design(s) did you use? measures all need to be modeled to enable re-use for IIR studies. Which (aspects) of these have the potential to be re-used? To Past IIR studies (published at IIiX and CHIIR) were analyzed with capture the variation within the IIR field, we use the broad respect to these design components for this workshop [23]. definition of research designs from Cheek [7]: “the way in There have been efforts to collect and make available some of which a research idea is transformed into a research project or these IIR research components. The Repository of Assigned Search plan that can then be carried out in practice by a researcher or Tasks (RepAST)5 collects, analyzes, and shares search tasks taken research team”. from publications of IIR studies. RepAST contains bibliographic • Re-use What previously created materials did you re-use? data and abstracts from approximately 750 published papers, as well This can cover all aspects, such as research designs, software, as a list of author-identified search task types (e.g., complex, simple, interfaces, data, scales, and specific survey questions. How subject, known-item, factual), and the full text of any assigned did you decide what to re-use? How did you discover the ma- search tasks reported in the papers [10]. As such, RepAST serves as terials that you re-used? Which problems did you encounter a library of tasks. Members of the IIR community are encouraged to searching for them? compare and contrast task descriptions as well as reuse the tasks in • Documentation What aspects of your study could be re- the collection. While RepAST is valuable, it has been underutilized used and how have you documented and represented them to to date, likely due to a lack of awareness. enable re-use? What aspects were fully documented in your Issues related to re-use in IIR have also been discussed at various publication? What aspects do you feel should be documented workshops, including the Supporting Complex Search Tasks (SCST) outside the main publication? workshops in 2015 and 2017 [1, 11], which were organized based on the experiences of running the iSBS shared task. In particular, the 5 https://ils.unc.edu/searchtasks/search.php 6 See, for example, [8] and https://qdr.syr.edu/ Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on Human Information Interaction and Retrieval Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019) (CHIIR 2019), 14 March 2019, Glasgow, UK 4 WORKSHOP ACTIVITIES To Re-use is to Re-write: Experiences with Re-using IIR Experiment The BIIRRR 2019 was a highly interactive, full-day workshop, which Software. Hall [14] describes his experiences with developing and combined accepted presentations, discussion lead-ins, and break- re-using software for both of experimental workflow and search out discussions. user interfaces in IIR experiments. ESS, the software for implement- ing experimental workflows, saw a considerable amount of re-use 4.1 Workshop Program over a five-year period by the main developer. However, re-use by other academics has been limited due to missing documentation We started the day with a full round of introductions of all partici- and the complexity of using it. Additionally after five years of in- pants, asking them to identify their interest in the workshop. The termittent development and expansion the maintainability of the morning program7 contained presentations of six accepted expe- software has significantly decreased. The second software compo- rience papers, organized to elicit more interaction and discussion. nent described by Hall, PyIRE, was a workbench for developing After each paper, a short question & answering round followed search user interfaces to be used in conjunction with ESS. The huge to collect critical aspects for re-use that were mentioned by the variety in required UI components between the different IIR experi- presenters. After the paper presentations, participants discussed ments led to a large amount of re-writing and very limited re-use. the experiences and interplay of the presented work and positions. As a result, Hall argues for the importance of good documentation The afternoon was organized around break-out sessions ded- for re-use purposes, and suggests that future systems of this kind icated to discussion of the workshop themes and the identified should focus on either ease-of-use or flexibility, but not both. concrete re-use issues from the paper presentations. The workshop closed with a concrete mission statement and a clear plan for future INEX iTrack revisited: Exploring the potential for re-use. Similar to work. the previous paper, Pharo [25] presents the experiences of another IIR track: the INEX iTracks conducted over a seven-year period. He 4.2 Workshop Papers provides a comprehensive overview of how the different BIIRRR Six experience papers were presented at the workshop. They ranged 2019 themes changed over the seven-year run of the INEX iTrack, from describing a concrete IIR study [17], the development of a such as terminology, research design, methodology, and resources. re-usable dataset [31], the development and challenges of re-usable Interestingly only a few changes were with a relatively stable frame- software for IIR experiments [14], the re-use aspects of two inter- work over the years were observed. Also most components are still active evaluation tracks [24, 25] and an analysis of IIR experiments available (on demand) and the provided corpora have been re-used published at previous IIiX and CHIIR conferences [23]. We briefly in the past. However, no concrete responsibility for the storage and summarize each of the experience papers below and relate them to maintaining of IIR track data is given and due to changing resources the re-use themes covered in the workshop. and people this data often disappears or is only partly available through previous organizers or participants. A single access point The Multi-Stage Experience: the Simulated Work Task Approach or repository could help overcoming this problem. to Studying Information Seeking Task Stages. Huurdeman et al. [17] demonstrate the complexity of IIR experiments by pointing to the Experiences with the 2013-2016 CLEF Interactive Information Re- several aspects and decisions need to be considered during the trieval Tracks. Petras et al. [24] present an overview of the four IIR preparation and implementation of an IIR study. In order to re- tracks at the CLEF (Conference and Labs of the Evaluation Forum) construct and possibly re-use previous work a detailed description conference, including the original planning, the set-up and lessons and documentation is needed that is often missing due to space learned for each consecutive track from 2013-2016. The paper men- limitations in conference papers. In general it was pointed out that tions that for later re-use, we need to document in much greater checklists and protocols as provided by Borlund [5] are essential detail (1) the collection characteristics for keeping collections up-to- and helpful tools that should be reported and provided for re-use. date, but stable at the same time, (2) the users’ cultural background However, for their own material the authors identified several chal- for more context information, (3) the underlying information seek- lenges with respect to the maintenance of system components as ing model and (4) the data gathering tools. Another aspect that is well as legal and privacy issues when it comes to the storage and rarely considered are guidelines on updating information systems, transmission of research data. but still allow backward comparability (to a previous design and user data gathered on it). The paper suggests an external repository Data Sets for Spoken Conversational Search. Trippas and Thomas for system software components, research designs and other mate- [31] compare their experiences creating two data sets for spoken rials to enable researchers benefiting from created components and conversational search by focusing on differences in terminology, to avoid problems when researchers move institutions and re-use methodology (e.g., transcription protocols), and research designs rights are not properly assigned. (e.g, in the form of tasks provided to the participants). They found that despite their involvement in the creation of both data sets, they Elements of IIR Studies: A Review of the 2006-2018 IIiX and CHIIR still had to be careful with regard to comparing them and identifying Conferences. Petras et al. [23] present an analysis of 145 papers where re-use would be possible. They therefore caution about reuse published at past IIiX and CHIIR conferences with the goal of iden- in interactive studies “despite careful design and description, and tifying aspects of IIR studies that would need to be documented despite close similarity in protocol”. for potential later re-use. It found 10 research design components 7 See the workshop website http://toinebogers.com/biirrr2019/program/ for the detailed and an additional 7 contextual aspects in a first coding round. The workshop program. preliminary analysis of papers reveals that these details were not Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2019), 14 March 2019, Glasgow, UK Bogers, T. et al. always reported in the text—this was also confirmed by the other repositories would require less effort to set up, they allow for more experience papers published at the workshop. specialized searching and the burden of maintenance can be spread over multiple partners. This collection of repositories and resources 4.3 Breakout Sessions could then be linked together explicitly by gathering them in a network and implicitly by using the DOI of the papers the different The break-out discussions were organized based on the interest in components featured in. In time, perhaps the iRepository could the different themes from the workshop participants and identified be then be re-imagined as a wrapper on top of these repositories during the paper discussions. Comments for all sessions were col- that allows researchers to search through all the different special- lected through a shared document and organized at the end of the ized repositories at once, search for components by specifying a workshop. paper, or by offering a wizard-like functionality that could advise Interoperability. The first breakout session focused on the issue in putting together an IIR study by combining components from of interoperability between systems and components for supporting the different repositories. IIR experiments. Participants in the group had in the past developed Another idea for stimulating resource re-use in the IIR com- both generic and experiment-specific systems for conducting IIR munity could be to produce a manifesto on how to perform more experiments and / or had used such systems. A critical discussion responsible and reusable IIR research, similar in spirit to the Leiden emerged with respect to potential re-use of system (components). Manifesto [16] on responsible use of research metrics and evalua- Partly the group agreed that the effort required for re-use exceeded tion. A BIIRRR manifesto could similarly advocate for issues like the benefit of being able to re-use existing software. One particular transparency, documentation, accessibility, and best practices for issue was the need for continual maintenance of software, which resource sharing and re-use. is much harder when there are dependencies on re-used software, Another proposed initiative was to continue the series of ex- which is often no longer maintained. perience paper presentations at a future BIIRRR workshop. The However, after further discussion it became clear that specific, experience papers were well-received because of the transparency generic elements that have the potential for re-use: pre- and post- and the behind-the-scenes look they offered on recent research task survey instruments and generic components such as those projects, in addition to the educational purpose they served for for logging user–system interactions. The focus on these was in young researchers attending the workshop. Future experience pa- part driven by the maintenance issue, as these kinds of elements pers could also come with the requirement to unearth and make can be re-used across many studies, allowing for the development available as many of the resources used in the study as possible of a community around the elements. Such a community would upon acceptance to the workshop. Another possibility that was ensure long-term availability and stability of the re-used elements, discussed could be to investigate the possibilities of a special issue significantly reducing the risk associated with re-use. dedicated to the experience of conducting specific IIR studies. We Based on the discussion outcomes and the participants experi- also discussed if and how documentation and sharing of resources ences with building IIR study systems, the decision was made to test could be included as part of the main CHIIR acceptance process. Fi- how easily possible it is to transfer components between systems. nally, participants floated the idea of a dedicated “Most reproducible After some discussion the pre- and post-task survey instruments paper” award at CHIIR to reward researchers for the transparency, were chosen as exemplary test components. The discussion then documentation and accessibility of their research. focused on the technical and administrative organisation of such a test. IIR Study Elements. The third breakout session discussed report- Since the lack of awareness has been identified as one major ing elements for IIR studies based on the presented review paper barrier on IIR material re-use, the idea of setting up a simple infor- [23]. The group members taking part in the session agreed that the mation page that draws together all the available tools / frameworks suggested list of elements is appropriate for describing IIR stud- / toolkits / components and similar for building IIR studies was dis- ies, and may even be flexible enough to summarize more general cussed. information behaviour studies. The first part of the discussion focused on concrete elements Community involvement & advocacy. The second breakout ses- in the proposed set. During the day, alternative and additional el- sion focused on the issue of community involvement and advocacy: ements were suggested, such as time and domain. Terminological what are the best ways of advocating for IIR resource re-use and discussions in the session also concluded that the broader term reproducibility. The group members taking part in this breakout document corpus should replace the test collection element. A discus- session agreed that setting up new and improving existing reposito- sion on whether to add additional knowledge about the researchers ries for IIR components—tasks, instruments, document collections, performing a study concluded that to record their name and affil- system components—would be a great way of stimulating sharing iation was sufficient. It became apparent that the developed list and re-use. One of the original BIIRRR 2018 goals was to brainstorm needed to be revised and added to - possibly with the help of the about single ‘iRepository’ that encompasses all of these components wider IIR community. in one place. However, after more discussion on this topic in the The discussion then moved to methodological aspects of an el- lead-up to BIIRRR 2019 and in this breakout session, participants ement schema for documentation. For example, it was noted that agreed that it would probably be more beneficial to start by set- it is necessary to standardize and have a controlled set of element ting up and/or improving specialized repositories for specific types terms and fields, but that this must not be at the cost of creativ- of components, such as the RePAST for search tasks. Specialized ity in describing methods and other experimental components. A Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on Human Information Interaction and Retrieval Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019) (CHIIR 2019), 14 March 2019, Glasgow, UK discussion on whether to create a taxonomy for each of the the will serve both as a practical and educational resource for class elements was postponed to future work. Similarly, ideas to experienced and new IIR researchers. create a hierarchy for study types or a decision tree were deemed • Technical / Infrastructure Finally, two technical initiatives to challenging for the current state. A guide for documentation have been planned to support the practical aspects of re-use. may nevertheless arise from the development of the element set. First, some of the organisers plan to work on improving the The possibility for the schema to be grouped in relationship to a RepAST repository. Second, a group of developers plan to theoretical (IIR) framework, such as the ones presented by Tague- develop and test formats for exchanging pre- and post-study Sutcliffe [28], Ingwersen and Järvelin [18] and Kelly [19] or APA’s survey instruments between systems. guidelines on how to report on studies and research designs was also discussed.8 A number of long-term issues were identified that in future might The last part of the discussion focused on using the element be addressed. One is data citability and options such as having a documentation schema once it would be established. The group data-oriented journal for the SIGIR community or a special section brainstormed how the schema could be instantiated with examples in SIGIR Forum. For CHIIR 2021 we consider proposing a tutorial to demonstrate its usefulness. A systematic review of the CHIIR around reuse, focusing on e.g. reusing tasks using RepAST, or build- conference papers appeared to be a good starting point for validat- ing an IIR experiment with Coagmento[27]. We are also considering ing the schema. Asking CHIIR authors at forthcoming conferences a multi-day workshop to discuss documentation guidelines and the to document their studies with the provided schema would have the development of distributed resource portals and to plan further additional advantage of introducing the schema to the IIR commu- community dissemination activities. nity. The documentation schema could also be used as a teaching Finally, we are aiming for an article in an appropriate journal to instrument, either to help students in summarizing IIR studies or articulate our vision, detail the knowledge gained from the BIIRRR even to help students consider the important elements of a study. 2018 survey, outcomes of the 2018 and 2019 workshops, results of the IIR study re-use analysis, and the concrete, practical frameworks 5 OUTCOMES & CONTINUING ACTIVITIES developed based on the workshops’ results. One of the main aims of the BIIRRR workshops has always been to deliver concrete outcomes and encourage continuing activities REFERENCES around IIR re-use. Based on the discussions in BIIRRR 2019 the [1] Nicholas Belkin, Toine Bogers, Jaap Kamps, Diane Kelly, Marijn Koolen, and Emine Yilmaz. 2017. Second Workshop on Supporting Complex Search Tasks. In following five areas were identified and necessary workstasks as- Proc CHIIR 2017. ACM, New York, NY, 433–435. signed: [2] Libby Bishop and Arja Kuula-Luumi. 2017. Revisiting Qualitative Data Reuse: A Decade On. Sage Open 7, 1 (2017), 2158244016685136. • CHIIR Manifesto First of all, some of the workshop organizers [3] Toine Bogers, Maria Gäde, Luanne Freund, Mark Hall, Marijn Koolen, Vivien plan to write a CHIIR manifesto, perhaps as a CHIIR 2020 Petras, and Mette Skov. 2018. Workshop on Barriers to Interactive IR Resources Re-use. In Proc CHIIR 2018. ACM, New York, NY, 382–385. perspectives paper, for responsible and reusable IIR research [4] Toine Bogers, Maria Gäde, Mark Hall, Luanne Freund, Marijn Koolen, Vivien in the form of guidelines for how to conduct meaningful Petras, and Mette Skov. 2018. Report on the Workshop on Barriers to Interactive IIR research that supports reuse and reproducibility of IIR IR Resources Re-use (BIIRRR 2018). SIGIR Forum 52, 1 (Aug. 2018), 119–128. [5] Pia Borlund. 2003. The IIR evaluation model: a framework for evaluation of studies. This manifesto will not only serve as a commitment interactive information retrieval systems. Inf. Res. 8 (2003). by its authors to re-use, but also as an educational tool for [6] Ben Carterette, Paul Clough, Mark Hall, Evangelos Kanoulas, and Mark Sanderson. 2016. Evaluating Retrieval over Sessions: The TREC Session Track 2011-2014. researchers entering the field. In Proceedings of the 39th International ACM SIGIR Conference on Research and • Elements for re-use Another crucial worktask that will be un- Development in Information Retrieval (SIGIR ’16). ACM, New York, NY, USA, dertaken is the definition of IIR study elements as well as the 685–688. https://doi.org/10.1145/2911451.2914675 [7] Julianne Cheek. 2008. Research design. In The SAGE Encyclopedia of Qualitative development of a categorization framework for categorizing Research Methods, Lisa M. Given (Ed.). SAGE. that can be related to other frameworks like [18, 19, 28] and [8] Louise Corti. 2011. The European Landscape of Qualitative Social Research guidelines. Archives: Methodological and Practical Issues. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research 12, 3 (2011). https://doi.org/10.17169/fqs-12. • BIIRRR’20 In order to continue the fruitful exchange we will 3.1746 also submit another BIIRRR workshop proposal for CHIIR [9] Colin Elman, Diana Kapiszewski, and Lorena Vinuela. 2010. Qualitative Data Archiving: Rewards and Challenges. PS: Political Science and Politics 43, 1 (2010), 2020, but with a small name change to Building towards IIR 23–27. http://www.jstor.org/stable/25699288 Resource Reuse to rather focus on the positive (Building) than [10] Luanne Freund and Barbara M. Wildemuth. 2014. Documenting and studying on the negative (Barriers). We want to keep the experience the use of assigned search tasks: RepAST. Proceedings of the American Society for Information Science and Technology 51, 1 (2014), 1–4. papers 1) as a way for researchers to both reflect on their [11] Maria Gäde, Mark M. Hall, Hugo Huurdeman, Jaap Kamps, Marijn Koolen, Mette own previous work, 2) as case studies to discuss challenges Skove, Elaine Toms, and David Walsh. 2015. Report on the First Workshop on and potential solutions and 3) as educational tools for early- Supporting Complex Search Tasks. SIGIR Forum 49, 1 (June 2015), 50–56. [12] Maria Gäde, Mark Michael Hall, Hugo C. Huurdeman, Jaap Kamps, Marijn Koolen, career researchers. Some suggestions are to include a request Mette Skov, Toine Bogers, and David Walsh. 2016. Overview of the SBS 2016 for experience paper authors to make as much of their re- Interactive Track. In Working Notes of the CLEF 2016 Conference (CEUR Work- shop Proceedings), Krisztian Balog, Linda Cappellato, Nicola Ferro, and Craig sources available as possible. Macdonald (Eds.), Vol. 1609. CEUR-WS.org, 1024–1038. • Resource Portal A simple portal will be developed that col- [13] Maria Gäde, Mark Michael Hall, Hugo C. Huurdeman, Jaap Kamps, Marijn Koolen, lects IIR resources and links to toolkits, frameworks. This Mette Skov, Elaine Toms, and David Walsh. 2015. Overview of the SBS 2015 Interactive Track. In Working Notes of the CLEF 2015 Conference (CEUR Workshop Proceedings), Linda Cappellato, Nicola Ferro, Gareth J. F. Jones, and Eric SanJuan 8 American Psychological Association - https://www.apastyle.org/ (Eds.), Vol. 1391. CEUR-WS.org. Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2019), 14 March 2019, Glasgow, UK Bogers, T. et al. [14] Mark Michael Hall. 2019. To Re-use is to Re-write: Experiences with Re-using CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019). IIR Experiment Software. In Proceedings of the CHIIR 2019 Workshop on Barriers CEUR-WS. to Interactive IR Resources Re-use (BIIRRR 2019). CEUR-WS. [24] Vivien Petras, Marijn Koolen, Maria Gaede, and Toine Bogers. 2019. Experiences [15] Mark Michael Hall, Hugo C. Huurdeman, Marijn Koolen, Mette Skov, and David with the 2013-2016 CLEF Interactive Information Retrieval Tracks. In Proceedings Walsh. 2014. Overview of the INEX 2014 Interactive Social Book Search Track. of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR In Working Notes of the CLEF 2014 Conference (CEUR Workshop Proceedings), 2019). CEUR-WS. Linda Cappellato, Nicola Ferro, Martin Halvey, and Wessel Kraaij (Eds.), Vol. 1180. [25] Nils Pharo. 2019. INEX iTrack revisited: Exploring the potential for re-use. In CEUR-WS.org, 480–493. Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources [16] Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. Re-use (BIIRRR 2019). CEUR-WS. 2015. Bibliometrics: the Leiden Manifesto for Research Metrics. Nature News 520, [26] Nils Pharo, Thomas Beckers, Ragnar Nordlie, and Norbert Fuhr. 2011. Overview of 7548 (2015), 429. the INEX 2010 Interactive Track. In INEX ’10: Proceedings of the Ninth International [17] Hugo C. Huurdeman, Jaap Kamps, and Max L. Wilson. 2019. The Multi-Stage Workshop of the Initiative for the Evaluation of XML Retrieval, Shlomo Geva, Jaap Experience: the Simulated Work Task Approach to Studying Information Seeking Kamps, Ralf Schenkel, and Andrew Trotman (Eds.). Springer, Berlin, Heidelberg, Task Stages. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive 227–235. IR Resources Re-use (BIIRRR 2019). CEUR-WS. [27] Chirag Shah. 2010. Coagmento-a collaborative information seeking, synthesis [18] Peter Ingwersen and Kalervo Järvelin. 2005. The Turn: Integration of Information and sense-making framework. Integrated demo at CSCW 2010 (2010). Seeking and Retrieval in Context. Vol. 18. Springer Science & Business Media. [28] Jean Tague-Sutcliffe. 1992. The pragmatics of information retrieval experimenta- [19] Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval tion, revisited. Information Processing & Management 28, 4 (1992), 467–490. Systems with Users. Foundations and Trends in Information Retrieval 3, 1–2 (2009), [29] Anastasios Tombros, Birger Larsen, and Saadia Malik. 2005. The Interactive 1–224. Track at INEX 2004. In INEX ’04: Proceedings of the Third International Workshop [20] Diane Kelly and Cassidy R Sugimoto. 2013. A Systematic Review of Interactive of the Initiative for the Evaluation of XML Retrieval, Norbert Fuhr, Mounia Lalmas, Information Retrieval Evaluation Studies, 1967–2006. Journal of the American Saadia Malik, and Zoltán Szlávik (Eds.). Springer, Berlin, Heidelberg, 410–423. Society for Information Science and Technology 64, 4 (2013), 745–770. [30] Elaine G. Toms and Mark M. Hall. 2013. The CHiC Interactive Task (CHiCi) [21] Ragnar Nordlie and Nils Pharo. 2012. Seven years of INEX interactive retrieval at CLEF 2013. In Working Notes of the CLEF 2013 Conference (CEUR Workshop experiments–lessons and challenges. In International Conference of the Cross- Proceedings), Pamela Forner, Roberto Navigli, Dan Tufis, and Nicola Ferro (Eds.), Language Evaluation Forum for European Languages. Springer, 13–23. Vol. 1179. CEUR-WS.org. [22] Paul Over. 2001. The TREC Interactive Track: An Annotated Bibliography. [31] Johanne Trippas and Paul Thomas. 2019. Data Sets for Spoken Conversational Information Processing & Management 37, 3 (2001), 369–381. Search. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR [23] Vivien Petras, Toine Bogers, and Maria Gaede. 2019. Elements of IIR Studies: Resources Re-use (BIIRRR 2019). CEUR-WS. A Review of the 2006-2018 IIiX and CHIIR Conferences. In Proceedings of the