A preliminary review of 145 IIiX and CHIIR papers on experimental IIR studies was performed in order to identify elements of the methodological approach of IIR studies, which are important to document for later access and re-use. The papers' ACM publication metadata was also analyzed for reporting methodological details. The analysis suggests 17 reporting elements and finds that current metadata practices are not suficient documentation for this purpose.
• Human-centered computing → User studies; Usability testing; Walkthrough evaluations; Laboratory experiments; Field studies; Empirical studies in HCI;
"Maturation of a research specialty relies on the ability to replicate research, provide standards for measurement and analysis, and understand past endeavors." — Kelly and Sugimoto [16, p. 746] The annual CHIIR conference series and the bi-annual conference series that spawned it, IIiX, have now seen a total of eight diferent editions over a ten-year period. During that time, the number of research studies, interactive information retrieval (IIR) systems, and the size of the community as a whole have steadily increased. Many research disciplines typically experience some form of convergence over such a period of time—a standardization of definitions and methodological approaches. But is this true for IIR?
A systematic review of IIR evaluation studies by Kelly and Sugimoto—covering the period of 1967-2006—found this not to be the case: “IIR studies [...] rely on a wide variety of methods and measures, perhaps because of the complexity of evaluating user behavior and system interfaces simultaneously.” [16, p. 746]. With a dedicated conference such as CHIIR—typically a sign of a maturing discipline—have we shifted to such as a phase of maturity? Or does the “complexity” of IIR studies preclude the formation of methodological standards?
This short paper represents the first phase of a systematic review of IIR experimental studies published at the IIiX and CHIIR conferences from 2006 to 2018. It is part of a larger research project that aims at providing a framework for the documentation, access, and re-use of research designs from experimental IIR studies. This is in line with the starting quote of this paper: replication, reproduction, and even just re-use of experimental designs and research data will provide a basis for the validation of research outcomes, a way to visualize the progress of the discipline over time, and support more collaborative research—research groups working jointly on the same research questions—and more comparative research—diferent research groups work on confirming patterns and hypothesis.
The results reported here are the first outcomes in an inductive coding project, where statements about research methodologies are extracted from papers describing experimental IIR research. The idea is to identify the elements of IIR studies that are necessary to describe the study and provide enough methodological context for access and re-use, which will then feed into a reporting and documentation data model—left for future work. This paper describes some first insights on reporting elements for IIR studies and presents a brief content analysis of reporting elements in IIR publications based on ACM CCS concepts and author-provided keywords. The goal is to invite the IIR community to discuss these identified elements, which would then be used for a second coding round and the development of a data model.
Our study uses Kelly and Sugimoto’s systematic review as a starting point, but changes their approach and coverage in several aspects. The IIiX conferences started after their analysis period, so this review looks at a more recent period in IIR research with perhaps a more well-defined research community bound by conference publications. While they focused on evaluation studies, this study incorporates all types of experimental IIR studies. Other significant adaptations will be discussed in section 3. 2
Earlier work on IIR illustrates the wide range of study designs and
evaluation approaches [
The first BIIRRR workshop [
Systematic reviews or meta-analyses focusing on methods and
metrics have either been applied to whole disciplines [
This study takes an inductive grounded theory approach, where
data collection and analysis was done simultaneously and the
categories—the methodological components of IIR studies—were
extracted from the data using an open coding approach [
While IIR studies are reported on in many venues [
Next, we formulated the inclusion and exclusion criteria for deciding which publications to include in our analysis. We decided to restrict our analysis to what we considered “archetypal” experimental IIR studies, although many more research strands exists in IIR. This decision served to exclude papers from the analysis that were concerned with theoretical, methodological, or technical concerns rather than with studying people’s interactive information search behaviors directly. We excluded these because we do not believe they would be as useful in defining reporting standards for later re-use of IIR research designs.
2For six poster or demo papers from the 2012 IIiX conference, no full-text version was available from the ACM Digital Library. For three of these papers, a PDF version was found on the authors’ personal website or in their institutional repository; for the other three papers, the assessment was performed using only the abstract.
To match our definition of an experimental IIR study, subjects must engage in either searching, browsing or another related information access interaction and should be observed during the study. This excludes studies where searchers only report on their search behavior (e.g., surveys or interviews) or studies, which only report on contextual aspects of the information search process (such as people’s information needs) without also studying specific interactions.
Our criteria were substantially broadened from Kelly and Sugimoto’s selection criteria to include IIR studies that not only evaluated information systems, but also studied the (system-mediated/supported) information search behavior of humans in general. This expansion of scope was intended to capture the widest possible range of research approaches in empirical IIR studies so that all the relevant facets for a reporting standard can be identified. Table 1 summarizes the inclusion and exclusion criteria for our study and highlights similarities to their guidelines. If a criterion is not explicitly mentioned, then we did not use it in this study. For example, while test subjects in Kelly and Sugimoto’s review were restricted to adults, we did not apply this criterion.
All conference publications were divided among the paper authors for the selection phase. All papers were read by at least one author until a clear decision could be made on the nature of the study and the relevant methodological details were extracted. This included at least the abstract, methodology, and evaluation sections of every paper. While the papers were read for selection, the first coding to extract elements of IIR studies was also performed. For unclear cases, all authors discussed the coding decisions. During the coding, guidelines for the inclusion and exclusion criteria were specified and sharpened when an ambiguous case demanded it. Papers coded before this were subsequently re-coded after a final revision of the guidelines. 3.2
During coding, the authors took note of methodological components of experimental IIR studies, which were mentioned in the papers and would lend themselves to be potentially re-used by other studies. After all papers were coded, these elements were pulled together in a list and categorized by type. Each element type was described and examples were added. This categorization of elements will be utilized in the second phase—after community discussion and agreement—to extract the specific research design elements from each included IIR study. 3.3
For each included IIR study, we extracted metadata from the publications in order to check how much of the methodological approaches or research design elements were reported by their authors in a structured way to enable systematic retrieval. As content metadata, we considered the ACM Computing Classification System (CCS) 4 classes, the General Terms and implicit subject descriptors as well as the author-provided keywords.
The purpose of the study should be to analyze “people’s information search behaviors, their use of interfaces and search features, and their interactions with systems”3. This includes analyses of people’s interaction behavior on one particular system (focus is on the searchers’ characteristics), IIR system or feature evaluations (focus is on the system characteristics) and studies of diferent interaction techniques (focus is on interaction characteristics).
Excluded are studies, which do not study humans interacting with information systems. Humans as users of an information system must be engaged in the study as subjects and must have an interaction with the researchers.
The study should be empirical and attempt to use at least some aspects of the scientific method.
Studies only using log data from a natural environment (uncontrolled log data) are excluded.
Studies targeting methodological issues are excluded. This includes studies on general evaluation design, relevance assessment for IR evaluation, task or scale design, eye-tracking equipment and study design, etc.
Subjects must engage in either searching, browsing or another related information access interaction (such as results selection) and will be observed during the study. This excludes studies where searchers only report on their search behavior (e.g., surveys or interviews) or studies, which only report on contextual aspects of the information search process (such as people’s information needs) without also studying specific interactions.
If tasks are included in the study, they can be both natural or simulated and assigned.
Studied information systems should have a search component as their main focus. This excludes filtering and recommendation systems, email or document management systems, expert systems, decision support systems or other systems unless the focus of the study is explicitly on search components in these systems.
The CCS updated its classes in 2012 from its 1998 version. While classes had alphanumeric notations in the 1998 version, these notations were dropped and only the class descriptions were maintained.
The 1998 version was mapped to the 2012 version and all publications in the ACM Digital Library were changed accordingly on their reference pages. The PDF files of the full text publications, which were the basis for the analysis, were not changed in the ACM Digital Library, so that old and new class descriptions were mixed. Long after the version change, authors used the older version (observed in our sample up until the 2014 IIiX conference). We chose to distinguish between the two versions, because diferent assignment behaviors from authors using the diferent versions could be observed (see section 4.2). Authors also used diferent representation formats5, which we normalized for the analysis.
The 16 ACM General Terms6 and implicit subject descriptors for proper names were provided by ACM as additional controlled terms to add to the already controlled CCS 1998 version. Since this list is too broad and the practice of using these terms was discontinued for CHIIR after 2014, we did not include these in our final analysis.
The author-provided keywords are those terms that authors apply to their publications as metadata to be searchable in the ACM Digital Library. We included them as they had the most potential to include methodological details on a study.
During the content analysis, we generated a frequency distribution of the utilized CCS classes. The author-provided keywords 5Even though the ACM provides a template here: https://dl.acm.org/ccs/ccs.cfm. 6See the list and instructions for application here: https://www.acm.org/publications/ computing-classification-system/1998acmcss were slightly normalized (capitalization removed, plural and singular forms and identical strings with dash / without dash normalized) before their frequency distribution was also generated. A manual scan extracted those keyword strings that provided details on methodological components. 4 4.1
For all eight analyzed conferences, a total of 432 publications were considered. Of these publications, 84 were excluded from the detailed analysis as per our formal exclusion criteria, because they contained keynote abstracts, workshop and tutorial overview papers as well as doctoral colloquium overview and abstract papers. Out of the remaining 348 long and short or poster/demo papers, 145 papers reported on an experimental IIR study based on our inclusion criteria. This means that only 41.7% out of all IIiX and CHIIR submissions report on an archetypal experimental IIR study (as defined by our criteria). We believe that an analysis of the remaining papers would provide more interesting insights into the nature of the IIR research community, but this remains outside of the scope of this study. 4.2
Out of 145 analyzed papers, only 91 provided CCS classification statements - naming 146 individual CCS classes. It is noticeable that after 2014, CCS statements were much more sporadic (71 until 2014, 20 after 2014). Until 2014, usually just one CCS class was provided in a paper (avg. 1.4 per paper), whereas after 2014, more than one class descriptions are listed (avg. 3 per paper) if they are provided, which is much rarer7. Until 2014, all but one8 of the 121 assigned CCS classes were assigned from the H. Information Systems class, mostly H3.3 Information Search and Retrieval (68) with its subclasses, H5.2 User Interfaces (13) and H3.7 Digital Libraries (11). The few papers using the 2012 ACM CCS version also mostly assigned subclasses from the Information Systems class (19 out of 35 individual classes) or the Human-Centered Computing class (11). The class assignments are not surprising, however, particular classes do not seem to correlate with a particular IIR study type in our sample. While the HCI design and evaluation methods class provides subclasses, which would describe study types, they were hardly used.
With 113 papers incorporating 477 individual author-provided keywords (avg. 4 per paper), this metadata option has more potential to include details on the research designs. The keywords follow a typical long-tail distribution: "user study"9 or variations of “information retrieval”, “information search” or “information seeking” were most often assigned (both 19 times). IIR (11), exploratory search (7) and eye-tracking (7) round out the top 5 most frequent keywords. The data collection method (among them: case study, ethnography, evaluation, experimentation, eye-tracking, log analysis, user study) was provided 51 times. Keywords often mentioned the conditions or features that were studied (in participants or systems). Not every assignment included details on the research design and most of them only described one or at most two aspects of the methodology. 7We speculate that this may be related to the switch from the 1998 to the 2012 CCS version, the sample showed that the 1998 version was still used in papers until 2014. Possibly, the behavior change could also be attributed to the interactive tool, which allows authors to choose classes from a visualized 2012 CCS. 8One additional class description of J.3 Computer Applications: Life and Medical Sciences was assigned. 9A class “User Studies” is also available in the CCS. 4.3
This first coding analysis found ten element categories for research designs and an additional seven categories for contextual information, which should be reported when documenting experimental IIR studies for re-use. Some of these were categorized with subcategories. Table 2 lists all identified categories for research design elements and provides examples.
Contextual information elements, which should also be included, are: (1) the goal of the study (e.g. system evaluation), (2) the research questions, (3) the studied information systems, (4) the studied conditions, (5) whether the study was part of a larger project, (6) the research data produced (and its location if stored in a repository), and (7) research publications, which used the study data or results.
These elements include the categories suggested in the BIIRRR
workshop [
The review of 145 conference papers on IIR research and the subsequent categorization suggests at least 17 description elements, which would need to be documented for appropriate access and re-use. The current content metadata options provided by ACM publications do not provide this level of detail. We therefore believe that the next step for any form of standardization in experimental IIR research would be to agree on a common reporting standard for IIR studies. The suggested elements will have to be validated in a second more detailed coding round, when all 145 IIiX and CHIIR papers will be coded with this classification scheme. We defer this to future work. Once this has been done for the small scale study, other relevant conferences and journals should be included.
In order to promote re-usability of elements of past IIR studies and frameworks, we believe it would be beneficial for the IIR community to start expecting and/or requiring a more detailed set of reporting standards, for example as part of the submission guidelines for future CHIIR conferences or as part of the provided structured metadata in an ACM formatted publication. A critical aspect is also the input of the research community, which needs to agree to and adapt the reporting guidelines.
If these research design elements are necessary for an adequate documentation, such a reporting requirement could also harmonize the discipline’s methodological practices and define concrete expectations for research quality in IIR experimental studies. Establishing reporting guidelines in a bottom-up community process could work towards this goal.
Other disciplines using qualitative and often heterogeneous
data—such as the social sciences—have dealt with quality
measures and re-use issues for research design aspects for many years
[