There is increasing interest in spoken conversational search-multiturn interactions with a search engine, spoken in natural languagebut until recently there was little public data to support research. We describe our experiences building two data sets for spoken conversational search: the Microsoft Information-Seeking Conversation set (“MISC”) and the Spoken Conversational Search set (“SCSdata”). Each data set contains recordings of spoken interactions between two people collaborating on web search tasks, but relatively small diferences in protocol have led to observably diferent data. We discuss some consequences of these diferences, and describe attempts to reproduce analyses from one set to the other.
The increasing capability for natural-language, voice interactions with computers poses a range of research and engineering questions. To address these questions we need corresponding data—for example, recordings of conversations with information-gathering agents. Unfortunately, current systems cannot maintain a lengthy exchange, have trouble tracking context, and are largely unaware of nonverbal communication and of users’ emotional state. In 2016–17 two separate groups tried to bridge the gap by recording informationseeking conversations between people, looking for structures which would help build new systems or evaluate old ones [c.f. 5, 9, 19].
The Microsoft Information-Seeking Conversation data (MISC) is a
set of recordings of spoken conversation between human “seekers”
and “intermediaries” [
The study. The overall setup for both the MISC and SCSdata recordings is shown in Figure 1. Tasks were assigned to a “seeker”, who was responsible for assembling information and deciding a ifnal answer. They were connected over an audio link to an “intermediary”, who stood in for a future software agent (SCSdata participants were located in the same room). The intermediary had unrestricted access to the web, including search engines. We recorded video and audio from both participants.
The data. The MISC data includes audio and video signals; transcripts; prosodic and linguistic signals; entry questions on demographics and personality; and post-task surveys on emotion, engagement, and efort. Screen recordings are also available, as is data on afective and physiological signals.
Reuse and reusability. We designed the MISC data with regard to our own future research, but intended from the start that it could be used by other researchers. Our participants consented to possible reuse and sharing, and were informed of their right to withdraw consent at any time, including post-hoc. The study was approved by our internal ethics review board.
Although MISC includes a good deal of derived data, we have
chosen to include the raw data wherever possible so as to enable
(a) replication and (b) further unanticipated analyses. For example,
we include the raw audio, from which we derived the included
transcripts; and we include these transcripts, from which we derived
data on word use. The only processing of the “raw” video and audio
has been to segment by task. The full text of each pre-experiment
and post-task question is also included. This policy has already
enabled reuse inside our research group: for example, work by
McDuf et al . [
We used standard instruments and standard processing tools
where available:
• To help interpret physiological and afectual signals, we
used the UPSS Impulsive Behaviour Scale [
We believed there may be a diference in behaviour and
selfreports depending on the complexity and dificulty of the
task, so we varied these in a controlled manner. We also
wanted tasks that elicited an emotional response, which
ruled out those from most past collections; instead we
selected tasks from the Repository of Assigned Search Tasks
(RepAST)2. Participants addressed the tasks using the open
web, which may make it hard to reproduce some results but
did allow intermediaries to use the full range of web search
features.
• We measured efort with the NASA task load index (TLX) [
This is a commonly-used and well-validated scale which we
were able to adopt with minimal modification (only omitting
the question on physical efort). Post-hoc tests validated this
modified scale (Krippendorf’s α = 0.84 [
Inquiry and Word Count (LIWC) [
We are happy to make many of our processing scripts available for other researchers—a small number use in-house tools—although again there have been no requests so far.
Reporting and availability. We described our protocol in detail in
our first publication [
The MISC data is available online at http://aka.ms/MISCv1. 1.2
The Spoken Conversational Search data set (SCSdata) contains the
utterance transcriptions of a spoken information seeking process
between two actors. To the best of our knowledge, SCSdata was
the first data set which was created in this experimental setup. It is
also the first SCS data set which received labelling of the actions or
utterances, albeit only for the first three turns [
The SCSdata was created to investigate the interaction behaviour
between the two actors, including helping us to understand
questions such as; what is the impact of audio-only interactions for
search?; how are information-dense documents transferred in an
audio-only setting?; what are the components or actions of an
information-seeking process via audio, and what is the impact of
query complexity on the interactions and interactivity in spoken
conversational search? The SCSdata has been used in research
published by the creators of the data set [
The study. The SCSdata was created in a controlled laboratory
study at RMIT University. We recorded the spoken interactions
between seeker and intermediary (as explained in Section 1.1). We
then transcribed the recordings with transcription principles and
protocols described by Trippas et al. [
The data. The data includes the transcriptions of the audio signals, the codebook and labels for the first three utterances, and the backstories used in the setup. Other data such as the audio, video, pre- and post-task questionnaires are not available due to ethics regulations.
The data is maintained by an author of this paper (Trippas).
Reuse and reusability. The SCSdata reuses nine backstories based
on TREC Q02, R03, and T04 as described by Bailey, Mofat, Scholer,
and Thomas [
Participants completed a pre-test questionnaire before starting
the study. This pre-test questionnaire gathered demographic data
such as age, gender, highest level of education, employment, and
computer and search engine usage. Participants were also asked to
complete a modified version of the Search Self-Eficacy Scale [
The SCSdata was designed with our own research questions in mind, while optimising the transcriptions and labelling for future use. We believe that the labelled data set is very valuable for the research community. The data set was recently updated and it is planned to update the data set with the full labelling annotations and label creation methodology in the near future.
Reporting and availability. We described our experimental setup
in the preliminary data analysis paper [
Other details such as the procedure of the study or questionnaire results have not yet been published.
The SCSdata is available online via https://jtrippas.github.io/ Spoken-Conversational-Search/. 2
In recent, unpublished work, one of us (Trippas) has developed a code schema for annotating utterances in spoken conversational search. Initial development used SCSdata, but since MISC is very similar it has been reused to validate the schema. We ofer below some observations on re-using MISC and SCSdata, based on this experience.
It is clearly valuable to have two data sets collected with such similar protocols, and for similar purposes. Coding conversations relies on having lengthy, naturalistic exchanges, and both SCSdata and MISC have several exchanges running to ten minutes. Both sets distinguish the “seeker” and “intermediary” roles, allowing direct comparison, and both include transcripts which could be coded more or less directly. However, some diferences across the data sets did hamper reuse, or led to unexpected findings. 2.1
First, while the SCSdata was manually transcribed, the MISC data is about ten times larger but has only been transcribed with a commercial speech-to-text system. Although the automatic speech recognition (ASR) system was state of the art, it was still prone to errors. (One common error was to inject “speech” when a participant was typing, as if the ASR was confused by keyboard noise.) These errors were discovered because a close reading was needed to label the MISC utterances for the validation of an annotation schema. The diference in transcription techniques also gave a different notion of utterance or turn: in SCSdata these are divided manually, while in MISC they are separated by pauses in the audio signal. Utterance-level statistics may not be directly comparable.
The sets also difer in the pre- and post-task questions. The MISC questions and responses are part of the released data, and the published description includes descriptive statistics and basic validity checks. We hope this is useful for future work. The SCSdata protocol also added many pre- and post-task items (see section 1.2), on overlapping themes but with diferent instruments. These have not been examined to date so they may or may not be useful or comparable. Future SCSdata releases will not include this data.
Some apparently small diferences between the SCSdata and MISC protocols have led to observable diferences in the collected data. SCSdata participants were expressly prohibited from reading out the task statement verbatim and had to verbalise their information request; MISC participants were given no instruction on this matter. As a result, the MISC data include seekers reading out and repeating the task statements, verbatim. More importantly, once both participants have the same statement, the roles of “seeker” and “intermediary” are blurred and the two act much more like peers. This has influenced the interactions in MISC, and the distribution of conversational moves.
The two protocols also difered at the end of each task. For MISC, “seekers” were asked to record an answer: this was meant partly to encourage participants to properly complete each task, and partly so researchers could look for diferences in answer correctness or completeness4. SCSdata participants were not asked to record an answer, but were asked to say “stop search” when they were satisfied with the found information and could answer the information need. This again led to diferences in behaviour, such as MISC “seekers” confirming spelling in order to write down the answer.
These diferences were an unexpected nuisance, as even with such similar protocols it required some work to understand and account for the substantial diferences in data. However, familiarity with the data meant that once we had observed the diferences, they were easy to understand. A close reading of the published descriptions would have given the same hints. Further, it is likely that the diferences were in fact useful for the validation, as they gave more variety and tested the coding schema in slightly diferent exchanges.
We also note some smaller diferences. For the SCSdata recordings, a researcher was in the room with the participant. The MISC researchers were not. This may have led to some diferences in the data, although we have not yet explored this. There is also a diference in audio quality. The audio files from the SCSdata are poor, because they were recorded through a video camera. Using those recordings was never part of the experimental setup.
Finally, there are details of the protocol which may have resulted in minor diferences between the sets. MISC featured a warm-up task, while SCSdata did not; MISC participants used a Windows PC, while SCSdata participants used a Mac; and MISC intermediaries started with Bing, SCSdata intermediaries with Google, although all were allowed to switch to any other site. 2.2
There has been some inconsistency in terminology. First the two
actors of the SCSdata were referred to as the “user” (the participant
with the search task) and “retriever” (the participant with the search
engine) [
Other terminology is not standard. Trippas et al. used “spoken conversational search” to emphasise the spoken channel, as opposed to multi-turn interactions with e.g. typing or selecting buttons. For the same scenario, Thomas et al. used the phrase “information-seeking conversation” to encourage a broader understanding encompassing negotiation and clarification, not just a traditional query/response “search” model. Other terms again are used elsewhere in the literature. Presumably in the near future this terminology, as well as the names of the diferent roles, will be standardised. 2.3
As explained in section 1.2, the tasks used for the SCSdata were
reused from research by Bailey et al. [
Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on Human Information Interaction and Retrieval Johanne Trippas and Paul Thomas
Two sets of spoken conversational searches—SCSdata and MISC— were collected independently, by diferent teams, in diferent geographical locations, to support diferent research. It is fortunate that the data sets are similar enough so that we can make direct comparisons, and use one set to verify observations from the other.
Despite being collected with very similar goals and methods, relatively small diferences in protocol made observable diferences to the data and we have had to be careful with reuse and comparisons. This was made much easier by our familiarity with the data; another researcher could quite reasonably choose these two data sets, compare them, and have dificulty. That this is possible despite careful design and description, and despite close similarity in protocol, may perhaps caution us about reuse in interactive studies generally.
We were however helped by the decision to explicitly allow the release of MISC’s raw data (not just, e.g., transcripts). Because audio was available, the transcription errors could be detected. Unfortunately ethical clearance precludes a similar release for SCSdata, and this may limit reuse.
Communication between two people is very culture-specific [
We thank Daniel McDuf, Mary Czerwinski, and Nick Craswell for their efort assembling MISC, and Penny Analytis for auditing the SCSdata transcriptions. We are grateful to our participants for their time.