In this paper we describe our participation in the Interactive Social Book Search task at CLEF 2015. We focus our analysis on differences in search behaviour between native and non-native speakers of English. The analysis is based on both questionnaire and log data. 49 participants out of the 192 total participants are native speakers and the remaining 143 participants are nonnative speakers. In general results show surprisingly few differences in search behaviour between native and non-native speakers. Non-native speakers spent more time on both the focused and the open task than the native speakers, but no significant differences were found in relation to number of queries, query length, depth of results inspection, number of books added to the book-bag, or length of notes explaining why a book was added to the book-bag.
This paper describes our participation in the interactive track of the Social Book Search lab organized at CLEF 2015.
Both in 2014 and this year, the Interactive track (ISBS) recruited test participants from different countries. During the in-lab sessions and through informal postexperiment feedback, we experienced a variety of language-related comments and questions from non-native English speaking test participants. As the book search experiment was conducted exclusively and completely in English, language proficiency could have had an effect on those participants’ book search behavior. Accordingly, we find it interesting to explore what role the context variable language proficiency plays in interactive social book search.
The literature on multilingual search is mainly focused on cross-language
information retrieval (CLIR), such as, e.g., the work done in 2010 by Nie [
The structure of this paper is as follows. We start in Section 2 by describing the methodology. Section 3 analyzes differences in search behaviour between native and non-native speakers of English in relation to the dependent variables time, search behavior, engagement, and book-bag usage. Finally, Section 4 discusses the results and concludes the paper. 2
The overall goal of the ISBS task is to investigate how book searchers use
professional metadata and user-generated content at different stages of the search process
(for an overview of the ISBS track see [
A total of 192 participants were recruited. Participants came from 36 different countries and participants’ mother tongues included 30 different languages including Afrikaans, Amharic, Arabic, Bulgarian, Bengali, Creole or Pidgin, Danish, German, Greek, English, Spanish, Persian (Farsi), Filipino, French, Hungarian, Icelandic, Italian, Malay, Dutch, Norwegian, Polish, Portuguese, Romanian, Russian, Tamil, Turkish, and Chinese. Based on the questionnaire responses, participants were grouped into either native speakers of English or non-native speakers. Native speakers were defined as participants who either had English as their mother tongue (“What is your mother tongue?”) or as their home language (“What language do you speak at home?”). According to this definition, 49 participants out of the 192 total participants (25.5%) are native speakers and the remaining 143 participants are non-native speakers (74.5%). 3
In this section we analyze differences in search behaviour between native and nonnative speakers of English in relation to the dependent variables time, search behavior, engagement, and book-bag usage. 3.1
We analyzed the search log data to examine whether there were any differences between native and non-native speakers’ search behaviour with regards to the dependent variable time. First we looked at time spent on the two different types of tasks. Results show that non-native speakers (M = 0:15:20.29) spent more time on the focused task than native speakers (M = 0:11:06.29). This difference was statistically significant according to an independent-samples t-test (t(104.714) = 2.78, p < .01), with equal variances not assumed (F = 4.62, p < .05). On the open task, non-native speakers also spent more time (M = 0:10:16.97) than native speakers (M = 0:07:49.04), but this difference was not significant according to an independent-samples t-test (t(190) = 1.14, p = .25). When comparing time spent on the open and the focused tasks, results show a significant but weak correlation between time spent on the open and focused tasks: r(191) = 0.15, p < .05. Further, there is a significant difference in time spent on the focused task (M = 0:09:39.22) and the open tasks (M = 0:14:15.46) according to paired t-test (t(191) = 4.11, p < .01).
Secondly, we looked at time in relation to fatigue. On average, people spend more time on a task if it is the first one they perform. If participants started with the focused task, they spent more time on the focused task (M = 0:16:17.15) than if they started with the open task followed by the focused task (M = 0:12:44.75). This difference was statistically significant according to an independent-samples ttest (t(146.605) = 2.26, p < .05), with equal variances not assumed (F = 4.25, p < .05). Likewise, if participants started with the open task, they spent more time on the open task (M = 0:09:52.61) than if they started with the focused task followed by the open task (0:09:21.26). This difference was not significant, however, according to an independent samples t-test (t(191) = 0.27, p = .79).
To examine whether there were any interaction effects between language proficiency and the order in which the two tasks were completed, we also ran a factorial ANOVA. For the focused task, the two main effects—native vs. non-native and task ordering—were not qualified by a significant interaction between the two factors (F (1,188) = 0.12, p = .73), indicating that the ordering effects were the same for the two language conditions. For the open task, there was no significant interaction either (F (1,188) = 0.001, p = .97). This suggests that language proficiency (or lack thereof) has no significant influence on the fatigue they experience due to task ordering for either task.
Thirdly, we looked at time in relation to the two different interfaces. Participants spent more time searching on the focused task using the multi-stage interface (0:16:01.67) than using the baseline interface (0:12:31.45), and the difference is significant according to independent-samples t-test (t(190) = 2.35, p < .05). Likewise, on the open task participants spent more time searching using the multistage interface (0:12:05.35) than using the baseline interface (0:07:16.10). This difference was statistically significant according to an independent-samples t-test (t(120.973) = 2.57, p < .05), with equal variances not assumed (F = 7.14, p < .01). In general, the results show that participants spent more time searching the multi-stage interface than with the baseline interface probably because the multistage interface is more complex.
To examine whether there were any interaction effects between language proficiency and the interface used by the participant, we ran a factorial ANOVA. For the focused task, the two main effects—native vs. non-native and interface—were not qualified by a significant interaction between the two factors (F (1,188) = 0.07, p = .80), indicating that the interface effects were the same for the two language conditions. For the open task, there was no significant interaction either (F (1,188) = 1.20, p = .27). This suggests that language proficiency (or lack thereof) did not influence how long people spent using the two different interfaces. 3.2
In both interfaces it was possible to issue queries by typing keywords into the search box. It is not unlikely that a lower proficiency in English would cause searchers to have to try multiple queries and query reformulations to achieve the same results as native speakers. While non-native speakers do formulate more queries on average (M = 12.17) than native speakers (M = 10.88), this difference was not significant according to an independent-samples t-test (t(190) = 1.08, p = .28). In contrast, one could expect native speakers to be better able to formulate longer, more precise queries due to their increased command of English. Native speakers did submit longer queries on average (M = 1.91) than non-native speakers (M = 1.85), although this difference was not significant either according to an independentsamples t-test (t(190) = 0.53, p = .59).
Another element of search behavior, the number of search results inspected, was analyzed as well to investigate whether language proficiency has an effect on how deep searchers dive into the search results. Due to the formatting of the search logs, we converted the results page numbers viewed to the number of results shown on those pages. Here, we made the assumption that every result on a viewed result page was judged for relevance. The average number of inspected results for each user was extracted from the log-data. Results show no significant difference between native (M = 29.37) and non-native (M = 27.43) speakers of English in the number of results inspected according to an independent-samples t-test (t(190) = 0.57, p = .57). 3.3
After participants had completed both search tasks, they were asked to complete an
engagement scale [
Finally, we looked at whether native and non-native speakers used the book-bag functionality differently. In the open, non-goal task participants were asked to add interesting books to the book-bag and add a note explaining why they selected each of the books. In the focused, goal-oriented task participants were asked to select a book for each of five sub-tasks and add an explanatory note. The results showed no significant difference between native (M = 8.08) and non-native (M = 8.34) speakers of English in the number of books added to their book bag according to an independent-samples t-test (t(190) = 0.47, p = .64). There was no significant difference between native (M = 1.76) and non-native (M = 2.81) speakers of English in the number of notes written according to an independent-samples t-test (t(190) = 1.73, p = .09). Similarly, looking at the length of notes there was no significant difference between native (M = 5.59) and non-native (M = 5.63) speakers of English in the number of words in notes written according to an independent-samples t-test (t(190) = 0.24, p = .98). We might expect non-native speakers to be more reluctant to add (lengthy) notes, because they would be less comfortable adding notes in a second language. However, this is not the case. On the contrary: it seems as if the non-native speakers are more active in both adding books and explanatory notes to the book-bag, but the differences are not significant. 4
This paper presents the preliminary results on differences in search behaviour
between native and non-native English speakers in the context of interactive social
book search. Earlier studies [
We would like to thank Birger Larsen for helping recruit test participants.