We report work in progress for visualizing and quantifying learning during search. Users initiate a search session with a Pre-Search Knowledge state. During search, they undergo a change in knowledge. Upon conclusion, users attain a PostSearch Knowledge state. We attempt to measure this dynamic knowledge-change from a stationary reference point: Expert Knowledge on the search topic Using word-embeddings of searchers' written summaries, we show that w.r.t. Expert Knowledge, there is observable and quantifiable diference between the Pre-Search knowledge (Pre-Exp distance) and Post-Search knowledge (Post-Exp distance).
KPrneo-wSeleadrcghe tual multiple choice questions (MCQs). The answer
options can be a mixture of fact-based responses (TRUE,
FALSE, or I DON’T KNOW ), [3, 4] or recall-based
responses (I remember / don’t remember seeing this
information) [
PKonsot-wSeleadrgche Post‒Exp Distance EKxnpoewrltedge bHyowgeuveesrs,wMoCrkQ.s aTlhloewtrheisrpdonadpepnrtosatochanlsewtsersceoarrrcehcetrlys
write natural language summaries or short answers,
Figure 1: Conceptual framework of Search-as-Learning. before and after the search [
Gwizdka) orcid: 0000-0001-7864-7726 (N. Bhattacharya); 0000-0003-2273-3996 (J. Gwizdka)
© 2020 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmUmoRns WLiceonrsekAsthtriobuptioPnr4o.0cIneteerdnaitniognasl ((CCC EBYU4R.0)-.WS.org)
Think of what you already know on the topic of this search and list as many phrases or words as you can that come to your mind. For example, if you know about side effects, please do not just type the phrase “side effects” ,but rather type “side effects” and then list the specific side effects you know about. Please list only one word or phrase per line and end each line with a comma.
Now that you have completed this search task, think of the information that you found and list as many words or phrases as you can on the topic of the search task. This will be short ANSWERS to the search questions.
For example, if you were searching for side effects, please do not just type the phrase “side effects”, but rather type “side effects” and then list the specific side effects you found. Please list only one word (or phrase) per line and end each line with a comma. angular distance( , ) = arccos
(1)
⋅
( ‖ ‖ ‖ ‖ )
/
mentally showed better results than average-pooling. Search, and Expert Knowledge (Fig. 1). word2vec
contains 300 dimensional vectors for about 100 billion
words (tokens) from the Google News dataset, and is
2. Experimental Design claimed to be the most stable word-embedding [13].
GloVe ofers multiple pre-trained word embeddings;
We analyze data from the user-study reported in [
Results
imum) if one of the input vectors was a zero vector.
This makes sense because zero vectors are obtained We hypothesize that participants’ learning during search only if participants’ responses do not contain any signs can be assessed from the ‘diference’ in their Pre-Search of knowledge (e.g., “none” or “i dont know”). and Post-Search responses. Since diferent participants To visualize the high-dimensional vectors of various may have diferent initial and final knowledge states, knowledge states, we employed the t-SNE algorithm. we measured it from a stationary reference-point: the This algorithm projects a set of high-dimensional obexpert knowledge. Calculating such diferences be- jects on a 2D plane in such a way that similar objects tween pieces of natural language texts is challenging, are modelled by nearby points, and dissimilar objects and is an active research topic. Word embedding is a are modelled by distant points. Using this algorithm, popular method of computing semantic similarity (or we obtained 2D representations of the Pre-Search, Postdistances) between two pieces of natural language texts. Search, and Expert Knowledges (Fig. 3, left column). A word embedding algorithm produces a numeric, high- The visualization shows an almost clear separation dimensional vector for each word, which is assumed between the Pre-Search (red circle) and Post-Search to encapsulate the ‘meaning’ of the word. In this work, (green square) knowledge states, with Expert Knowlwe leverage two popular pre-trained word-embedding edge (blue star) residing near the Post-Search knowlmodels: word2vec [11], and GloVe [12], to compute edge states. This is a visual confirmation and support ‘diferences’ or ‘distances’ between Pre-Search, Post- to the hypothesis that participants gain knowledge durPre-search Knowledge Embedding Post-search Knowledge Embedding
Expert Knowledge Embedding
Participant-Task Pair Pre – Exp Distance Post – Exp Distance
Participant-Task Pair Pre – Exp Distance Post – Exp Distance
[0 = min distance, 1= max distance] ing search, and move ‘closer’ to the Expert Knowledge the sum of the positive diference ranks (Σ +) and the state at the end of a search. sum of the negative diference ranks (Σ −). Since Σ −
The Euclidean and Angular distances between Pre- was greater than Σ + in all the tests, the diference beSearch and Expert (Pre-Exp distance), and Post-Search tween Pre-Exp and Post-Exp distances is negative. This and Expert (Post-Exp distance), are shown in the mid- means that the majority of participants had lower Postdle and right columns, respectively, in Fig. 3. For both Exp distance than Pre-Exp distance (i.e. they moved distance metrics, the majority of the participants have closer to expert knowledge at the end of the task). The lower Post-Exp distances than Pre-Exp distances (i.e. magnitude of a phenomenon is measured by efect size, their Post-Search response is less distant, or more simi- which ranges from 0 (no efect) to 1 (maximum efect). lar to, Expert Knowledge). These metrics were calcu- All the tests had efect sizes greater than 0.8, signifying lated between the high dimensional embedding vectors, that searching online had a strong efect on minimizing which supports the fact that the 2D visualizations (left the distance between participants’ knowledge level and column) showing the clear separation between Pre- and expert knowledge.
Post-Search Knowledge levels is not merely by random chance. Interestingly, for few participants, the Post-Exp distance was higher than the Pre-Exp distance. This 4. Conclusion and Future Work possibly demonstrates a ‘loss’ in knowledge level: users were closer to Expert Knowledge before the search, and We showed that word embeddings have promise for moved away from Expert Knowledge after the search. visualizing and quantifying vocabulary-based learn
We further tested whether these visual diferences ing during search. Clear separation between user’s between Pre-Exp and Post-Exp distances were statis- Pre-Search and Post-Search knowledge states was seen tically significant. Since the distance values were not and measured using simple distance metrics. Possinormally distributed, we employed the non-parametric ble future directions include predicting these learning Wilcoxon Signed-Rank test, which is used for compar- metrics from search-interactions measures. Another ing paired or related samples. (Tbh)[e3r6e]sults are presented direction is to experiment with contextual embeddings in Table 1. We can see that across diferent c(hdo)iecyees-torafckin(ge.[g1.3,5B]ERT). We also plan to investigate individual difword(ce)meybee-dtrdaicnkgins,gt[h1e0r3e] were significant diferences ferences in learning during search. between the Pre-Exp and Post-Exp distances. Thus, the results are not due to choice of particular word em- 4.0.1. Acknowledgements bedding models. The directionalities of the diferences in the Wilcoxon Signed-Rank test are expressed using
Wilcoxon SR Test all tests significant at p < .05 ΣR+ = 20.0, ΣR– = 1205.0 word2vec 95% CI: -2.76 to -1.82 Effect Size: 0.84 GloVe 6B 50d 8.678(.±226.39) 5.124(.±681.29) 9ΣE5Rff%e+ctC=SI:3iz-74e..0:0,03.8tΣo2R-2–.4=81188.0 GloVe 6B 100d 9.348(.±926.55) 5.465(.±171.42) 9ΣE5Rff%e+ctC=SI:3iz-04e..0:4,06.8tΣo3R-2–.7=91195.0 GloVe 6B 300d 12.1151(.±937.18) 7.206(.±811.72) 9ΣE5Rff%e+ctC=SI:2iz-95e..0:7,09.8tΣo3R-3–.6=51196.0 GloVe 42B 300d 12.1171(.±734.10) 7.096(.±661.80) 9ΣE5Rff%e+ctC=SI:2iz-95e..0:9,02.8tΣo3R-3–.7=91196.0 13.24 (±3.16) ΣR+ = 28.0, ΣR– = 1197.0 coGnloVsuel8t4a0nBt3f0o0rd exper1t2-.6v9ocabula8r.3y67(.c±711r.7e9a)tio9n5;%aCnI: d-5.6Y7 iton-g3.-48 long Zhang, for contributing to experimenEftfeaclt Sdizaet:a0.8c3ollection. The research was partially funded by IMLS Award #RE-04-11-0062-11 to Jacek Gwizdka.