Search engines are normally not designed to support human learning intents and processes. The eld of Search as Learning (SAL) aims to investigate the characteristics of a successful Web search with a learning purpose. In this paper, we analyze the impact of text complexity of Web pages on predicting knowledge gain during a search session. For this purpose, we conduct an experimental case study and investigate the in uence of several text-based features and classi ers on the prediction task. We build upon data from a study of related work, where 104 participants were given the task to learn about the formation of lightning and thunder through Web search. We perform an extensive evaluation based on a state-of-the-art approach and extend it with additional features related to textual complexity of Web pages. In contrast to prior work, we perform a systematic search for optimal hyperparameters and show the possible in uence of feature selection strategies on the knowledge gain prediction. When using the new set of features, state-of-the-art results are noticeably improved. The results indicate that text complexity of Web pages could be an important feature resource for knowledge gain prediction.
ing was that the time spent on text-based Web pages had a greater impact on knowledge gain than time spent Conventional information retrieval systems are usually on video-based Web pages. Gadiraju et al. [7] explored designed to satisfy an information need. The research the in uence of behavioral features on the learning outarea Search as Learning (SAL), on the other hand, deals come, and found a positive correlation between the avwith the assumption that search sessions can also be erage complexity of user queries and their knowledge driven by a learning intention. Research in the area of gain. Recently, some approaches have been suggested SAL is not only concerned with the ranking of search that combine several types of features [8, 9]. For examresults, but also with the detection or prediction of ple, Otto et al. [9] studied the e ect on knowledge gain the learning intention or even the knowledge state and prediction, when complexity and linguistic features are knowledge gain [1, 2]. complemented with multimedia features. They achieved
Vakkari [3] presented a survey of features which indi- slight improvements by adding multimedia features, e.g., cate the user’s knowledge and learning needs, but also representing the amount of image and video data on the knowledge gain during the search process. More recently, screen or the image type (infographics, outdoor photoga wide variety of features were considered, including raphy, etc.). resource-based (based on text or multimedia content) A crucial aspect of learning is the appropriateness or behavioral features. For example, Syed and Collins- of the text for the reader. In his survey, CollinsThompson [4] have considered document retrieval fea- Thompson [10] has summarized studies that deal with tures to improve learning outcome for short- and long- the automatic assessment of the reading di culty of term vocabulary learning. Collins-Thompson et al. [5], texts. Hancke [11] has previously analyzed lexical, synon the other hand, have studied di erent query types tactic, and morphological features for German, while and found a correlation between the variety of intrinsic Kurdi et al. [12] investigated features that allow for conquery types and knowledge gain. Pardi et al. [6] further clusions about the complexity of English texts. examined the time spent on Web pages with primarily In this paper, we investigate the in uence of text comtextual or video content and learning outcome. One nd- plexity of Web pages on knowledge gain prediction in a comprehensive experimental case study. For this purpose, we present a large set of text-based features of various types and, furthermore, analyze the impact of di erent classi ers and feature selection strategies on knowledge gain prediction. First, the experimental results show that state-of-the-art results [9] can be signi cantly improved and, second, that the textual complexity of Web pages can be a valuable predictor for the classi cation of knowledge gain. Our contributions can be summarized as follows:
To measure knowledge gain, the participants were asked • A large set of features describing textual complex- to solve a 10-item multiple choice test one week beity of Web pages is presented. fore (t1) and immediately after (t2) the Web search. The • We conduct an extensive, systematic evaluation knowledge gain is subsequently de ned as the di erence including multiple classi ers, hyperparameter between the numbers of correct answers of t2 and t1. The analysis and optimization, as well as feature se- potential range of values for the knowledge gain is therelection strategies and analyze their impact on fore [− 10, 10]. The average value in t1 was 5.24 ± 1.80 knowledge gain prediction. respectively 7.46 ± 1.43 in t2. The average knowledge • We demonstrate that the state-of-the-art-results gain was 2.22 ± 1.78 and lies in the range of [− 3, 6]. can be improved, even when only considering textual complexity features. 2.3. Feature Extraction The remainder of this paper is structured as follows: In In the study, the participants performed free Web Section 2 the experimental setup and the process of ex- searches, such that realistic search and browsing behavtraction is described. Experimental results are reported ior could be recorded. Since we focus on the textual in Section 3 and the impact of text complexity features is complexity of the visited pages, other page types like analyzed. Finally, a summary of the main results and an search engine result pages and video-based contents are outlook is given in section 4. ltered. For this purpose, we used a keyword-based approach and omitted pages which contained the following keyterms in their URL: "google.", "youtu", "ecosia", "RDSIn2. Experimental Setup and dex","universitaetsbibliothek", "meteoros", "webcam" and Text-based Features "learningsnacks". For all remaining pages, we extracted all displayed text without further processing. This can We use data from a study [13] in which participants were lead to the fact that e.g., tables or advertisements are asked to acquire knowledge about the formation of thun- in the analyzed texts. We decided against any further der and lightning. The topic has already proven useful in preprocessing in order to minimize the bias in the data previous work [14, 15]. It is a phenomenon that is gen- set. erally known and requires both factual and procedural knowledge. On the Web, many sources exist on the sub- 2.4. Website Features ject, explaining it in diverse ways (texts, graphics, videos, etc.). The participants were asked to do a Web search for To assess the complexity of text on Web pages, we extract a maximum of 30 minutes; but were allowed to end the eight di erent types of features: search earlier if they felt they had learned everything important. We could use data from = 104 participants (88 female, 16 male, average age of 22.7 ± 2.7 years), for which the visited Web pages were downloaded during the experiment. The participants were recruited over a local recruitment portal composed of students from the University of Tübingen. Students were compensated with 16e per person for participating in the study. None of the participants had former expertise in meteorology. • syntactical features • readability scores • part of speech (POS) density • lexical richness • lexical variation • lexical sophistication • syntactic constituents features • connectives 2.1. Technical Setup of the Study Since the study was conducted in German, we mainly rely on the Common Text Analysis Platform (CTAP) tool [16], While plenty of data were collected during the study which currently provides 218 di erent complexity fea(data sources such as eye and mouse tracking informa- tures for the German language. In total, we extract 248 tion), here, we focus on the text content of the visited Web features from each Web page. Below we give a short depages. During the Web search, all visited Web pages of scription of each feature group. For a complete overview the participants were tracked and recorded via the "Scrap- consider the appendix3. bookX" (1.5.14)1 and "ScrapbookXAutosave" (1.4.3)2 plu- The syntactic features group consists of basic text gins. statistics such as the number of letters, syllables, words, and sentences. Moreover, the average length of each
2https://github.com/danny0838/ refox-scrapbook-autosave
element is considered, like sentence length in letters or
word length in syllables, as well as the standard deviation. phrases per T-unit, but also words per T-unit or noun
in the text based on Kurdi [12]’s observation that there The second group of features consists of well-known may be a connection between more di cult texts and readability scores that aim to estimate the skills a reader more complex tenses. To extract the tenses, we use the must have to understand the text. The features are based tool of Dönicke [21]. on combinations of the syntactic features (automated The last group
Connectives (according to readability index (ARI), Coleman-Liau index, Flesch
subsets of de ned parts of speech like conjunctions three or more syllables (SMOG index). For example, the (and, or, etc.) or adverbs (in contrast, therefore, etc.). formula for ARI is as follows: ARI = 4.71 · |characters| + 0.5 · |sentences| − 21.43 |words| |words|
are calculated for each Web page visited during the search numerical value on a scale of 1-14 (1: Kindergarten, 14: sessions. Since the participants accessed a di erent num
The POS density group re ects the density of di erent word types like adjectives or verbs in the website text. It is based on the tokenization of the text and calculates the di erent number of word types (e.g., adjectives or verbs) in relation to all tokens, e.g., densityadjectives = |adjectives| |tokens|
the number of non-duplicated tokens is set in relation to all tokens. In addition to the fraction ttoykpeenss , various variations such as the logarithm or square root are applied to the numerator and denominator.
The lexical variation group examines the subset of lexical words (LW) consisting of nouns, verbs, adjectives and adverbs. The class puts the number of individual components in relation to the number of lexical words, e.g., the lexical variation lv_adjectives for adjectives: lv_adjectives = |adjectives| |LW|
ferent frequency lists [18, 19]. All words of the Web page text are assigned to sets of all words AW, lexical words
jectives and adverbs) and functional words FW (i.e., not
quency lists (per million words) of AW, LW and FW is consequently used as a feature. Furthermore, the Karlsruhe Childrens Text (KCT) [20] list is used to determine the average and minimum age of active use of AW, LW and FW.
that determine the number of di erent syntactic constituents, like noun phrases, relative clauses or T-units. = 104 records for our analysis. for knowledge gain prediction. ber of Web pages, we compute the average, the minimum and the maximum for each feature for each participant.
As a result, we obtain a total of 3 · 248 = 744 features
In this section, we report results for knowledge gain prediction using features for text complexity. For a fair comparison, we use the same evaluation setting including hyperparameter optimization for all experiments. In the same way, we replicate the results of Otto et al. [9]4 with our evaluation procedure. 3.1. Knowledge Gain Definition
common approach [7, 8, 9] to assign each search session to one of three classes = {, , ℎ} based on the Standard Deviation Classi cation approach. is z-normalized (^) according to equation 1.
^ = − (1)
of all knowledge gain measures . Then, for every znormalized knowledge gain ^ the class is assigned as follows: () :=
Moderate, if − ⎨ ⎧ Low, ⎩ High, if ^ < − 2
1 if ^ > 21
^ 1 1 2 ≤ ≤ 2
logging led to missing HTML data for nine participants which were crawled at a later date. We rely on the data crawled during the original experiment, leading to Normalization Feature Selection
(optional)
ture of the 80% to the interval [0, 1]. This is an essential step for some of the classi ers, e.g., Support Vector Machine. The 20% test set is then normalized by the minimum and maximum of the 80% for evaluation. It is possible that the values lie outside the interval of [0, 1]. However, we decide against clipping in order to not lose any information due to normalization. Figure 1 provides an overview of our proposed evaluation. In our evaluation we use the implementation of Scikit-learn [23]. 3.3.1. Hyperparameter Optimization 3.2. Metrics
Figure 1: Overview of our evaluation method. A 5-fold cross- depends on the chosen hyperparameters. However, since validation is performed and for each split the features are the training, validation and test data change in each iterafirst normalized, optionally selected/reduced and the hyper- tion due to cross-validation, these cannot be determined parameters of the respective classifier are optimized on the once and used for the entire evaluation. Therefore, to 80% train/validation data. The test data are scaled with the obtain valid results, we perform an optimization of the minimum and maximum of the train/validation data and op- hyperparameters in each of the ve iterations. We utilize tionally the features are filtered. Finally, the classifier opti- Optuna [24] for an Bayesian search to e ciently nd a mized on the train and validation data is used to predict the good con guration and limit the number of runs to 500 knowledge gain on the test data set. to reduce the computational cost. From the 80% of the data coming from the 80:20 split of the cross-validation, another 80:20 split is performed, where 80% is training data and 20% is validation data. We set the maximization This yields the following class distribution: |XLow|=40, of the weighted F1 score as the optimization objective. |XModerate|=39, |XHigh|=25. This is to prevent the class imbalance from making the underrepresented class High less important, as it would be, for example, with overall accuracy. 3.3.2. Feature Selection
recall, 1 score, and accuracy. These are de ned as follows:
precision = TP (2) of input features (more is not always better). For example, TP + FP in the Random Forest algorithm, a subset of the features
TP is selected several times to create weak classi ers and recall = TP + FN (3) there is no guarantee that "good features" will prevail.
precision · recall For this reason, we want to reduce the number of feaF1 score = 2 · precision + recall (4) tures while trying to preserve valuable features. Again, it is important to separate the feature selection from the accuracy = TP + TN (5) test data, which changes in each iteration. As with hy
TP + TN + FP + FN perparameter optimization, we use the further split into where TP are the values correctly classi ed as positive, training and validation data to do this. It follows that the TN are the values correctly classi ed as negative, and FP selected features may change in each iteration. For the are the values incorrectly classi ed as positive and FN selection of the features to be used for this evaluation, are the values incorrectly classi ed as negative. we rely on two strategies: 3.3. Experimental Setup Cross-validation is a good way to evaluate the classi cation result, since every feature vector acts as a test sample in one fold. We thus choose a 5-fold cross-validation with 80% train/validation and 20% test set split. This results in ve elements per class in each test set in each iteration of the cross-validation. 1. 2-based Feature Selection: This method examines whether a feature has a statistically significant relationship to knowledge gain. While one feature is analyzed for a relationship, all other features are ignored. The features with the highest values based on the 2-test are selected. 2. Tree-based Feature Selection: Features without a direct correlation to the knowledge gain 3.3.3. Classifiers
K-Nearest Neighbors [28], Multi-layer Perceptron [29], and Support Vector Machine [30]. The objective is to experimentally determine the best con guration in order to nd the maximum potential for knowledge gain prediction, given the set of features.
can be important predictors in combination with 3.4. Classifier Performance other features. For this reason, we employ a treebased approach using a Random Forest classi er. In Table 1, we compare the performance for all classi ers. This is tted to the training data and then ana- As baselines, we list the results for weighted guessing lyzed to see which features were most heavily (WG), which is the mean of each metric for 10,000 ranused in the decision. The values with the high- domly generated vectors consisting of class labels with est importance are selected. The goal is to select respect to the class distribution, and the original reported valuable features for the classi cation even with- results from Otto et al. [9] (Otto*). For a fair compariout direct correlation. son with our features, we reproduced the results using the features from Otto et al. [9] with our pipeline (Otto).
Furthermore, to analyze the performance for a feature set as diverse as possible, we combined the features of Otto et al. [9], and our proposed feature set for evaluation (Otto+our). For the cumulative predictions for all
ve iterations of cross-validation, the precision, recall, and F1 score are calculated for each class (Low, Moderate, and High), as well as the average of these metrics over all classes, and the overall accuracy.
First, it is notable that the reproduced results of Otto et al. [9] (Otto) are better compared to their reported result (Otto*). The results of the Multi-layer Perceptron (MLP) provide a 5.9% higher F1 score (34.9% compared to 40.8%). However, in direct comparison to the reproduced result with a Random Forest (RF), the original Table 2 results are better. It is striking, that the improved out- The optimized hyperparameters per fold 1, ...5 for the come stems mainly from better predictions from the class Random Forest classifier for our features. High. A closer look reveals that the recall scores for the 1 2 3 4 5 tree-based classi ers Adaboost (Ada), Decision Tree (DT) and Random Forest (RF) are comparatively low. These estimators 242 299 154 150 223 algorithms seem to preferentially predict the more repre- mmaaxx__fdeeapttuhres s2q2rt lo1g72 sq8rt lo1g72 s1q7rt sented classes for the features of Otto et al. [9] and accept criterion entr. gini sqrt entr. gini a worse result for the underrepresented class High. This min_n_split 6 3 7 7 4 impression is enforced by the fact that for all feature min_n_leaf 5 8 3 8 7 sets the F1 score (f1) for the three classi ers is signi cantly worse for the class High than for the classes Low and Moderate. This is not the case for any of the other classi ers. hyperparameters for each fold 1, ..., 5 are shown in
Nevertheless, Random Forest (RF) and Adaboost (Ada) Table 2. No pattern can be discovered in the parameters, perform best for the other feature sets (our and Otto+our). they are very di erent in shape. This could possibly be The RF using the features of textual complexity (our) related to the heterogeneity of the data and the weakness yields a slightly better macro F1 score (42.0%) than the of the features for prediction.
MLP using the features of Otto (41.2%). In addition, the RF achieves an overall accuracy of 43.3% while the MLP only achieves 40.6%. The best result is obtained by the 3.5. Feature Selection Adaboost classi er for Otto+our with 45.2% macro F1 In Table 1, it is observable that the classi cation result score and 46.2% overall accuracy. Examining the results for the Random Forest classi er (RF) performs worse for the Random Forest algorithm for all three feature sets, for the combination of features (Otto+our) than for the we notice that the F1 scores of all three classes for the complexity-only features (our). It seems that considcombination of features are strictly between the F1 scores ering more features does not necessarily improve the of the individual feature sets. At the same time, the F1 classi cation quality. The result for the Random Forest scores for the combination of features are all better than classi er (RF) for the textual complexity (our) features for the individual sets for Adaboost. We assume that for ∈ {1, 3, 5, ..., 99} is shown in Figure 2. It can the Random Forest algorithm is a ected by too many be seen, that the classi cation result is achieved with (diverse) features. Adaboost can weight the features dif- fewer features, regardless of the feature selection stratferently and thus utilize the strengths of both feature egy. With the 2-based selection method, the result is sets. also achieved with fewer features, but later than with
Another observation is that the F1 scores of all feature the tree-based method. This makes sense in so far as the sets for the K-Nearest Neighbors (KNN) algorithm are signi cantly higher for the class Moderate than for the sctlarastseegsiLesowwiathndLoHwig(ho.r THhigehre)fkonreo,wwleedsguesgpaeicnt dthiaetrsmeaurcchh 50 Our, Random Forest chi2 tmhoercel.aFsusrMthoedremraotree,owf eoucranfeoabtuserrevseisthhaitgthhecoFm1spcaorreedfotor 45 tarlelefeatures the classes Low and High, independent of the classi er. % On closer inspection, we found that often instances of in40 the class Low are classi ed as High and vice versa. If rcoe we put the classi cation result for the classes Low and FS135 High together, i.e., a new class Not Moderate, we would vag get 74.1%, 70.8% and 73.1% F1 score for the classi ers 30 MLP, RF and SVM, respectively, for this new class. It seems like the complexity features are useful to detect if someone does not have a Moderate increase in knowledge 25 0 20 40 60 80 gain. We plan to investigate this interesting aspect in the Number of Selected Features future. Figure 2: Average F1 scores of the Random Forest classifier
For our textual complexity features, the best result was using ∈ {1, 3, 5, ..., 99} of our features for the 2-based obtained with the Random Forest classi er. In each itera- (chi2) and the tree-based (tree) Feature Selection strategy. tion of the 5-fold cross-validation, an independent hyper- The result for all features is indicated with the dotted line. parameter optimization was performed. The optimized 2-based method considers the features independently of each other, and only measures the individual correla- 50 Otto, Random Forest chi2 tion of a feature with knowledge gain. In contrast, the tree tree-based strategy selects features based on their impor- in%40 all features tance for an upstream Random Forest. Thus, the baseline ihg level can already be reached with = 19 features. ssH30
Cross-validation is used for evaluation as described lca above (Section 3.3.1). Similarly, feature selection is per- fro formed ve times. However, this implies that the features rcoe20 chosen in each iteration of the cross-validation may di er, FS1 which complicates the analysis of which features most vag10 in uence the classi cation result. We therefore propose to highlight the features that were selected in at least 0 0 10 20 30 40 50 60 70 three out of ve iterations. Since the classi cation result Number of Selected Features of the Random Forest was already achieved with = 17 Figure 3: F1 scores for the class High for the features of features, we report the features based on this con gu- Otto et al. [9] for ∈ {1, 3, 5, ..., 79} features for the ration. The features and their frequencies are shown in 2-based (chi2) and the tree-based (tree) Feature Selection Table 3. Three features were selected at least three times, strategy. The result for all features is indicated with the dotbut none were selected in every iteration of the cross- ted line. validation. All three were aggregated by the minimum, indicating that the Web page with the lowest textual complexity is most important for the classi cation result. 4. Conclusions This strengthens the impression that the features or the aggregations (Minimum, Maximum and Average) are too In this paper, we have investigated the impact of textual weak to provide a strong prediction of the knowledge complexity of Web pages on knowledge gain during a gain. In the future, we aim to include more features and Web search. The experimental results demonstrated that nd aggregations that are more suitable to re ect search the state of the art can be improved by only considering patterns. the textual complexity of Web pages. The results also
In the last section, it was observed that the F1 score showed that a systematic assessment of di erent hyperfor the class High is signi cantly below the values for parameter settings, feature selection, and several classithe classes Low and Moderate, regardless of the feature ers is important – in particular, since the correlations set. We performed feature selection before hyperpa- between features and the target outcome are relatively rameter optimization and repeated the evaluation with weak. During the evaluation, it became apparent that ∈ {1, 3, 5, ..., 79} features. Figure 3 shows how the as little as 17 features per iteration of cross-validation F1 score for the class High changes with a subset of the would have been su cient to achieve the result. Furtherfeatures of Otto et al. [9]. The green curve describes the more, we found that a moderate knowledge gain can be F1 scores based on the tree-based feature selection strat- predicted relatively well, but, interestingly, the distincegy, which tries to select the most important features tion between successful and unsuccessful Web search for classi cation. It is noticeable that almost any tested does not work well (in terms of knowledge gain). The subset would have been more suitable than using the full reasons for this e ect have to be investigated in more feature set. Moreover, the curve does not change from detail. = 65 onward (same observation for the classes Low Although we have obtained state-of-the-art results, and Moderate), which suggests that the tree-based fea- there are some limitations. In this case study, we anature selection strategy does not consider many features lyzed only the data of a study on knowledge acquisition at all. about a speci c science topic, the formation of thunder
Part of this work is nancially supported by the Leibniz Association, Germany (Leibniz Competition 2018, funding line "Collaborative Excellence", project SALIENT [K68/2017]). storms. Consequently, limited conclusions can be drawn about general Web searches and the results need to be con rmed or extended by future studies. In this sense, the reported results need to be reproduced for (a) di erent types of learning tasks (e.g., procedural knowledge) and (b) conceptual learning tasks in other domains (e.g., non-science topics).
In the future, we would like to deepen our understanding of what behavioral patterns characterize e ective Web searches, for instance, by examining how the sequence of Web pages (and their characteristics) in uence learning success. An intuitive assumption is, for example, that a successful learning session consists of Web pages of increasing complexity. Furthermore, we have considered the textual complexity of the entire Web page, but not in every case is the Web page content read in its entirety. In future work we would like to focus more on the actual seen during Web search.
Lastly, we focused on text-based Web pages in this case study. However, many of the Web searches were not unimodal but multimodal. Consequently, further investigations will need to include further complexity measures such as visual complexity of the Web pages or videos. 135. URL: https://www.jbe-platform.com/content/ ence Abstracts, Stanford University Library, 2011, journals/10.1075/itl.165.2.01col. doi:https://doi. p. 8. URL: http://xtf-prod.stanford.edu/xtf/view? org/10.1075/itl.165.2.01col. docId=tei/ab-003.xml. [11] J. Hancke, S. Vajjala, D. Meurers, Readability classi- [20] R. Lavalley, K. Berkling, S. Stüker, Preparing chilcation for german using lexical, syntactic, and dren’s writing database for automated processmorphological features, in: M. Kay, C. Boitet ing, in: K. M. Berkling (Ed.), Language Teach(Eds.), COLING 2012, 24th International Confer- ing, Learning and Technology, Satellite Workence on Computational Linguistics, Proceedings shop of SLaTE-2015, LTLT@SLaTE 2015, Leipzig, of the Conference: Technical Papers, 8-15 Decem- Germany, September 4, 2015, ISCA, 2015, pp. 9– ber 2012, Mumbai, India, Indian Institute of Tech- 15. URL: http://www.isca-speech.org/archive/ltlt_ nology Bombay, 2012, pp. 1063–1080. URL: https: 2015/lt15_009.html.
//aclanthology.org/C12-1065/. [21] T. Dönicke, Clause-level tense, mood, voice and [12] M. Kurdi, Lexical and syntactic features selection modality tagging for german, in: K. Evang, for an adaptive reading recommendation system L. Kallmeyer, R. Ehren, S. Petitjean, E. Sey arth, based on text complexity, in: ICISDM ’17, 2017. D. Seddah (Eds.), Proceedings of the 19th Interna[13] J. von Hoyer, G. Pardi, Y. Kammerer, P. Holtz, tional Workshop on Treebanks and Linguistic TheoMetacognitive judgments in searching as learning ries, TLT 2020, Düsseldorf, Germany, October 27-28, (sal) tasks: Insights on (mis-) calibration, multime- 2020, Association for Computational Linguistics, dia usage, and con dence, in: Proceedings of the 2020, pp. 1–17. URL: https://doi.org/10.18653/v1/ 1st International Workshop on Search as Learning 2020.tlt-1.1. doi:10.18653/v1/2020.tlt-1.1. with Multimedia Information, SALMM ’19, Associa- [22] E. Breindl, A. Volodina, U. H. Waßner, Handtion for Computing Machinery, New York, NY, USA, buch der deutschen Konnektoren 2, De Gruyter, 2019, p. 3–10. doi:10.1145/3347451.3356730. 2014. URL: https://doi.org/10.1515/9783110341447. [14] R. Mayer, R. Moreno, A split-attention e ect in doi:doi:10.1515/9783110341447. multimedia learning: Evidence for dual processing [23] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, systems in working memory, Journal of Educational B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, Psychology 90 (1998) 312–320. R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, [15] F. Schmidt-Weigand, K. Scheiter, The role of spatial D. Cournapeau, M. Brucher, M. Perrot, E. Duchdescriptions in learning from multimedia, Com- esnay, Scikit-learn: Machine learning in Python, put. Hum. Behav. 27 (2011) 22–28. URL: https:// Journal of Machine Learning Research 12 (2011) doi.org/10.1016/j.chb.2010.05.007. doi:10.1016/j. 2825–2830.
chb.2010.05.007. [24] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, [16] X. Chen, D. Meurers, CTAP: A web-based tool Optuna: A next-generation hyperparameter optisupporting automatic complexity analysis, in: mization framework, in: A. Teredesai, V. Kumar, D. Brunato, F. Dell’Orletta, G. Venturi, T. François, Y. Li, R. Rosales, E. Terzi, G. Karypis (Eds.), ProP. Blache (Eds.), Proceedings of the Workshop on ceedings of the 25th ACM SIGKDD International Computational Linguistics for Linguistic Complex- Conference on Knowledge Discovery & Data Minity, CL4LC@COLING 2016, Osaka, Japan, Decem- ing, KDD 2019, Anchorage, AK, USA, August 4ber 11, 2016, The COLING 2016 Organizing Commit- 8, 2019, ACM, 2019, pp. 2623–2631. URL: https: tee, 2016, pp. 113–119. URL: https://www.aclweb. //doi.org/10.1145/3292500.3330701. doi:10.1145/ org/anthology/W16-4113/. 3292500.3330701. [17] M. Zie e, E ects of display resolution on visual [25] L. Breiman, Random forests, Mach. performance, Hum. Factors 40 (1998) 554–568. Learn. 45 (2001) 5–32. URL: https://doi.org/ URL: https://doi.org/10.1518/001872098779649355. 10.1023/A:1010933404324. doi:10.1023/A: doi:10.1518/001872098779649355. 1010933404324. [18] M. Brysbaert, M. Buchmeier, M. Conrad, A. Jacobs, [26] Y. Freund, R. E. Schapire, A decision-theoretic genJ. Bölte, A. Böhl, The word frequency e ect: a eralization of on-line learning and an application review of recent developments and implications to boosting, J. Comput. Syst. Sci. 55 (1997) 119–139. for the choice of frequency estimates in german., URL: https://doi.org/10.1006/jcss.1997.1504. doi:10.
Experimental psychology 58 5 (2011) 412–24. 1006/jcss.1997.1504. [19] E. L. Aiden, J. Michel, Culturomics: Quantitative [27] L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone, analysis of culture using millions of digitized books, Classi cation and Regression Trees, Wadsworth, in: 6th Annual International Conference of the Al- 1984. liance of Digital Humanities Organizations, DH [28] E. Fix, J. L. Hodges, Discriminatory analysis - non2011, Stanford, CA, USA, June 19-22, 2011, Confer- parametric discrimination: Consistency properties, International Statistical Review 57 (1989) 238. [29] F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms, American Journal of Psychology 76 (1963) 705. [30] C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (1995) 273–297. URL: https://doi.org/10.1007/BF00994018. doi:10.1007/ BF00994018.