We present the concept of an intelligent tutoring system which combines web search for learning purposes and state-of-theart natural language processing techniques. Our concept is described for the case of teaching information literacy, but has the potential to be applied to other courses or for independent acquisition of knowledge through web search. The concept supports both, students and teachers. Furthermore, the approach integrates issues like AI explainability, privacy of student information, assessment of the quality of retrieved information and automatic grading of student performance.
1. Motivation work and support the creation of own experiments.
On the other hand, many educational institutions alInformation literacy is a core skill for the digital age. ready conduct their courses, exercises, and examinaIn modern education and work environments it is of tions online. This means that student assessments are growing importance as knowledge work is increas- already available in digital, machine-readable form, ingly based on large and rapidly changing knowledge ofering a wide range of analysis options. Focusing sources. Search and organization of knowledge is a on information literacy, a course typically consists of constant requirement. Higher education teaches in- teaching material in text form and the course particiformation literacy sometimes in dedicated courses and pants themselves practice information skills and genoften only within another course. Studies show that erate text in online research and essays. However, the the level is low: E.g. students have dificulties in us- evaluation of free texts such as essays, references and ing operators in search terms, organize literature and research methodology still requires intensive manual tend not to know appropriate sources to find scientific work. literature. Consequently, we deal with the question which
The potential of Artificial Intelligence (AI) in higher methods of Natural Language Processing (NLP) can education still needs to be explored and innovative support coaching of information competency and how applications need to be developed. Can computers they can be applied for the teacher, and for the stusupport teaching staf in coaching information com- dent. The focus is on the combination of various deep petency? – The research area “AI in Education” ad- learning approaches to automatically help students to dresses the application and evaluation of AI meth- accelerate the learning process by automatic feedback, ods in the context of education and training. One but also to support teachers by pre-evaluating free text of the main focuses of this research is to analyze and suggesting corresponding scores or grades. and improve teaching and learning processes. On the one hand, deep learning – learning in multi-layered (“deep”) artificial neural networks – has become a cen- 2. Related Work tral component of AI research and numerous libraries or frameworks1 have been created that simplify the
the search has led to positive learning efects. In a similar fashion, providing feedback and suggestions around a search activity can support the reflection on search tools’ usage, on one’s information needs and on the goals of the task at hand. In his keynote at ECIR 20202 C. Shah envisioned the next decade of research in search and recommendation where modeling 2https://ecir2020.org/keynote-speakers/ the tasks is central to raise the quality of the results. it “constitutes a composite set of knowledge, skills, Defined tasks around the learning of information liter- attitudes, competencies and practices that allow efacy are a good example of context where recommen- fectively access, analyze, critically evaluate, interpret, dations can be made more relevant to the process. In- use, create and disseminate information and media telligent Tutoring Systems (ITS) follow a long tradi- products with the use of existing means and tools on a tion of environments where AI supports learning. The creative, legal and ethical basis. It is an integral part of most widespread didactic situation where ITS has been so-called “21st century skills” or “transversal compeemployed, is as exercises where a direct feedback (e.g. tencies”” [18]. In higher education, these information in form of a score or recommendation) is ofered fol- skills are highly relevant for students. Nevertheless, lowing interactions with a dedicated system. Multiple the information literacy of students is often measured example intelligent tutoring systems exist and many as low [19]. Courses on basic scientific work cover sevfollow the model of [2]. Our research aims to observe eral domains of information literacy. Often, there is a the work of the students instead of requiring exercise strong focus on searching skills, correct citing and asspecific actions. sembling short abstracts based on scientific texts. The
State-of-the-art research and the basis for the devel- practice of teaching information literacy skills has not opment of an NLP system to coach information com- developed much towards digital formats. Some open petency include sentiment analysis [3], topic identi- online courses exist [20], but there is no use of AI tools ifcation [4], named entity recognition [5], text sum- yet. marization [6], word sense disambiguation [7] and in- An example is the ILO-MOOC (informationliterformation retrieval [8]. A major scientific challenge acy.eu). It allows to study in a self-paced manner. The is the explainability of system outputs [9]. The NLP feedback for students is show right or wrong after anmethods can be combined with knowledge graphs to swering multiple choice questions. Another example include ontology-based knowledge coding in the pro- is at IUBH University where bachelor students with cesses [10]. This can be enhanced by including visu- a diverse background are trained on the basics of scializations that represent both the inputs of the stu- entific work . While the focus of the assignments is in dent and the results of the NLP analysis. Lachner [11] the production of written texts, it involves all aspects shows that graph representations can support the un- of information literacy. The course is made for both derstanding of a topic. To represent ontologies or com- remote and on-site attendance and involves various plex topics, knowledge graphs such as [12] help to communication channels, many of them happening on identify context. the web. The resulting competencies expect an inde
For the processing in deep learning architectures, pendent and self-confident scientific work which may sequences of words are encoded into vector spaces be strongly supported by an automatic evaluation. in order to perform computations in neural networks.
Tools for text vectorization are Word2Vec , GloVe [13] or fastText [14]. The concept of the skip-thought 4. Proposed System Architecture vectors [15], universal sentence encoder (USE) and bidirectional encoder representations from transformers (BERT) [16] are methods also supporting sentence embeddings in the semantic vector space. [17] investigates and compares state-of-the-art deep learning techniques for automatic short answer grading.
Their experiments demonstrate that systems based on BERT [16] performed best for English and German. On their German data set they report a Mean Average Error of 1.2 points, i.e. 31% of the student answers are correctly graded and in 40% the system deviates by 1 out of 10 points.
Our concept proposes an integration within the web activities of the learner attending an assignment task which includes searching, reading, evaluating, and writing: Using JavaScript or web extensions, the text and timestamps of the search results, of the viewed publications, and of the input text can be used as features for the NLP models. Based on the assignment’s objective, the feature vectors generated from the student’s behavior and text is processed by our NLP models. The models were trained by annotated text data from previous course members (model solution, already graded works, other annotated works) to generate textual feedback. 3. Information Literacy Courses The concept comprises several tasks for which support for students can be provided. For the sake of The term Information Literacy is often used synony- brevity we illustrate two of them: mous with Media Literacy. According to the UNESCO A core task in scientific work and, thus, in teaching information literacy is web search. Students are of- retrieved by adapting the BERT fine-tuning architecten required to search for documents fulfilling certain ture to extract named entities as proposed in [23] and requirements e.g. within a closed collection of docu- [24]. The confidence score can be retrieved by mapments. An AI system observes the search terms input ping the predicted scores to classes and output a vector by the student and compares the strategies to identi- that contains a probability for each class. ifed objectives. The system tracks the actions of the Further feedback to both teachers and learners can students regarding search terms, observed documents, be given by the visualizations of individual states and headers and time spent. It then suggests the most configurations of the system. Given the exemplary appropriate steps toward reaching better results. In tasks, the trial and error processes can be shown in the example of searching in a closed collection with a diferent paths along a timeline that can be shared by pre-defined goal, suggestions for further search terms the student with a teacher to allow for better feedback. leading to relevant documents can be made.Text vec- Also the categorization and identification of correct torization and a deep learning model-based classifi- steps are to be shown to the learner using clear visucation can be used for keyword extraction [21]. As alizations to help understanding the decision making. such, the student learns in the direct interaction with The analysis of the evolving students’ work gatha search system and improves skills based on previous ers data that should not necessarily be shown to felactivities by providing automatic feedback. low students or teachers: The data is made of personal
Another exemplary task in teaching information lit- trial and error processes and is to be considered as prieracy is related to academic writing. Students are vate. Other content, e.g. from chat rooms and forums, asked to assemble a short summary and synthesis of can be considered as public. It is adequate that a bot several papers. The system supports them in analyz- can provide answers using information that all chat ing the writing, recognizing the parts in a certain pa- members have seen (e.g. lecture scripts, assignments’, per, checking whether the short summary is adequate posts). Similarly, submitted assignments’ text data can and without plagiarism. Siamese neural networks can be automatically graded based on existing information be used to detect similarities there [22]. The system such as earlier assignments or expert texts. also uses NLP to analyze the coherence of the text.
Here an AI based system also gives context-dependent 5. Conclusion and Future Work suggestions on how to improve the text. The suggestions provided can refer to documents, showing a title, We have described the architecture of an intelligent tua time when the student saw it, and a link to the docu- toring system which combines web search and natural ment as last accessed. This is efective for the learner language processing techniques on the basis of inforas the reading is kept in memory. Such support within mation competency. After implementing the system the writing process can help students more than a the- and training the machine learning models, the system oretical unit on academic writing. needs to be evaluated and optimized for students and
In our suggested AI based information literacy tu- teachers regarding usability and eficiency for which toring system, word sequences in search terms, re- several courses exist. With the help of metrics, we retrieved documents, and reference documents are en- duce the error rate for training the models. Finally, we coded into vector spaces in order to perform compu- intend to speedup the system. Throughout the impletations in deep learning architectures. A fine-tuning mentation usability tests are repeatedly performed to architecture, such as BERT, which has proven itself in ensure the quality of the proposed system. many NLP tasks, provides the basis of our system: It is We plan to apply the architecture within several based on a pre-trained deep learning model, which — courses, adjust the tasks to be automatically measursupplemented by a linear regression layer — is adapted able and annotate corpora of articles so that an auto our specific tasks, e.g. grading short summaries or tomatic evaluation yields productive feedback. Using retrieved documents from the web search, and the pa- this system, we expect to answer the following quesrameters of the embeddings are tuned accordingly. A tions: How to adequately capture the students’ activdata set with labeled and graded documents and sum- ity, select information to store and evaluate it, and how maries from old information literacy courses serves to to ofer support which is timely and relevant for the optimize the model for predicting scores. learning process.
To achieve a steeper learning curve and to guarantee explainability, we suggest two methods: (1) Highlighting keywords and (2) displaying the confidence score of the system’s output. The keywords can be
USA, 2015. [1] A. Lazonder, Do two heads search better than [16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, one? Efects of student collaboration on web BERT: Pre-training of Deep Bidirectional Transsearch behaviour and search outcomes, BJET 36 formers for Language Understanding, in: (2005) 465–475. ACL:HLT, 2019. [2] K. VanLehn, The Behavior of Tutoring Systems, [17] J. Sawatzki, T. Schlippe, M. Benner-Wickner, I. J. Artificial Intelligence in Education 16 (2006) Deep Learning Techniques for Automatic Short 227–265. Answer Grading, in: submitted to: COLING, [3] H. Liu, Sentiment Analysis of Citations Using ACL, 2020.
Word2vec, arXiv:1704.00177 (2017). [18] The Moscow Declaration on Media and Informa[4] M. Lamba, M. Margam, Metadata Tagging tion Literacy, 2020. https://iite.unesco.org/mil, of Library and Information Sciences Theses: 2020-05-02.
Shodhganga (2013-2017), 2018. doi:10.5281/ [19] A. Hebert, Information literacy skills of firstzenodo.1475795. year Library and Information Science graduate [5] X. Li, J. Feng, Y. Meng, Q. Han, F. Wu, J. Li, A Uni- students: An exploratory study, Evidence Based ifed MRC Framework for Named Entity Recogni- Library and Information Practice 13 (2018) 32–52. tion, arXiv:1910.11476 (2019). [20] S. Nowrin, L. Robinson, D. Bawden, Multi[6] A. Padmakumar, A. Saran, Unsupervised Text lingual and Multi-cultural Information Literacy: Summarization Using Sentence Embeddings, in: Perspectives, Models and Good Practice, Global Technical Report, University of Texas at Austin, Knowledge, Memory and Communication (2019). 2016. [21] Y. Zhang, M. Tuo, Q. Yin, L. Qi, X. Wang, T. Liu, [7] Y. Wang, M. Wang, H. Fujita, Word Sense Dis- Keywords Extraction with Deep Neural Network ambiguation: A comprehensive knowledge ex- Model, Neurocomputing 383 (2020) 113 – 121. ploitation framework, Knowledge-Based Sys- [22] E. Hambi, F. Benabbou, A Multi-Level tems (2020). Plagiarism-Detection-System Based on Deep [8] H. Zhang, G. Cormack, M. Grossman, Learning Alg., in: IJCSNS, 2019.
M. Smucker, Evaluating Sentence-Level [23] K. Hakala, S. Pyysalo, Biomedical Named EnRelevance Feedback for High-Recall Information tity Recognition with Multilingual BERT, in: Retrieval, Inf. Retr. J. (2020). BioNLP-OST, 2019. [9] H. Liu, Q. Yin, W. Wang, Towards Explainable [24] E. Taher, S. Hoseini, M. S., Beheshti-NER: PerNLP: A Generative Explanation Framework for sian named entity recognition Using BERT, in: Text Classification, in: A. Korhonen, D. Traum, ICNLSP, 2019.
L. Màrquez (Eds.), ACL, 2019. [10] D. Gromann, L. Anke, T. Declerck, Special Issue on Semantic Deep Learning, Semantic Web 10 (2019) 815–822. [11] A. Lachner, C. Neuburg, Learning by writing explanations: Computer-based Feedback about the Explanatory Cohesion Enhances Students’
Transfer, Instructional Science 47 (2019) 19–37. [12] M. Jaradeh, A., Oelen, M. Prinz, M. Stocker,
S. Auer, Open research knowledge graph: A system walkthrough, in: TPDL, 2019. [13] J. Pennington, R. Socher, C. Manning, Glove:
Global Vectors for Word Representation., in:
EMNLP, 2014. [14] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov,
Enriching Word Vectors with Subword Information, Transactions of the Association for Computational Linguistics 5 (2017) 135–146. [15] R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel,
A. Torralba, R. Urtasun, S. Fidler, Skip-Thought