=Paper=
{{Paper
|id=Vol-2699/paper24
|storemode=property
|title=NLP for Student and Teacher: Concept for an AI based Information Literacy Tutoring System
|pdfUrl=https://ceur-ws.org/Vol-2699/paper24.pdf
|volume=Vol-2699
|authors=Paul Libbrecht,Thierry Declerck,Tim Schlippe,Thomas Mandl,Daniel Schiffner
|dblpUrl=https://dblp.org/rec/conf/cikm/LibbrechtDS0S20
}}
==NLP for Student and Teacher: Concept for an AI based Information Literacy Tutoring System==
https://ceur-ws.org/Vol-2699/paper24.pdf
NLP for Student and Teacher: Concept for an AI based
Information Literacy Tutoring System
P. Libbrechta , T. Declerckb , T. Schlippea , T. Mandlc and D. Schiffnerd
a IUBH Fernstudium, Bad Reichenhall, Germany
b DFKI GmbH, Saarbrücken, Germany
c University of Hildesheim, Hildesheim, Germany
d DIPF | Leibniz Institute for Educational Research and Information, Frankfurt, Germany
Abstract
We present the concept of an intelligent tutoring system which combines web search for learning purposes and state-of-the-
art natural language processing techniques. Our concept is described for the case of teaching information literacy, but has
the potential to be applied to other courses or for independent acquisition of knowledge through web search. The concept
supports both, students and teachers. Furthermore, the approach integrates issues like AI explainability, privacy of student
information, assessment of the quality of retrieved information and automatic grading of student performance.
1. Motivation work and support the creation of own experiments.
On the other hand, many educational institutions al-
Information literacy is a core skill for the digital age. ready conduct their courses, exercises, and examina-
In modern education and work environments it is of tions online. This means that student assessments are
growing importance as knowledge work is increas- already available in digital, machine-readable form,
ingly based on large and rapidly changing knowledge offering a wide range of analysis options. Focusing
sources. Search and organization of knowledge is a on information literacy, a course typically consists of
constant requirement. Higher education teaches in- teaching material in text form and the course partici-
formation literacy sometimes in dedicated courses and pants themselves practice information skills and gen-
often only within another course. Studies show that erate text in online research and essays. However, the
the level is low: E.g. students have difficulties in us- evaluation of free texts such as essays, references and
ing operators in search terms, organize literature and research methodology still requires intensive manual
tend not to know appropriate sources to find scientific work.
literature. Consequently, we deal with the question which
The potential of Artificial Intelligence (AI) in higher methods of Natural Language Processing (NLP) can
education still needs to be explored and innovative support coaching of information competency and how
applications need to be developed. Can computers they can be applied for the teacher, and for the stu-
support teaching staff in coaching information com- dent. The focus is on the combination of various deep
petency? – The research area “AI in Education” ad- learning approaches to automatically help students to
dresses the application and evaluation of AI meth- accelerate the learning process by automatic feedback,
ods in the context of education and training. One but also to support teachers by pre-evaluating free text
of the main focuses of this research is to analyze and suggesting corresponding scores or grades.
and improve teaching and learning processes. On the
one hand, deep learning – learning in multi-layered
(“deep”) artificial neural networks – has become a cen- 2. Related Work
tral component of AI research and numerous libraries
or frameworks1 have been created that simplify the Lazonder [1] showed that searching and talking about
the search has led to positive learning effects. In a
Proceedings of the CIKM 2020 Workshops, October 19–20, Galway, similar fashion, providing feedback and suggestions
Ireland
email: p.libbrecht@iubh-fernstudium.de ( P. Libbrecht);
around a search activity can support the reflection on
declerck@dfki.de ( T. Declerck); t.schlippe@iubh.de ( T. Schlippe); search tools’ usage, on one’s information needs and
mandl@uni-hildesheim.de ( T. Mandl); schiffner@dipf.de ( on the goals of the task at hand. In his keynote at
D. Schiffner) ECIR 20202 C. Shah envisioned the next decade of re-
orcid:
© 2020 Copyright for this paper by its authors. Use permitted under Creative search in search and recommendation where modeling
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)
1 To name a few: TensorFlow, Keras , Caffe, PyTorch 2 https://ecir2020.org/keynote-speakers/
the tasks is central to raise the quality of the results. it “constitutes a composite set of knowledge, skills,
Defined tasks around the learning of information liter- attitudes, competencies and practices that allow ef-
acy are a good example of context where recommen- fectively access, analyze, critically evaluate, interpret,
dations can be made more relevant to the process. In- use, create and disseminate information and media
telligent Tutoring Systems (ITS) follow a long tradi- products with the use of existing means and tools on a
tion of environments where AI supports learning. The creative, legal and ethical basis. It is an integral part of
most widespread didactic situation where ITS has been so-called “21st century skills” or “transversal compe-
employed, is as exercises where a direct feedback (e.g. tencies”” [18]. In higher education, these information
in form of a score or recommendation) is offered fol- skills are highly relevant for students. Nevertheless,
lowing interactions with a dedicated system. Multiple the information literacy of students is often measured
example intelligent tutoring systems exist and many as low [19]. Courses on basic scientific work cover sev-
follow the model of [2]. Our research aims to observe eral domains of information literacy. Often, there is a
the work of the students instead of requiring exercise strong focus on searching skills, correct citing and as-
specific actions. sembling short abstracts based on scientific texts. The
State-of-the-art research and the basis for the devel- practice of teaching information literacy skills has not
opment of an NLP system to coach information com- developed much towards digital formats. Some open
petency include sentiment analysis [3], topic identi- online courses exist [20], but there is no use of AI tools
fication [4], named entity recognition [5], text sum- yet.
marization [6], word sense disambiguation [7] and in- An example is the ILO-MOOC (informationliter-
formation retrieval [8]. A major scientific challenge acy.eu). It allows to study in a self-paced manner. The
is the explainability of system outputs [9]. The NLP feedback for students is show right or wrong after an-
methods can be combined with knowledge graphs to swering multiple choice questions. Another example
include ontology-based knowledge coding in the pro- is at IUBH University where bachelor students with
cesses [10]. This can be enhanced by including visu- a diverse background are trained on the basics of sci-
alizations that represent both the inputs of the stu- entific work. While the focus of the assignments is in
dent and the results of the NLP analysis. Lachner [11] the production of written texts, it involves all aspects
shows that graph representations can support the un- of information literacy. The course is made for both
derstanding of a topic. To represent ontologies or com- remote and on-site attendance and involves various
plex topics, knowledge graphs such as [12] help to communication channels, many of them happening on
identify context. the web. The resulting competencies expect an inde-
For the processing in deep learning architectures, pendent and self-confident scientific work which may
sequences of words are encoded into vector spaces be strongly supported by an automatic evaluation.
in order to perform computations in neural networks.
Tools for text vectorization are Word2Vec , GloVe [13]
or fastText [14]. The concept of the skip-thought 4. Proposed System Architecture
vectors [15], universal sentence encoder (USE) and
Our concept proposes an integration within the web
bidirectional encoder representations from transform-
activities of the learner attending an assignment task
ers (BERT) [16] are methods also supporting sentence
which includes searching, reading, evaluating, and
embeddings in the semantic vector space. [17] in-
writing: Using JavaScript or web extensions, the text
vestigates and compares state-of-the-art deep learn-
and timestamps of the search results, of the viewed
ing techniques for automatic short answer grading.
publications, and of the input text can be used as fea-
Their experiments demonstrate that systems based on
tures for the NLP models. Based on the assignment’s
BERT [16] performed best for English and German. On
objective, the feature vectors generated from the stu-
their German data set they report a Mean Average Er-
dent’s behavior and text is processed by our NLP mod-
ror of 1.2 points, i.e. 31% of the student answers are
els. The models were trained by annotated text data
correctly graded and in 40% the system deviates by 1
from previous course members (model solution, al-
out of 10 points.
ready graded works, other annotated works) to gen-
erate textual feedback.
3. Information Literacy Courses The concept comprises several tasks for which sup-
port for students can be provided. For the sake of
The term Information Literacy is often used synony- brevity we illustrate two of them:
mous with Media Literacy. According to the UNESCO A core task in scientific work and, thus, in teaching
information literacy is web search. Students are of- retrieved by adapting the BERT fine-tuning architec-
ten required to search for documents fulfilling certain ture to extract named entities as proposed in [23] and
requirements e.g. within a closed collection of docu- [24]. The confidence score can be retrieved by map-
ments. An AI system observes the search terms input ping the predicted scores to classes and output a vector
by the student and compares the strategies to identi- that contains a probability for each class.
fied objectives. The system tracks the actions of the Further feedback to both teachers and learners can
students regarding search terms, observed documents, be given by the visualizations of individual states and
headers and time spent. It then suggests the most configurations of the system. Given the exemplary
appropriate steps toward reaching better results. In tasks, the trial and error processes can be shown in
the example of searching in a closed collection with a different paths along a timeline that can be shared by
pre-defined goal, suggestions for further search terms the student with a teacher to allow for better feedback.
leading to relevant documents can be made.Text vec- Also the categorization and identification of correct
torization and a deep learning model-based classifi- steps are to be shown to the learner using clear visu-
cation can be used for keyword extraction [21]. As alizations to help understanding the decision making.
such, the student learns in the direct interaction with The analysis of the evolving students’ work gath-
a search system and improves skills based on previous ers data that should not necessarily be shown to fel-
activities by providing automatic feedback. low students or teachers: The data is made of personal
Another exemplary task in teaching information lit- trial and error processes and is to be considered as pri-
eracy is related to academic writing. Students are vate. Other content, e.g. from chat rooms and forums,
asked to assemble a short summary and synthesis of can be considered as public. It is adequate that a bot
several papers. The system supports them in analyz- can provide answers using information that all chat
ing the writing, recognizing the parts in a certain pa- members have seen (e.g. lecture scripts, assignments’,
per, checking whether the short summary is adequate posts). Similarly, submitted assignments’ text data can
and without plagiarism. Siamese neural networks can be automatically graded based on existing information
be used to detect similarities there [22]. The system such as earlier assignments or expert texts.
also uses NLP to analyze the coherence of the text.
Here an AI based system also gives context-dependent
suggestions on how to improve the text. The sugges-
5. Conclusion and Future Work
tions provided can refer to documents, showing a title, We have described the architecture of an intelligent tu-
a time when the student saw it, and a link to the docu- toring system which combines web search and natural
ment as last accessed. This is effective for the learner language processing techniques on the basis of infor-
as the reading is kept in memory. Such support within mation competency. After implementing the system
the writing process can help students more than a the- and training the machine learning models, the system
oretical unit on academic writing. needs to be evaluated and optimized for students and
In our suggested AI based information literacy tu- teachers regarding usability and efficiency for which
toring system, word sequences in search terms, re- several courses exist. With the help of metrics, we re-
trieved documents, and reference documents are en- duce the error rate for training the models. Finally, we
coded into vector spaces in order to perform compu- intend to speedup the system. Throughout the imple-
tations in deep learning architectures. A fine-tuning mentation usability tests are repeatedly performed to
architecture, such as BERT, which has proven itself in ensure the quality of the proposed system.
many NLP tasks, provides the basis of our system: It is We plan to apply the architecture within several
based on a pre-trained deep learning model, which — courses, adjust the tasks to be automatically measur-
supplemented by a linear regression layer — is adapted able and annotate corpora of articles so that an au-
to our specific tasks, e.g. grading short summaries or tomatic evaluation yields productive feedback. Using
retrieved documents from the web search, and the pa- this system, we expect to answer the following ques-
rameters of the embeddings are tuned accordingly. A tions: How to adequately capture the students’ activ-
data set with labeled and graded documents and sum- ity, select information to store and evaluate it, and how
maries from old information literacy courses serves to to offer support which is timely and relevant for the
optimize the model for predicting scores. learning process.
To achieve a steeper learning curve and to guaran-
tee explainability, we suggest two methods: (1) High-
lighting keywords and (2) displaying the confidence
score of the system’s output. The keywords can be
References Vectors, in: NIPS, MIT Press, Cambridge, MA,
USA, 2015.
[1] A. Lazonder, Do two heads search better than [16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,
one? Effects of student collaboration on web BERT: Pre-training of Deep Bidirectional Trans-
search behaviour and search outcomes, BJET 36 formers for Language Understanding, in:
(2005) 465–475. ACL:HLT, 2019.
[2] K. VanLehn, The Behavior of Tutoring Systems, [17] J. Sawatzki, T. Schlippe, M. Benner-Wickner,
I. J. Artificial Intelligence in Education 16 (2006) Deep Learning Techniques for Automatic Short
227–265. Answer Grading, in: submitted to: COLING,
[3] H. Liu, Sentiment Analysis of Citations Using ACL, 2020.
Word2vec, arXiv:1704.00177 (2017). [18] The Moscow Declaration on Media and Informa-
[4] M. Lamba, M. Margam, Metadata Tagging tion Literacy, 2020. https://iite.unesco.org/mil,
of Library and Information Sciences Theses: 2020-05-02.
Shodhganga (2013-2017), 2018. doi:10.5281/ [19] A. Hebert, Information literacy skills of first-
zenodo.1475795. year Library and Information Science graduate
[5] X. Li, J. Feng, Y. Meng, Q. Han, F. Wu, J. Li, A Uni- students: An exploratory study, Evidence Based
fied MRC Framework for Named Entity Recogni- Library and Information Practice 13 (2018) 32–52.
tion, arXiv:1910.11476 (2019). [20] S. Nowrin, L. Robinson, D. Bawden, Multi-
[6] A. Padmakumar, A. Saran, Unsupervised Text lingual and Multi-cultural Information Literacy:
Summarization Using Sentence Embeddings, in: Perspectives, Models and Good Practice, Global
Technical Report, University of Texas at Austin, Knowledge, Memory and Communication (2019).
2016. [21] Y. Zhang, M. Tuo, Q. Yin, L. Qi, X. Wang, T. Liu,
[7] Y. Wang, M. Wang, H. Fujita, Word Sense Dis- Keywords Extraction with Deep Neural Network
ambiguation: A comprehensive knowledge ex- Model, Neurocomputing 383 (2020) 113 – 121.
ploitation framework, Knowledge-Based Sys- [22] E. Hambi, F. Benabbou, A Multi-Level
tems (2020). Plagiarism-Detection-System Based on Deep
[8] H. Zhang, G. Cormack, M. Grossman, Learning Alg., in: IJCSNS, 2019.
M. Smucker, Evaluating Sentence-Level [23] K. Hakala, S. Pyysalo, Biomedical Named En-
Relevance Feedback for High-Recall Information tity Recognition with Multilingual BERT, in:
Retrieval, Inf. Retr. J. (2020). BioNLP-OST, 2019.
[9] H. Liu, Q. Yin, W. Wang, Towards Explainable [24] E. Taher, S. Hoseini, M. S., Beheshti-NER: Per-
NLP: A Generative Explanation Framework for sian named entity recognition Using BERT, in:
Text Classification, in: A. Korhonen, D. Traum, ICNLSP, 2019.
L. Màrquez (Eds.), ACL, 2019.
[10] D. Gromann, L. Anke, T. Declerck, Special Issue
on Semantic Deep Learning, Semantic Web 10
(2019) 815–822.
[11] A. Lachner, C. Neuburg, Learning by writing
explanations: Computer-based Feedback about
the Explanatory Cohesion Enhances Students’
Transfer, Instructional Science 47 (2019) 19–37.
[12] M. Jaradeh, A., Oelen, M. Prinz, M. Stocker,
S. Auer, Open research knowledge graph: A sys-
tem walkthrough, in: TPDL, 2019.
[13] J. Pennington, R. Socher, C. Manning, Glove:
Global Vectors for Word Representation., in:
EMNLP, 2014.
[14] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov,
Enriching Word Vectors with Subword Informa-
tion, Transactions of the Association for Compu-
tational Linguistics 5 (2017) 135–146.
[15] R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel,
A. Torralba, R. Urtasun, S. Fidler, Skip-Thought