In recent years we have seen a significant proliferation of intelligent personal assistant devices in different use domains. If, on the one hand, the number of conversation-based interactions is growing, on the other hand, the design of chatbots able to assist users in complex decisionmaking tasks is still a significant challenge. To this aim, we need to design chatbots that can give final users proper advice to accomplish their goals. In this paper, we propose a strategy that aims to involve domain experts as the only people who, with their competencies, can train the chatbots to provide helpful suggestions. To test our idea, we present two chatbots designed to help teachers create new courses or caregivers define rules for monitoring older people's indexes of active life. In our approach, we ask domain experts to find or write wiki pages describing the learning objects to compose a course or activities to monitor older people's behaviours. Then, the model we defined extracts from the wiki pages the metadata the chatbot will use to filter the learning objects or the monitoring rules to adopt. To finish, we present a few preliminary tests demonstrating comforting indications about our model.
Driven by the success of intelligent personal assistant devices such as Amazon Alexa and Google Assistant, chatbots are emerging as new interactive solutions in different use domains. Also known as Conversational User Interfaces (CUIs) or Conversational Agents (CAs), these apps demonstrate attractive strategies for implementing language-based interactions and delivering a better user experience (UX).
Based on the work in [
From a technical point of view, designing a chatbot to support short interactions is relatively more straightforward. For example, the use of chatbots is becoming commonplace on the websites of businesses. They help customers find a suitable service or product, answer their questions, and allow them to make a booking or submit an order. In contrast, if we need to design a chatbot-based long interaction, we have to deal with more complex challenges, and in this field, the chatbot's potential has yet to be leveraged.
In particular, this paper aims to explore how to create chatbots to embed into interfaces used for complex decision-making tasks. Here, we are interested in developing chatbots that act as assistants or coaches of users for supporting long-term engagements. Some long-term chatbots exploit the duration of the relation to gradually present a rich set of content, such as a complex story or a game, or to gradually build skills and capabilities in the user, such as in educational, fitness or therapy chatbots.
To maintain conversations with users as long as possible, can happen that the chatbot has to suggest doing something or making a decision. This conversational recommender facility allows chatbots to present personalised content through discussions to help users accomplish a specific goal. Strategies to design this type of chatbot require an interdisciplinary approach that often transcends technology and focuses on requirements regarding user experience, domain-specific rules and issues. These requirements can be explained by domain experts who are not technical experts but know the context of work well and have the competence to train the chatbots to provide helpful suggestions.
For this reason, we aim to involve non-technical users in developing chatbots and their recommendation facilities. The proposed approach leverages specific machine-learning techniques to analyse Wikipedia pages (also named wiki pages), in which domain experts can specify requirements and competencies that the chatbot can use to formulate its suggestions.
According to these considerations, in the next Section, we present an overview of relevant studies
we used to motivate the adoption of conversation agents in two user domains. The first is a chatbot
integrated with a Learning Management System used to assist teachers in creating digital courses. A
second chatbot has been designed to act as an older people's assistant, providing functionalities to
combat the typical loneliness that can affect their quality of life. Section 3 describes our strategy, based
on End User Development (EUD) [
Exploiting the proposed typology in [
The literature presents many studies regarding using chatbots in the educational domain [
Currently, to facilitate the creation of a course, teachers can use authoring tools such as Absorb2, Learnopoly3 or Elucidat4. These tools help teachers develop, launch and review an e-learning course, but they cannot support them throughout creating a new digital course. Our idea is to give a chatbot the task of finding existing Learning Objects (LOs) [15-17]. Through the chatbot suggestions, teachers can use these reusable and interoperable LOs as building blocks for producing a course more quickly and efficiently. Once discovered the LOs, teachers need to find a strategy for combining and sequencing them to ensure that learning resources can be appropriately assembled in a course that can meet the teachers' objectives and requirements.
At the beginning of the interaction, the chatbot requests information related to the course to be created. This information regards its difficulty, the number and duration of the lessons, the language, and the course topics. (Figure 1).
Regarding topics, skills and competencies, the chatbot allows adding items to the corresponding list if the user needs it. Once the teacher has entered the course information, the parsing system checks the orthographic and performs the translation into English (since the dataset is in English) via the DeepTranslator library5.
Subsequently, using the topics, the difficulty, type and duration, the chatbot can suggest which learning objects teachers can combine to create the new digital course. Figure 2 shows how a teacher selected two LOs to specify the necessary resources. If a LO contains exercises, the teacher can set to repeat the activity if the student fails to finish a LO in the desired time or according to an established rating. 5 Deep translator. https://pypi.org/project/deep-translator/ - Last access: 2023-05-01.
The conversational agent we designed takes the name of Charlie. Charlie, implemented using Google DialogFlow6, acts as a medical advisor, friend and carer for autonomous seniors living alone.
To simplify interaction, Charlie provides limited choices that are easily understood and accessible via the mobile phone or tablet screen. The dialogues are generally of short duration to avoid tiring the user. Key features depicted in Figure 3 include the possibility of giving daily automatic notifications, healthy recommendations and offering activity reminders about important events, such as the need to take medicine. Charlie can also present news, weather forecast, or different forms of entertainment, including a memory-based game and quizzes. Another feature includes active listening to help older people improve their mood. The users indicate their thoughts, and then the assistant asks for information regarding their emotional situation. Through telling a story, Charlie establishes a pretext to communicate with the user and ask for information to develop a conversation. This functionality allows the users to consider Charlie in a friendly way, pushing them to confide and express themselves without worrying about communicating with artificial intelligence.
Moreover, another agent's goal is to monitor the indexes of active life specifically defined to keep the trend of older adults' physical-cognitive state under control. To this aim, we need to involve older person's caregivers or relatives to configure the digital agent. As explained in the following Sections, the idea is to help them compose monitoring rules that can define the active life indexes to be controlled.
Reusing learning objects (LOs) rather than their reinvention aims to save time and effort [18]. Moreover, from a quality point of view, the more a resource is reused, the more likely it is to be of high quality simply because more people will have been exposed to it and have had the opportunity to provide feedback [19]. To this aim, we need to involve domain experts who can use their experience of solving problems in the past to build on and create new solutions in new situations.
Most current e-learning platforms are closed systems that hardly share and reuse materials because they are made in proprietary formats. To solve the problems of sharing and reusing teaching materials in other e-learning systems, many international organisations established e-learning standards. The Sharable Content Object Reference Model (SCOR [20]) is recognised as the most popular one, and the IEEE-Standards Association has approved its Learning Object Metadata (LOM) [21]. Another standard, the Dublin Core7, emerged over the years to facilitate the sharing and reuse of learning materials which establishes metadata policies and provides suggestions for using the LOs.
All these standard protocols allow teaching materials for different learning management systems to be shared, reused, and integrated, but none agree on which metadata could be used for describing LOs. Some studies [22] recommend a minimal metadata set representing an LO. However, even if no specific rules are indicated, surveys in [23, 24] have shown that Dublin Core is suitable for describing the bibliographic side of digital resources, and LOM allows the best representation of the pedagogical aspects.
Due to its nature, we decided to use LOM for our project, but a problem remains. How to exploit this metadata to suggest proper LOs and combine them in a final course. According to works proposed in [25-27], our idea is to use specific machine-learning (ML) techniques to analyse a wiki page associated with each LO. Expert teachers are involved in creating these wiki pages to describe the content covered by the LO and, in particular, its requirements and final competencies [28]. Then, the model extracts metadata from the wiki pages and defines a sequence of LOs according to their prerequisites. Figure 4 depicts a LO related to using Scratch, a block-based visual programming language aimed at learning coding basics for students ages 8 to 16.
The model we defined aims at parsing the wiki pages to extract relevant concepts that describe the semantics of the linked LO. Using the LOM metadata, we can associate 72 features to each LO and 9 descriptive areas for categorising the information content in the teaching resource. The final goal is to learn if a LO "A" is a prerequisite for a LO "B".
To this aim, we need to create a "prerequisite" attribute by using a set of metadata such as (1) the age of acquisition of a concept, (2) the age of acquisition of correlated concepts, (3) the length of a concept description, (4) the number of mathematic expressions presented on each wiki page, and (5) the frequency of concept visualisations. With the Age of Acquisition (AoA) of a concept, we refer to the work presented in [29]. The study collects AoA ratings for 1,957 Italian content words (adjectives, nouns, and verbs), asking participants to estimate the age at which they thought they had learned the word as a result of a Web survey procedure. With AoA of correlated concepts, we mean the AoA average value of the concepts that appear on the wiki page that describes a given concept. This second type of feature aims to model the relationship between pairs of concepts. In particular, it evaluates if a concept appears as a sub-string in the title or the description of the other concept.
Then, we applied our model to a dataset of resources used to teach computer programming skills and computational thinking. The dataset contains 554 LOs, and we asked to 11 students of the Department of Computer Science at the University of Milano to find or write a wiki page for each of these LOs to describe them. Then we enrolled 15 teachers recruited by the Social Thingum company8 to create new digital courses to test the effectiveness and efficiency of the suggestions provided by the chatbot. In detail, our chatbot suggests which LOs to take into account by analysing the information provided by the teacher about the topics, the difficulty, type and duration of the course to create. Then, using Sentence-BERT (S-BERT)[30], a machine-learning model based on Transformers, the chatbot computes the semantic similarity between input data and LO metadata. After discovering the better LOs, the chatbot used the "prerequisite" attribute to suggest how to combine them for creating a course according to the teacher's needs.
Due to the limited number of records, in analysing the reliability and accuracy of our model, we achieved only an average F1 of 0.68 over the test set. Nevertheless, we are confident this result could be a good indication of the extraction model of the prerequisites that can be used for suggesting a sequence of proper LOS to consider for creating a course. 3.4.
As said before, we designed a chatbot in the healthcare domain to help caregivers and relatives to monitor a set of indexes that describe the active life of their dear ones. To identify these indexes, domain experts need to determine what to monitor for taking under control the trend of older people's physicalcognitive state. To this aim, we focused on how to write rules that caregivers can establish depending on data gathered by Charlie.
Unfortunately, the problem with this strategy is that the caregivers are often "lost in the data sea" and do not know which parameters to check for monitoring the senior's behaviour. To help caregivers, we designed a model recommending which rules to adopt. Essentially, the idea is to develop functionality to compute predictions and the consequent rules by extracting knowledge from the data representing older adults' behaviours.
In particular, we are investigating an approach that produces a set of readable and understandable IF-THEN rules [31], which are easily interpretable. The rules are suggested by using BERT for generating embedding representations helpful in capturing the semantic similarity between the elderly's profile and a set of metadata describing the activities to monitor. The elderly profile is established using a specific screening model we defined integrating the MoCA - Montreal Cognitive Assessment9 model, the MMSE - Mini-Metal State Examination test [32] and the GPCog - General Practitioner Assessment of Cognition10 provided by experts of the ASST (Aziende Socio Sanitarie Territoriali - Territorial SocioHealth Companies) of Crema (Italy).
Screening requires the caregiver to carry out a brief cognitive questionnaire on the patient, administering quick tests to investigate the main cognitive functions (attention, memory, language, perception, executive functions) and identify signs of possible deterioration. This neuropsychological evaluation makes it possible to define a set of metadata to be associated with the older person. Then, for each activity that the caregiver can indicate for the assisted person (news to provide, notifications, reminders, quizzes, games, active listening or storytelling activities), the doctors of the ASST involved in our project specified a description, objectives, possible benefits to do it, requirements, and motivations by using a wiki page. According to the elderly's profiles and activity descriptions, our model provides caregivers with a recommendation service that suggests how they can create the rules. In detail, the model analyses the wiki page associated with each activity and then extracts from it information that can better fit the older adult's profile.
At the moment, we are in the preliminary phases of this work, so we do not have results helpful for evaluating the reliability and accuracy of our model. Currently, we are training the model and populating the dataset of activity wiki pages involving doctors of the ASST of Crema.
In this paper, we have presented two chatbots specifically designed to investigate how to involve domain experts in the training process of the recommendation facility that the chatbot can use to help users in complex decision-making tasks.
The idea is to use intelligent assistants to help teachers create new courses or caregivers define rules for monitoring older people's indexes of active life. To enable the chatbots to suggest correctly, they need to be trained with the help of domain experts. These experts are not technical people but know the use domains and have the competence to train the chatbots to provide helpful suggestions.
Our EUD approach is based on asking domain experts to find or write wiki pages to describe the learning objects or activities to use for monitoring older people's behaviours. Then, the model we defined extracts from the wiki pages the metadata the chatbot uses to filter the learning objects or the monitoring rules to adopt.
For suggesting them, we decided to use Sentence-BERT, a machine-learning model based on Transformers. BERT discovers the semantic similarity between the wiki page's metadata and the properties the teachers used to specify the new course to create or the properties characterising older people's profiles.
At the moment, we carried out only a few preliminary tests, with results that attest to reasonable indications of the extraction model. These tests mainly concern the strategies used to involve domain experts in training the recommendation facility of a chatbot in the educational field. Nevertheless, we are aware of some limitations that affect our study. The main issue concerns the sample size of participants in our preliminary tests. Recruiting a few users does not allow us to present a complete statistical confirmation and validation of the reliability of the collected data. For this reason, further research aims to extend the study by involving more users with a broader context of use in each area of interest where chatbots can be used. [14] Chaves, A. P., & Gerosa, M. A. (2020). How Should My Chatbot Interact? A Survey on Social Characteristics in Human–Chatbot Interaction Design. International Journal of Human– Computer Interaction, 1-30. [15] Ruiz, J. G., Mintzer, M. J., & Issenberg, S. B. (2006). Learning objects in medical education.
Medical teacher, 28(7), 599-605. [16] Deschênes, M. (2020). Recommender systems to support learners' Agency in a learning Context: a systematic review. International Journal of Educational Technology in Higher Education, 17(1), 50. [17] Urdaneta-Ponte, M. C., Mendez-Zorrilla, A., & Oleagordia-Ruiz, I. (2021). Recommendation systems for education: systematic review. Electronics, 10(14), 1611. [18] Downes, S. (2001). Learning objects: Resources for distance education worldwide.
International Review of Research in Open and Distance Learning, 2 (1). [19] Jones, R. (2004). Designing adaptable learning resources with learning object patterns. Journal of Digital Information, 6 (1), Article No. 305. [20] SCORM (2003). Sharable Courseware Object Reference Model, Retrieved June 7, 2007, from, http://www.adlnet.gov/downloads/downloadpage.aspx?ID=243 [21] LOM (2002). Final LOM Draft Standard, Retrieved June 7, 2007, from, http://ltsc.ieee.org/wg12/20020612-Final-LOM-Draft.html [22] Cohen, E., & Nycz, M. (2006). Learning objects and e-learning: An informing science perspective. Interdisciplinary Journal of E-Learning and Learning Objects, 2(1), 23-34. [23] Dagienė, V., Jevsikova, T., & Kubilinskienė, S. (2013). An integration of methodological resources into learning object metadata repository. Informatica, 24(1), 13-34. [24] Hoebelheinrich, N., Biernacka, K., Brazas, M., Castro, L. J., Fiore, N., Hellstrom, M., ... & Whyte, A. (2022). Recommendations for a minimal metadata set to aid harmonised discovery of learning resources [25] Karpova, M., Shmelev, V., & Dukhanov, A. (2015, October). An automation of the course design with use of learning objects with evaluation based on the Bloom taxonomy. In 2015 9th International Conference on Application of Information and Communication Technologies (AICT) (pp. 138-142). IEEE. [26] Gasparetti, F., De Medio, C., Limongelli, C., Sciarrone, F., & Temperini, M. (2018).
Prerequisites between learning objects: Automatic extraction based on a machine learning approach. Telematics and Informatics, 35(3), 595-610. [27] Xiao, K., Bai, Y., & Wang, Z. (2022). Extracting Prerequisite Relations among Concepts From the Course Descriptions. International Journal of Software Engineering and Knowledge Engineering, 32(04), 503-523. [28] Liang, C., Wu, Z., Huang, W., & Giles, C. L. (2015, September). Measuring prerequisite relations among concepts. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 1668-1674). [29] Elsherif, M. M., Preece, E., & Catling, J. C. (2023). Age-of-acquisition effects: A literature review. Journal of Experimental Psychology: Learning, Memory, and Cognition. [30] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [31] Ghiani, G., Manca, M., Paternò, F., Santoro, C.: Personalisation of context-dependent applications through trigger-action rules. ACM Trans. Comput. Hum. Interact. 24(2), 33 pages (2017). [32] Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). "Mini-mental state": a practical method for grading the cognitive state of patients for the clinician. Journal of psychiatric research, 12(3), 189-198.