GUapp is an ecosystem for job-postings search and recommendation for the Italian public administration. Its main goal is to match user skills and requests with job positions available on the Gazzetta Uficiale website, ofering recommendation services in a conversational setting. Guapp's dialogues are modelled employing a domain-specific Knowledge Graph, which improves the users' natural language interaction with the app as well as the user experience. Thanks to that, the search and recommendation process becomes incremental and the user can dynamically provide her preferences at each stage of the interaction. In this paper, we present GUapp and its overall architecture, besides the functioning of the conversational agent that dialogues with the user by exploiting a custom-designed Knowledge Graph. We also show a running example that outline how GUapp models users and provides them efective recommendations through natural language conversations.
impacts the digital experience of users when they need to find interesting items in a large set of possible options [1]. That is the case of looking for a book to read, a smartphone to buy, a TV series to watch, and so on.
The G U a p p platform has been designed and developed to find and discover job positions among job ofers in the Italian public administration1. This problem has been investigated in the literature from two diferent perspecFrom the Information Retrieval side, there are systems CEUR htp:/ceur-ws.org ISN1613-073 © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop Proceedings (CEUR-WS.org) 1All the documents are freely available online at https://www. gazzettaufficiale.it/30giorni/concorsi that generally do not take into account the user’s past preferences, and only retrieve the most relevant documents based on the current user query. Recommender
ing services and items tailored to the specific users’ characteristics and requests. In our case, the list of job calls is daily updated and provided to the user depending on her past preferences.
G U a p p ofers a natural-language based interaction through a chatbot, which allows the user to define her interests, describe her skills, and filter out results that did not match with her requirements. Moreover, our system leverage the felt issues of cold-start and the possible lack of items to suggest with a Conversational Agent which interacts with users exploiting a domain-specific Knowledge Graph (KG), diferently from the previous version of the tool [2]. The G U a p p KG is obtained by merging some sub-graphs from state-of-the-art solutions like Dbpedia2 and new triples generated from data scraped from external sources such as the ISTAT3 website. From the latter, we have taken information about profession hierarchies and fields to which jobs belong. Furthermore, we have built an ontology on which the retrieved facts rely. This KG allows our system to search for new semantically linked user preferences, besides engaging a negotiation phase when the proposed calls do not match all the user requirements. Exploiting the KG relations, GUapp can search for jobs that do not perfectly suit the user preferences but still remaining close to her interests. On this line, conversations can reach a finer grained level of de2https://www.dbpedia.org/ tails on job aspects to recommend to users and enhance its expressiveness as well.
natural [19]. Accordingly, a CoRS let the system build the user profile during the interaction, allowing her to express preferences, by a human-like dialog. The aforementioned task is well suited to Knowledge-based and KG-based RSs. An example can be found in [20], where a comprehensive KG is built upon a specific domain to lead dialogues and recommendations. CoRS have proved to be very efective both from the recommendation and Human-Computer Interaction perspective and they have been used in diferent domains [ 21, 22]. However, to the best of our knowledge, the G U a p p system represents the first attempt of implementing a CoRS for the jobrecommendation task.
The proposed work is between two principal research areas: recommender systems and conversational agents.
Recommender Systems (RSs) support the user during the decision making process when she has to decide among a large set of diferent options. RSs are grouped into two main categories: Collaborative Filtering (CF) and Content-based (CB). The CF exploits the user community in order to identify items potentially interesting for a given individual, following the intuition that similar users like similar items. Conversely, CB systems try to 3. The G U a p p ’s Architecture suggest items matching user preferences with items descriptions [1]. G U a p p implements a CBRS. Actually, the In Figure 1 G U a p p ’s architecture is sketched, composed of main goal of the RS behind GUapp is to match the user six main components: the Orchestrator, the Chatbot, the requests/preferences with the textual description of the Recommender System, the User Profiler, the Crawler and job proposal. A lot of studies investigated the specific the Knowledge Graph. task of job recommendation [3]. In the past, the most used approaches for job recommendation were based on boolean search and filtering [ 4]. Later, the attention has been focused on the problem of catching the user preferences and building a user profile. In [ 5] the authors propose a system that builds the user profile by passively detecting click-stream and read-time behaviour of users.
Malinowski et al. [4] proposes a strategy based on multislot user profile in which diferent information are stored: demographic data, job experiences, languages, and IT skills. Similarly, in G U a p p tool this kind of information is acquired by mean of a KG-driven conversation. Recom- Figure 1: The GUapp’s Architecture and its flow of data. mending items by means of a set of rules which verify whether the user tastes are satisfied or not was dealt The Orchestrator manages the interaction between the at first by the Knowledge based Recommender Systems diferent components of the system. For example, it in[6]. These systems perform reasoning on ontologies and vokes the RS when user asks for receiving a list of job Knowledge Bases (KBs) to find items that match the user positions based on the information stored in her profile. preferences. For instance, Carrer-Neto et al. [7] opted In detail, this module leverages the overall flow of data sethat movies belonging to the same ontological classes lecting the appropriate component to solve specific tasks. of the items that the user liked in the past has to be rec- We will consider the case on which the Recommender ommended. Diferently, Tarus et al. [ 8] exploit both System needs to start a negotiation phase with the user. ontological features and collaborative filtering to provide The Orchestrator will collect data semantically related to recommendations in the field of learning resources for her profile by querying the Knowledge Graph, that will some learner targets. be exploited to find other items interesting to the user.
On the other hand, Conversational Agents are software The Chatbot is the component of G U a p p which allows agents that use natural language to interact with the user. users to interact through natural language. It is impleThey can be classified in two main classes: end-to-end mented by using DialogFlow4, a Google platform for and modular systems [9]. The former typically exploits designing and integrating conversational user interfaces, Deep Learning techniques for learning a dialog model and it mainly leads conversations with users. For this from a set of past conversations [10, 11, 12, 13, 14]. Mod- purpose, the chatbot is equipped with an Intent Recogular systems adopt a pipeline-based agent which is com- nizer and an Entity Recognizer, allowing the system to posed of a set of modules, each with a specific function understand several user requests and retrieve from her [15, 16, 17, 18]. The main diference between a Conversa- essential data for the recommending task. That is a crutional Recommender System (CoRS) and a traditional RS cial step in order to identify the back-end services to be is the interaction with the user that is more eficient and invoked for accomplishing the request. TheIntent Recog- job features that are crucial for the recommendation task, nizer analyzes the natural language request searching for besides granting the chatbot to leverage fine-grained specific goals such as collecting the user preferences, pro- conversations and negotiations with users. To make the viding new job recommendations, or negotiating with the system more specialized on the job-opening domain, we user. The Entity Recognizer is invoked in order to check plan to enrich our KG of further facts related to the job whether the sentence contains mentions to real-world calls an positions recommendation. entities. It is implemented through Dialogflow as well, and it is powered by the KG entities. It adopts a fuzzy- 4. Building the Knowledge matching strategy in order to identify real-world entities Sources in the user sentence. In the case the match succeeds, the recognized entity is returned. The more detailed a collection of data about features of
The User Profiler instead collects all the users prefer- items is, the higher is the accuracy of the recommendaences. In detail, we store all the data that they provide tions provided by the system. Following this intuition during conversations like favourites job locations, pro- and given the conversational configuration on which fessions etc. When the user signs into the app for the GUapp is built, we found owning a well-structured source ifrst time, her profile is empty (i.e. cold-start situation), of information an essential requisite. The system can but thanks to the chatbot the user talks with G U a p p about provide more fine-grained recommendations by using her skills, wishes, and ambitions, then the system is able a knowledge source that defines aspects users evaluate to rank job calls by exploiting the provided information. to match their interests. For instance, job location and The User Profile is actively updated to the new user in- profession that a person could cover represent two main puts, guaranteeing the recommendations to be adaptive. features that people consider while seeking a new job. At
The Recommender System is another core component the same time, skills and experiences are crucial informaof G U a p p . It exploits an Elasticsearch5 index, which stores tion to retrieve the most proper job position for the user. all the information scraped by the Crawler enriched with The employment of a Knowledge Graph further allows all the linked entities of our KG. In particular, for each the system to build semantically explicit user profiles. job call, we automatically search for mentions of the KG That is results in highly interpretable recommendations, entities or the ontological categories. The Elasticsearch besides leading eficient negotiations in case there is no documents will store the entity/category label related item that satisfies all the user requirements. to each discovered mention with a confidence score, es- On this line, we opted to enrich GUapp with a coltimated with the BM25 algorithm, which shows how lection of Linked Open Data (LOD), suitable for the job much the labelling process outcomes are reliable. The recommendation task, since their availability in strucrecommendations will be the job call closest to the user tured non-proprietary formats under an open license. preferences with the highest confidence score. Unfortunately, there are no KG and Ontologies already
The main intuition behind this model is the possibility available which outline the hierarchies of job professions to make dialogues as interactive and eficient as possible, and their belonging fields. Accordingly, we have started allowing the system to negotiate with users whether to implement a new LOD resource that perfectly fits the results do not completely match their preferences. GUapp intents, besides being also available for other re
To be updated with all new jobs that the market ofers, lated purposes. To the best of our knowledge, the GUapp G U a p p implements a Crawler that daily extracts the job KG identifies the first attempt of structuring relations bepositions from Gazzetta Uficiale , the oficial journal of tween professions and their application fields under the record of the Italian government. It directly communi- guidelines of the LOD protocols. Moreover, it integrates cates with the Orchestrator for storing all the obtained also all the data related to the job recommending task data into the Elasticsearch instance as new documents. obtained from other state-of-the-art solutions, like cities,
Finally, we have provided to the system a domain- regions, and countries provided by Dbpedia. specific Knowledge Graph built upon several sources, like We first collected all the RDF statements about locaDbpedia and the ISTAT website, as stated before. The tions from the Dbpedia project and the associated onlatter identifies a collection of raw data that are not in tology to create a KG that was complete and consistent KG form. As a consequence, we have modelled a new for this work. For this purpose, we have exploited the Ontology on which the overall KG can rely. It defines OpenLink Virtuoso6, a Dbpedia SPARQL endpoint that alrelations and hierarchies about professions, domain of lowed us to perform diferent SPARQL queries to collect competences, locations and so forth and it wisely inte- the interested data. Regarding professions and appligrates the raw information with the already structured cation fields, we opted to create a new ontology from ones. This allows G U a p p to be highly knowledgeable about scratch, assembling this information from highly reliable 5https://www.elasticsearch.com 6https://dbpedia.org/sparql/ sources. We found that the ISTAT website, managed by knowledge base and the ontology are used in order to an Italian research institute for statistics, totally accom- handle the negotiation. Given the conversational nature plish this requirement. It stored a complete hierarchy of of the system, a preliminary step for preference elicitation professions organized for sectors, application fields, and is needed. In this phase, the agent asks the user some relservices in a tree data structure navigable through web evant information about her preferences. In particular, it pages for each position. For example, at the higher level, asks about the geographical area and the field of interest. we can found distinctions between intellectual, technical, This first conversation phase follows a well-defined diaand ofice jobs while descending into the graph groups logue flow. Let us consider a scenario in which Claudio, a like scientific, health, and managing positions are out- user who is graduated as Computer Engineer, is looking lined. This taxonomy reflects what the GUapp ontology for jobs in the computer engineering field. Therefore, he asserts about professions. The leaves of the ISTAT tree visits the GUapp website and finds the section related describe all the positions currently recognized in our so- to the conversational agent. After a standard welcome ciety, like computer scientists and computer engineers, message, the agent asks for the geographical area Clauwhich compose some of the GUapp KG facts. dio is interested in. In this case, Claudio writes that he is
All this data are automatically retrieved from the previ- interested to work in Rome. The second crucial question ously mentioned website exploiting the Crawler routines. that the agent asks to Claudio is about the job position They not only collect all the job calls of the day, but they he would like to cover. He answers stating that he is also scrape all the information that populates the KG. interested in a job as Computer Engineer. At this point, Then, an owl file is generated for the GUapp ontology the agent tries to map Claudio’s responses to entities and and all the RDF triples are realized to form the KG. For class in the KG. In this particular case, the agent underinstance, the Computer Engineer profile belongs to the stands that Claudio is interested in jobs proposal in Rome GUapp class Electronic Engineer, a subclass of Engineer- (that is linked to the entity gpr:roma) regarding a position ing and of the higher Intellectual and highly specialized of computer engineer (which refers to the ontological Scientific profession class. These facts are finally inte- class gpo:computerEngineer ). As a result, by exploiting grated with those obtained from Dbpedia and form the the Elasticsearch index, the agent starts searching for overall GUapp KG. At the current state, our KG is lim- jobs that match the Claudio’s requirements and creates ited to the Italian language and it has data restricted to a list of possible recommendations that perfectly match professions, fields, and locations. Nevertheless, thanks his preferences. The recommendations are ranked based to its semantic structure, we plan to expand it by adding on the relevance score described in section 3. Therefore, several languages besides including other job position the agent provides only the first job proposals in the features like user skills and job goals. ranked list in order to not impact negatively the user ex
Our system relies on this knowledge source mainly for perience. If Claudio asks for more results, the agent will implementing two functionalities. The first one is the explore the ranked list in order to provide other possilabelling phase of the crawled dataset that makes the rec- ble recommendations until he is satisfied or no more job ommendation possible. In detail, the job calls retrieved proposal is available. If Claudio found an interesting job by the Crawler are in the form of unstructured text, so call, the interaction ends, otherwise the scenario is more we found it necessary to index the documents on a search interesting since the agent is not able to provide other engine like Elasticsearch. Looking for the KG labels in solutions that perfectly match the user’s needs. In that each job call, we enriched all the collected texts with the case, the agent needs more information to understand if GUapp linked entities, which is helpful in performing the user is willing to travel or how flexible is with respect recommendations given the users’ preferences. Instead, to the job place. We refer to this phase as negotiation. the second functionality allows the system to perform Figure 2 shows an example of the behavior of the agent the preference elicitation and the negotiation steps dur- in the negotiation and how it exploits the ontology for ing a conversation. Lead by the GUapp KG entities and predicting other possible recommendations. At the end categories, our conversational agent realizes dialogues of the preference elicitation phase shown in the previous deeply related to the recommendation domain. It also example, the agent creates a ranked list of possible job grants to manage two highly felt issues in the RecSys recommendations in the computer engineering field in community, like the cold start problem and the absence Rome. Thus it provides to Claudio the first result that of items to suggest. is job ofer as a researcher ofered by the University La Sapienza that is an instance of the Computer Engineering 5. Recommending Jobs through class in the ontology. Let us assume that Claudio does not Dialogues ifnd the job proposal interesting and he asks for more results. For the sake of simplicity, we assume that no more In this section, we present an example of the ontology- alternatives are available. In order to provide other soludriven conversation with some details about how the tions, the agent needs more information about Claudio. In particular, since there could be possible job ofers in want the system to search for similar jobs or areas of slightly diferent fields, it asks about the user’s flexibility work. In this case, the agent will ask if the user is willing with respect to the job place. Following the example, the to travel and, according to the answer, it will provide job agent asks Claudio ”I can recommend you similar jobs in proposals more or less distant from the original request. slightly diferent fields. How are you flexible in this sense?” . All the information about geographical entities is orgaThe agent maps the Claudio’s answer to the number of nized into the knowledge base. This allows the system to edges that it could navigate backward in the ontology. navigate the graph and find possible job ofers that are For instance, if Claudio answers ”Not so much”, it means in the same geographical area. that he is not interested in jobs that difer too much from the field he proposed previously. For this reason, the 6. Conclusion and Future Work answer is mapped to a maximum of 2 backward hops in In this paper we presented G U a p p , a platform for searchthe ontology. In case Claudio replies with ”Quite flexible” , ing jobs in the Italian public administration. We have the agent maps the answer to a number of 3 backward outlined the architecture designed to make possible the hops. This mapping has been empirically defined. Since job recommendation task in a conversational setting, beour ontology is composed of 6 levels, 2 backward hops sides describing the overall structure of the G U a p p KG and allow the system to provide job recommendations that its ontology. Thanks to that, G U a p p consists of a recombelong to a similar domain as the previous ones. In this mender system that suggests relevant jobs to the users, sense, the agents navigates backward the ontology start- and one of the most interesting aspects is the integration ing from the class gpo:computerEngineer and reaching the of a KG that helps driving the dialogue, making the interparent node gpo:electronicEngineeringAndTLC. Starting action more natural and pushed at a finer grained level. from the inner node reached after the backward phase, a Indeed, the preference elicitation becomes incremental, forward step is needed to reach the leaves of the sub-tree. with the possibility of refining and improving the user At this point, following the example, the agent reaches requests. We are planning of introducing other several the leaf gpo:electricalEngineer that, for simplicity, is the ifne-grained features for making the ecosystem competionly node in the current sub-tree associated to possible tive with other state-of-the-art solutions, testing the User job ofers in Rome. Also in this case, it creates a ranked Experience with an A/B test and setting up a platform in list following the same approach described previously order to share the dataset coming from G U a p p usage with and provides the user the most relevant alternatives. the industrial and academic community, with the respect
The same method described in this section can be used of the privacy concerns. in another possible scenario. Assuming that, during the negotiation phase, the user answers that she does not [12] A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji,
M. Mitchell, J.-Y. Nie, J. Gao, B. Dolan, A neu[1] D. Jannach, M. Zanker, A. Felfernig, G. Friedrich, ral network approach to context-sensitive generRecommender systems: an introduction, Cam- ation of conversational responses, arXiv preprint bridge University Press, 2010. arXiv:1506.06714 (2015). [2] V. Bellini, G. M. Biancofiore, T. D. Noia, E. D. Sci- [13] J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, ascio, F. Narducci, C. Pomo, Guapp: A conversa- A. Miller, A. Szlam, J. Weston, Evaluating Pretional agent for job recommendation for the ital- requisite Qualities for Learning End-to-End Diaian public administration, in: 2020 IEEE Confer- log Systems, arXiv:1511.06931 [cs] (2015). URL: ence on Evolving and Adaptive Intelligent Systems, http://arxiv.org/abs/1511.06931.
EAIS 2020, Bari, Italy, May 27-29, 2020, 2020, pp. [14] A. Bordes, Y.-L. Boureau, J. Weston, Learning 1–7. URL: https://doi.org/10.1109/EAIS48028.2020. end-to-end goal-oriented dialog, arXiv preprint 9122756. doi:10.1109/EAIS48028.2020.9122756. arXiv:1605.07683 (2016). [3] S. T. Al-Otaibi, M. Ykhlef, A survey of job rec- [15] J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, ommender systems, International Journal of the A. Miller, A. Szlam, J. Weston, Evaluating prerequiPhysical Sciences 7 (2012) 5127–5142. site qualities for learning end-to-end dialog systems, [4] J. Malinowski, T. Keim, O. Wendt, T. Weitzel, Match- arXiv preprint arXiv:1511.06931 (2015). ing people and jobs: A bilateral recommendation [16] T. Zhao, M. Eskenazi, Towards end-to-end learning approach, in: Proceedings of the 39th Annual for dialog state tracking and management using Hawaii International Conference on System Sci- deep reinforcement learning, arXiv:1606.02560 [cs] ences (HICSS’06), volume 6, IEEE, 2006, pp. 137c– (2016). URL: http://arxiv.org/abs/1606.02560, arXiv: 137c. 1606.02560. [5] R. Rafter, B. Smyth, Passive profiling from server [17] J. Williams, A. Raux, M. Henderson, The dialog logs in an online recruitment environment, in: state tracking challenge series: A review, Dialogue Workshop on Intelligent Techniques for Web Per- & Discourse 7 (2016) 4––33. sonalization at the the 17th International Joint Con- [18] B. Liu, I. Lane, An End-to-End Trainable Neuference on Artificial Intelligence, Seattle, Washing- ral Network Model with Belief Tracking for Taskton, USA, August, 2001, 2001. Oriented Dialog, Interspeech 2017 (2017) 2506–2510. [6] R. Burke, Knowledge-based recommender systems, URL: http://arxiv.org/abs/1708.05956. doi:10.21437/ Encyclopedia of library and information systems Interspeech.2017- 1326, arXiv: 1708.05956. 69 (2000) 175–186. [19] P. Wärnestål, User evaluation of a conversational [7] W. Carrer-Neto, M. L. Hernández-Alcaraz, recommender system, in: Proceedings of the 4th R. Valencia-García, F. García-Sánchez, So- IJCAI Workshop on Knowledge and Reasoning in cial knowledge-based recommender system. Practical Dialogue Systems, 2005, pp. 32–39. application to the movies domain, Expert [20] S. Haussmann, O. Seneviratne, Y. Chen, Y. Ne’eman, Syst. Appl. 39 (2012) 10990–11000. URL: J. Codella, C. Chen, D. L. McGuinness, M. J. Zaki, https://doi.org/10.1016/j.eswa.2012.03.025. Foodkg: A semantics-driven knowledge graph for doi:10.1016/j.eswa.2012.03.025. food recommendation, in: The Semantic Web [8] J. K. Tarus, Z. Niu, A. Yousif, A hybrid knowledge- - ISWC 2019 - 18th International Semantic Web based recommender system for e-learning based Conference, Auckland, New Zealand, October 26on ontology and sequential pattern mining, Future 30, 2019, Proceedings, Part II, 2019, pp. 146–162. Gener. Comput. Syst. 72 (2017) 37–48. URL: https: URL: https://doi.org/10.1007/978-3-030-30796-7_10. //doi.org/10.1016/j.future.2017.02.049. doi:10.1016/ doi:10.1007/978- 3- 030- 30796- 7\_10. j.future.2017.02.049. [21] F. Narducci, P. Basile, M. de Gemmis, P. Lops, G. Se[9] A. Bakarov, V. Yadrintsev, I. Sochenkov, Anomaly meraro, An investigation on the user interacdetection for short texts: Identifying whether your tion modes of conversational recommender syschatbot should switch from goal-oriented conver- tems for the music domain, User Modeling and sation to chit-chatting, in: International Confer- User-Adapted Interaction (2019) 1–34. doi:https: ence on Digital Transformation and Global Society, //doi.org/10.1007/s11257- 019- 09250- 7.
Springer, 2018, pp. 289–298. [22] A. Iovine, F. Narducci, G. Semeraro, Conversa[10] L. Shang, Z. Lu, H. Li, Neural responding ma- tional recommender systems and natural language: chine for short-text conversation, arXiv preprint A study through the converse framework, DearXiv:1503.02364 (2015). cision Support Systems (2020) 113250. doi:https: [11] O. Vinyals, Q. Le, A neural conversational model, //doi.org/10.1016/j.dss.2020.113250. arXiv preprint arXiv:1506.05869 (2015).