This paper proposes the use of Kibana and ElasticSearch for the efficient recommendation of job offers to students, in order to allow them to improve their professional career. Both Kibana and ElasticSearch tools are integrated in a self-developed web-system at UNED. By using Kibana, users can get a graphical representation of the job situation for a time period, which could indicate them what is the knowledge most wanted by companies. Also the system can let users search for information using keywords within the job offers stored in the system. This is intended to provide users with a picture of the job market, which allows them to focus their training on the most demanded professional profiles by employers, or to find hiring trends. For storing all the collected information, ElasticSearch has been deployed in order to perform agile queries from not-structured information, like job descriptions.
In recent years, the production of digital information has raised in the context of
employment. One of the key features of this growth is the existence of multiple massive
sources of information, including social networks (such as InfoJobs [
This variety of heterogeneous pieces of information gives birth to the Big Data paradigm, which has a great potential to revolutionize different dimensions of society.
Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
However, there are still research areas to explore to obtain a higher throughput. In the field of education, the Learning Analytics (LA) area covers this paradigm.
Our current work is focused on analyzing the students’ learning to improve it or to make recommendations, but not in the students’ learning process itself. More specifically, our attention is focused at the field of Academic Analytics (AA) in an institutional level, in order to help our University to analyze students’ learning profiles and academic performance. It is very important for an Institution or University to give their students the possibility to check the job offers available depending of the level of knowledge. Also, the administrative staff could carry out this task in order to analyze the requirements of companies. This can be a starting point for the Institution to perform additional studies in order to improve the employability of students.
Therefore, this paper proposes a self-developed web-system at UNED based on
Kibana and ElasticSearch. It can obtain information from several sources of job
offers, such as Infojobs and Monster [
Kibana [
From the back-end side, job offers are gathered by using both web-scraping
techniques and/or by means of APIs, depending of the availability of information in the
sources of information [
The structure of this paper is as follows: Section II depicts the data extraction and processing procedures. After that, Section III focuses on the data visualization phase for the recommendation of job offers to students. Finally, Section IV discusses our conclusions and suggests guidelines for future work. 2
The information hosted on the web is normally unstructured, or at most
semistructured –in the case that access mechanisms have been provided to developers.
Some websites take advantage of APIs (Access Programming Interfaces). These APIs
operate as a bridge between the users’ browser and the REST services and they allow
getting JSON responses to source’s invocations by using the field selectors. For
instance, the LinkedIn API has many restrictions which limit its use. For this reason,
this data collection method is not the most appropriate in some cases, as stated in [
As for our web-system, the administrator can manage the extraction and processing of the different sources of information, for instance, starting, stopping or modifying parameters such as the time between captures. Job offers will be stored in ElasticSearch for a more advanced management of job offers.
Once data is retrieved from its source, our self-developed web-system carefully
preprocesses information to guarantee the quality of the data collection and its later
aggregation [
On the other hand, we use a relational database to save the data provided by web crawling tools in a structured way. It has also been considered appropriate the storage of data in a powerful server of searches, in order to experiment with flexibility which functionalities and performance this type of product presents when storing and performing searches by text. Within this category we can mention Apache Solr, ElasticSearch, or MongoDB. In these kinds of databases, the information can be stored in the form of JSON documents, which are indexed so that access to information is much faster than with relational databases. These are open-source, and can be used for the most common programming languages by using REST services. They also offer additional tools that give value and complete the initial product, among others, data visualization and analysis tools. The most powerful and agile database is ElasticSearch, after studying all of them. ElasticSearch is based on Lucene, so it can be seen as a powerful search-engine for texts, which can be deployed easily in a distributed way. Additionally, it can handle RESTful services in an efficient way. 3
The final decision of using ElasticSearch compared to any other competitor was also
due to the need to implement a system that offered different graphs that allowed users
to interpret the labor market. This is where Kibana [
Kibana only queries data held in ElasticSearch, which acts as both the data store and the analytics engine. Fig. 2 show a visualization panel, a Dashboard, which has been fully-integrated from Kibana as a frame in our self-developed web-system. This panel is totally flexible and adaptable from the Kibana interfaces, as shown in Fig. 3. When any of the graphics of the panel in use is modified in Kibana, this is automatically updated in the associated frame integrated in our web-system.
As an example, Fig. 2 and Fig. 3 show a set of analytics for job offers collected from the different professional networks that our expert system supports. In both figures, we can observe the ten provinces with more job offers. In this case, Madrid and Barcelona are the cities with more job offers. The second graphic is related to number of categories and subcategories of job offers. The “commercial_y_ventas” category is the one with more job offers and, as the most popular subcategory, we have “comercial”. In Fig. 3, we can observe additional graphics, and other relevant information, such as the amount of job offers located in the system, and the date of the last offer captured by the system.
Finally, users can perform free-text searches by employing the powerful ElasticSearch engine from our application, so looking for relevant offers that could be interesting from his/her point of view. In Fig. 4, an example is shown. In this case, the user has performed a search using the “Informática” word, and our system has searched and presented all the related job offers located in the ElasticSearch database. Learning does not take place in an isolated context, but occurs in a social, cultural and working atmosphere. For this reason, it is very relevant for our students the analysis and aggregation of external sources of information from current professional networks. This will help students to guide their education and, therefore, their professional career. For this purpose, several professional sources of information, such as Infojobs and Monster, are available on the Internet. For this reason, a large amount of data is currently being obtained from these websites in the field of Big Data, and more specifically, in the topic of Academic Analytics (AA), as stated above.
This work proposes the use of Kibana and ElasticSearch for the recommendation
of job offers to students to improve their professional career. Both Kibana and
ElasticSearch tools are integrated in a self-developed web-system. By using the Kibana
interface, users can get a graphical representation of the job situation during a period
of time, which could indicate them what the most demanded knowledge by companies
is, and search relevant keywords that fit their necessities or preferences. This is
intended to provide users with a picture of the job market that allows them to focus their
training on the professional profiles most demanded by employers. To store the
collected information, ElasticSearch has been deployed for its ability to perform agile
visual queries from not-structured information, like job descriptions. Both Kibana and
ElasticSearch are fully-connected and integrated with our web system, used for
linking job offers and users’ preferences. The recommendation step matches the ontology
proposed in [
Authors would like to acknowledge the support of the European research project ERC-2015-STG-679528 POSTDATA, and the local project (2014I/PPRO/031) from UNED and Banco Santander; and the Region of Madrid for the support of E-Madrid Network of Excellence (S2013-ICE2715). The authors also acknowledge the support of SNOLA, officially recognized Thematic Network of Excellence (TIN2015-71669REDT) by the Spanish Ministry of Economy and Competitiveness.