Semantic enrichment methods may be used to identify relevant entities in textual documents. These extracted entities are part of knowledge graphs and thus linked by semantic relationships. This work explores the idea of navigating the semantic relationships among extracted entities as a way to search a text corpus. A modular software system (including document management, semantic enrichment, data consolidation, and data integration) has been designed, to o er a visual user interface for such navigation on top of an arbitrary corpus of textual documents. The software, called arca, has been used in a real use case: to search in the book catalogue of a publishing house. The evaluation carried out with a set of potential users has shown so far the feasibility and e ectiveness of the approach. Critical issues and potential limitations of the paradigm have also been found and are discussed.
Searching and exploring a vast text corpus has often arisen as a human need. Traditionally, a search is based on manually curated metadata classifying documents, for example, by arguments, authors, etc. In the digital era, the search moved from physical cabinets to databases. Albeit being a useful paradigm, the maintenance of metadata is an expensive process, which becomes progressively more expensive and less reliable with the increase of required detail. E orts of ? This work has been partly supported by projects ARCA (POR FESR Lazio 2014{2020 - Avviso pubblico \Creativita 2020", domanda prot. n. A0128-201717189) and STORYBOOK (POR FESR Lazio 2014-2020 - Avviso Pubblico \Progetti di Innovazione Digitale", domanda prot. n. A0349-2020-34437).
Copyright c 2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). This volume is published
and copyrighted by its editors. IRCDL 2021, February 18-19, 2021, Padua, Italy.
transitioning to electronic documents (either created natively as such or
digitized) have helped, enabling the direct text-based search of the content.
Semantic enrichment methods, as named-entity recognition and linking (NERL) [
NERL successfully enabled users to search and analyze text corpora more
e ectively [
For the sake of the analytic approach, we frame our e ort through a set of research questions. The questions elicited below are relevant to the application of KG-based approaches for the exploration of text corpora.
Q1. Would users, exploring a corpus of text, pro t from the semantic navigation of the associated KG of topics? Q2. What kind of user interface would e ectively support such navigation? Q3. What kind of users, scenarios, and tasks would bene t from this interaction paradigm? Q4. Do building and maintaining a semantic enrichment and KG creation pipeline necessarily involve high upfront costs and highly skilled developers? 1.2
To try to reply to the questions above, we designed the presented study. H1 (relevant to Q1 and Q2). Users will be able to explore a text corpus through a KG-based user interface which o ers the following main functions: (a) nd concepts through text search, (b) visually navigate the concepts and their relationships, and (c) show documents relevant to the selected concept. H2 (relevant to Q3). Given a corpus of text in a speci c domain, it will bene t both users with little knowledge of the domain (by supporting semanticallyrelevant discovery) and domain experts (by enabling topic-oriented visual organizations of the documents).
H3 (relevant to Q4). It is feasible to build a ready-to-use complete system, including both semantic enrichment pipeline and web-based front end, able, with only some con guration, to be applied to any speci c corpus to enable the KGbased exploration. 1.3
A comprehensive software system has been previously envisioned and proposed
[
In order to ensure the whole solution is useful for potential users it has been implemented and evaluated within a speci c case study: exploring the book catalog of a medium-sized publishing house specialized mainly in ancient history. The concrete case study o ered the context for fruitful exchange among the stakeholders that are often involved in scenarios of information retrieval and library search: who maintain the corpus (the publisher), who need to search the corpus (researchers of the eld and interested individuals), who develop the software solution (in this case the authors of the present study).
The remaining sections are organized as follows. Section 2 presents related work about visual information seeking. Section 3 reports the design process starting from identifying user requirements to the development and implementation of the nal interface. Section 4 describes the pipeline of the proposed system and the technologies used during the implementation. Section 5 reports the evaluation process and analyzes the ndings. Finally, Section 6 discusses future research directions. 2
In this section, relevant literature is brie y surveyed for relevant works, starting from traditional systems for visual information seeking to tools for semantic enrichment of unstructured text and visualization/exploration of semantic data as KGs, both in the general case and the speci c case of a corpus of books. 2.1
There has been a large amount of work in the literature about visual
information seeking [
There has hence been recently a lot of research on how to attach semantics to
unstructured data [
The GLOBDEF system [
Apache Stanbol 3 is a set of components able to o er various services for semantic enrichment, visualization of KG and the management of metadata. It is advantageous and can be integrated with our system, but on itself, it does not o er a ready-to-use system. 2.3
The extracted semantics can then be extremely useful for exploring the data, but
they are not xed and homogeneous like a set of prede ned metadata. Therefore,
data models and visual user interfaces need to deal with these complex and
heterogeneous data. The Semantic Web [
Many user interfaces for visualization and exploration of KGs exist, and new
ones are being developed every year, especially in the context of Semantic Web
and Linked Data technologies [
Metaphactory [
A customized version of Ontodia is included in the software system presented in this paper. The customization enables the use of such KG navigation to search in a text corpus of documents. 2.4
Many tools face the challenge of the exploration of the contents of a digital library. In particular, two are in the same direction of this work.
Yewno Discover [
is not adaptable and exible to di erent contexts of use, except with ad hoc adjustments.
Talk to Books4 is a tool by Google to explore ideas and discover books by getting quotes that respond to user's queries helping users nd interesting books that may not be available through keyword search, but does not admit the visual exploration of the RDF KG which allows users to discover, visually, connections between concepts and books. 3
The speci c use case of the publishing house o ered the opportunity to adopt a user-centred design approach to identify and re ne the system requirements. From informal interviews with publishing house representatives and a team of researchers in the same domain, an initial set of requirements has been identi ed: { the user should be able to search entities textually; { for an entity, the user should be able to see the relevant books; { the user should be able to navigate among entities, following semantic relationships between them; { for a document, the user should be able to access the basic information and be informed on how to obtain it (buy it from a bookstore, borrow it from a library, etc.); { any user should be able to perform the operations without the need of being taught how to, following established interaction patterns and metaphors. 4
The software system has been implemented and tested in the context of the speci c use case, but it is designed for general use. The aim is to o er a readyto-use package to explore visually any corpus of texts through a specialized KG. 4.1
The system is composed of a pipeline to build the KG and a web-based front end to search the corpus using the KG. The pipeline is composed of three steps: newly added documents of the corpus enter the pipeline; in the second step, semantic enrichment services extract information from the documents; in the third step, the generated data is consolidated locally in a way that it can also be integrated with additional data provided by external services. RDF is used to represent all the data items in the pipeline, employing existing vocabularies and ontologies whenever possible and creating new terms if needed.
4.2
The visual user interface is composed of two main components (see Fig. 1). The rst component contains the visualization and search of the entities contained in the KG. It is a customized version of the Ontodia workspace (described in Section 2.3). The second component shows the list of documents associated with the selected entity, o ering further interaction with them.
nent (see part 1 of Fig. 1, left side) has the following features.
Searching graph entities. The panel on the left can be used to search for entities in the knowledge graph, corresponding in the use case mainly with entities from DBpedia. For example, typing \Rome" the user gets all the entities containing that string in their label. One or more of the returned entities (e.g., the one corresponding to the city of Rome) may be loaded to the graph navigation panel through drag-and-drop.
Knowledge graph navigation. The central panel allows the user to navigate the KG. Starting from any shown entity, its connections can be expanded (hence adding the connected entities to the graph), either generically showing them all or selecting only some connections of a speci c semantic type5 (e.g., birth place). Furthermore, the connections among shown entities are shown by default, as they may be of interest. This panel is coordinated with the document list panel 5 By the informal term of semantic connection type we refer here to RDF properties (described below and shown in part 2 of Fig. 1, right side) so that the latter shows the list of documents which include, as a topic, the entity currently selected in the former.
Documents as entities. Apart from being shown in the document list panel, documents can also beo explored as entities themselves in the KG exploration. They are linked to their topics by two types of semantic connections: concept for any entity found in the text, top concept for the ones recognized as main topics for that text. This choice enables di erent ways to interact with the system: { starting from a document, exploring its topics and then possibly other documents from them (e.g., in Fig. 1, from the book The Tale of Cupid and Psyche to the topic Rome and then to the book Scutulata Pavimenta); { from shown entities, visualizing which documents are about two or more of them (e.g., in Fig. 1, the book The Tale of Cupid and Psyche is both about Rome and, speci cally, about Castel Sant'Angelo).
Kinds of entities. Di erent colours are used as an aid to distinguish three broad sets of entities: { DBpedia entities not found in the corpus are in blue; { DBpedia entities found at least once in the corpus are in green; { documents are in red.
Document list. The document list panel (part 2 of Fig. 1, rigth side), which can be shown or hidden as needed, shows the list of documents associated with the entity currently selected in the graph exploration panel, i.e., the documents whose extracted entities include that one. The documents may be shown ordered by year of publication or by relevance (if for that document it is a main topic or just a topic). By clicking on the info button of a book, a modal window with further information on the document is opened. The information includes the list of snippets, i.e., all the textual contexts of the document in which the selected concept has been found. 5
As anticipated, the system has been applied to the use case of a publishing house specialized in ancient history. Speci cally, it has been tested on a corpus of 112 books, a subset of the full catalog of the editor. The evaluation has been carried out with the help of a group of researchers who are experts of ancient history and speci cally of the topics covered by the set of books. The evaluation of the system has been divided into three phases. In the rst phase, the researchers, as domain experts, assessed the quality of the semantic enrichment process. In the following two phases, the system as a whole has been evaluated through user tests. In both phases, users interact with the visual user interface, in the rst phase without given constraints, in the second phase with a set of tasks and a more structured setup. 5.1
The researchers have read deeply ten books contained in the system, and for each book have evaluated the quality of: { the correspondence of the top concepts with the main topics; { the correspondence of the concepts with entities mentioned in the book; { the disambiguation of the words.
The ndings were analyzed qualitatively, as the purpose was to test the feasibility of the whole system rather than to evaluate the NERL method per se. The quality has been deemed su cient to be used e ectively. Nevertheless, some issues emerged. They are described in section 5.4 along with ideas to approach them. 5.2
In the second phase of the evaluation, two users were invited to experiment using the user interface without any speci c constraints.
Three types of reactions have been observed: { positive surprise in nding and verifying relationships between concepts that they were already aware of; { amazement in nding new unknown relationships; { displeasure in not nding expected relationships, due to the lack of content. Overall, from this rst observation of the user interface use, the results have been that the navigation was smooth and stimulating in the exploration of the contents. 5.3
In the second part of the evaluation, six users participated in a task-oriented evaluation.
Before starting with the compilation of the questionnaire, users were invited to watch four video tutorials in the appropriate section of the interface to familiarize themselves with the arca commands and functions. They were given a questionnaire containing task-instructions interspersed with the related questions, some of them open ended, some of them requiring a number on a Likert scale from 1 to 5. The questionnaire was divided into four sections: (I) three basic tasks followed each one by a numeric question on the di culty; (II) ve identical macro-tasks (decomposed in seven sub-tasks) each one of them consisting in a set of search-exploration steps starting from a di erent topic, chosen by the user, interspersed by two questions after each sub-task, one numeric and one open ended, to evaluate the data quality of the explored results; (III) four general open ended questions; (IV) other three pairs of questions, one numeric and one open ended, to evaluate the technical aspects of the interface.
Using and aggregating Likert scale scores was helpful in quantitatively summarizing the sentiment of the users, albeit the number of users involved in the test is too small to look for statistical relevance.
I To answer the rst section of the questionnaire, users were assigned three precise activities to perform, and on the basis of the results obtained, it been assessed the ease of use of the system and the satisfaction obtained by the exploration of resources. From Figure 2a it can be seen that 100% of the test considered the navigability of arca from simple to very simple. In Figure 2b the 87% consider consistent the books related to the selected entities and in Figure 2c 83% is satis ed with the list of search results connected to a concept. II In the second section of the questionnaire, instead of assigning speci c tasks, each of the six users was given the freedom to choose ve di erent concepts from which to start their own research and ve di erent books to explore at own choice. For each chosen concept and book, the questions assessed the quality of the explored search results. From this group of questions, it emerged that for the users, the obtained results are mostly coherent with what they expect. III In the third section, four general questions were asked, which are shown below along with an outline of user responses.
Among the most useful features, users that have tested the interface have identi ed: { the possibility of identifying and exploring links between concepts which makes the research method immersive and stimulating; { the search for books starting from concepts and, above all, starting from other books; { the fact that it is an open system and modulated on improvable and replaceable components; { it is a potentially very large container of editorial products; { the ability to save search paths and export them as SVG and PNG images; { the display of integrated data (texts, images, links); { the direct connection with the catalogs of the publishing house; { the possibility of verifying the concepts extracted from the corpus of books through the \info" button of the component containing the books; { the division of the concepts extracted from books into concepts and top concepts.
Among the weaknesses of arca at the actual stage, users have identi ed: { sometimes errors of disambiguation and restitution of entities not always relevant; { sometimes errors in extracting concepts from the text; { little information contained in the system to satisfy curiosity in exploring a larger catalog of books to discover more connections; { the system o ers multiple features that must be properly explained to allow the user a complete browsing experience.
better system? The features that, for the users, could improve the system are: { integration with other KGs; { improvement of the entity extraction module; { improve the order and organization of the connected elements to the entities;
Users have noted that it might be interesting to implement the following features: { o er the possibility to save the search history; { create personal bibliographic lists with the results of the searches in a dedicated space on the board (exportable and downloadable); { create a support system that can help the users to know in a more easily and understandable way, how to take full advantage of all the features of arca IV In the fourth section, there are questions to assess the technical aspects of the interface. The Likert scale questions and answers reveal that the loading times and the organization of the visualization of the results can be improved. In the motivation following the Likert scale questions, users told that they have experienced slowdowns in particular in viewing the book catalog, and in the loading of video tutorials. The system obtained a more than satisfactory performance in recommending relevant editorial products and received a high score in terms of usability; the simplicity of use; user satisfaction with the results shown; consistency of the contents with the s domain of the publishing house; attractiveness of the system.
Nonetheless, the evaluation of the quality of the entity extraction (Section 5.1) highlighted some issues that need to be addressed in future versions and considered in related works, like the wrong resolution of acronyms and abbreviations and the incorrect disambiguation of entities due to the absence of the correct entity in the DBpedia KG.
Another feedback is that some users complained about a relatively small amount of information in the internal KG (built with the concepts of 112 books and the related metadata). It is expected that when the catalog of books is numerically more signi cant, the chance of discovering new information and connections while browsing the KG will increase.
Finally, based on initial observations, it has been seen that using the system at rst glance can be di cult without viewing the video tutorials. As a lighter alternative to video tutorials, an help component could be implemented to accompany the users in the rst searches and thus make them independent in exploiting all the possibilities of exploration that the system o ers. 6
Conclusions and future work arca is an innovative system based on the visual semantic search that allows the exploration of a text corpus. Through a knowledge graph-based navigation, users can start from any relevant entity and reach other entities related to it, discovering in which books or articles each entity is present and evaluating which of these results are useful for their research. The user studies conducted so far con rm the amenability of the proposed system to domain experts who were able to perform non-trivial tasks of search and exploration, tasks that would be more cumbersome to execute with the search tools they are used to. Feedback gathered from users suggest that the proposed exploration mechanism tends to amplify the user experience by also o ering opportunities for further study and discovery of sources, themes, and materials, which have the potential of enriching the research process with new ideas.
Through the presented user study, many desiderata have been collected, and they can be used to guide further development and experimentation in this context. Furthermore, in order to extend the evaluation to more users, a more extensive indirect observation study has been planned. In addition to the questionnaire, the analysis will be further completed with objective data gathered by tracking users' activity through interaction logs.
As a potential future direction of research and development, the scope of the semantic enrichment process could be broadened to other document elements, such as images and captions, enriching KG exploration.