DAICA - Digital Assistant Investigating Cultural Assets Lothar Hotz1 , Dan Cristea2 , Justyna Pietrzak3 , Martin Povazay4 , Brigitte Rauter4 , and Daniela Buleandra5 1 HITeC e.V. c/o University of Hamburg, Germany, hotz@informatik.uni-hamburg.de 2 University Alexandru Ioan Cuza and Romanian Academy, Romania, dcristea@info.uaic.ro 3 Eleka Ingeniaritza Linguistikoa S.L., Spain, justyna@eleka.net 4 P.Solutions Informationstechnologien GmbH, Austria, {martin.povazay,brigitte.rauter}@psolutions.at 5 SIVECO Romania SA, Romania, Daniela.Buleandra@siveco.ro Abstract. Besides web pages, the web offers access to an immense va- riety of digitized source material, inventories and catalogues hosted by libraries and archives relevant for humanities and social sciences (HSS) studies. In practice, a remote access to HSS information is consider- ably hampered by several barriers: Researchers interested in a specific topic do not know which institution harbors information related to a specific topic; Data collections are equipped with unique user interfaces and offer different data structures; Language barriers impede informa- tion exploitation; Retrieval mechanisms do not provide intelligent access to semantically related information. In this paper, we describe an Dig- ital Assistant Investigating Cultural Assets (DAICA) for research and information procurement in HSS, guided by the vision of a digital in- formation space of cultures. The DAICA will support HSS studies by autonomously identifying appropriate resources and presenting topical investigation results. In particular, the DAICA will integrate technol- ogy and provide a solution for analysing historical digitized documents, performing semantical search in deep data structures, automatic transla- tion, extending a search by meaningful relations, creating summaries of identified resources, and providing user interactions for complex search results. Keywords: Semantic search, machine translation, summarization, op- tical character recognition, cultural heritage 1 Introduction The web offers an immense variety of digital or digitized source material, in- ventories and catalogues relevant for studies in humanities and social sciences (HSS). Institutions such as libraries, museums, archives, local and regional au- thorities, parliamentary and media documentation services provide a wealth of 2 information, organized on local websites and in principle accessible for inquiries and research from anywhere in the Internet, however not by ordinary search engines - they reside in the ”deep web” and require special access methods. Resources exist in a multitude of media types: unstructured and structured text, with and without metadata, full text accessible or not, images, and videos. Media comprise historical or cultural texts, biographical information, newspa- pers articles, maps of cities, regions, and countries, paintings, photographs, and much more. In addition to varying data formats, the user interfaces to these re- sources differ widely, both regarding language and functionality. Hence, research is difficult, and the outcomes are all too often incomplete and insufficient. Typ- ically, tedious manual investigations are needed to study a specific topic: one has to get into contact with many possible sources, get acquainted with access modalities, retrieve data, commonly by language-specific key-word search, deter- mine relevance across language barriers, and summarise the resulting material. In HSS studies, the task is further aggravated because of diverse cultural re- gions, each with its own history and tradition, and hence, often with a different understanding of seemingly identical terms. This situation is faced by scholars, students, journalists, public, and media if they want to investigate abroad data sources, not only data of their local libraries. In principle the same applies to investigations by business and com- mercial aggregators. There is clearly the need to pave the way for barrier-free access, i.e., support in accessing distributed repositories, in translation services, in fast interpretation of new unknown resources, and collaboration with other interested users about this matter to cultural data. Fortunately, basic technology for multilingual and semantically enhanced search in multimedia databases is available. But in order to effectively sup- port HSS studies, techniques have to be adapted and integrated. For example, relevance criteria including temporal and geographical proximity, cultural vicin- ity, or taxonomical distance of terms must be taken into account for a semantic search to be effective. Optical character recognition (OCR) must be invoked for searching scanned documents; Machine translation must be used for iden- tifying, linking and presenting relevant multilingual data, summarisation and visualization and user-device interaction facilities must be provided for complex and multifaceted data. Altogether, these methods can allow effective retrieval of historical and cultural data from the deep web. As a technological innovation, this paper presents a concept of an intelligent HSS research assistant called Digital Assistant Investigating Cultural Assets, DAICA, which will be used by scholars of the humanities and by the interested public or the industry having the need for investigation a specific topic. Sum- marisations will be returned of semantically related documents, articles, texts not only of web-pages but also of other data sources. This paper describes the use cases of a DAICA (Section 2), a new search procedure in Section 3 integrating semantic search, machine translation, OCR, entity discorvery, and summarization. These technologies are integrated in a configurable framework (Section 4 and Section 5). 3 2 A Use Case of DAICA Figure 1 presents a use case of a digital assistant presented in this paper with user and background interactions processed by DAICA. Main features are the proactive discovering of the user’s interest topics (“A1”), the background actions based on localization if the users is moving (“A2”) and on OCR for interpreting original ancient documents. By further observing the user’s writing activities, DAICA computes related sources in the web and, hence, supports the user by his investigation activities (“A3”). Fig. 1. Use Case “New Type of Assisted HSS Study” 4 3 A New Integrated Search Procedure A main task of DAICA is the transparent integration of semantic search, machine translation, and all other capabilities such as summarization, OCR, and entity discovery (see Figure 2 which illustrates the complete process). A user spells out implicitly or explicitly the search specification (query) in his/her language, the source language, in the example German. DAICA translates this query and its ontological enhancements into the target language (here Romanian) of a data source and starts the search. The results and their enhancements through links and summarisation are again translated back into the source language. In the following section, the details about the involved components are given. Fig. 2. Process of a search session with ontological enhancement, DAICA instantiation (see next section) retrieval, and machine translation 5 4 The DAICA Framework In order to obtain sustainable, reusable results the objective of DAICA is to build a general framework which will be used for the development of complex spe- cialised architectures, each accommodating different capabilities, selected dur- ing configuration sessions. These capabilities realize the basic technologies for investigation tasks, i.e., optical character recognition (OCR) for enabling the translation of text images into words, search which takes the meaning of queries and documents into account, machine translation for interpreting documents in foreign languages, summarisation for getting a quick overview of a document or article, entity and link discovery for identifying important persons and subjects in a text. However, it is not evident which capabilities to use at which time, or, if capabilities come in variants, which version is best for a given investigation task. Therefore, DAICA is defined as a framework, its infrastructure, and a suitable interface technology which can be used to interactively assemble architectures for selecting suitable components which implement the capabilities for a given investigation task. Various kinds of users, ”aggregators”, will use this kit to build DAICA configurations that best support their own needs or meet the investigation requirements of others. DAICA instantiations will constitute another layer of outputs resulting from DAICA interactions. A DAICA instantiation (or instance) represents the combination between a DAICA configuration and a specific set of acquired re- sources from different data sources (usually referring to a specific topic a certain user has worked on). These resources will be accumulated by a user (or a commu- nity of users) during a series of work sessions with DAICA. Hence, all interactions typically with specific investigation goals and directed to specific resources, are stored, catalogued and offered for post-research re-use, for the benefit of their creators or future users. Hence, DAICA instantiations are collections of resources with respect to a topic. Examples of DAICA configurations can be: – DAICA-1: Capability to process contemporary German, Spanish, English and Romanian, with OCR, indexing, external linking of name entities, sum- marisation, and translation between these four languages; – DAICA-2: Processing German and Romanian texts from 1850 to present date, OCR including the Gothic German and the transitional Cyrillic alpha- bet used in Romania in the middle of the XIX-th century, indexing and exter- nal linking of name entities, time expressions, summarisation, and translation between these two languages. Examples of DAICA instantiations can be: – Based on DAICA-2: Links to the bibliographical sources in the Library of Hamburg and Academy Library of Bucharest, knowledge-base with dated entries related to the migration in Germany and Romania in the XIX-th century; 6 – Based on DAICA-1: Links of German academic libraries with information from Basque archives, in relation to investigations accomplished by German scholars in the Basque Country in XIX century. Hence, an instantiation summarizes all information about a specific topic, e.g., content identified in some libraries, notes made by the user. Furthermore, if made public, other users can make use of and refine such previously created instantiations through the DAICA instantiation retrieval mechanism. As such, the DAICA instantiations are the base for building a community discussing and further developing cultural topics. DAICA uses a number of already existent technologies, which will be adapted to comply with the actual requirements. The DAICA capabilities include the following features: – Specification and customization of the investigation task (proactive and trig- gered search specifications); – Specification of data sources, their access, and their content in the form of metadata schemas, languages, or ontologies and used terminology, hence enabling access to foreign data and content without the need of manually traversing user interfaces or interpreting a library structure (by data source profiles); – Analysis of ancient digitized documents (by pattern recognition and OCR, word spotting); – Deep semantic search through data sources (by indexing and semantic search); – Automatic translation of queries and resources (by machine translation); – Linking of resources with expressive relationships on the basis of semantic entities (by entity identification, reference resolution, detection of temporal and spatial relations); – Creating summaries of the identified resources (by automatic summarisa- tion); – Friendly end-user interfaces for the visualization of complex search results and dependencies in the Web for different types of devices: laptop, tablet, smart phone (by innovative visualization and user-device interaction); – Easy configuration of new processing architectures to support a wide range of thematic investigations (by configuration facilities); – Projects profiling for storing, retrieving and sharing of resources as DAICA instances (by instantiation facilities). In summary, these features will be integrated in a generally applicable and cus- tomizable technological framework that will allow easy configuration of new ar- chitectures in order to help researchers and other categories of users to perform assisted cultural HSS investigations. Once installed in a DAICA platform, the framework can be used by aggregators (libraries, research institutes, administra- tion) to configure new applications that will allow public users to get access to new data or to administer previously curated DAICA instantiations. 7 5 Technologies for DAICA 5.1 Existing investigation tools The widely used Aleph integrated library system provides academic, research, and national libraries with the efficient, user-friendly tools and workflow sup- port they need to meet the increasing requirements of the industry today and in the future. Built on an Oracle database, Aleph runs on a range of operat- ing systems. Employing system-wide XML technology, Aleph offers third-party integration through an XML gateway. The product is based on industry stan- dards, offering the ultimate in resource-sharing capabilities, full connectivity, and seamless interaction with other systems and databases. Another solution used in libraries is DigiTool, which enables academic li- braries and library consortia to manage and provide access to digital resources, both those that are created for use within the institution and those that are collected and maintained by the library for the benefit of the public. Since many resources have a public exposure on the Web, other existing inves- tigation tools or techniques which can be used for searching are the Web search engines and crawlers. Some open source or commercial tools (which can influ- ence the solution) are: Datapark, ebhath, Eureka, Indri, ISearch, IXE, Lucene, Managing Gigabytes (MG), MG4J, mnoGoSearch, MPS Information Server, Na- mazu, Nutch, Omega, OmniFind IBM Yahoo! Ed., OpenFTS, PLWeb, SWISH- E, SWISH++, Terrier, WAIS/ freeWAIS, WebGlimpse, XML Query Engine, XMLSearch, Zebra, BBDBot and Zettair. Besides search itself, one technique to be used when combining HSS data from multiple sources is data integration. State of the art approaches for data integration have adopted a schema-first (e.g., ETL, enterprise integration), a schema-never (e.g., search engines), or a schema-later (e.g., dataspaces) method- ology. Such tools provide the basic search and data access interfaces to library content. For DAICA, libraries operating those tools can and will be integrated through data source profiles. Furthermore, the provided search facilities of the tools will be used by the semantic search capability to perform keyword-based search. A lot cultural assets are currently published through EUROPEANA. EURO- PEANA bases its search functionalities on who, what, where, when and corre- sponding restrictions for media type, language, country, and provider. DAICA will base the search on semantic ontologies and multilingual access, thus, fa- cilitating the document access for users. However, through the envisioned data source profiles, EUROPEANA can be integrated in the DAICA framework and, thus, be part of a DAICA investigation. 5.2 OCR, pattern recognition For identification and retrieval of digitzed but not yet recognized documents, DAICA includes OCR (Optical Character Recognition) tools. The main chal- lenges which have to be faced are the following: 8 – OCR must be based on a variety of historical fonts and spellings. – Document images may have poor quality and may require image enhance- ment. – Character and word recognition may be ambiguous due to noise. – Word, sentence and semantic context must be exploited for disambiguation. There exist several commercial and open source OCR tools, which perform high- quality OCR (up to 99%) for standard fonts and low-noise conditions [1]. On the other hand, character and word recognition results may be quite poor (be- low 80%) without prior knowledge of the font and without exploiting context information. Hence, DAICA applies several innovative techniques to achieve high-quality OCR. First, OCR tasks are supported by their semantic context using meta-data and ontologies. Hence, ambiguities can be significantly reduced. For example, ambiguous readings can be refuted if the semantic distance (computed from an ontology such as WordNet) to the investigation topic exceeds a threshold. As a second innovative technique, applicable to manuscripts or unusual fonts, DAICA will allow word spotting based on patterns supplied by the investigator. This way, occurrences of similar patterns can be retrieved from a document. A third technique, mainly applicable to handwritten documents, will be the use of an advanced text-line finder which can cope with varying line orientations. Thus, the approach for DAICA will be mainly based on existing OCR tools of the partners and open source tools, as well as low-level and context-supported computer vision and manuscript analysis [2,3,4]. 5.3 Semantic search A central goal of the DAICA is to provide support for studies of cultural heritage by extending keyword-based search to a much broader search based on semantic relations. A semantic search has the advantage of narrowing down ambiguous word meanings, especially across language barriers, and allowing proactive back- ground search for related information. This goal can be achieved by a variety of techniques which try to take the intention of the user and the meaning of a query into account when searching in data sources. There exist several approaches for semantic search as documented, for ex- ample, in the surveys, [5,6,7]. DAICA lays the focus on exploiting ontological information which are used in two fundamental ways: (i) to define, refine and expand the query topic; (ii) to find semantically related information in data sources. Several publicly accessible implementations of semantic search approaches exist, including QWant, GoPubMed, Swoogle, and Google’s Knowledge Graph, which deal with specific kinds of ontological representations. These techniques, however, do not meet the requirements for the intelligent agent conceived in this work: (i) DAICA will have to access a large number of heterogeneous con- tent structures used in the archiving institutions for cultural heritage or similar 9 data aggregations. Some may be supported by full-fledged Semantic Web on- tologies, others by customized categorization schemes. In consequence it will be necessary to invoke ontology alignment in some form; (ii) Search will be mul- tilingual, crossing language barriers between the user and information sources; (iii) In DAICA, the user can define a query by several kinds of topic descriptions, ranging from keywords, annotated images, graphic patterns, to coherent texts. Hence several heterogeneous measures for semantical distance will play a part, for example taxonomical distance, relatedness by names, time or geographical location, or chains of ontological structures; (iv) The user will be supported by proactive search, i.e., by autonomous background explorations through entity and link discovery in user’s text writing; (v) Access to DAICA will be possible via mobile devices, and rendition of results will include summarisation. The software for individual techniques is mostly available either as open source or detained by the authors. The main task for the DAICA is to conceive and integrate a tool combining the techniques in a user-friendly way. 5.4 Ontology management In our approach, ontologies play an important role for obtaining meaningful search results in support of a user’s investigation. All essential DAICA function- alities resort to ontologies, in particular semantic search, language translation, interpreted OCR, entity discovery, topical linking, and summarisation. Ontolo- gies may provide concept names and definitions in terms of relations to other concepts, for example generalization, specialization, synonyms and antonyms. Standardized properties relate entities to important search criteria, such as lo- cation and time. Due to the highly heterogeneous data sources of the cultural heritage and diverse evolved standards, investigations with DAICA have to cope with mul- tiple ontologies in different languages, ranging from carefully designed OWL ontologies to simple databases characterized by metadata schemes. In order to determine the relevance of resources for a user query, DAICA must be able to align these ontologies with the semantics of the query. Several methods for query answering based on multiple and multilingual ontologies have been devel- oped in the past decade, see [8,9,10] for surveys. Typically, there is a matching (or alignment) step where correspondences between heterogeneous ontologies are determined, and an interpretation step, where information relevant for a query is extracted. In the DAICA infrastructure, ontology matching and interpretation will be performed for ontologies based on standards such as Dublin Core or Schema.org, on controlled vocabularies (WordNet and thesaurus vocabularies), and on exist- ing biographical data standards and classifications. In several countries data sources are described by authority files of standardized metadata, in Germany: Gemeinsame Normdatei, beacon files, gazetteer data and links, files for Common public corporation data (GKD, Gemeinsame Körperschaftsdatei) with company and institution names, registers with personal names (PND), and Common norm 10 data file (SWD, Schlagwortnormdatei) with commonly used tag words, cate- gories, and subject headings. Mass data with named entity tagging and recogni- tion data will enhance the scope of results and open up semantic relations and links to more resources. 5.5 DAICA instantiation retrieval A DAICA instantiation represents a DAICA configuration and the resources acquired by a user or a community of users having close scientific interests for a specific topic using this configuration. As such, a DAICA instantiation represents a complete investigation case which is both, a useful documentation for the investigator and a valuable resource for similar investigations of other users. It is the objective of DAICA to support all users of the DAICA community by a case base of instantiations and case-based retrieval mechanisms. Case-based information retrieval is a well-established technology, see [11,12] for surveys. While case-based retrieval has been originally conceived for feature- based object representations, applications to relational structures have proved quite successful [13]. More recently, case-based retrieval was further enhanced by ontology-based representations and corresponding similarity measures [14]. During the development of DAICA, a special theoretical attention will be given to an ontological organisation of the collections of DAICA instantiations. For example, issues of interest here are: demarcation strategies (when is it that two instances have to be considered as identical or distinct?), inheritance (is it that an instantiation A inherits parts of descriptions, sources, links, etc. from an instance B?). 5.6 Machine translation, multilingual processing in combination with semantic search and summarisation Developing of efficient machine translation is a long-lasting and multi-level pro- cess. DAICA uses a mixture of the mature technologies of statistical, example- based and and rule-based machine translation (SMT and RBMT). As basic fea- tures, DAICA includes: – Resource collection (semi-automatic parallel corpora extraction and dictio- nary building, with special emphasis on lesser-resourced languages and in- domain registers). These data will be used for training and tuning machine translation modules. – Development of the query translation module. Previous experience and ex- pertise of the partners will be used for adapting existing methods to language pairs of DAICA. SMT is language-independent, and the same toolkit can be used for any pair of languages provided specific single language texts and parallel texts for all language translation pairs exist. But the state of the development rule-based machine translation (RBMT) varies, depending on the language pair. 11 – DAICA will use the Apertium software [15]. Apertium is a classical shallow- transfer or transformer system, released under GNU Licence. Apertium in- cludes dictionaries for language pairs involving Spanish. The Apertium MT engine consists of the pipelined modules for morphological anlaysis, part-of- speech tagger, and text generators as well as Statistical Machine Translation (SMT) based on the Moses toolkit [16]. Hence, our summariser is multilingual at the architectural level, meaning that it incorporates a pipeline of modules which has the same structure irrespective of the language of the processed document. However, initial elements of this chain (among which, the tokeniser, the POS-tagger, the lemmatiser, the NP-chunker, the clause splitter, the name entity recogniser, and the anaphora resolver) are strongly language dependent. In the former ATLAS project6 , summarisers for Bulgarian, German, Greek, Polish and Romanian have been built, meaning that our general summarisation architecture has been adapted for all these languages by assembling basic-levels NLP modules supplied by partners. For DAICA, we will build summarisers for German, English, and Romanian, by re-using (and, where necessary, also enhanc- ing) the German and Romanian basic components and including open-source modules for English. The situation is somehow different when the language of the documents is old. Romanian or, for instance, has changed dramatically over time. Not only the lexica, grammar and syntax have evolved, but also the alphabet has changed from Old Cyrillic to Latin, with a mixture of the two, called the Cyrillic Transition Alphabet, used for a period in the middle of the XIX-th century. Based on previous work [17,18] we will study on diachronic Romanian morphology. 5.7 Entity and link discovery Recognition of entity mentions in texts (names of people, moments of time, countries, locations, events, organizations) and their correct interpretation in context is an issue of primary importance in DAICA. These mentions should open access gates to entries in the collection of accessible resources. Examples for points of interests are: – Identify entity mentions in metadata field values and full texts and, if neces- sary, do their ontological interpretations, e.g., identify temporal entities and historical dates and events; – Identify relevant relations between entities such as relations between in- stances: is-in (at