Both, historical as well as contemporary images depicting particular aspects of a certain culture, or a certain cultural practice, are widely available in a variety of formats. Typically, historical-cultural images can be found in museums, national archives and libraries both in analogue and digital formats. The Europeana image collection serves as a valuable example of a digital image collection available to the public, that is freely accessible and searchable. A signi cant amount of the collection, however, does not contain a rich semantic description of the cultural and social aspects represented in the images that would go beyond some metadata like author and title. To enable the enrichment of these images, we propose a project which brings together experts from digital humanities, linguistics, arti cial intelligence and semantic web technology. The project aims at analysing the contents of the images with a combination of computer vision, natural language processing and manual curation to represent them with a more descriptive and representative controlled vocabulary. This combination of di erent types of expertise to address the problem enables us to closely collaborate and learn from each other by taking di erent roles and perspectives. So far, the collaboration has contributed to the understanding of the detailed requirements from the digital humanities and socio-linguistic perspectives for the representation and processing of cultural images using semantic web technologies, such as multi-disciplinary thesauri, ontologies, computer vision and AI.
Fred R. Barnard famously once claimed that \A picture is worth a thousand words". This assumption resonates well with the aims of the collaborative project discussed in this paper. By cultural image, we understand any image (artwork, photograph, sketch, etc) that depicts cultural artefacts, practices, or social situations, where we particularly focus on food-related content in the context of the project described in more detail below. The historical dimension is just one of several di erent ones under the umbrella of culture and particularly alludes to places, persons or events. Several books have indeed been written about historical images (pictures, paintings, photographs, sketches etc) such as the Mona Lisa by Leonardo da Vinci. Except for a few Nobel artworks, no matter how much is written about them or captured in the metadata, there remains ample information contained in the images themselves which is left uncaptured in words. In many museums, cultural heritage and public archives, there are images which vividly and meticulously represent the cultural, social, political and other aspects of a society in time. The modern-day digital technology, however, heavily relies on the exchange of information in a form of text to present or represent things. Until recently, often only a few keywords or descriptive sentences were used to annotate, search and retrieve images. This has created a huge gap between the symbolic \thousand words" and the few keywords and descriptors associated with the images.
In this paper, we present a collaborative research endeavour which aims to
unpack the number of details contained in cultural images and represent them with
carefully selected, semantically interlinked multi-faceted ontologies, taxonomies
and thesauri. To this end, a group of researchers from the digital humanities
(socio-cultural linguists, archival curators) and computer scientists (computer
vision and AI experts and ontology engineers) bring the ChIA5 project
(accessing & analysing cultural images with new technologies) together, to carry out
pilot research in the area [
5 https://chia.acdh.oeaw.ac.at/ 57/143
The cradle of this collaborative project is an inspiring, informal brainstorming session held between a digital humanist, who is an expert in the area of lexicographic collections, a computer science expert and an ontology and knowledge engineer. The discussion identi ed several pertinent, to-date unresolved challenges related to historical collections including cultural images, pictures, photographs, sketches etc., collected from several centuries and stored in libraries, archives and museums. The main challenge identi ed during the discussion was that the majority of these resources lack any rich semantics (description) except for some available metadata, for example, date, author and title. Despite this fact, these collections, particularly historical images, bear rather rich aspects describing the detailed social, cultural, political, economic, etc. interaction of di erent societies. These aspects are not represented in a written form except for images well studied by experts. This led to the under-exploitation of the resources as a whole for research, educational and business purposes. Considering the richness of digital collections and the impact they have for understanding the culture and interaction between societies at di erent times, it became clear that digital humanities was concerned with the long-standing question of how to make these resources better accessible and available by drawing on the expertise from di erent disciplines. The challenging factor until now, however, is concerned with the questions of how to organise cultural image resources automatically, how to analyze their content and the aspects and represent them in a more detailed, interlinked and interrelated manner. It also focuses on how to support systematic search and retrieval of these resources using rich semantics which can go beyond searches based on facets such as author and title.
Since the inception of this research question, we conducted studies [
This opens up the path to the investigation of best knowledge organisation methods and tools that support semantic annotation/enrichment, and semantic search of historical images.
With the three major research questions emanating from the rst question, we developed a collaboration between four major disciplines: digital humanities, socio-cultural linguistics, computer science (computer vision and AI in particular) and knowledge organisation and semantic web technologies. This unique collaboration between the experts (Section 3) enabled us to deeply understand the problem at each corner of the interaction and helped us to tackle the technical details outlined in Section 4. 3
ChIA is an ongoing collaborative project and it is very important to give a detailed description of the composition of the team and how the team collaborates. 3.1
There are four core members in the ChIA collaboration, broadly categorised as digital humanists (two) and computer scientists (two). The digital humanist group mainly focuses on the analysis and representation of cultural images to enhance better access for users of the systems. Their main research interests include the information is contained in cultural images, the aspects of a society the images represent, how such aspects are represented and how the information contained in the images is represented using di erent languages and formats. The two digital humanists have further specialisation in their interest. The rst member is a linguist by profession who is interested in conducting a socio-cultural and socio-linguistic analysis of cultural images and design thinking methods on how societies represent complex historical and cultural events using digital images. The second member is a manager and content-coordinator and also focuses on providing metadata and platforms for collecting, organising and presenting cultural images on the Europeana-local-Austria platform. The rst focuses on the research aspect, the second focuses on the implementation and provision of the actual service.
The second group, computer scientists, also has two members specializing in di erent domains in computer science. One of them is an AI and computer vision expert with the objective of looking at cultural images as opposed to contemporary image collections to nd out how details contained in an image can be extracted by computer vision tools. Finally, we have the fourth member who specialises in knowledge representation, focusing on how complex social and cultural aspects can be represented across disciplines using ontologies, taxonomies 59/143 and thesauri. The main objective of this team from the technical point of view is to analyse cultural images and extract as much detail as possible and represent such detail with semantically rich, interlinked and interoperable system and to support its exploitation via chatbots.
There are two principal investigators in this project, one from the digital
humanities and the other from the technical computer science. The most
important element next to the research question is enabling a thorough understanding
among the team members. According to research on a diversi ed
multidisciplinary team [
The team is not only professionally diversi ed but also physically dispersed with only two of the team members working in the same building while the third team member works in the same country but in a di erent city. The fourth team member is in Dublin, Ireland. This physical distance introduced challenges and opportunities. The main challenge is that the team members are not able to meet and update their day-to-day activities. However, di erent channels are used to plan tasks and update progress. The most important one is the weekly project meeting which is held via Skype. This meeting allows us to update our progress, to plan our next tasks and serves as a podium to ask a question about topics we do not understand. This gives us a great opportunity to have live interactions among the team members. We have understood that there are at least three types of members in the team: those who provide up-to-date information daily, sending updates and any relevant information about the project, those who keep the work going by experimenting and building prototypes to test the ideas that are circulated among the team and those who prepare, plan and evaluate the overall direction and progress of the project. They keep the meetings running smoothly, supervise the adherence of the work to relevant standards etc. The four members usually play one or more of these roles interchangeably during our meetings or throughout the week. Our o cial communication channel is via email and our document sharing platform is google drive. We prefer to use google as it supports sharing and collaborative editing of project-related documents and supports easy and quick interaction among the project members wherever they are. Also, a project space on slack has been set up to communicate important information to team members and the advisory board. 60/143
One of the major drawbacks of the online communication system is that it is very easy to lose track of the agenda while members try to explain complex concepts. To reduce this, we further introduced internal review meetings every six months to come together physically to review our progress and discuss complex matters in person. The other limitation is the lack of su cient time to discuss and understand complex issues within a single meeting. This is a di cult scenario and when such complex issue arises, it is discussed o ine in one-to-one settings. This makes it easier to understand the issues in detail and further allow for an hunhastened and friendly discussion. In general, the physical distance between the members has contributed to slightly reduce the progress of our understanding of other members perspectives and interests. 4
One of the main deriving research questions is bridging the gap between the information packed in the images and the explicit annotation of the content of the images using ontologies. The interaction between the team members to understand the problem and to work toward the solution required the following interaction (Fig. 1) to be held all the time among the team members. These interactions represent various levels of research collaboration within the project. For the sake of brevity, the interaction is explained by taking the dataset preparation phase of the project. 4.1
The Digital Humanities (DH) quadrant is dedicated to identifying cultural
images that are widely used by users of the platform. The digital humanist uses the
collection to search and retrieve cultural images. The total image base consists
of data originating from many di erent repositories created by a considerable
number of cultural heritage professionals during several decades [
The SL quadrant, in turn, interacts with the DH to understand what kind of
cultural images are available in the collection and how they are represented in
the metadata so far. This includes understanding the conceptualisation of the
cultural aspects of societies [
The main question in the Semantic Web (SW) quadrant is to identify useful
ontologies to represent the rich information contained in the images to support
semantic annotation, reasoning and semantic search. This will enable the project
62/143
to provide a rich, semantically interlinked open data of the image collection
which will be consumed by the AI system to build an interactive chatbot. The
SW interacts with the CV quadrant to represent the output of the computer
vision using ontologies. The collaboration between the SW and SL quadrant
also bene ts from previous collaborative research that has been carried out in
[
The Computer Vision (CV) quadrant focuses on extracting as much detailed
information from the digital cultural image collections identi ed by the DH. This
quadrant consumes existing computer vision tools [
All the four quadrants have a common goal which is to understand the requirements of the other quadrants and making their research question and requirements clear to the other members. The circle in the middle of the collaboration diagram (Fig. 1) represents the information we all need to have in common about the project. We believe that as the more interaction we have among ourselves and advance the project, we will be able to push the radius of the circle to learn and collaborate with more overlapping interest from the other team members. Our objective is to push the inner circle to the edges of the outer circle in all directions. Until that happens every member of each team will have a blind spot which requires the consultation of the responsible member of the quadrant. For example, the SL may not fully understand how the SW quadrant works. In such cases, depending on the problem, the SL may rely on the SW expert to carry out the task. Such tasks may remain the blind spot of the SL.
Much of the CV work is technical and requires time and e ort to fully understand it. This could be another blindspot for the team members. We look forward to having training on computer vision and its use of algorithms to familiarise ourselves with the concept. We also look forward to having more technical training on how to build a chatbot system and on how to evaluate its outputs. This will also help us to bridge the gap between the technical part of the project and the research (theoretical) aspects of the project. 5
In this paper, we explored the collaboration among four experts from di erent academic backgrounds working together to address long-standing digital humanities questions. The collaboration from the inception of the project to its current 63/143 implementation stage is demonstrated. We looked into the major research questions and how the team collaborated to address each of these research questions. Collaboration between technical computer scientists and digital humanities experts is not usually smooth. In this project, we demonstrate how we managed to maintain the collaboration at its best by enabling smooth interaction between the team members. The lessons we have learned from this collaboration and which we continue to learn is that it is very important to start with a sound digital humanities research question and understand the problem from many di erent sides before rushing to a technical solution. While working with the digital humanists to understand the problem with an example, it is also important to introduce the technical requirements early to ensure that the digital humanists capture all the necessary information which is required to complete the technical solution and most importantly all the requirements to evaluate the system at the end. Finally, we conclude that a picture can indeed say more than a thousand words, however, with the right set of digital tools, controlled vocabularies, thesauri and ontologies we should be able to successfully access implicit knowledge via human and machine analysis.
Acknowledgements: This research is funded by the Austrian Academy of Sciences under the funding scheme: go!digital Next Generation (GDNG 2018051). The ChIA project is carried out in collaboration with the Adapt Centre, DCU. The ADAPT SFI Centre for Digital Media Technology is funded by Science Foundation Ireland through the SFI Research Centres Programme and is cofunded under the European Regional Development Fund (ERDF) through Grant # 13/RC/2106. 64/143