This work presents a neurosymbolic AI system for support of research and discovery activities. The system is powered by metaphactory and builds on the open scholarly knowledge graphs SemOpenAlex, LPWC and CS-KG. Researchers are supported through a range of AI methods (generative natural language interface to KGs, retrieval with semantic templates, neighborhood exploration with KG embeddings) ofering convenient means to solve research tasks like scientific artifacts overview, publication search and getting recommendations. This work gives an overview of the system design, architecture, data landscape and supported functionalities. A concrete implementation of this system is used in a research project exploring the application of AI methods for electric traction drives design.
CEUR ceur-ws.org A
Research and scientific discovery are based on complex cognitive, creative and scientific humancentred processes that can hardly be fully formalized, defined or verified solely by a machine. The majority of these processes are relying on methodologies and approaches that are based on learned experience, knowledge and reasoning abilities.
Nowadays, the involvement of AI in creative processes has established as a new reality, particularly in the form of AI-powered research services and assistants. While methods of AI and machine learning have been used for hypothesis testing or finding optimal solutions (e.g., in proteins, algorithms and games), the human has always been defining the methodology, application rules and evaluation. At the same time, humanity has accumulated and continues to generate a vast amount of knowledge and data that can now be explored through AI.
Integration between generative AI and knowledge graphs (KGs) has proven to be beneficial. LLMs, being black-box models, and KGs, which explicitly store rich factual knowledge, are, however, both incomplete, but in a diferent way. LLMs contain commonsense knowledge, but might be based on outdated training data; they can generate an unlimited number of coherent and fluent texts, suggesting creative hypotheses, but these are not always factually correct. On the other hand, KGs can be viewed as sources of reliable and trustworthy information.
Although supporting research tasks involves some creativity, it is equally important to produce reliable and trustworthy results. We ensure a certain level of reliability and trust by developing a research support system that produces results grounded on knowledge available in established scientific knowledge graphs, and allows the user to use natural language search and conversational AI functionalities on demand.
The proposed research assistance system, further referred as KIRA Research Support System, has been designed and implemented in the context of the KIRA project, which focused on optimized control of electric traction drives, to support project partners in their aspirations for methods improvement in a particular research context. The motivation, therefore, was to design trustworthy scenarios for project collaboration, decision making and task solving on top of knowledge graphs, and make them accessible to the users. The system was available to about 50 users from 10 organizations during the project, and was developed from their direct needs and feedback, as well as via continuous improvement of methods, allowing extrinsic evaluation. Moreover, to allow further development of collaborative Human-AI discovery, based on reinforcement learning and multi-agent approaches, a stable research support system baseline, which can be later populated with AI agents, was required.
Our contributions can be thus summarized as follows: • a KG-powered neurosymbolic AI system for research activities support; • a data landscape, integrating several scientific KGs in a single system to provide broader explorative experience; • a spectrum of research assistance tasks, mapped to neurosymbolic AI methods and applications, which has been developed iteratively through involvement of the users, continuous validation and feedback from them.
Doing research involves a lot of inspiration and planning, spread across creative and routine activities. Scientific communication is also an important part of it, promoting ideas transfer and exchange, collaborative discovery and results dissemination. In the KIRA System, we are supporting both the tasks, facilitating research activities (i.e. project collaboration), and the tasks, related to experiment planning (i.e. exploratory search of connections and ways to improve methods for particular tasks). The tasks, which are currently supported, are listed below formulated as user stories, representing requirements for the implementation of the system, as described in Sec. 5). A detailed description of using the system to explore scientific artifacts is provided in Fig. 3, Appendix A. Besides tasks and knowledge graphs access, each user of the system has an individual dashboard with a personalized overview of relevant information, covering afiliation, topics of interest, papers and team members.
Find Publications, Topics, Authors, Datasets As a user, I would like to ask questions in natural language about publications on particular topics, their authors, afiliations, etc., and get back results from the knowledge graph, for example, List the ten most influential authors publishing about Amyloid-beta precursor protein or What are the papers about holograms, calibration and augmented reality published in the last 2 years?. For some cases, I would like to summarize the results with keeping provenance from the knowledge graph, and get statistics about paper metadata distribution.
Foster Collaboration within the Project As a user, I would like to have an overview over the project, including its goal, funding, runtime, outcomes, partners and involved persons, as well as existing relations between persons in the system to other entities in the system. Also, I would like to identify synergies between a certain partner and other partners, related to project-relevant tasks or methods, i.e. Who of the partners is working on the same tasks as me?. Get Recommendations As a user I would like to get personalized recommendations for scientific key content, i.e. If I have a specific Datasets I want recommendations which Task and Methods I can use for it. I want to be recommended connections between them that may not yet exist, being informed that embedding-based link prediction has been used to forecast new links. Get Overview of Scholarly Artifacts Scientific artifacts can be Datasets, Methods, Tasks, Models, Publications, Repositories, etc. As a user, I would like to have an overview of existing AI-Methods and obtain a detail view of a specific AI-Method. I want to see existing connections for scientific artifacts, i.e. Which methods are improving the given method?, What is the state-ofthe-art (leaderboard) for the given task?, Which metrics are used to evaluate a task or method? Explore Concepts’ Neighbourhood As a user, I would like to find similar methods, tasks, datasets, and topics via neighborhood exploration and see them visualized in the vector space. Compare Methods As a user, I want to compare a pair methods by its application across methods, tasks, materials, and datasets, and observe the scientific artifacts, co-occurring with each of the methods, as well as artifacts, with are specific for and shared by both methods, i.e. What is the diference in application of recurrent neural network and long short-term memory across tasks. Also, being aware of possible hallucinations, I want to prompt an LLM to do a featured methods comparison, based on the information retrieved from the scholarly KG.
Neurosymbolic systems combine the power of human expert knowledge (symbolic) with the
language understanding capabilities (neural) of large neural networks such as LLMs. As the
authors highlight in [
The closest neurosymbolic system providing access to scholarly knowledge is the Open
Research Knowledge Graph (ORKG) [
The data landscape of our system is populated with knowledge graphs, representing diverse
scientific content. Firstly, publications metadata are covered by SemOpenAlex [
Furthermore, we created the KIRA Project Knowledge Graph (KIRA KG) modelling the involved partners and persons as well as their relationships, fields of expertise, project obligations and outcomes. This graph also stores user profiles that are generated from user interaction with the system artifacts, i.e. bookmarking relevant papers, ranking recommendations, adding research topics.
SemOpenAlex. SemOpenAlex is an extensive RDF knowledge graph that contains over 26 billion triples about scientific publications and their associated entities from all scientific domains. Containing more than 256 million publications, it can serve as a central hub for interlinking within the scientific domain.
Linked Papers With Code. LPWC is an RDF knowledge graph that comprehensively models the research field of machine learning. It contains information about more than 400,000 machine learning publications and describes the tasks addressed, the datasets utilized, the methods implemented, and the evaluations conducted, along with their results. It is tightly interlinked to SemOpenAlex with more than 1,2 million links between works and authors, making it a perfect ift for the KIRA system.
Computer Science Knowledge Graph. CS-KG contains facts and relations about research entities automatically extracted from scientific publications. Describing relationships between Tasks, Methods, Materials and Metrics, the facts are further annotated with information about frequency, co-occurrences, extraction time and extraction method.
KIRA Project Knowledge Graph. Based on well-established ontologies like FOAF and schema.org, the KIRA Project KG aims to reflect the research project including its key participants, their relations and artifacts produced in the context of the project. The data of this KG is hence not focused on actual scientific concepts but rather on data that helps to foster collaboration between the relevant actors in the project.
The central hub of our data integration eforts are publications in SemOpenAlex. Both CS-KG and LPWC contain types for publications and provide references to SemOpenAlex publications either in the form of identifiers or as explicit owl:sameAs links (cf. Fig. 1). The facts in the KIRA Project KG are created on the basis of SemOpenAlex publications as well to ensure tight interlinking. Following these links, we can combine relevant content from multiple graphs, for example to find synergies and overlapping fields of expertise for project participants.
The main purpose of the presented system is to provide research assistance on top of scholarly knowledge graphs. This assistance encompasses several aspects: 1. Research support: (1) automating scientific routines, i.e. publications search, methods overview and comparison, etc., (2) facilitating research activities with AI methods; 2. Access to relevant knowledge bases and publication databases: (1) natural language interfaces to knowledge graphs, (2) domain exploration and methods application in a specific research context, (3) context-specific recommendations of scholarly artifacts; 3. Personalized experience: (1) a personal dashboard for every system user, (2) paper recommendations based on personal dashboard and user’s interaction with the scholarly artifacts, (3) feedback and user activity tracking with the interaction knowledge graph; 4. Support of collaboration through the knowledge graph: (1) identifying synergies through partners via mapping activities in the project, (2) enabling exchange on specific topics, (3) sharing relevant content that is created or found during the project.
The KIRA Research Support System is implemented on top of the metaphactory platform, having research tasks implemented in dedicated apps with the usage of built-in metaphactory semantic search, querying and templating functionality, as well as integrating external services (large language models, graph embeddings).
The metaphactory platform ofers simple extension points to build, bundle and deploy
lightweight ”apps” along with the platform without the need of changing the platform
binary or re-compiling the platform. Sources of an ”app” can be put under the source control and
a packaged version of an ”app” can be deployed as a docker container volume along with the
platform.[
The architecture of the KIRA Research Support System is shown in Fig. 2. The integration of the respective knowledge graphs is provided in Sec. 4.2, and connections between tasks, datasets and methods are described in Table 1.
The KIRA Research Support System is a metaphactory App which hosts a plethora of resources and tools. Each of the research tasks mentioned in Sec. 2 is a component of that App, employing a combination of neurosymbolic AI methods.
To make these connections explicit, we provide a mapping between tasks, datasets and methods in Table 1.
Semantic search over knowledge graphs is an advanced method for retrieving the unambiguous concepts and its relationships with a focus on concept meaning, rather than the surface form of a word. Focusing on the user’s information need (intent), metaphactory semantic search framework allows the user to define complex information needs in an incremental way. The UI components supporting search definition to capture this information need and express it as a SPARQL query to be executed against the graph database. At the next step, it is possible to refine the set of search results using facets, and visualize the search results in diferent ways.
Semantic templates in metaphactory handle formatting and rendering of the structured data (semantic search results) into final markup by applying features of the template language. The template language instructs where to render variables and provides basic flow control blocks. Thus, retrieved knowledge is surfaced as pages, dashboards and diagrams, presenting semantic search results to the end users.
There is a plethora of methods to solve the problem of translating user’s information needs,
expressed in natural language, to query languages, in particularly, SPARQL. In the system,
we implemented a zero-shot approach, based on augmenting LLMs with an RDFS ontology
definition, required to construct the query, during prompting. This approach extends existing
in-context learning approaches, proposed in [
Knowledge graph embedding methods aim to learn dense vector representations of graph’s
nodes and edges to support knowledge graph completion and similarity search[
Similarity measures are based on a distance function and can be computed for diferent types
of information objects (strings, numbers, tuples, vectors, images). In comparison to speech
recognition, where edit-based measures (i.e. word error rate and its derivatives, based on
Levenshtein distance) are more relevant, machine learning approaches (i.e. k-means algorithm
in distance-based clustering, nearest neighbor search) learn similarities between objects in the
vector space, actively employ token-based and hybrid measures (Euclidean distance, cosine
similarity, Jaccard index, etc.)[
The implemented knowledge graph-based system on top of the metaphactory platform aims to support researchers in their everyday research tasks and routines. This system has originated from the KIRA project, which goal was to optimize control of electric traction drives with AI methods. From this perspective, we have populated the system with the applications, leveraging AI neurosymbolic methods, in particularly, by adding neural (large language models and vector representations) to the metaphactory built-in symbolic methods (semantic search and semantic templates). The central part of the system are the scholarly knowledge graphs, and the project interaction graph, that altogether serve the users’ needs in research assistance and foster collaboration through the knowledge graph.
In the future work, we aspire to design and integrate multi-agent workflows to address the
challenges and limitations of deep learning models[
This work has been funded by the German ministry BMWK under project KIRA FKZ: 19I21030I. X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, J. Tang, Agentbench: Evaluating llms as agents, 2023. URL: https://arxiv.org/abs/2308.03688. arXiv:2308.03688. [15] X. Tang, Q. Jin, K. Zhu, T. Yuan, Y. Zhang, W. Zhou, M. Qu, Y. Zhao, J. Tang, Z. Zhang, A. Cohan, Z. Lu, M. Gerstein, Prioritizing safeguarding over autonomy: Risks of llm agents for science, 2024. URL: https://arxiv.org/abs/2402.04247. arXiv:2402.04247.
This task allows the user to see existing connections between scientific artifacts. These connections are extracted from the scholarly graphs, integrated on the KIRA system, and surfaced with semantic templates.
Firstly, from the KIRA system start page the user selects the task Get Overview of Scientific Artifacts, and secondly, the knowledge graph for exploration: CS-KG, LPWC, or SemOpenAlex.
Depending on the graph, the user can search across diferent combinations of artifacts, such as Method, Task, Material, Metric, Repository, Dataset, Evaluation, Paper, Conference, and some other artifacts.
Let user search for genetic algorithm method. After choosing the method from the search results, the user is taken to the artifact overview page. In Fig. 3, connections between scientific artifacts across CS-KG and SemOpenAlex are shown for this particular method.