While Large Language Models (LLMs) are powerful for information extraction, the reliability of their output remains a challenge, making human supervision essential. We introduce SynSem-Align, a support tool where LLMs and humans collaborate on knowledge extraction. Our approach integrates three core components: (1) Ontology-Driven filtering to suggest relevant extraction patterns, (2) Syntactic Candidate Mining to precisely identify knowledge candidates using a CKY-based approach over dependency structures, and (3) ParaphraseBased Equivalence Filtering using an LLM for semantic validation. This integrated workflow enables users to transparently and reliably construct knowledge graphs, demonstrating a practical path towards verifiable knowledge extraction that balances automation with human oversight.
While Large Language Models (LLMs) have significantly advanced automated knowledge extraction,
ensuring the reliability and verifiability of their outputs remains a key challenge [
In this paper, we present SynSem-Align, a tool for verifiable, human-supervised knowledge extraction. SynSem-Align externalizes the extraction logic as explicit, selectable patterns, rather than relying on implicit, model-internal processing. In contrast to black-box approaches with LLMs that attempt to generate triples directly from prompts embedding both the source text and ontology constraints, our tool first enumerates multiple candidate triples and then refines them using ontology-based filtering and paraphrase-based equivalence checks. This combination ensures that extraction is not only guided by linguistic constraints but also accompanied by reasoning steps that can be inspected and corrected. A distinctive aspect of SynSem-Align is that it explicitly acknowledges inherent ambiguities: even after refinement, alternative knowledge representations may remain. Instead of suppressing this diversity, SynSem-Align provides mechanisms to expose it to human judgment. Through an interactive interface for visual verification, users can examine how textual spans map onto candidate triples and adjudicate among competing interpretations. This design allows knowledge graphs to be constructed in a manner that is transparent, auditable, and domain-adaptable.
To complement these methodological contributions, we provide a demonstration that walks through SynSem-Align’s workflow step by step. To ensure reproducibility and foster further exploration, the source code is publicly available on GitHub1.
A fundamental task in knowledge extraction is to make explicit, for a specific span of text, what knowledge can justifiably be extracted and why. In our view, the core dificulty is not a simple opposition between syntax and semantics but a residual gap between the results of syntactic and semantic analyses and the knowledge graph that we finally construct. Syntactic and semantic analyses constrain the space of possible meanings, yet the set of valid interpretations remains diverse, and that diversity is often domain dependent even for expressions that share the same surface structure.
Consider the metonymic phrase ‘Land of Smiles’. The ‘A of B’ construction typically ofers and as candidates for relations in the knowledge graph, for example attribution, origin, or possession; in this instance a natural interpretation is that the residents in frequently smile (). This illustrates that even text spans that share the same syntactic ‘A of B’ structure can yield diverse semantic relationships between their terms. Furthermore, even after applying both syntactic and semantic constraints, the mapping from the analyzed text to a knowledge graph may still be indeterminate. If we admit a latent variable that does not appear in the span, the partial expression ‘A of B’ can license multiple candidate relations, for example (, ), (, ), (, ), and others. This indicates that, despite the restrictions provided by syntax and semantics, a substantial set of candidate knowledge graphs remains. Since the appropriate choice can vary by domain or field, tools are needed that make the mapping from text to knowledge graph relations explicit and verifiable on the basis of consistent reasoning.
End to end LLMs may learn such correspondences implicitly, but their black box nature prevents us from verifying how a particular triple was licensed by a particular piece of text, and it prevents users from trusting or correcting the outcome. We therefore emphasize transparency and controllability. Users should see which part of the text supports which candidate relation, and what reasoning connects them. In practice, we operationalize this with paraphrase based validation anchored at the span: if a syntactic candidate can be rephrased into a natural and semantically equivalent sentence aligned with the same text span, it is retained; otherwise, it is discarded. This procedure enumerates the diversity of interpretations that remain after syntax and semantics have constrained the space, and it justifies each retained interpretation in a way that can be inspected, trusted, and corrected by humans.
The dominant approach for knowledge extraction currently leverages the zero-shot or few-shot
capabilities of Large Language Models (LLMs) [
Our work takes a diferent stance by grounding the extraction process in a pre-constructed pattern
base governed by humans. This pattern base is semi-automatically derived from text-to-graph datasets
such as Text2KGBench [
SynSem-Align enhances transparency and makes explicit the assumptions underlying LLM reasoning
by explicitly controlling syntactic and semantic judgments, thereby supporting users in making final
decisions regarding knowledge extraction. This stance stands in contrast to the Auto-KG Agent [
SynSem-Align structures the extraction process into explicit, inspectable steps. The workflow systematically enumerates syntactic candidates, prunes implausible ones through ontology based and paraphrase based validation, and reserves the final decision for the human user. This staged design ensures that the diversity of possible interpretations is preserved, implausible ones are pruned, and the outcome remains transparent and auditable.
The process begins with dependency parsing, which identifies grammatical relations and decomposes
the syntactic units(Figure 1). This analysis enables the system to process complex constructions such as
coordination [
The resulting structure feeds into a CKY-based charting algorithm [11], which combines sentence fragments and systematically matches them against the pre-constructed pattern base (Figure 2). The pattern base consists of reusable templates such as ‘[X1] is [Y1]’ or ‘[X1] of [Y1],’ which explicitly link surface syntax to potential semantic relations. Through this process, the system enumerates all triples that the surface syntax can license, anchored to explicit and reusable patterns rather than opaque model inferences.
The outcome is a broad but systematically organized set of syntactic candidates, which inherently inclN映uO画dU「eNs bとなoりth p l aのu s i b leトaトnロ」d s pのu r i o u s監i督n t e r pと r e t a tio脚n本s . A tはt h i s sta宮g崎e , th e駿g o a l iでs exhPRaOuPsNtive ePnROuPmNeratAiUoXn、,
NOUN ADP PROPN ADP NOUN ADP NOUN ADP leavmionvige, thneexttadosokr of dNiOscardingToitomro plausNiOble canddiirdecatotres toTOsubsescqreuenewnritterrefineHmA ent staMgiyaezsak.i Hayao DE 映画 「となり の トトロ」 の NOUN NOUN ADP PROPN ADP movie next door NO Totoro NO (Tonari no Totoro, instance, movie) (Hayao Miyazaki, director ,Tonari no Totoro ) 監督 NOUN director と ADP TO 脚本 は NOUN ADP screen writer HA 宮崎 PROPN Miyazaki 駿 PROPN Hayao で
AUX
DE & (Hayao Miyazaki, screen writer,Tonari no Totoro )
Because the syntactic candidate set is inherently diverse, it inevitably contains both plausible and spurious interpretations. This diversity is deliberately preserved, since subsequent refinement ensures that only semantically valid interpretations remain.
First, ontology based type and relation constraints rule out triples inconsistent with domain knowledge (Figure 3). For example, the system maps ‘My Neighbor Totoro’ to the concept film and ‘director’ to the relation director, discarding implausible triples such as ‘film directed person’. 映画「となりのトトロ」の 監督と 脚本は 宮崎駿で、 movie ’Tonari no Totoro’ -NO director-TO screen writer-HA Miyazaki Hayao-DE
Even after syntactic enumeration and refinement through ontology and paraphrase validation, multiple semantically valid candidates may remain. These reflect residual ambiguity that cannot be resolved automatically, and thus the decisive step is delegated to the human user.
The remaining candidates represented as abstract patterns such as ‘[X1] is [Y1]’ or ‘[X1] of [Y1]’ are presented in an explicit and inspectable form. The user visually inspects these alternatives and selects the pattern that best captures the intended meaning of the input sentence.
This explicit human decision directly determines the extracted knowledge and ensures that the reasoning process remains auditable: users can see which syntactic candidates were generated, which were excluded, and why the final pattern was chosen. The division of responsibilities is therefore clear: the system enumerates and refines candidates, the LLM provides only bounded auxiliary validation,
- movie ’tonari no Totoro’ -NO director-TO screen writer-HA Miyazaki Hayao-DE - Parallel elements: “director”, “screen writer” (e.g., "A", "B", "C") Judgment Criteria: 1. Similarity Check: - The head (central word) of each parallel element must be the same part of speech (e.g., all nouns or all verbs). - Phrase structures should be aligned as much as possible
(e.g., all “noun + particle”).
- Swapping the order of the parallel elements must still produce a natural Japanese sentence. - Swapping only the parallel elements should not significantly change the overall meaning of the sentence (the main roles/semantic structure are preserved).
- Output “True” only if both criteria are satisfied,
output “False” if either criterion is not met. - The output must be **only** the following JSON format.
Do **not** include any explanations or comments. and the final disambiguation rests with the human expert. In this way, SynSem-Align guarantees that knowledge extraction remains transparent, controllable, and accountable, with interpretive authority explicitly assigned to the user.
We have presented SynSem-Align, a human supervised knowledge extraction tool. By efectively combining an ontology driven filter, a transparent syntactic matching engine based on dependency parsing and CKY, and LLM assisted semantic validation, our tool ofers a transparent and verifiable workflow for supervised extraction.The explicit, pattern based logic, controlled by the user and verified through semantic checks, provides a robust framework for building reliable knowledge graphs, successfully balancing automation with essential human oversight.
ChatGPT (OpenAI, 2025) was used only for English language checks (rephrasing, grammar, and style). All scientific content was created by the authors, who take full responsibility for the final manuscript.
This research was partially supported by JSPS KAKENHI Grant Number JP23K28152. Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 3394–3403. URL: https://aclanthology.org/N19-1343/. doi:10.18653/v1/N19-1343. [11] T. KASAMI, An eficient recognition and syntax analysis algorithm for context-free languages, Science Report, Air Force Cambridge Research Laboratory (1965).