The automation of business processes is advancing rapidly, particularly in domains governed by complex rulebased documentation such as financial contracts. These documents are highly structured and semantically rich, which makes them well suited for modeling with formal ontologies. Although large language models ofer promising capabilities for information extraction, their efectiveness is limited by the challenge of designing consistent prompts across contract types due to a lack of standardized semantic definitions. In this paper, we explore how embedding ontological structures and constraint specifications into prompts can improve the accuracy and reusability of information extraction systems, using confirmation notices for investment transactions as a case study. Our findings show that incorporating semantic constraints into prompts improves performance in language model-based information extraction, highlighting the potential of combining Semantic Web technologies with language models to support accurate and maintainable information extraction from financial contracts.
CEUR ceur-ws.org
The demand for business process automation is rapidly increasing, particularly in the domain of legal
contracts [
Recently, the advent of large language models (LLMs) has renewed interest in ontology-based
approaches [
Financial contracts, such as investment agreements, derivatives, and loans, are a compelling domain
for Semantic Web technologies [
related fields ontology + arithmetic rule expression include in prompt
replace
Contract (unstructured)
LLM
Knowledge
Graph
Investment Management System
Hence, this paper investigates how ontology design [
We make two key contributions. First, we show that incorporating ontologies and inter-property constraints into prompts improves LLM extraction performance, particularly for logically related properties. This contrasts with traditional approaches, where ontologies are typically used only for validation. Second, we find that less formal, LLM-adapted formats can outperform more structured representations, revealing a gap between conventional ontology design and the practical efectiveness of language models.
Our approach is grounded in the hypothesis that the incorporation of ontological structures and interproperty constraints into prompts can enhance the accuracy and consistency of LLM-based information extraction. To evaluate this, we design a controlled experimental framework that systematically varies prompts used to guide extraction across two dimensions: representation style and constraint inclusion.
We use a curated corpus of 19 financial contract documents, consisting of 9 purchase confirmation notices and 10 sale confirmation notices 1. Although these documents originate from custodial banking operations, they exhibit features that are often observed in financial contracts and notification documents more broadly, making them a useful sample for developing and assessing information extraction methods. These documents have the following notable characteristics. First, they vary considerably in format across firms, as there is no standardized template. Second, key information such as trade dates may be phrased in diverse ways. Third, such key information is often embedded within full sentences or tables and is intermingled with additional information. As a result, the documents are unstructured in nature, presenting diverse and realistic challenges for information extraction.
Each document contains 11 target properties. Of these, five are classified as independent(Fund Code, Trade Date, Settlement Date, Base Currency, and Settlement Currency), and six are classified as dependent properties that participate in arithmetic relationships:
Order Quantity × Unit Price = Gross Amount,
Gross Amount + Fee = Settlement Amount (Settlement Currency).
Settlement Amount (Settlement Currency) represents the converted value of Settlement Amount (Base Currency) after currency exchange. 1Confirmation notices are widely used in custodial banking operations, where they are exchanged between securities firms and custodial banks as standard records of securities transactions. We edited the content while retaining key structural properties such as inter-field relationships.
We construct prompts with four components: a task description, an OCRed contract, a list of properties to extract, and an output format (Figure 2). We vary only the property specification and output format across the twelve prompt schemas described below. The task description and document remain fixed. We parse LLM’s responses accordingly and evaluate accuracy by exact match against the ground truth.
We create 12 prompt schemas varying along two dimensions: representation style (natural language
or ontology-based), and inclusion of constraints (none or rules). This yields four schema categories:
(i) natural language without rules, (ii) natural language with verbalized arithmetic constraints, (iii)
ontology-based without constraints, and (iv) ontology-based with constraints (Figure 3). Within category
(iii), we explore variations in ontological modeling across three aspects: class structure (separate classes
for subscription/redemption — Subclassed vs. a unified class — Unified), transaction type representation
(explicit property values — Tagged Type vs. natural language comments — Commented Type), and
currency representation (enumerating allowed values — Enum Currency vs. comment‑based guidance
— Commented Currency). For category (iv), we evaluate four styles of constraint representation: natural
language comments (Rule-Comment), formal SHACL rules (Rule-SHACL) [
These design variations allow us to evaluate how structural abstraction and constraint formalism influence LLM extraction performance, as well as to assess the sensitivity of outcomes to representational choices. Because small changes in prompts can lead to substantial shifts in LLM behavior, this diversity enables a robust evaluation of both the efectiveness and generality of diferent modeling strategies.
We use the gemma3n:e4b model [21] with a 32k token context and a fixed temperature of 0.1. We ran each document–schema pair five times to account for randomness in the LLM output. We evaluated extraction accuracy using exact matching on each target property.
The source code used in our experiments is available at https://github.com/rkondo3/ut_trust_open.
The left panel of Figure 4 shows the overall extraction accuracy for the four primary schema categories. Ontology-based representations outperformed natural language in both settings: without constraints, accuracy improved from 66.2% to 78.6% (+12.4%); and with constraints, from 78.9% to 81.4% (+2.5%). Similarly, adding rule-based constraints improved accuracy across representation styles: from 66.2% to 78.9% (+12.7%) for natural language, and from 78.6% to 81.4% (+2.8%) for ontology. The highest accuracy was achieved when both ontology and rules were included, which confirms their complementary efects.
The right panel of Figure 4 shows that all ontology + rule variants exceeded 80% accuracy, outperforming ontology-only and natural language baselines. Even the SHACL-based variant performs well, which suggests that structured knowledge improves extraction without manual tuning. Among these, the unstructured Rule-Comment schema achieved the highest accuracy (83.8%), whereas the structured SHACL variant was the lowest among the four (79.6%). This suggests that formal rigor does not always yield better LLM performance. With improved design, intermediate formats may ofer a better balance between structure and model interpretability.
Our results also demonstrated that ontology structuring choices afected LLM performance. Unifiedclass schemas outperformed subclassed schemas, which aligns with Semantic Web practice that favors abstraction when distinctions are minimal [22]. Over-modeling may obscure patterns that LLMs could otherwise learn. For currency, comment-guided formats (e.g., “use ISO 4217 codes”) outperformed formal definitions, which suggests that LLMs benefit from explicit prompts and motivates runtime embedding of linked ontology definitions.
Finally, the plain natural language schema without ontology or constraints performs worst (66.2%), confirming the substantial value of incorporating structural information—whether through ontological modeling, constraint specifications, or both.
Figure 5 provides a detailed breakdown of the property-level extraction accuracy. Independent properties such as Fund Code, Base Currency, and Settlement Currency were extracted with high accuracy across all schema categories (84% - 95%). Moreover, the average performance for the four approaches shows only moderate variations, ranging from 76% to 82% overall accuracy. These results suggest that for straightforward information retrieval, even simple natural language prompts are generally suficient, and the added structure of ontologies or rules ofers limited additional benefit.
Among the independent properties, date-related properties (Trade Date, Settlement Date) exhibited relatively higher dificulty. In particular, all approaches had low accuracy for Settlement Date (44% - 63%). When settlement date information was unavailable, these prompts frequently returned the trade date value instead of indicating missing information, thereby reducing extraction accuracy. This behavior suggests that, although ontological structure provides beneficial semantic guidance, it may also introduce systematic biases toward semantically related but distinct property values.
Dependent properties such as Order Quantity, Unit Price, Gross Amount, and Settlement Amounts proved relatively challenging for the NL w/o Rules approach with only 55% average accuracy. By contrast, the inclusion of either ontology or rules significantly improved results, raising accuracy to levels comparable to independent fields (76–82%). This underscores the importance of structural guidance in enabling LLMs to interpret relational constraints.
Using a diverse set of schema variants evaluated on financial documents, we demonstrated that prompts enriched with structural guidance through ontological modeling and inter-property constraints consistently outperformed purely natural language descriptions. This framework opens the possibility of quantitatively assessing ontological and constraint modeling choices through task-based evaluation.
R.H. is supported by JST FOREST Program (JPMJFR216Q), JST PRESTO Program (JPMJPR2469), Grantin-Aid for Scientific Research (KAKENHI, JP24K03043) and the UTEC-UTokyo FSI Research Grant Program. R.K. is funded by JST, ACT-X Grant Number JPMJAX23CA, Japan. During the preparation of this work, the author(s) used ChatGPT, Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. We also thank Edanz for editing the draft.
During the preparation of this work, the author(s) used ChatGPT, Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. 5, https://www.madofftrustee.com/images/upload/11-02540_Lion_Global_Investors_Beckerlegge_ Declaration_Exh5_120-0005.pdf, 2022. Accessed 2025-09-01. [21] Gemma Team, Gemma 3n: Model documentation, 2025. [22] N. F. Noy, D. L. McGuinness, Ontology Development 101: A Guide to Creating Your First Ontology, Technical Report KSL‑01‑05 & SMI‑2001‑0880, Stanford Knowledge Systems Laboratory and Stanford Medical Informatics, 2001. URL: http://protege.stanford.edu/publications/ontology_ development/ontology101.pdf, technical Report.