An Experiment with LLM in Contract Extraction⋆ Nhan Le1,*,† , Tran Cao Son1,† 1 New Mexico State University Abstract This paper reports on our experiment with LLM in the development of a module, which translates a contract in natural language into a set of contract clauses and is intended to serve as a component of an contract execution monitoring system. The paper briefly discusses the overall design of the system and the need for an automatic contract extraction module. It then includes a short review of the contract specification language and describes the methods used in contract extraction and preliminary evaluation of the use of LLM. Keywords Supply Chains, Contracts, LLM 1. Introduction In a supply chain, entities (or agents) agree on producing, purchasing, and supplying each other certain products. Each product is associated with some cost and description. Agreements often have deadlines for product delivery and payment options etc. Agreements between agents can be formalized as a set of contract clauses. Agents are obligated to come up with plans that help them to fulfill their obligations. However, the success of an agent’s plan heavily depends on the other agents’ willingness and successes. Therefore, agents need the capabilities to identify potential issues that can affect the execution of their plans and identify courses of actions for dealing with these issues. This article is part of a project aimed at developing formalization and reasoning methodologies that make it possible to draw formal conclusions about the properties of supply chains. The project leverages the domain-agnostic characterization of such properties described in [3, 4]. The focus of the present paper is the development of a supply chain monitoring system with the components as shown in Figure 1. It assumes that agents have contracts among themselves which are specified via high-level description language ℒ𝑐 (see next section for more detail). The system will be used by each agent participating in the supply chain network, allowing the agent to monitor the progress of the part of the chain that involves the agent. It has five components: the reasoning engine, the planning module, the diagnosis module, the explanation module, and an interface to other agents within the supply chain. The reasoning engine is responsible for answering queries from the user. It will invoke the planning or diagnosis module whenever it is necessary. The reasoning module will activate the explanation module to generate an explanation for every query. The diagnosis module is responsible for identifying potential issues affecting the successful execution of the supply chain at the node. The planning module is responsible for computing plans that help mitigate those issues. The system will interact with an ontology solver for reasoning about the concerns of the agent and help the agent to determine the trustworthiness of other agents. The proposed system will also include an interface for the agent to work with other agents within the supply chain. This interface is responsible for updating the progress of contracts among agents. It will work with the reasoning module to identify clauses within contracts that can no longer be satisfied or might not be satisfied. Workshop on Symbolic and Neuro-Symbolic Architectures for Intelligent Robotics Technology (SYNERGY) co-located with the 21st International Conference on Principles of Knowledge Representation and Reasoning (KR2024), November 2–8, 2024, Hanoi, Vietnam. * Corresponding author. † These authors contributed equally. $ tiennhan@nmsu.edu (N. Le); stran@nmsu.edu (T. C. Son) € https://www.cs.nmsu.edu/~tson/ (T. C. Son)  0000-0003-3689-8433 (T. C. Son) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Figure 1: Overal Architecture of a Contract Monitoring Agent A preliminary implementation of this system (in simulation mode) is described in [1]. Most of the modules are logic programs under answer set semantics [2]. An agent receives its contract specification and monitors the changes in the environment and will respond to queries related to a contract 𝐶 such as (i) can 𝐶 be successfully completed? (ii) has any agent violated a clause in 𝐶? (iii) will any concern of the agent, as defined in the CPS Ontology1 [3, 4], be (un)satisfied? etc. Furthermore, the question is “what needs to be done to mitigate a disruption that arises?” One of the main inputs of the system is the contract specification, which has to be manually encoded. Our aim in conducting the experiment described in this paper is to learn how this process can be automated with LLM. Specifically, we investigate the following problem: Given: A text representing a contract between two entities, for example, the text “[From [5]] XYZ Homes builds eight to nine 2,000 square foot homes each month for new home buyers. Each new home requires 16,000 board feet of Number 2 Common grade lumber. In order to complete eight to nine homes, XYZ Homes must purchase 144,000 board feet of Number 2 Common grade lumber each month. Lumber Yard A is the preferred supplier of this lumber. XYZ Homes contracts with Lumber Yard A for the required lumber. The two sides agree on the following: 1. Lumber Yard A will produce a total of 144,000 board feet of lumber for XYZ Homes. 2. Lumber Yard A guarantees to schedule the transport and delivery of 14-16 tractor trailers worth of lumber in one month to XYZ Homes. 3. The lumber delivered to XYZ Homes will be at or above Number 2 Common grade. 4. The agreed upon cost of lumber is at $122,000 for 144,000 board feet and the transport and delivery cost will be at or below $500,000 for 144,000 board feet.” Goal: Translate the above text into a set of clauses (precise definition in the next section), for example, the following set of clauses encoding the contract between XYZ Homes and Lumber Yard A. Let 𝐿 and 𝐻 denote Lumber Yard A and XYZ Homes, respectively. The set of clauses specified by the above paragraph is as follows: 𝐶1 : 𝐿 responsible_for 𝑏𝑜𝑎𝑟𝑑(144𝐾, 𝑄) ∧ 1 ≤ 𝑄 when 𝑏𝑦_𝑤𝑒𝑒𝑘 4 (1) 𝐶2 : 𝐿 responsible_for 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑(144𝐾, 𝑄) ∧ 2 ≤ 𝑄 when 𝑏𝑦_𝑤𝑒𝑒𝑘 4 (2) 𝐶3 : 𝐻 responsible_for 𝑝𝑎𝑦𝑚𝑒𝑛𝑡(122𝐾, 𝑏𝑜𝑎𝑟𝑑) when 𝑏𝑦_𝑤𝑒𝑒𝑘 4 (3) 𝐶4 : 𝐻 responsible_for ∃𝑋 ≤ 500𝐾.[𝑝𝑎𝑦𝑚𝑒𝑛𝑡(𝑋, 𝑠ℎ𝑖𝑝𝑝𝑖𝑛𝑔)] when 𝑏𝑦_𝑤𝑒𝑒𝑘 4 (4) We will next review ℒ𝑐 , a language for representing and reasoning about contracts that is also the target language. Afterwards, we will describe the process of extracting contract clauses from text using LLM. We then evaluate the results and discuss the issues as well as the potentials of the approach. 1 Part of the National Institute of Standards and Technology’s Cyber-Physical Systems (NIST CPS) Framework. 2. Representing and Reasoning about Contracts ℒ𝑐 , proposed in [5], is a language for representing and reasoning about contracts. The language is built on action domains of individual agents [6]. In ℒ𝑐 , each contract has two parts. The public part is shared between parties of the contract and includes the clauses that they agree. The private part is internal to each party and details the concerns of the party which are related to the contract. Given two agents 𝐴 and 𝐵. Assume that 𝐷𝐴 and 𝐷𝐵 are the action domain of 𝐴 and 𝐵, respectively, i.e., 𝐷𝑥 (𝑥 ∈ {𝐴, 𝐵}) encodes the set of fluents that agent 𝑥 is aware of and the actions which can be executed by 𝑥. We assume that 𝐴 and 𝐵 use the same language in encoding the fluents, i.e., a property shared between 𝐴 and 𝐵 will have the same representation in 𝐷𝐴 and 𝐷𝐵 . The public part of a contract between 𝐴 and 𝐵 over 𝐷𝐴 and 𝐷𝐵 consists of clauses of the form: ref_id : 𝑎𝑔𝑒𝑛𝑡 responsible_for 𝑔𝑜𝑎𝑙 when 𝑡𝑖𝑚𝑒_𝑒𝑥𝑝𝑟𝑒𝑠𝑠𝑖𝑜𝑛 (5) where • ref_id is an identifier of the clause; • 𝑎𝑔𝑒𝑛𝑡 ∈ {𝐴, 𝐵}; • 𝑔𝑜𝑎𝑙 is a fluent formula constructed over fluents appearing in 𝐷𝐴 ∪ 𝐷𝐵 ; and • 𝑡𝑖𝑚𝑒_𝑒𝑥𝑝𝑟𝑒𝑠𝑠𝑖𝑜𝑛 is a formula representing a time constraint of one the following forms: 𝑎𝑙𝑤𝑎𝑦𝑠 | 𝑒𝑣𝑒𝑛𝑡𝑢𝑎𝑙𝑙𝑦 | 𝑝𝑒𝑟_𝑢𝑛𝑖𝑡[𝑛 . . . 𝑚] | 𝑏𝑦_𝑢𝑛𝑖𝑡 𝑛 (6) where 𝑢𝑛𝑖𝑡 can be any time unit such as day, week, etc., 𝑛 and 𝑚 are integers, 𝑛 ≤ 𝑚, and [𝑛 . . . 𝑚] denotes the range [𝑛, 𝑛 + 1, . . . , 𝑚]. Given the public part of a contract 𝐶 between 𝐴 and 𝐵, the private part of 𝐶 for either agent 𝐴 or 𝐵 is represented by statements of the form ref_id : 𝜌 (7) where ref_id is a reference identifier 𝐶 and 𝜌 is a requirement. It is assumed that the ontology 𝒪 associates each requirement with one or more concerns (of the agent) from the concern forest defined in the CPS Framework, or any customized concern forest specific to the agent. Formally, a contract 𝒞 between two agents 𝐴 and 𝐵 constructed over two actions domains 𝐷𝐴 and 𝐷𝐵 is a triple (𝐶, 𝑃𝐴 , 𝑃𝐵 ) where • 𝐶 is a set of clauses of the form (5); and • 𝑃𝐴 (resp. 𝑃𝐵 ) is a set of statements of the form (7) for 𝐴 (resp. 𝐵). Here, (𝐶, 𝑃𝐴 ) or (𝐶, 𝑃𝐵 ) is the contract under 𝐴 or 𝐵’s perspective, respectively, and it will be used by 𝐴 or 𝐵 to evaluate the progress of the contract. Observe that 𝐴 (resp., 𝐵) does not necessarily know about 𝑃𝐵 (resp., 𝑃𝐴 ). Given a contract 𝒞 = (𝐶, 𝑃𝐴 , 𝑃𝐵 ) between two agents 𝐴 and 𝐵. It suffices for this paper to state that the semantics will allow us to determine whether a clause of 𝒞 is satisfied (or successfully executed) given an action sequence that have been executed by the agents and the observations that the agents can collect. Interested readers are referred to the paper [5] for a detailed presentation of the language. 3. Extracting Clauses From a Contract We follow the general guidance for prompt engineering given in2 to extract the clauses from the text. In this paper, the focus is on the public part of a contract. This is done in two steps. 2 https://learn.deeplearning.ai/courses/chatgpt-prompt-eng/lesson/2/guidelines: ChatGPT Prompt Engineering for Developers https://platform.openai.com/docs/guides/prompt-engineering/six-strategies-for-getting-better-results: Prompt engineering Step 1: Extracting Basic Information of a Contract. A contract (in supply chain) is between entities (companies) and includes products, categories of products, deadlines, payment terms, etc. We call all this information as contract details and begin with asking LLM to extract the contract details. The prompt for this step is as follows: # Generate c o n t r a c t s d e t a i l s def g e n e r a t e _ c o n t r a c t _ d e t a i l s ( c o n t r a c t ) : " " " Generate contract d e t a i l s . " " " prompt = f " " " Giving a c o n t r a c t above , p l e a s e g e n e r a t e t h e f o l l o w i n g t h i n g s : 1 / Name o f a l l p a r t i e s i n t h e c o n t r a c t 2 / P r o d u c t / s e r v i c e b e t w e e n c o m p a n i e s and i t s q u a n t i t y i f presented in the c o n t r a c t 3/ C at e g o r ie s / grade of the product / s e r v i c e i f presented 4 / Time d e l i v e r y e s t i m a t e 5 / Payment t h a t e a c h company h a s t o make U s e t h e f o l l o w i n g f o r m a t , w i t h no e x t r a t e x t , no e x t r a n o t e f r o m t h e model : 1 / Name o f s u p p l i e r and b u y e r − Supplier : − Buyer : < buyer > 2 / P r o d u c t / s e r v i c e and q u a n t i t y i s p r o v i d e d by t h o s e p a r t i e s − Product / s e r v i c e : − Quantity : < quantity > 3/ C at e g o r ie s / grade of product : − Grade : < g r a d e a t o r above > 4 / E s t i m a t e t i m e d e l i v e r y by week : − Time ( w e e k s ) : < t i m e d e l i v e r y > 5 / Payment c o s t f ro m b u y e r : − Payment c o s t : < payment c o s t i f any > − T r a n s p o r t and d e l i v e r y c o s t : < s h i p p i n g c o s t i f any > I f t h e r e a r e n o t h i n g i n t h e c o n t r a c t t h a t can p r o v i d e t h i s i n f o r m a t i o n , t h e n w r i t e \ " No i n f o r m a t i o n p r o v i d e d . \ " { contract } """ return g e t _ c o m p l e t i o n ( prompt ) For the contract given in the introduction, the output from this step is as follows: 1 / Name o f s u p p l i e r and b u y e r : − S u p p l i e r : Lumber Yard A − Buyer : XYZ Homes 2 / P r o d u c t / s e r v i c e and q u a n t i t y i s p r o v i d e d by t h o s e p a r t i e s : − P r o d u c t / s e r v i c e : Number 2 Common g r a d e l u m b e r − Quantity : 144 ,000 board f e e t 3/ Categories / grade of product : − Grade : Number 2 Common o r a b o v e 4 / E s t i m a t e t i m e d e l i v e r y by week : − Time ( weeks ) : 1 month ( a p p r o x i m a t e l y 4 weeks ) 5 / Payment c o s t from b u y e r : − Payment c o s t : $122 , 0 0 0 − T r a n s p o r t and d e l i v e r y c o s t : a t o r below $500 , 0 0 0 Step 2: Construction of Clauses. After the extraction of the basic information, we ask LLM to construct the clauses according to the language ℒ𝑐 . The prompt for this step aims at providing LLM with a brief explanation of how these elements function in the contract, along with some example clauses. This is as follows. def g e t _ c l a u s e s ( c o n t r a c t _ d e t a i l s ) : " " " Generate contract clauses . " " " prompt = f " " " A clause in a c o n t r a c t d e s c r i b e s the r e s p o n s i b i l i t i e s that a p a r t y must f u l f i l l . P l e a s e g e n e r a t e c l a u s e s f o r t h e p r o v i d e d c o n t r a c t u s i n g t h e f o l l o w i n g g u i d e l i n e s , w i t h no a d d i t i o n a l i n f o r m a t i o n o r t e x t , no e x t r a t e x t , no e x t r a n o t e f r o m m o d e l : ’ ’ ’ Each c l a u s e s h o u l d i n c l u d e : 1) < clause_number > : A d i s t i n c t i d e n t i f i e r f o r the clause , f o r m a t t e d a s " C< c l a u s e _ n u m b e r > " . 2 ) < p a r t y _ r e s p o n s i b l e > : The name o f t h e p a r t y r e s p o n s i b l e for f u l f i l l i n g the o b l i g a t i o n s outlined in the clause . 3) < r e s p o n s i b i l i t y >: A b r i e f d e s c r i p t i o n of the r e s p o n s i b i l i t y . This could include d e l i v e r i n g a product o r s e r v i c e , m e e t i n g q u a l i t y s t a n d a r d s , o r making payments , etc . . . 4 ) < d e a d l i n e > : The t i m e l i n e o r s c h e d u l e f o r f u l f i l l i n g t h e r e s p o n s i b i l i t y . The d e a d l i n e s h o u l d b e s p e c i f i e d i n t e r m s o f f r e q u e n c y ( e . g . , monthly , w e e k l y ) o r a s p e c i f i c week number ( e . g . , by \ _week < week_number > ) . ’ ’ ’ Each c l a u s e s h o u l d b e f o r m a t t e d a s f o l l o w s , w i t h no a d d i t i o n a l i n f o r m a t i o n o r t e x t , no e x t r a t e x t , no e x t r a n o t e from model : C< c l a u s e _ n u m b e r > : < p a r t y _ r e s p o n s i b l e > r e s p o n s i b l e f o r < r e s p o n s i b i l i t y > when < d e a d l i n e > . " U s e t h e s e e x a m p l e c l a u s e s t o g u i d e y o u r f o r m a t t i n g , don ’ t do e x a c t l y l i k e t h e s e examples : − " C1 : A r e s p o n s i b l e f o r p r o d u c e d b o a r d ( 1 4 4 K , Q ) and 2 < Q when by week 4 − " C2 : A r e s p o n s i b l e f o r d e l i v e r e d ( 1 4 4 K , Q ) and 2 < Q when by week 4 − " C3 : B r e s p o n s i b l e f o r payment ( 1 2 2 K , b o a r d ) when by week 4 − " C4 : B r e s p o n s i b l e f o r payment ( X , s h i p p i n g ) when by week 4 P l e a s e f o r m a t t h e c l a u s e s f o r t h e c o n t r a c t d e t a i l s and contract text provided : ‘ ‘ ‘ contract details : { contract_details } ‘ ‘ ‘ ‘ ‘ ‘ contract text : { contract_text } ‘ ‘ ‘ """ return g e t _ c o m p l e t i o n ( prompt ) Combining with the results of Step 1, the output of the above prompt with the contract in the introduction is as follows: C1 : Lumber Yard A r e s p o n s i b l e f o r p r o d u c i n g Number 2 Common g r a d e lu mb er ( 1 4 4 , 0 0 0 b o a r d f e e t ) when by week 4 . C2 : Lumber Yard A r e s p o n s i b l e f o r d e l i v e r i n g Number 2 Common g r a d e lu mb er ( 1 4 4 , 0 0 0 b o a r d f e e t ) t o XYZ Homes when by week 4 . C3 : XYZ Homes r e s p o n s i b l e f o r making payment ( $122 , 0 0 0 ) f o r t h e d e l i v e r e d l u m b e r when by week 4 . C4 : XYZ Homes r e s p o n s i b l e f o r making payment ( $500 , 0 0 0 o r below ) f o r t r a n s p o r t and d e l i v e r y c o s t s o f l u m b e r when by week 4 . 4. Evaluation For our experiments, we utilized ChatGPT-3.5-turbo3 and a locally hosted Llama-3 model by Ollama4 as our LLMs for the first phase, in which LLMs extract contract details such as supplier names, product quantities, delivery times, and cost, as well as representing responsibilities between parties. We generated 11 contracts and tested them using the previously described prompts. The results were promising, with the LLMs producing clauses that accurately captured the information and conditions specified in the prompts. We observe that both ChatGPT and Llama3 demonstrated similar performance in extracting clauses from contracts. A detailed breakdown of results is provided in the appendix, showing that the majority of clauses matched the ground truth. (See Appendix for details of the contracts and responses by ChatGPT and Llama3). To avoid the well-known problem of haluzination and increase our trust in the output, we experiment with the idea of using a different method to verify the accuracy of the generated clauses. To test this idea, we employed DSPy, a framework for algorithmically optimizing language model prompts and weights5 whose ‘chain-of-thought’ feature is really appealing. In principle, DSPy allows for a systematic evaluation of clauses generated by the LLMs and provides justification for the consistency of the clauses and the contract. Thus, DSPy could enhance the reliability of the contract extraction process. A DSPy module consists of a signature which specifies the input/output behavior of the module. In our case, the inputs consist of contract text, entities, and questions, while the output is the corre- sponding answer (Yes/No and the reasons for the answer). We then utilized the DSPy module named dspy.ChainOfThought, which instructs the LM to think step by step before generating the response based on the signature. In our first experiment with DSPy for full clause verification, we observe that the system will always answer with ‘Yes’ and provide a reason for its answers. We provided the DSPy framework with a training dataset comprising 10 contracts, each with their respective entities, questions, and answers. We then change some of the clauses, making them inconsistent with the contract, and rerun the experiment. Unexpectedly, DSPy would validate even incorrect clauses, making it unreliable for this task. As such, 3 https://platform.openai.com/docs/guides/text-generation/reproducible-outputs: Text generation models 4 Downloaded from Ollama https://ollama.com/library/llama3 5 https://dspy-docs.vercel.app/docs/intro: DSPy document. the results are not good as the system would respond with ‘yes’ for almost all input clauses regardless of whether they are consistent with the contract. This only suggests that off-the-shelf scripts from the system do not really work or verifying clauses, which are fairly complex, against the contracts is not a simple task for DSPy. To reduce the complexity of the verifying process using DSPy, we experiment with the result of the first step of contract extraction, i.e., we test whether the entities generated by the LLM matched those in the contracts using DSPy. Again, we provide the DSPy framework with a training dataset comprising 10 contracts, each with their respective entities, questions, and answers. In summary, DSPy Signature and the module for DSPy chain of thought are as follow: c l a s s C o n t r a c t D e t a i l s ( dspy . S i g n a t u r e ) : " " " E x t r a c t d e t a i l s f r om a c o n t r a c t . " " " c o n t r a c t = dspy . I n p u t F i e l d ( d e s c = " T e x t o f t h e c o n t r a c t " ) e n t i t y = dspy . I n p u t F i e l d ( d e s c = " E n t i t y o f t h e c o n t r a c t " ) q u e s t i o n = dspy . I n p u t F i e l d ( d e s c = " Q u e s t i o n a b o u t e n t i t y " ) answer = dspy . O u t p u t F i e l d ( d e s c = " E n t i t y e x i s t s i n t h e c o n t r a c t . Answer y e s o r no . " ) DSPy chain of thought as follow: c l a s s G e n e r a t e C o n t r a c t D e t a i l s ( dspy . Module ) : def _ _ i n i t _ _ ( s e l f ) : super ( ) . _ _ i n i t _ _ ( ) s e l f . g e n e r a t e _ d e t a i l s = dspy . ChainOfThought ( C o n t r a c t D e t a i l s ) def f o r w a r d ( s e l f , c o n t r a c t , e n t i t y , q u e s t i o n ) : prediction = s e l f . generate_details ( contract=contract , entity =entity , question=question ) return dspy . P r e d i c t i o n ( answer = p r e d i c t i o n . answer ) The results of this experiment are much better than the results of the previous experiment (i.e., verifying full clauses). The results showed that DSPy correctly identified mismatches between the LLM output and the contract in most cases. For instance, when the LLM generated incorrect names for the supplier or buyer, DSPy was able to flag these errors. This confirmed the utility of DSPy for simpler verification tasks, while more advanced methods would be needed for full clause verification. Below, we report the results of this experiment using the same supply chain contract between XYZ Homes and Lumber Yard A. We intentionally changed some entities to be different from the original output of the LLLM to see whether DSPy can recongize it (in this case, the name of the supplier and buyer) so the input for this experiment is the contract and the following list: 1 / Name o f s u p p l i e r and b u y e r : − S u p p l i e r : XYA company − Buyer : QWER Homes 2 / P r o d u c t / s e r v i c e and q u a n t i t y p r o v i d e d by s u p p l i e r : − P r o d u c t / s e r v i c e : Number 10 Common g r a d e l u m b e r − Q u a n t i t y : 1 5 0 , 0 0 0 b o a r d f e e t p e r month 3/ Categories / grade of product / s e r v i c e : − Grade : At o r a b o v e Number 2 Common g r a d e 4 / E s t i m a t e t i m e d e l i v e r y by week : − Time ( weeks ) : 4 −7 weeks 5 / Payment c o s t from b u y e r : − Payment c o s t : $122 , 0 0 0 − T r a n s p o r t and d e l i v e r y c o s t : a t o r below $500 , 0 0 0 The output that DSPy returns is as follows. The first five items present the entities that DSPy recognizes from the contract. The following paragraph provides the differences between the input (the entities we would like to verify) and DSPY’s outputs and the supporting evidence of the differences. B o o t s t r a p p e d 4 f u l l t r a c e s a f t e r 5 e x a m p l e s i n round 0 . C o n t r a c t d e t a i l s : The e x t r a c t e d d e t a i l s from t h e c o n t r a c t a r e : 1. Entity : ∗ S u p p l i e r : Lumber Yard A ∗ Buyer : XYZ Homes 2 . P r o d u c t / S e r v i c e and q u a n t i t y p r o v i d e d by s u p p l i e r : ∗ P r o d u c t / s e r v i c e : Number 2 Common g r a d e l u m b e r ∗ Q u a n t i t y : 1 4 4 , 0 0 0 b o a r d f e e t p e r month ( n o t 1 5 0 , 0 0 0 a s mentioned in the e n t i t y s e c t i o n ) 3 . Categories / grade of product / s e r v i c e : ∗ Grade : At o r a b o v e Number 2 Common g r a d e 4 . E s t i m a t e t i m e d e l i v e r y by week : ∗ Time ( weeks ) : Not s p e c i f i e d i n t h i s c o n t r a c t , b u t i m p l i e d t o be one month b a s e d on t h e r e q u i r e m e n t f o r Lumber Yard A t o s c h e d u l e t r a n s p o r t and d e l i v e r y o f 14 −16 t r a c t o r t r a i l e r s worth o f l u m b e r . 5 . Payment c o s t from b u y e r : ∗ Payment c o s t : $122 , 0 0 0 f o r 1 4 4 , 0 0 0 b o a r d f e e t ∗ T r a n s p o r t and d e l i v e r y c o s t : At o r below $500 , 0 0 0 f o r 1 4 4 , 0 0 0 board f e e t The e n t i t y m e n t i o n e d i n t h e c o n t r a c t d o e s n o t match t h e c o n t e n t o f t h e e n t i t y s e c t i o n . The c o r r e c t e n t i t i e s a r e Lumber Yard A ( s u p p l i e r ) and XYZ Homes ( b u y e r ) , b u t t h e s e e n t i t i e s do n o t match QWER Homes ( b u y e r ) and XYA company ( s u p p l i e r ) m e n t i o n e d i n t h e entity section . T h e r e f o r e , one o r b o t h o f t h e s e s e c t i o n s c o n t a i n i n c o r r e c t information . 5. Conclusions We report on a preliminary development of a module for contract clause extraction using LLM tech- nologies. Our experiences indicate that LLM could provide help for our task. However, it is also clear that there is no free lunch. LLM is reasonable in identifying entities and relations for a restricted class of texts (contracts) if the prompts are carefully developed. Furthermore, checking the output from LLM also requires additional work. In the future, we plan to identify different sources of contracts, which might contain contracts that are not in the scope of this paper (supply chain), and experiment with LLM to check the breadth of LLMs. Furthermore, we will continue with our validation of generated clauses using DSPy and integrate this module into our simulation system. Acknowledgement. Portions of this publication and research effort are made possible through the help and support of NIST via cooperative agreement 70NANB21H167. References [1] L. Tran, T. C. Son, D. Flynn, M. Balduccini, A Multi-Agent Simulation for Supply Chains Contract Execution, 2024. [2] M. Gelfond, V. Lifschitz, Logic programs with classical negation, in: D. Warren, P. Szeredi (Eds.), Logic Programming: Proceedings of the Seventh International Conference, 1990, pp. 579–597. [3] E. Griffor, C. Greer, D. A. Wollman, M. J. Burns, Framework for cyber-physical systems: volume 1, overview, 2017. [4] E. Griffor, C. Greer, D. A. Wollman, M. J. Burns, Framework for cyber-physical systems: Volume 2, working group reports, 2017. [5] D. Flynn, C. Nadeau, J. Shantz, M. Balduccini, T. C. Son, E. Griffor, Formalizing and Reasoning about Supply Chain Contracts between Agents, in: 25th PADL, volume 13880, 2023. [6] M. Gelfond, V. Lifschitz, Action Languages, Electronic Transactions on Artificial Intelligence 3 (1998). 6. Appendix: Contracts and Generated Clauses Here are all 11 contracts and their clauses that we used: 1. A text representing a contract between two entities, for example, the text “[From [5]] XYZ Homes builds eight to nine 2,000 square foot homes each month for new home buyers. Each new home requires 16,000 board feet of Number 2 Common grade lumber. In order to complete eight to nine homes, XYZ Homes must purchase 144,000 board feet of Number 2 Common grade lumber each month. Lumber Yard A is the preferred supplier of this lumber. XYZ Homes contracts with Lumber Yard A for the required lumber. The two sides agree on the following: a) Lumber Yard A will produce a total of 144,000 board feet of lumber for XYZ Homes. b) Lumber Yard A guarantees to schedule the transport and delivery of 14-16 tractor trailers worth of lumber in one month to XYZ Homes. c) The lumber delivered to XYZ Homes will be at or above Number 2 Common grade. d) The agreed upon cost of lumber is at $122,000 for 144,000 board feet and the transport and delivery cost will be at or below $500,000 for 144,000 board feet. • C1: Lumber Yard A responsible for producing Number 2 Common grade lumber (144,000 board feet) when by week 4. • C2: Lumber Yard A responsible for delivering Number 2 Common grade lumber (144,000 board feet) to XYZ Homes when by week 4. • C3: XYZ Homes responsible for making payment ($122,000) for the delivered lumber when by week 4. • C4: XYZ Homes responsible for making payment ($500,000 or below) for transport and delivery costs of lumber when by week 4. 2. ABC Electronics produces 10-12,000 electronic devices each month. Each device requires 50 high-quality capacitors. To complete their production, ABC Electronics must purchase 500,000 capacitors each month. Component Supplier B is the preferred supplier of these capacitors. a) Component Supplier B will produce a total of 500,000 capacitors for ABC Elec- tronics. This constraint addresses the functionality concern of ABC Electronics. b) Component Supplier B guarantees to schedule the transport and delivery of 10-12 shipments of capacitors in one month to ABC Electronics. This constraint addresses the time to market concern. c) The capacitors delivered to ABC Electronics will be at or above the specified high-quality grade. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of capacitors is $150,000 for 500,000 units and the transport and delivery cost will be at or below $50,000 for 500,000 units. This constraint addresses the cost concern." • C1: Component Supplier B responsible for produced(500K, high-quality capaci- tors) when by week 4. • C2: ABC Electronics responsible for purchase(500K, high-quality capacitors) when by week 4. • C3: Component Supplier B responsible for delivered(500K, high-quality capaci- tors) ∧ 10-12 < Q when by week 1 month. • C4: Component Supplier B responsible for ∃ X < $50K. [transport and deliv- ery(X)] when by week 1 month. • C5: ABC Electronics responsible for payment($150K, high-quality capacitors) when by week 1 month. 3. DEF Pharmaceuticals manufactures 20-25 batches of medicine each month. Each batch requires 200 liters of a specific chemical compound. To complete their production, DEF Pharmaceuticals must purchase 5,000 liters of the chemical compound each month. Chemical Supplier C is the preferred supplier of this compound. a) Chemical Supplier C will produce a total of 5,000 liters of the chemical compound for DEF Pharmaceuticals. This constraint addresses the functionality concern of DEF Pharmaceuticals. b) Chemical Supplier C guarantees to schedule the transport and delivery of 5-6 shipments of the chemical compound in one month to DEF Pharmaceuticals. This constraint addresses the time to market concern. c) The chemical compound delivered to DEF Pharmaceuticals will meet the specified purity standards. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of the chemical compound is $100,000 for 5,000 liters and the transport and delivery cost will be at or below $20,000 for 5,000 liters. This constraint addresses the cost concern." • C1: Chemical Supplier C responsible for producing chemical compound (Q) for 5,000 liters when by week 4. • C2: DEF Pharmaceuticals responsible for purchasing 5,000 liters of chemical compound from Chemical Supplier C when by week 4. • C3: Chemical Supplier C responsible for delivering 5-6 shipments of chemical compound to DEF Pharmaceuticals within a one-month period (estimated to be delivered in 4-6 shipments) when by week 4. • C4: DEF Pharmaceuticals responsible for making payment of $100,000 for the chemical compound and ensuring transport and delivery cost does not exceed $20,000 for 5,000 liters when by week 4. 4. GHI Automotive produces 50-60 cars each month. Each car requires 1,000 pounds of high-grade steel. To complete their production, GHI Automotive must purchase 60,000 pounds of high-grade steel each month. Steel Supplier D is the preferred supplier of this steel. a) Steel Supplier D will produce a total of 60,000 pounds of high-grade steel for GHI Automotive. This constraint addresses the functionality concern of GHI Automotive. b) Steel Supplier D guarantees to schedule the transport and delivery of 6-7 ship- ments of steel in one month to GHI Automotive. This constraint addresses the time to market concern. c) The steel delivered to GHI Automotive will meet the specified high-grade stan- dards. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of steel is $300,000 for 60,000 pounds and the transport and delivery cost will be at or below $40,000 for 60,000 pounds. This constraint addresses the cost concern." • C1: Steel Supplier D responsible for producing 60000 pounds of high-grade steel when by week 6. • C2: GHI Automotive responsible for purchasing 60000 pounds of high-grade steel from Steel Supplier D when by week 6-7. • C3: Steel Supplier D responsible for delivering 60000 pounds of high-grade steel to GHI Automotive when by week 6-7. • C4: Steel Supplier D responsible for ensuring the quality and grade of the pro- duced steel meets the specified high-grade standards when by week 6-7. • C5: Steel Supplier D responsible for keeping transport and delivery costs at or below $40000 for 60000 pounds of steel when by week 6-7. • C6: GHI Automotive responsible for paying a total cost of $300000 for 60000 pounds of high-grade steel when by week 6-7. 5. JKL Furniture manufactures 200-250 pieces of furniture each month. Each piece requires 50 board feet of premium hardwood. To complete their production, JKL Furniture must purchase 12,500 board feet of premium hardwood each month. Wood Supplier E is the preferred supplier of this hardwood. a) Wood Supplier E will produce a total of 12,500 board feet of premium hardwood for JKL Furniture. This constraint addresses the functionality concern of JKL Furniture. b) Wood Supplier E guarantees to schedule the transport and delivery of 2-3 ship- ments of hardwood in one month to JKL Furniture. This constraint addresses the time to market concern. c) The hardwood delivered to JKL Furniture will meet the specified premium grade standards. This constraint addresses several concerns including physical, reliabil- ity, quality, and trustworthiness. d) The agreed upon cost of hardwood is $50,000 for 12,500 board feet and the transport and delivery cost will be at or below $10,000 for 12,500 board feet. This constraint addresses the cost concern." • C1: Wood Supplier E responsible for producing 12,500 board feet of premium hardwood when by week 4. • C2: JKL Furniture responsible for receiving 2-3 shipments of premium hardwood when by week 4. • C3: Wood Supplier E responsible for delivering premium hardwood to JKL Furni- ture meeting specified premium grade standards when by week 4. • C4: JKL Furniture responsible for paying $50,000 for 12,500 board feet of premium hardwood and at or below $10,000 for transport and delivery cost when by week 4. 6. MNO Textiles produces 5-6,000 garments each month. Each garment requires 10 yards of premium fabric. To complete their production, MNO Textiles must purchase 60,000 yards of premium fabric each month. Fabric Supplier F is the preferred supplier of this fabric. a) Fabric Supplier F will produce a total of 60,000 yards of premium fabric for MNO Textiles. This constraint addresses the functionality concern of MNO Textiles. b) Fabric Supplier F guarantees to schedule the transport and delivery of 6-7 ship- ments of fabric in one month to MNO Textiles. This constraint addresses the time to market concern. c) The fabric delivered to MNO Textiles will meet the specified premium qual- ity standards. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of fabric is $240,000 for 60,000 yards and the transport and delivery cost will be at or below $30,000 for 60,000 yards. This constraint addresses the cost concern." • C1: Fabric Supplier F responsible for producing 60,000 yards of premium fabric when by week 4. • C2: MNO Textiles responsible for purchasing 60,000 yards of premium fabric from Fabric Supplier F when by week 4. • C3: Fabric Supplier F responsible for delivering the produced fabric to MNO Textiles in 6-7 shipments within one month (weeks 1-4) when by week 4. • C4: Fabric Supplier F responsible for meeting the premium quality standards of the delivered fabric when by week 4. • C5: MNO Textiles responsible for paying $240,000 for 60,000 yards of premium fabric and $30,000 or less for transport and delivery costs when by week 4. 7. PQR Foods produces 30-35,000 packaged meals each month. Each meal requires 0.5 pounds of a specific ingredient blend. To complete their production, PQR Foods must purchase 17,500 pounds of the ingredient blend each month. Ingredient Supplier G is the preferred supplier of this blend. a) Ingredient Supplier G will produce a total of 17,500 pounds of the ingredient blend for PQR Foods. This constraint addresses the functionality concern of PQR Foods. b) Ingredient Supplier G guarantees to schedule the transport and delivery of 3-4 shipments of the ingredient blend in one month to PQR Foods. This constraint addresses the time to market concern. c) The ingredient blend delivered to PQR Foods will meet the specified quality stan- dards. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of the ingredient blend is $70,000 for 17,500 pounds and the transport and delivery cost will be at or below $10,000 for 17,500 pounds. This constraint addresses the cost concern." • C1: Ingredient Supplier G responsible for produced(17500) when by week <4>. • C2: PQR Foods responsible for payment(70000) when by week <4>. • C3: Ingredient Supplier G responsible for delivered(17500, specified quality stan- dards) when by week <4-7>. • C4: Ingredient Supplier G responsible for X < 10000. [payment(X, transport and delivery)] when by week <4-7>. 8. STU Beverages produces 40-45,000 bottles of drinks each month. Each drink requires one custom glass bottle. To complete their production, STU Beverages must purchase 45,000 custom glass bottles each month. Bottle Supplier H is the preferred supplier of these bottles. a) Bottle Supplier H will produce a total of 45,000 custom glass bottles for STU Beverages. This constraint addresses the functionality concern of STU Beverages. b) Bottle Supplier H guarantees to schedule the transport and delivery of 4-5 ship- ments of bottles in one month to STU Beverages. This constraint addresses the time to market concern. c) The glass bottles delivered to STU Beverages will meet the specified custom quality standards. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of glass bottles is $180,000 for 45,000 bottles and the transport and delivery cost will be at or below $25,000 for 45,000 bottles. This constraint addresses the cost concern." • C1: Bottle Supplier H responsible for produced (45,000) when by week 4. • C2: STU Beverages responsible for payment (180,000, custom glass bottles) when by week 4. • C3: Bottle Supplier H responsible for delivered (custom glass bottles) and meeting custom quality standards when by week 4. • C4: Bottle Supplier H responsible for scheduled transport and delivery of 4-5 shipments of custom glass bottles when by week 4. 9. VWX Cosmetics produces 20-25,000 cosmetic products each month. Each product requires one custom-designed packaging box. To complete their production, VWX Cosmetics must purchase 25,000 custom-designed packaging boxes each month. Pack- aging Supplier I is the preferred supplier of these boxes. a) Packaging Supplier I will produce a total of 25,000 custom-designed packaging boxes for VWX Cosmetics. This constraint addresses the functionality concern of VWX Cosmetics. b) Packaging Supplier I guarantees to schedule the transport and delivery of 2-3 shipments of packaging boxes in one month to VWX Cosmetics. This constraint addresses the time to market concern. c) The packaging boxes delivered to VWX Cosmetics will meet the specified custom design standards. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of packaging boxes is $50,000 for 25,000 boxes and the transport and delivery cost will be at or below $5,000 for 25,000 boxes. This constraint addresses the cost concern." • C1: Packaging Supplier I responsible for produced(25000) when by week 4. • C2: VWX Cosmetics responsible for payment(50000) when by week 4. • C3: Packaging Supplier I responsible for delivered(25000,custom-designed pack- aging boxes) when by week 4. • C4: Packaging Supplier I responsible for Q < 5000. [transport and delivery(Q)] when by week 4. • C5: Packaging Supplier I responsible for meets(custom design standards) when by week 4. 10. YZA Tech produces 5-6,000 computer systems each month. Each system requires 4 high-performance processors. To complete their production, YZA Tech must pur- chase 24,000 high-performance processors each month. Component Supplier J is the preferred supplier of these processors. a) Component Supplier J will produce a total of 24,000 high-performance processors for YZA Tech. This constraint addresses the functionality concern of YZA Tech. b) Component Supplier J guarantees to schedule the transport and delivery of 4-5 shipments of processors in one month to YZA Tech. This constraint addresses the time to market concern. c) The processors delivered to YZA Tech will meet the specified high-performance standards. This constraint addresses several concerns including physical, reliabil- ity, quality, and trustworthiness. d) The agreed upon cost of processors is $480,000 for 24,000 units and the transport and delivery cost will be at or below $60,000 for 24,000 units. This constraint addresses the cost concern." • C1: Component Supplier J responsible for producing 24,000 high-performance processors when by week 4. • C2: YZA Tech responsible for payment of $480,000 for 24,000 units when by week 4. • C3: Component Supplier J responsible for delivering 4-5 shipments of high- performance processors to YZA Tech when every month (weeks 1-4). • C4: YZA Tech responsible for accepting high-performance processors meeting the specified standards when by week 4. • C5: Component Supplier J responsible for ensuring transport and delivery cost at or below $60,000 for 24,000 units when by week 4. 11. BCD Machinery produces 15-18,000 industrial machines each month. Each machine requires 75 heavy-duty bearings. To complete their production, BCD Machinery must purchase 1,350,000 heavy-duty bearings each month. Bearing Supplier K is the preferred supplier of these bearings. a) Bearing Supplier K will produce a total of 1,350,000 heavy-duty bearings for BCD Machinery. This constraint addresses the functionality concern of BCD Machinery. b) Bearing Supplier K guarantees to schedule the transport and delivery of 13-15 shipments of bearings in one month to BCD Machinery. This constraint addresses the time to market concern. c) The bearings delivered to BCD Machinery will meet the specified heavy-duty standards. This constraint addresses several concerns including physical, reliabil- ity, quality, and trustworthiness. d) The agreed upon cost of bearings is $675,000 for 1,350,000 units and the transport and delivery cost will be at or below $75,000 for 1,350,000 units. This constraint addresses the cost concern." • C1: Bearing Supplier K responsible for producing 1,350,000 heavy-duty bearings when by weeks 13-15. • C2: BCD Machinery responsible for purchasing 1,350,000 heavy-duty bearings from Bearing Supplier K when by weeks 13-15. • C3: BCD Machinery responsible for payment of $675,000 for 1,350,000 units and transport and delivery cost at or below $75,000 when by weeks 13-15.