<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>An Experiment with LLM in Contract Extraction⋆</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Nhan Le</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Tran Cao Son</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>New Mexico State University</institution>
          ,
          <country country="US">USA</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>This paper reports on our experiment with LLM in the development of a module, which translates a contract in natural language into a set of contract clauses and is intended to serve as a component of an contract execution monitoring system. The paper briefly discusses the overall design of the system and the need for an automatic contract extraction module. It then includes a short review of the contract specification language and describes the methods used in contract extraction and preliminary evaluation of the use of LLM.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Supply Chains</kwd>
        <kwd>Contracts</kwd>
        <kwd>LLM</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>In a supply chain, entities (or agents) agree on producing, purchasing, and supplying each other certain
products. Each product is associated with some cost and description. Agreements often have deadlines
for product delivery and payment options etc. Agreements between agents can be formalized as a set of
contract clauses. Agents are obligated to come up with plans that help them to fulfill their obligations.
However, the success of an agent’s plan heavily depends on the other agents’ willingness and successes.
Therefore, agents need the capabilities to identify potential issues that can afect the execution of their
plans and identify courses of actions for dealing with these issues.</p>
      <p>
        This article is part of a project aimed at developing formalization and reasoning methodologies
that make it possible to draw formal conclusions about the properties of supply chains. The project
leverages the domain-agnostic characterization of such properties described in [
        <xref ref-type="bibr" rid="ref12 ref13 ref18 ref3 ref4">3, 4</xref>
        ]. The focus of
the present paper is the development of a supply chain monitoring system with the components as
shown in Figure 1. It assumes that agents have contracts among themselves which are specified via
high-level description language ℒ (see next section for more detail). The system will be used by each
agent participating in the supply chain network, allowing the agent to monitor the progress of the
part of the chain that involves the agent. It has five components: the reasoning engine, the planning
module, the diagnosis module, the explanation module, and an interface to other agents within the
supply chain. The reasoning engine is responsible for answering queries from the user. It will invoke
the planning or diagnosis module whenever it is necessary. The reasoning module will activate the
explanation module to generate an explanation for every query. The diagnosis module is responsible
for identifying potential issues afecting the successful execution of the supply chain at the node. The
planning module is responsible for computing plans that help mitigate those issues. The system will
interact with an ontology solver for reasoning about the concerns of the agent and help the agent to
determine the trustworthiness of other agents. The proposed system will also include an interface for
the agent to work with other agents within the supply chain. This interface is responsible for updating
the progress of contracts among agents. It will work with the reasoning module to identify clauses
within contracts that can no longer be satisfied or might not be satisfied.
      </p>
      <p>
        A preliminary implementation of this system (in simulation mode) is described in [
        <xref ref-type="bibr" rid="ref1 ref10 ref16 ref7">1</xref>
        ]. Most of the
modules are logic programs under answer set semantics [
        <xref ref-type="bibr" rid="ref11 ref17 ref2 ref8">2</xref>
        ]. An agent receives its contract specification
and monitors the changes in the environment and will respond to queries related to a contract  such
as (i) can  be successfully completed? (ii) has any agent violated a clause in ? (iii) will any concern
of the agent, as defined in the CPS Ontology 1 [
        <xref ref-type="bibr" rid="ref12 ref13 ref18 ref3 ref4">3, 4</xref>
        ], be (un)satisfied? etc. Furthermore, the question is
“what needs to be done to mitigate a disruption that arises?”
      </p>
      <p>One of the main inputs of the system is the contract specification, which has to be manually encoded.
Our aim in conducting the experiment described in this paper is to learn how this process can be
automated with LLM. Specifically, we investigate the following problem:</p>
      <p>
        Given: A text representing a contract between two entities, for example, the text “[From
[
        <xref ref-type="bibr" rid="ref14 ref5">5</xref>
        ]] XYZ Homes builds eight to nine 2,000 square foot homes each month for new home
buyers. Each new home requires 16,000 board feet of Number 2 Common grade lumber. In
order to complete eight to nine homes, XYZ Homes must purchase 144,000 board feet of
Number 2 Common grade lumber each month. Lumber Yard A is the preferred supplier of
this lumber. XYZ Homes contracts with Lumber Yard A for the required lumber. The two
sides agree on the following:
1. Lumber Yard A will produce a total of 144,000 board feet of lumber for XYZ Homes.
2. Lumber Yard A guarantees to schedule the transport and delivery of 14-16 tractor
trailers worth of lumber in one month to XYZ Homes.
3. The lumber delivered to XYZ Homes will be at or above Number 2 Common grade.
4. The agreed upon cost of lumber is at $122,000 for 144,000 board feet and the transport
and delivery cost will be at or below $500,000 for 144,000 board feet.”
Goal: Translate the above text into a set of clauses (precise definition in the next section),
for example, the following set of clauses encoding the contract between XYZ Homes and
Lumber Yard A. Let  and  denote Lumber Yard A and XYZ Homes, respectively. The set
of clauses specified by the above paragraph is as follows:
1 :  responsible_for (144, ) ∧ 1 ≤  when _ 4 (1)
2 :  responsible_for (144, ) ∧ 2 ≤  when _ 4 (2)
3 :  responsible_for (122, ) when _ 4 (3)
4 :  responsible_for ∃ ≤ 500.[(, ℎ)] when _ 4 (4)
We will next review ℒ, a language for representing and reasoning about contracts that is also the target
language. Afterwards, we will describe the process of extracting contract clauses from text using LLM.
We then evaluate the results and discuss the issues as well as the potentials of the approach.
1Part of the National Institute of Standards and Technology’s Cyber-Physical Systems (NIST CPS) Framework.
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. Representing and Reasoning about Contracts</title>
      <p>
        ℒ, proposed in [
        <xref ref-type="bibr" rid="ref14 ref5">5</xref>
        ], is a language for representing and reasoning about contracts. The language is built
on action domains of individual agents [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. In ℒ, each contract has two parts. The public part is shared
between parties of the contract and includes the clauses that they agree. The private part is internal to
each party and details the concerns of the party which are related to the contract.
      </p>
      <p>Given two agents  and . Assume that  and  are the action domain of  and , respectively,
i.e.,  ( ∈ {, }) encodes the set of fluents that agent  is aware of and the actions which can be
executed by . We assume that  and  use the same language in encoding the fluents, i.e., a property
shared between  and  will have the same representation in  and . The public part of a contract
between  and  over  and  consists of clauses of the form:
ref_id :</p>
      <p>responsible_for  when _
where
• ref_id is an identifier of the clause;
•  ∈ {, };
•  is a fluent formula constructed over fluents appearing in  ∪ ; and
• _ is a formula representing a time constraint of one the following forms:
 |  | _[ . . . ] | _ 
where  can be any time unit such as day, week, etc.,  and  are integers,  ≤ , and
[ . . . ] denotes the range [,  + 1, . . . , ].</p>
      <p>Given the public part of a contract  between  and , the private part of  for either agent  or 
is represented by statements of the form
(5)
(6)
(7)
ref_id : 
where ref_id is a reference identifier  and  is a requirement. It is assumed that the ontology 
associates each requirement with one or more concerns (of the agent) from the concern forest defined
in the CPS Framework, or any customized concern forest specific to the agent. Formally, a contract 
between two agents  and  constructed over two actions domains  and  is a triple (, , )
where
•  is a set of clauses of the form (5); and
•  (resp. ) is a set of statements of the form (7) for  (resp. ).</p>
      <p>Here, (, ) or (, ) is the contract under  or ’s perspective, respectively, and it will be used
by  or  to evaluate the progress of the contract. Observe that  (resp., ) does not necessarily know
about  (resp., ).</p>
      <p>
        Given a contract  = (, , ) between two agents  and . It sufices for this paper to state
that the semantics will allow us to determine whether a clause of  is satisfied (or successfully executed)
given an action sequence that have been executed by the agents and the observations that the agents
can collect. Interested readers are referred to the paper [
        <xref ref-type="bibr" rid="ref14 ref5">5</xref>
        ] for a detailed presentation of the language.
      </p>
    </sec>
    <sec id="sec-3">
      <title>3. Extracting Clauses From a Contract</title>
      <p>We follow the general guidance for prompt engineering given in2 to extract the clauses from the text.
In this paper, the focus is on the public part of a contract. This is done in two steps.
2https://learn.deeplearning.ai/courses/chatgpt-prompt-eng/lesson/2/guidelines: ChatGPT Prompt Engineering for Developers
https://platform.openai.com/docs/guides/prompt-engineering/six-strategies-for-getting-better-results: Prompt engineering
Step 1: Extracting Basic Information of a Contract. A contract (in supply chain) is between
entities (companies) and includes products, categories of products, deadlines, payment terms, etc. We
call all this information as contract details and begin with asking LLM to extract the contract details.
The prompt for this step is as follows:
# G e n e r a t e c o n t r a c t s d e t a i l s
def g e n e r a t e _ c o n t r a c t _ d e t a i l s ( c o n t r a c t ) :
" " " G e n e r a t e c o n t r a c t d e t a i l s . " " "
prompt = f " " "
G i v i n g a c o n t r a c t above , p l e a s e g e n e r a t e t h e f o l l o w i n g t h i n g s :
1 / Name o f a l l p a r t i e s i n t h e c o n t r a c t
2 / P r o d u c t / s e r v i c e b e t w e e n c o m p a n i e s and i t s q u a n t i t y i f</p>
      <p>p r e s e n t e d i n t h e c o n t r a c t
3 / C a t e g o r i e s / g r a d e o f t h e p r o d u c t / s e r v i c e i f p r e s e n t e d
4 / Time d e l i v e r y e s t i m a t e
5 / Payment t h a t e a c h company has t o make
Use t h e f o l l o w i n g format , w i t h no e x t r a t e x t , no e x t r a n o t e from
t h e model :
1 / Name o f s u p p l i e r and b u y e r
− S u p p l i e r : &lt; s u p p l i e r &gt;
− Buyer : &lt; buyer &gt;
2 / P r o d u c t / s e r v i c e and q u a n t i t y i s p r o v i d e d by t h o s e p a r t i e s
− P r o d u c t / s e r v i c e : &lt; p r o d u c t / s e r v i c e &gt;
− Q u a n t i t y : &lt; q u a n t i t y &gt;
3 / C a t e g o r i e s / g r a d e o f p r o d u c t :</p>
      <p>− Grade : &lt; g r a d e a t o r above &gt;
4 / E s t i m a t e t i m e d e l i v e r y by week :
− Time ( weeks ) : &lt; t i m e d e l i v e r y &gt;
5 / Payment c o s t from b u y e r :
− Payment c o s t : &lt; payment c o s t i f any &gt;
− T r a n s p o r t and d e l i v e r y c o s t : &lt; s h i p p i n g c o s t i f any &gt;
I f t h e r e a r e n o t h i n g i n t h e c o n t r a c t t h a t can p r o v i d e t h i s
i n f o r m a t i o n , t h e n w r i t e \ " No i n f o r m a t i o n p r o v i d e d . \ "
{ c o n t r a c t }
" " "
return g e t _ c o m p l e t i o n ( prompt )</p>
      <sec id="sec-3-1">
        <title>1 / Name o f s u p p l i e r and buyer :</title>
        <p>− S u p p l i e r : Lumber Yard A
− Buyer : XYZ Homes</p>
        <p>For the contract given in the introduction, the output from this step is as follows:
2 / P r o d u c t / s e r v i c e and q u a n t i t y i s p r o v i d e d by t h o s e p a r t i e s :
− P r o d u c t / s e r v i c e : Number 2 Common g r a d e lumber
− Q u a n t i t y : 1 4 4 , 0 0 0 board f e e t
4 / E s t i m a t e t i m e d e l i v e r y by week :
− Time ( weeks ) : 1 month ( a p p r o x i m a t e l y 4 weeks )</p>
      </sec>
      <sec id="sec-3-2">
        <title>5 / Payment c o s t from buyer : − Payment c o s t : $122 , 0 0 0 − T r a n s p o r t and d e l i v e r y c o s t : a t or below $500 , 0 0 0</title>
        <p>Step 2: Construction of Clauses. After the extraction of the basic information, we ask LLM to
construct the clauses according to the language ℒ. The prompt for this step aims at providing LLM
with a brief explanation of how these elements function in the contract, along with some example
clauses. This is as follows.
def g e t _ c l a u s e s ( c o n t r a c t _ d e t a i l s ) :
" " " G e n e r a t e c o n t r a c t c l a u s e s . " " "
prompt = f " " "
A c l a u s e i n a c o n t r a c t d e s c r i b e s t h e r e s p o n s i b i l i t i e s t h a t a
p a r t y must f u l f i l l . P l e a s e g e n e r a t e c l a u s e s f o r t h e p r o v i d e d
c o n t r a c t u s i n g t h e f o l l o w i n g g u i d e l i n e s , w i t h no a d d i t i o n a l
i n f o r m a t i o n o r t e x t , no e x t r a t e x t , no e x t r a n o t e from model :
’ ’ ’ Each c l a u s e s h o u l d i n c l u d e :
1 ) &lt; c l a u s e _ n u m b e r &gt; : A d i s t i n c t i d e n t i f i e r f o r t h e c l a u s e ,
f o r m a t t e d a s " C&lt; c l a u s e _ n u m b e r &gt; " .
2 ) &lt; p a r t y _ r e s p o n s i b l e &gt; : The name o f t h e p a r t y r e s p o n s i b l e
f o r f u l f i l l i n g t h e o b l i g a t i o n s o u t l i n e d i n t h e c l a u s e .
3 ) &lt; r e s p o n s i b i l i t y &gt; : A b r i e f d e s c r i p t i o n o f t h e
r e s p o n s i b i l i t y . T h i s c o u l d i n c l u d e d e l i v e r i n g a p r o d u c t
o r s e r v i c e , m e e t i n g q u a l i t y s t a n d a r d s , o r making payments
, e t c . . .
4 ) &lt; d e a d l i n e &gt; : The t i m e l i n e o r s c h e d u l e f o r f u l f i l l i n g t h e
r e s p o n s i b i l i t y . The d e a d l i n e s h o u l d be s p e c i f i e d i n t e r m s
o f f r e q u e n c y ( e . g . , monthly , w e e k l y ) o r a s p e c i f i c week
number ( e . g . , by \ _week &lt; week_number &gt; ) . ’ ’ ’
Each c l a u s e s h o u l d be f o r m a t t e d a s f o l l o w s , w i t h no
a d d i t i o n a l i n f o r m a t i o n o r t e x t , no e x t r a t e x t , no e x t r a
n o t e from model :
C&lt; c l a u s e _ n u m b e r &gt; : &lt; p a r t y _ r e s p o n s i b l e &gt; r e s p o n s i b l e f o r &lt;</p>
        <p>r e s p o n s i b i l i t y &gt; when &lt; d e a d l i n e &gt; . "
Use t h e s e example c l a u s e s t o g u i d e you r f o r m a t t i n g , don ’ t do
e x a c t l y l i k e t h e s e e x a m p l e s :
− " C1 : A r e s p o n s i b l e f o r p r o d u c e d b o a r d ( 1 4 4 K , Q) and 2 &lt; Q
when by week 4
− " C2 : A r e s p o n s i b l e f o r d e l i v e r e d ( 1 4 4 K , Q) and 2 &lt; Q when by
week 4
− " C3 : B r e s p o n s i b l e f o r payment ( 1 2 2 K , b o a r d ) when by week 4
− " C4 : B r e s p o n s i b l e f o r payment ( X , s h i p p i n g ) when by week 4
P l e a s e f o r m a t t h e c l a u s e s f o r t h e c o n t r a c t d e t a i l s and</p>
        <p>c o n t r a c t t e x t p r o v i d e d :
‘ ‘ ‘ c o n t r a c t d e t a i l s : { c o n t r a c t _ d e t a i l s } ‘ ‘ ‘
‘ ‘ ‘ c o n t r a c t t e x t : { c o n t r a c t _ t e x t } ‘ ‘ ‘
" " "
r e t u r n g e t _ c o m p l e t i o n ( prompt )</p>
        <p>Combining with the results of Step 1, the output of the above prompt with the contract in the
introduction is as follows:
C1 : Lumber Y a r d A r e s p o n s i b l e f o r p r o d u c i n g Number 2 Common g r a d e
l u m b e r ( 1 4 4 , 0 0 0 b o a r d f e e t ) when by week 4 .</p>
        <p>C2 : Lumber Y a r d A r e s p o n s i b l e f o r d e l i v e r i n g Number 2 Common g r a d e
l u m b e r ( 1 4 4 , 0 0 0 b o a r d f e e t ) t o XYZ Homes when by week 4 .
C3 : XYZ Homes r e s p o n s i b l e f o r making payment ( $ 1 2 2 , 0 0 0 ) f o r t h e
d e l i v e r e d l u m b e r when by week 4 .</p>
        <p>C4 : XYZ Homes r e s p o n s i b l e f o r making payment ( $ 5 0 0 , 0 0 0 o r b e l o w ) f o r
t r a n s p o r t and d e l i v e r y c o s t s o f l u m b e r when by week 4 .</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Evaluation</title>
      <p>For our experiments, we utilized ChatGPT-3.5-turbo3 and a locally hosted Llama-3 model by Ollama4 as
our LLMs for the first phase, in which LLMs extract contract details such as supplier names, product
quantities, delivery times, and cost, as well as representing responsibilities between parties. We
generated 11 contracts and tested them using the previously described prompts. The results were
promising, with the LLMs producing clauses that accurately captured the information and conditions
specified in the prompts. We observe that both ChatGPT and Llama3 demonstrated similar performance
in extracting clauses from contracts. A detailed breakdown of results is provided in the appendix,
showing that the majority of clauses matched the ground truth. (See Appendix for details of the
contracts and responses by ChatGPT and Llama3).</p>
      <p>To avoid the well-known problem of haluzination and increase our trust in the output, we experiment
with the idea of using a diferent method to verify the accuracy of the generated clauses. To test this
idea, we employed DSPy, a framework for algorithmically optimizing language model prompts and
weights5 whose ‘chain-of-thought’ feature is really appealing. In principle, DSPy allows for a systematic
evaluation of clauses generated by the LLMs and provides justification for the consistency of the clauses
and the contract. Thus, DSPy could enhance the reliability of the contract extraction process.</p>
      <p>A DSPy module consists of a signature which specifies the input/output behavior of the module.
In our case, the inputs consist of contract text, entities, and questions, while the output is the
corresponding answer (Yes/No and the reasons for the answer). We then utilized the DSPy module named
dspy.ChainOfThought, which instructs the LM to think step by step before generating the response
based on the signature.</p>
      <p>In our first experiment with DSPy for full clause verification, we observe that the system will always
answer with ‘Yes’ and provide a reason for its answers. We provided the DSPy framework with a
training dataset comprising 10 contracts, each with their respective entities, questions, and answers. We
then change some of the clauses, making them inconsistent with the contract, and rerun the experiment.
Unexpectedly, DSPy would validate even incorrect clauses, making it unreliable for this task. As such,
3https://platform.openai.com/docs/guides/text-generation/reproducible-outputs: Text generation models
4Downloaded from Ollama https://ollama.com/library/llama3
5https://dspy-docs.vercel.app/docs/intro: DSPy document.
the results are not good as the system would respond with ‘yes’ for almost all input clauses regardless
of whether they are consistent with the contract. This only suggests that of-the-shelf scripts from the
system do not really work or verifying clauses, which are fairly complex, against the contracts is not a
simple task for DSPy.</p>
      <p>To reduce the complexity of the verifying process using DSPy, we experiment with the result of the
ifrst step of contract extraction, i.e., we test whether the entities generated by the LLM matched those
in the contracts using DSPy. Again, we provide the DSPy framework with a training dataset comprising
10 contracts, each with their respective entities, questions, and answers.</p>
      <p>In summary, DSPy Signature and the module for DSPy chain of thought are as follow:
c l a s s C o n t r a c t D e t a i l s ( dspy . S i g n a t u r e ) :
" " " E x t r a c t d e t a i l s from a c o n t r a c t . " " "
c o n t r a c t = dspy . I n p u t F i e l d ( d e s c = " T e x t o f t h e c o n t r a c t " )
e n t i t y = dspy . I n p u t F i e l d ( d e s c = " E n t i t y o f t h e c o n t r a c t " )
q u e s t i o n = dspy . I n p u t F i e l d ( d e s c = " Q u e s t i o n a b o u t e n t i t y " )
answer = dspy . O u t p u t F i e l d ( d e s c = " E n t i t y e x i s t s i n t h e c o n t r a c t .</p>
      <p>Answer y e s o r no . " )
DSPy chain of thought as follow:
c l a s s G e n e r a t e C o n t r a c t D e t a i l s ( dspy . Module ) :
def _ _ i n i t _ _ ( s e l f ) :
super ( ) . _ _ i n i t _ _ ( )
s e l f . g e n e r a t e _ d e t a i l s = dspy . ChainOfThought ( C o n t r a c t D e t a i l s )
def f o r w a r d ( s e l f , c o n t r a c t , e n t i t y , q u e s t i o n ) :
p r e d i c t i o n = s e l f . g e n e r a t e _ d e t a i l s ( c o n t r a c t = c o n t r a c t , e n t i t y
= e n t i t y , q u e s t i o n = q u e s t i o n )
return dspy . P r e d i c t i o n ( answer = p r e d i c t i o n . answer )</p>
      <p>The results of this experiment are much better than the results of the previous experiment (i.e.,
verifying full clauses). The results showed that DSPy correctly identified mismatches between the LLM
output and the contract in most cases. For instance, when the LLM generated incorrect names for the
supplier or buyer, DSPy was able to flag these errors. This confirmed the utility of DSPy for simpler
verification tasks, while more advanced methods would be needed for full clause verification. Below,
we report the results of this experiment using the same supply chain contract between XYZ Homes and
Lumber Yard A. We intentionally changed some entities to be diferent from the original output of the
LLLM to see whether DSPy can recongize it (in this case, the name of the supplier and buyer) so the
input for this experiment is the contract and the following list:
1 / Name o f s u p p l i e r and b u y e r :
− S u p p l i e r : XYA company
− Buyer : QWER Homes
2 / P r o d u c t / s e r v i c e and q u a n t i t y p r o v i d e d by s u p p l i e r :
− P r o d u c t / s e r v i c e : Number 10 Common g r a d e lumber
− Q u a n t i t y : 1 5 0 , 0 0 0 b o a r d f e e t p e r month
3 / C a t e g o r i e s / g r a d e o f p r o d u c t / s e r v i c e :
− Grade : At o r above Number 2 Common g r a d e
4 / E s t i m a t e t i m e d e l i v e r y by week :
− Time ( weeks ) : 4 −7 weeks</p>
      <p>The output that DSPy returns is as follows. The first five items present the entities that DSPy
recognizes from the contract. The following paragraph provides the diferences between the input (the
entities we would like to verify) and DSPY’s outputs and the supporting evidence of the diferences.
B o o t s t r a p p e d 4 f u l l t r a c e s a f t e r 5 examples i n round 0 .</p>
      <p>C o n t r a c t d e t a i l s : The e x t r a c t e d d e t a i l s from t h e c o n t r a c t a r e :</p>
    </sec>
    <sec id="sec-5">
      <title>5. Conclusions</title>
      <p>We report on a preliminary development of a module for contract clause extraction using LLM
technologies. Our experiences indicate that LLM could provide help for our task. However, it is also clear
that there is no free lunch. LLM is reasonable in identifying entities and relations for a restricted class
of texts (contracts) if the prompts are carefully developed. Furthermore, checking the output from LLM
also requires additional work. In the future, we plan to identify diferent sources of contracts, which
might contain contracts that are not in the scope of this paper (supply chain), and experiment with
LLM to check the breadth of LLMs. Furthermore, we will continue with our validation of generated
clauses using DSPy and integrate this module into our simulation system.</p>
      <p>Acknowledgement. Portions of this publication and research efort are made possible through the help and
support of NIST via cooperative agreement 70NANB21H167.</p>
    </sec>
    <sec id="sec-6">
      <title>6. Appendix: Contracts and Generated Clauses</title>
      <p>Here are all 11 contracts and their clauses that we used:
c) The capacitors delivered to ABC Electronics will be at or above the specified
high-quality grade. This constraint addresses several concerns including physical,
reliability, quality, and trustworthiness.
d) The agreed upon cost of capacitors is $150,000 for 500,000 units and the transport
and delivery cost will be at or below $50,000 for 500,000 units. This constraint
addresses the cost concern."
• C1: Component Supplier B responsible for produced(500K, high-quality
capacitors) when by week 4.
• C2: ABC Electronics responsible for purchase(500K, high-quality capacitors)
when by week 4.
• C3: Component Supplier B responsible for delivered(500K, high-quality
capacitors) ∧ 10-12 &lt; Q when by week 1 month.
• C4: Component Supplier B responsible for ∃ X &lt; $50K. [transport and
delivery(X)] when by week 1 month.
• C5: ABC Electronics responsible for payment($150K, high-quality capacitors)
when by week 1 month.
3. DEF Pharmaceuticals manufactures 20-25 batches of medicine each month. Each batch
requires 200 liters of a specific chemical compound. To complete their production,
DEF Pharmaceuticals must purchase 5,000 liters of the chemical compound each
month. Chemical Supplier C is the preferred supplier of this compound.
a) Chemical Supplier C will produce a total of 5,000 liters of the chemical compound
for DEF Pharmaceuticals. This constraint addresses the functionality concern of
DEF Pharmaceuticals.
b) Chemical Supplier C guarantees to schedule the transport and delivery of 5-6
shipments of the chemical compound in one month to DEF Pharmaceuticals. This
constraint addresses the time to market concern.
c) The chemical compound delivered to DEF Pharmaceuticals will meet the specified
purity standards. This constraint addresses several concerns including physical,
reliability, quality, and trustworthiness.
d) The agreed upon cost of the chemical compound is $100,000 for 5,000 liters and
the transport and delivery cost will be at or below $20,000 for 5,000 liters. This
constraint addresses the cost concern."
• C1: Chemical Supplier C responsible for producing chemical compound (Q) for
5,000 liters when by week 4.
• C2: DEF Pharmaceuticals responsible for purchasing 5,000 liters of chemical
compound from Chemical Supplier C when by week 4.
• C3: Chemical Supplier C responsible for delivering 5-6 shipments of chemical
compound to DEF Pharmaceuticals within a one-month period (estimated to be
delivered in 4-6 shipments) when by week 4.
• C4: DEF Pharmaceuticals responsible for making payment of $100,000 for the
chemical compound and ensuring transport and delivery cost does not exceed
$20,000 for 5,000 liters when by week 4.
4. GHI Automotive produces 50-60 cars each month. Each car requires 1,000 pounds of
high-grade steel. To complete their production, GHI Automotive must purchase 60,000
pounds of high-grade steel each month. Steel Supplier D is the preferred supplier of
this steel.</p>
      <p>a) Steel Supplier D will produce a total of 60,000 pounds of high-grade steel for
GHI Automotive. This constraint addresses the functionality concern of GHI
Automotive.
b) Steel Supplier D guarantees to schedule the transport and delivery of 6-7
shipments of steel in one month to GHI Automotive. This constraint addresses the
time to market concern.
c) The steel delivered to GHI Automotive will meet the specified high-grade
standards. This constraint addresses several concerns including physical, reliability,
quality, and trustworthiness.
d) The agreed upon cost of steel is $300,000 for 60,000 pounds and the transport
and delivery cost will be at or below $40,000 for 60,000 pounds. This constraint
addresses the cost concern."
• C1: Steel Supplier D responsible for producing 60000 pounds of high-grade steel
when by week 6.
• C2: GHI Automotive responsible for purchasing 60000 pounds of high-grade steel
from Steel Supplier D when by week 6-7.
• C3: Steel Supplier D responsible for delivering 60000 pounds of high-grade steel
to GHI Automotive when by week 6-7.
• C4: Steel Supplier D responsible for ensuring the quality and grade of the
produced steel meets the specified high-grade standards when by week 6-7.
• C5: Steel Supplier D responsible for keeping transport and delivery costs at or
below $40000 for 60000 pounds of steel when by week 6-7.
• C6: GHI Automotive responsible for paying a total cost of $300000 for 60000
pounds of high-grade steel when by week 6-7.
5. JKL Furniture manufactures 200-250 pieces of furniture each month. Each piece
requires 50 board feet of premium hardwood. To complete their production, JKL
Furniture must purchase 12,500 board feet of premium hardwood each month. Wood
Supplier E is the preferred supplier of this hardwood.</p>
      <p>a) Wood Supplier E will produce a total of 12,500 board feet of premium hardwood
for JKL Furniture. This constraint addresses the functionality concern of JKL
Furniture.
b) Wood Supplier E guarantees to schedule the transport and delivery of 2-3
shipments of hardwood in one month to JKL Furniture. This constraint addresses the
time to market concern.
c) The hardwood delivered to JKL Furniture will meet the specified premium grade
standards. This constraint addresses several concerns including physical,
reliability, quality, and trustworthiness.
d) The agreed upon cost of hardwood is $50,000 for 12,500 board feet and the
transport and delivery cost will be at or below $10,000 for 12,500 board feet. This
constraint addresses the cost concern."
• C1: Wood Supplier E responsible for producing 12,500 board feet of premium
hardwood when by week 4.
• C2: JKL Furniture responsible for receiving 2-3 shipments of premium hardwood
when by week 4.
• C3: Wood Supplier E responsible for delivering premium hardwood to JKL
Furniture meeting specified premium grade standards when by week 4.
• C4: JKL Furniture responsible for paying $50,000 for 12,500 board feet of premium
hardwood and at or below $10,000 for transport and delivery cost when by week
4.
6. MNO Textiles produces 5-6,000 garments each month. Each garment requires 10 yards
of premium fabric. To complete their production, MNO Textiles must purchase 60,000
yards of premium fabric each month. Fabric Supplier F is the preferred supplier of
this fabric.</p>
      <p>a) Fabric Supplier F will produce a total of 60,000 yards of premium fabric for MNO</p>
      <p>Textiles. This constraint addresses the functionality concern of MNO Textiles.
b) Fabric Supplier F guarantees to schedule the transport and delivery of 6-7
shipments of fabric in one month to MNO Textiles. This constraint addresses the
time to market concern.
c) The fabric delivered to MNO Textiles will meet the specified premium
quality standards. This constraint addresses several concerns including physical,
reliability, quality, and trustworthiness.
d) The agreed upon cost of fabric is $240,000 for 60,000 yards and the transport
and delivery cost will be at or below $30,000 for 60,000 yards. This constraint
addresses the cost concern."
• C1: Fabric Supplier F responsible for producing 60,000 yards of premium fabric
when by week 4.
• C2: MNO Textiles responsible for purchasing 60,000 yards of premium fabric
from Fabric Supplier F when by week 4.
• C3: Fabric Supplier F responsible for delivering the produced fabric to MNO</p>
      <p>Textiles in 6-7 shipments within one month (weeks 1-4) when by week 4.
• C4: Fabric Supplier F responsible for meeting the premium quality standards of
the delivered fabric when by week 4.
• C5: MNO Textiles responsible for paying $240,000 for 60,000 yards of premium
fabric and $30,000 or less for transport and delivery costs when by week 4.
7. PQR Foods produces 30-35,000 packaged meals each month. Each meal requires 0.5
pounds of a specific ingredient blend. To complete their production, PQR Foods must
purchase 17,500 pounds of the ingredient blend each month. Ingredient Supplier G is
the preferred supplier of this blend.</p>
      <p>a) Ingredient Supplier G will produce a total of 17,500 pounds of the ingredient
blend for PQR Foods. This constraint addresses the functionality concern of PQR
Foods.
b) Ingredient Supplier G guarantees to schedule the transport and delivery of 3-4
shipments of the ingredient blend in one month to PQR Foods. This constraint
addresses the time to market concern.
c) The ingredient blend delivered to PQR Foods will meet the specified quality
standards. This constraint addresses several concerns including physical, reliability,
quality, and trustworthiness.
d) The agreed upon cost of the ingredient blend is $70,000 for 17,500 pounds and
the transport and delivery cost will be at or below $10,000 for 17,500 pounds.</p>
      <p>This constraint addresses the cost concern."
• C1: Ingredient Supplier G responsible for produced(17500) when by week &lt;4&gt;.
• C2: PQR Foods responsible for payment(70000) when by week &lt;4&gt;.
• C3: Ingredient Supplier G responsible for delivered(17500, specified quality
standards) when by week &lt;4-7&gt;.
• C4: Ingredient Supplier G responsible for X &lt; 10000. [payment(X, transport and
delivery)] when by week &lt;4-7&gt;.
8. STU Beverages produces 40-45,000 bottles of drinks each month. Each drink requires
one custom glass bottle. To complete their production, STU Beverages must purchase
45,000 custom glass bottles each month. Bottle Supplier H is the preferred supplier of
these bottles.</p>
      <p>a) Bottle Supplier H will produce a total of 45,000 custom glass bottles for STU</p>
      <p>Beverages. This constraint addresses the functionality concern of STU Beverages.
b) Bottle Supplier H guarantees to schedule the transport and delivery of 4-5
shipments of bottles in one month to STU Beverages. This constraint addresses the
time to market concern.
c) The glass bottles delivered to STU Beverages will meet the specified custom
quality standards. This constraint addresses several concerns including physical,
reliability, quality, and trustworthiness.
d) The agreed upon cost of glass bottles is $180,000 for 45,000 bottles and the
transport and delivery cost will be at or below $25,000 for 45,000 bottles. This
constraint addresses the cost concern."
• C1: Bottle Supplier H responsible for produced (45,000) when by week 4.
• C2: STU Beverages responsible for payment (180,000, custom glass bottles) when
by week 4.
• C3: Bottle Supplier H responsible for delivered (custom glass bottles) and meeting
custom quality standards when by week 4.
• C4: Bottle Supplier H responsible for scheduled transport and delivery of 4-5
shipments of custom glass bottles when by week 4.
9. VWX Cosmetics produces 20-25,000 cosmetic products each month. Each product
requires one custom-designed packaging box. To complete their production, VWX
Cosmetics must purchase 25,000 custom-designed packaging boxes each month.
Packaging Supplier I is the preferred supplier of these boxes.</p>
      <p>a) Packaging Supplier I will produce a total of 25,000 custom-designed packaging
boxes for VWX Cosmetics. This constraint addresses the functionality concern
of VWX Cosmetics.
b) Packaging Supplier I guarantees to schedule the transport and delivery of 2-3
shipments of packaging boxes in one month to VWX Cosmetics. This constraint
addresses the time to market concern.
c) The packaging boxes delivered to VWX Cosmetics will meet the specified custom
design standards. This constraint addresses several concerns including physical,
reliability, quality, and trustworthiness.
d) The agreed upon cost of packaging boxes is $50,000 for 25,000 boxes and the
transport and delivery cost will be at or below $5,000 for 25,000 boxes. This
constraint addresses the cost concern."
• C1: Packaging Supplier I responsible for produced(25000) when by week 4.
• C2: VWX Cosmetics responsible for payment(50000) when by week 4.
• C3: Packaging Supplier I responsible for delivered(25000,custom-designed
packaging boxes) when by week 4.
• C4: Packaging Supplier I responsible for Q &lt; 5000. [transport and delivery(Q)]
when by week 4.
• C5: Packaging Supplier I responsible for meets(custom design standards) when
by week 4.
10. YZA Tech produces 5-6,000 computer systems each month. Each system requires 4
high-performance processors. To complete their production, YZA Tech must
purchase 24,000 high-performance processors each month. Component Supplier J is the
preferred supplier of these processors.</p>
      <p>a) Component Supplier J will produce a total of 24,000 high-performance processors
for YZA Tech. This constraint addresses the functionality concern of YZA Tech.
b) Component Supplier J guarantees to schedule the transport and delivery of 4-5
shipments of processors in one month to YZA Tech. This constraint addresses
the time to market concern.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>L.</given-names>
            <surname>Tran</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. C.</given-names>
            <surname>Son</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Flynn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Balduccini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A</given-names>
            <surname>Multi-Agent Simulation for Supply Chains Contract Execution</surname>
          </string-name>
          ,
          <year>2024</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>M.</given-names>
            <surname>Gelfond</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Lifschitz</surname>
          </string-name>
          ,
          <article-title>Logic programs with classical negation</article-title>
          , in: D.
          <string-name>
            <surname>Warren</surname>
          </string-name>
          , P. Szeredi (Eds.),
          <source>Logic Programming: Proceedings of the Seventh International Conference</source>
          ,
          <year>1990</year>
          , pp.
          <fpage>579</fpage>
          -
          <lpage>597</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>E.</given-names>
            <surname>Grifor</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Greer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. A.</given-names>
            <surname>Wollman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. J.</given-names>
            <surname>Burns</surname>
          </string-name>
          ,
          <article-title>Framework for cyber-physical systems: volume 1, overview</article-title>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>E.</given-names>
            <surname>Grifor</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Greer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. A.</given-names>
            <surname>Wollman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. J.</given-names>
            <surname>Burns</surname>
          </string-name>
          ,
          <article-title>Framework for cyber-physical systems</article-title>
          : Volume
          <volume>2</volume>
          , working group reports,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>D.</given-names>
            <surname>Flynn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Nadeau</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Shantz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Balduccini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. C.</given-names>
            <surname>Son</surname>
          </string-name>
          , E. Grifor,
          <article-title>Formalizing and Reasoning about Supply Chain Contracts between Agents</article-title>
          ,
          <source>in: 25th PADL</source>
          , volume
          <volume>13880</volume>
          ,
          <year>2023</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>M.</given-names>
            <surname>Gelfond</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Lifschitz</surname>
          </string-name>
          , Action Languages,
          <source>Electronic Transactions on Artificial Intelligence</source>
          <volume>3</volume>
          (
          <year>1998</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <article-title>1. A text representing a contract between two entities, for example, the text “[From [5]] XYZ Homes builds eight to nine 2,000 square foot homes each month for new home buyers</article-title>
          .
          <source>Each new home requires 16</source>
          ,
          <article-title>000 board feet of Number 2 Common grade lumber. In order to complete eight to nine homes</article-title>
          ,
          <source>XYZ Homes must purchase 144</source>
          ,
          <article-title>000 board feet of Number 2 Common grade lumber each month. Lumber Yard A is the preferred supplier of this lumber. XYZ Homes contracts with Lumber Yard A for the required lumber. The two sides agree on the following: a) Lumber Yard A will produce a total of 144,000 board feet of lumber for XYZ Homes</article-title>
          .
          <article-title>b) Lumber Yard A guarantees to schedule the transport and delivery of 14-16 tractor trailers worth of lumber in one month to XYZ Homes. c) The lumber delivered to XYZ Homes will be at or above Number 2 Common grade</article-title>
          . d)
          <article-title>The agreed upon cost of lumber is at $122,000 for 144,000 board feet and the transport and delivery cost will be at or below $500,000 for 144,000 board feet</article-title>
          . •
          <article-title>C1: Lumber Yard A responsible for producing Number 2 Common grade lumber (144,000 board feet) when by week 4. • C2: Lumber Yard A responsible for delivering Number 2 Common grade lumber (144,000 board feet) to XYZ Homes when by week 4. • C3: XYZ Homes responsible for making payment ($122,000) for the delivered lumber when by week 4. • C4: XYZ Homes responsible for making payment ($500,000 or below) for transport and delivery costs of lumber when by week 4</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          2.
          <source>ABC Electronics produces 10-12</source>
          ,
          <article-title>000 electronic devices each month. Each device requires 50 high-quality capacitors. To complete their production</article-title>
          ,
          <source>ABC Electronics must purchase 500</source>
          ,
          <article-title>000 capacitors each month. Component Supplier B is the preferred supplier of these capacitors. a) Component Supplier B will produce a total of 500,000 capacitors for ABC Electronics</article-title>
          .
          <article-title>This constraint addresses the functionality concern of ABC Electronics. b) Component Supplier B guarantees to schedule the transport and delivery of 10-12 shipments of capacitors in one month to ABC Electronics. This constraint addresses the time to market concern.</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <article-title>d) The agreed upon cost of processors is $480,000 for 24,000 units and the transport and delivery cost will be at or below $60,000 for 24,000 units. This constraint addresses the cost concern."</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          <article-title>• C1: Component Supplier J responsible for producing 24,000 high-performance processors when by week 4</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          <article-title>• C2: YZA Tech responsible for payment of $480,000 for 24,000 units when by week 4</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          <article-title>• C3: Component Supplier J responsible for delivering 4-5 shipments of highperformance processors to YZA Tech when every month (weeks 1-4</article-title>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <article-title>• C4: YZA Tech responsible for accepting high-performance processors meeting the specified standards when by week 4</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          <article-title>• C5: Component Supplier J responsible for ensuring transport and delivery cost at or below $60,000 for 24,000 units when by week 4</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          11.
          <source>BCD Machinery produces 15-18</source>
          ,
          <article-title>000 industrial machines each month. Each machine requires 75 heavy-duty bearings. To complete their production</article-title>
          ,
          <source>BCD Machinery must purchase 1</source>
          ,
          <issue>350</issue>
          ,
          <article-title>000 heavy-duty bearings each month. Bearing Supplier K is the preferred supplier of these bearings. a) Bearing Supplier K will produce a total of 1,350,000 heavy-duty bearings for BCD Machinery</article-title>
          .
          <article-title>This constraint addresses the functionality concern of BCD Machinery. b) Bearing Supplier K guarantees to schedule the transport and delivery of 13-15 shipments of bearings in one month to BCD Machinery. This constraint addresses the time to market concern. c) The bearings delivered to BCD Machinery will meet the specified heavy-duty standards. This constraint addresses several concerns including physical, reliability, quality, and trustworthiness. d) The agreed upon cost of bearings is $675,000 for 1,350,000 units and the transport and delivery cost will be at</article-title>
          or below $
          <volume>75</volume>
          ,000 for 1,
          <issue>350</issue>
          ,000 units.
          <article-title>This constraint addresses the cost concern."</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <article-title>• C1: Bearing Supplier K responsible for producing 1,350,000 heavy-duty bearings when by weeks 13-15</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          <article-title>• C2: BCD Machinery responsible for purchasing 1,350,000 heavy-duty bearings from Bearing Supplier K when by weeks 13-15</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          <article-title>• C3: BCD Machinery responsible for payment of $675,000 for 1,350,000 units and transport</article-title>
          and delivery cost at or below $
          <volume>75</volume>
          ,000 when by weeks
          <volume>13</volume>
          -
          <fpage>15</fpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>