<?xml version="1.0" encoding="UTF-8"?>
<TEI xml:space="preserve" xmlns="http://www.tei-c.org/ns/1.0" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.tei-c.org/ns/1.0 https://raw.githubusercontent.com/kermitt2/grobid/master/grobid-home/schemas/xsd/Grobid.xsd"
 xmlns:xlink="http://www.w3.org/1999/xlink">
	<teiHeader xml:lang="en">
		<fileDesc>
			<titleStmt>
				<title level="a" type="main">Verifying Temporal Trust Logic using CTL Model Checking</title>
			</titleStmt>
			<publicationStmt>
				<publisher/>
				<availability status="unknown"><licence/></availability>
			</publicationStmt>
			<sourceDesc>
				<biblStruct>
					<analytic>
						<author>
							<persName><forename type="first">Nagat</forename><surname>Drawel</surname></persName>
							<email>drawe@encs.concordia.ca</email>
						</author>
						<author>
							<persName><forename type="first">Jamal</forename><surname>Bentahar</surname></persName>
							<email>bentahar@ciise.concordia.ca</email>
						</author>
						<author>
							<persName><forename type="first">Mohamed</forename><forename type="middle">El</forename><surname>Menshawy</surname></persName>
						</author>
						<author>
							<persName><forename type="first">Amine</forename><surname>Laarej</surname></persName>
							<email>laarej.amine@gmail.com</email>
						</author>
						<author>
							<persName><forename type="first">R</forename><surname>Cohen</surname></persName>
						</author>
						<author>
							<persName><forename type="first">M</forename><surname>Sensoy</surname></persName>
						</author>
						<author>
							<persName><forename type="first">T</forename><forename type="middle">J</forename><surname>Norman</surname></persName>
						</author>
						<author>
							<affiliation key="aff0">
								<orgName type="department">CIISE</orgName>
								<orgName type="institution">Concordia University</orgName>
								<address>
									<country key="CA">Canada</country>
								</address>
							</affiliation>
						</author>
						<author>
							<affiliation key="aff1">
								<orgName type="department">CIISE</orgName>
								<orgName type="institution">Concordia University</orgName>
								<address>
									<country key="CA">Canada</country>
								</address>
							</affiliation>
						</author>
						<author>
							<affiliation key="aff2">
								<orgName type="institution">Menoufia University</orgName>
								<address>
									<country>Egypt moh</country>
								</address>
							</affiliation>
						</author>
						<author>
							<affiliation key="aff3">
								<orgName type="department">CIISE</orgName>
								<orgName type="institution">Concordia University</orgName>
								<address>
									<country key="CA">Canada</country>
								</address>
							</affiliation>
						</author>
						<title level="a" type="main">Verifying Temporal Trust Logic using CTL Model Checking</title>
					</analytic>
					<monogr>
						<imprint>
							<date/>
						</imprint>
					</monogr>
					<idno type="MD5">D0C18516A7136AA1952B0265A8AFC34B</idno>
				</biblStruct>
			</sourceDesc>
		</fileDesc>
		<encodingDesc>
			<appInfo>
				<application version="0.7.2" ident="GROBID" when="2023-03-23T23:09+0000">
					<desc>GROBID - A machine learning software for extracting information from scholarly documents</desc>
					<ref target="https://github.com/kermitt2/grobid"/>
				</application>
			</appInfo>
		</encodingDesc>
		<profileDesc>
			<abstract>
<div xmlns="http://www.tei-c.org/ns/1.0"><p>Several formal trust frameworks have been introduced in the area of Multi-Agent Systems (MASs). However, the problem of model checking trust logics is still a challenging research topic that has not been sufficiently investigated yet. In this paper, we address this challenge by proposing a formal and fully automatic model checking technique for a temporal logic of trust. From the logical perspective, the starting point of our proposal is TCTL, a Computation Tree Logic of preconditional Trust that has been recently proposed. We extend this logic by introducing a new modality for conditional trust and describe the logical relationship between preconditional and conditional trust. From the formal verification perspective, we develop transformation-based algorithms fully implemented in a Java toolkit that automatically interacts with the NuSMV model checker. Our verification approach automatically transforms the problem of model checking TCTL into the problem of model checking CTL. We also develop a model checking algorithm for the conditional trust. We provide proofs of the soundness and completeness of our transformation algorithms. Finally, experiments conducted on a standard industrial case study of auto-insurance claim processing demonstrate the efficiency and scalability of our approach in verifying TCTL and conditional trust formulae.</p></div>
			</abstract>
		</profileDesc>
	</teiHeader>
	<text xml:lang="en">
		<body>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>Trust is regarded as being one of the key aspects behind the success and growth of applications based on Multi-Agent Systems (MASs). It has been the focus of many research projects, both theoretical and practical, in the recent years, particularly in domains where open multi-agent technologies are applied (e.g., Internet-based markets, information retrieval, etc.). The importance of trust in such domains arises mainly because it provides a social control that regulates the relationships and interactions among agents. However, despite the growing number of various multi-agent applications, they still encounter many challenges in the verification of agents' behaviors. The existence of many autonomous entities in such systems makes this verification difficult due to the increase in their complexity and heterogeneity. The main challenge that faces MASs is how to ensure the reliability of the trust relationships in the presence of misbehaving entities. Such entities not only create an exception for other agents, but also may obstruct their proper work <ref type="bibr" target="#b25">[26]</ref>. The fact that such systems usually operate in open and uncertain environments makes reasoning about trust and checking the existence of untrusted computations highly desired.</p><p>Many formalisms and approaches that facilitate the specifications of trust in MASs can be found in the literature. However, few approaches addressed trust from a high level abstraction viewpoint <ref type="bibr" target="#b11">[12,</ref><ref type="bibr" target="#b12">13,</ref><ref type="bibr" target="#b36">37]</ref>. Modal logic approaches provide powerful mechanisms that can be effectively used for trust reasoning. Such approaches yield a formal semantics to reason about trust properties in various applications such as security protocols, information sources, and e-markets. For instance, in <ref type="bibr" target="#b9">[10,</ref><ref type="bibr" target="#b22">23,</ref><ref type="bibr" target="#b30">31]</ref>, the authors proposed several logical frameworks for the concept of trust. Trust in such logics is mostly expressed as a combination of different modalities based on the logic of action and time <ref type="bibr" target="#b21">[22]</ref> and the BDI logic <ref type="bibr" target="#b7">[8]</ref>. In <ref type="bibr" target="#b27">[28]</ref>, a modal logic for reasoning about the interaction between belief, evidence and trust is presented. Other approaches are interested in analyzing trust in information sources <ref type="bibr" target="#b2">[3,</ref><ref type="bibr" target="#b9">10,</ref><ref type="bibr" target="#b10">11,</ref><ref type="bibr" target="#b26">27]</ref>. Moreover, some proposals have addressed trust in the context of computer security <ref type="bibr" target="#b20">[21,</ref><ref type="bibr" target="#b30">31]</ref>. Most of these approaches focus on the cognitive side of trust (i.e., trusted agents are capable of exhibiting beliefs, desires, and intentions properties). Hence, the trust is considered as a belief of an agent (the truster) involving ability and willingness of the trustee to perform some actions for the truster. Although these approaches are highly appropriate to reason about trust, their verification faces a fundamental limitation due to their reliance on the internal structure of the interacting agents. In fact, the distributed and open nature of MASs makes the capability of handling and verifying the trust interaction issues of such approaches arduous. That is, it is very difficult for one agent to completely trust others by making assumptions about their internal states.</p><p>As in <ref type="bibr" target="#b11">[12,</ref><ref type="bibr" target="#b12">13,</ref><ref type="bibr" target="#b36">37]</ref>, in this paper, we are considering a cognitive-independent view of trust where trust ingredients are seen from a non-epistemic angle. Trust in our work is defined from a high-level abstraction perspective without having to depend on individual agents' internal states. We represent trust as a direct relation from one agent, the truster, toward another agent, the trustee, where such a relation presupposes specific preconditions with respect to a particular content. Specifically, the truster considers the trustee as a trustworthy agent with regard to a specific content when the behavior of the trustee with respect to this content is as the truster expects, where this expectation is shared by the two participants. For example, the buyer trusts that the seller will deliver the ordered items upon the buyer's payment, where the payment consists the precondition and the order delivery is the expectation commonly shared. From the modeling and specification perspectives, we are considering the Trust Computation Tree Logic (TCTL) <ref type="bibr" target="#b12">[13]</ref>, an extension of the CTL logic <ref type="bibr" target="#b18">[19]</ref>. Equipped with reasoning postulates, TCTL does not only provide a formal basis for reasoning about trust states with preconditions, but it can also be seen as a formal modeling of the social trust interactions among agents.</p><p>As mentioned earlier, the fact that agents are autonomous and have to interact with each other within unreliable social environments entails that deciding whether to trust another agent (for instance to perform some actions) or not is a challenging task. For instance, agents may not be able to comply with their obligations (e.g., an agent may not send the agreed payment for goods received). This raises the need for developing efficient methodologies to handle their present and future behaviors in order to ensure the fulfillment of the system requirements. Currently, the technique of model checking <ref type="bibr" target="#b5">[6,</ref><ref type="bibr" target="#b6">7]</ref> has attracted several contributions with a significant industrial implication. Although these contributions addressed a number of multi-agent aspects such as social commitments <ref type="bibr" target="#b3">[4,</ref><ref type="bibr" target="#b13">14]</ref> and knowledge <ref type="bibr" target="#b29">[30,</ref><ref type="bibr" target="#b40">41]</ref>, model checking trust in multi-agent settings has not been sufficiently investigated yet. From this view, we aim in this work to contribute in the modeling and verification of trust systems. To do so, we show how trust is formally defined and present a formal and fully automatic model checking technique to verify trust properties. We consider two types of trust, analyze the relationship between them and develop transformation-based model checking techniques for verifying the properties that an agent requires to be achieved by the trusted agent. The two types are preconditional, where trust requires first the satisfaction of the precondition, and conditional, where trust is expressed with antecedents and consequences. The following example shows a typical situation in the context of electronic commerce where trust is a highly desired property.</p><p>Example 1. Let us consider the buyers-sellers relationships. The seller has to trust the buyer that the payment for the ordered goods will be sent, and the buyer has to trust the seller to ship the goods. For instance, the buyer requests to purchase one or more items from the seller (i.e., the trust relationship is established between the two parties). Once the former selects an item, the seller trusts that buyer with regard to the payment in order for the request to take place. When the requested items are paid, the seller confirms the order and starts the delivery process. Finally, the requested items are shipped and the buyer is notified.</p><p>Obviously, on-line interactions are characterized by uncertainty and, moreover, the anonymity of the interaction partners. Thus, there is no guarantee that this process will be surely satisfied in concrete applications. Therefore, the need for a logical language that can provide a certain level of abstraction with the ability to express the trust properties explicitly and a verification procedure to verify the trust interactions are of great significance. Figure <ref type="figure" target="#fig_0">1</ref> illustrates the overall approach of model checking TCTL, which consists of three different but integrated phases. In the first phase, we recall the logic TCTL and its formal model defined using the formalism of vector-based interpreted systems <ref type="bibr" target="#b12">[13]</ref>. In the second phase, we introduce our formal verification technique based on transforming the problem of model checking TCTL into the problem of model checking CTL. In the third phase, we implement our transformation technique in a Java toolkit that automatically interacts with the NuSMV model checker and report the verification results using a case study. Moreover, the paper also introduces conditional trust which is not expressed in TCTL, analyzes the logical relationship between preconditional and conditional trust and exploits this relationship to design a new algorithm to verify properties with conditional trust.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>𝑴</head><p>TCTL model represented as a Vector-based Interpreted System TCTL formula 𝝋</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>NuSMV model checker</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Verification results</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Automatic transformation to CTL model and formula</head><p>Transformed CTL model represented as a Kripke Structure 𝒇(𝑴)</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>𝒇(𝝋)</head><p>Transformed CTL formula The present article is organized as follows. We discuss the related work in Section 2. In Section 3, we recall the syntax and semantics of the TCTL logic framework, and introduce the conditional trust along with its syntax, semantics, and logical relationship with preconditional trust. We present our formal transformation algorithms to model check TCTL and conditional trust in Section 4 along with the soundness and completeness proofs. In Section 5, we present the case study and experimental results obtained using the tool that implements our transformation algorithms. We conclude and identify future research directions in Section 6.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">Literature Review</head><p>While the number of proposals on trust modeling is significant, they differ, however, in the topics they addressed and the systems they implemented. In this paper, we are primarily concerned with the issues of reasoning about and verifying trust in the context of MASs using the model checking approach, which has not been deeply investigated yet for trust systems. Various approaches on trust modeling have developed modal logics that capture the important elements of the cognitive theory of trust as proposed in <ref type="bibr" target="#b4">[5]</ref>. In these approaches, trust is considered as a mental attitude based on a specific set of goals and expressed in terms of certain beliefs. In this respect, Herzig et al. <ref type="bibr" target="#b22">[23]</ref> proposed a formal logical framework to formally reason about occurrent and dispositional trust in a multi-agent environment. The authors defined a logic that combines temporal, dynamic, belief and choice logics. Moreover, the proposed logic is extended with operators to allow reasoning about reputation in the scope of collective beliefs. In another work <ref type="bibr" target="#b23">[24]</ref>, Herzig and his colleagues simplified the previous logic presented in <ref type="bibr" target="#b22">[23]</ref> by considering a very simple kind of actions based on the concepts of propositional assignment. That is, the truth values of a propositional variable is assigned to either true or false by the corresponding agents' actions. The new logic provides a simple framework, and it is expressive enough to account for the cognitive theory of trust. In <ref type="bibr" target="#b27">[28]</ref>, Liu and Lorini presented a new dynamic logic called DL-BET for reasoning about the interaction between belief, evidence and trust. The authors introduced three modal operators where each of these concepts are respectively represented. In this logic, the trust operator semantics is interpreted using neighborhood semantics <ref type="bibr" target="#b31">[32]</ref>, which maps each world into a set of subsets of worlds. The authors provided a complete axiomatization for both the static component of the proposed logic and its dynamic extension. Other approaches have focused on trust in information sources. In this line, an early work presented BIT, a modal logic that extends the traditional doxastic logic with modalities for representing belief, information acquisition, and trust <ref type="bibr" target="#b26">[27]</ref>. In the BIT formalism, the trust operator is interpreted using neighborhood semantics <ref type="bibr" target="#b31">[32]</ref>. The logic is provided with a rigorous semantics to precisely characterize 1) the relationships among beliefs and information acquisition, and 2) how different trust properties are represented by considering various axioms of the logic. In a related work, Demolombe and Lorini <ref type="bibr" target="#b10">[11,</ref><ref type="bibr" target="#b30">31]</ref> have focused on analyzing trust in various features of information sources. That is, a certain agent is trusting another agent if the former believes that the information transmitted to them is reliable. In particular, in <ref type="bibr" target="#b30">[31]</ref>, the authors formalized some security properties and their relationships with trust such as integrity, availability and privacy by proposing a modal operator of obligation. Fuchs and colleagues <ref type="bibr" target="#b20">[21]</ref> also addressed trust in the context of computer security. More recent proposals have made the link between trust and argument-based frameworks <ref type="bibr" target="#b2">[3,</ref><ref type="bibr" target="#b33">34,</ref><ref type="bibr" target="#b38">39]</ref>. However, the above mentioned models are not suitable for verification mechanisms using model checking techniques because of their reliance on the internal structure of the interacting agents. Practically, the fact that MASs are deployed in open environments means that these agents are managed by different providers using different types of platforms. Thus, it is very difficult for one agent to completely trust others or to make assumptions about their internal states. Moreover, model checking neighborhood semantics-based modal logics is yet to be solved <ref type="bibr" target="#b16">[17,</ref><ref type="bibr" target="#b32">33]</ref>.</p><p>The closest approach to our work is the one presented by Singh in <ref type="bibr" target="#b36">[37]</ref> where the social perspective of trust has been put forward. Specifically, the author provided a formal semantics for trust with various logical constraints used to reason about trust from an architectural point of view. This logical model combines temporal modalities of linear temporal logic (LTL) <ref type="bibr" target="#b34">[35]</ref>, modality C for commitments <ref type="bibr" target="#b35">[36]</ref> and modality T for trust. Yet, unlike our approach that is based on interpreted systems structure with a grounded semantics, Singh's logic is interpreted using neighborhood semantics <ref type="bibr" target="#b31">[32]</ref> whose model checking is still an open problem <ref type="bibr" target="#b16">[17]</ref>.</p><p>From the model checking point of view, some authors adopt a direct method, which can be performed by either developing a proper model checker from scratch or by extending existing tools with new algorithms for the needed temporal modalities. For instance, in our previous work <ref type="bibr" target="#b12">[13]</ref>, we proposed a new logic-based framework for specifying and model checking preconditional trust in MASs. We introduced TCTL, a branching temporal logic of preconditional trust interpreted in a new vector-based version of interpreted systems that captures the trust relationship between the interacting parties. Reasoning postulates and new symbolic model checking algorithms are presented to formally specify and automatically verify the system under consideration against some desirable properties expressed in TCTL. Thus, a new model checker extending MCMAS, called MCMAS-T, dedicated to TCTL, along with its new input language VISPL (Vectorextended ISPL) have been created. However, TCTL is restricted to preconditional trust and does not support conditional trust, and the symbolic model checking algorithms, although efficient with flat models, showed limited efficiency when the verified models are not flat (i.e., include non-self loops). In this paper, in addition to the introduction of conditional trust, we investigate a different verification mechanism that does not suffer from the non-flat-models limitation. This has been achieved thanks to the high efficiency of CTL model checking to which the model checking of TCTL and conditional trust is transformed.</p><p>Another relevant work is presented by Aldini in <ref type="bibr" target="#b1">[2]</ref>, where a formal framework to evaluate the effectiveness and robustness of trust-based models in order to detect and then isolate different kinds of attacks has been introduced. Indeed, the author integrates trust modeling with distributed systems. In this work, the system properties are expressed using a trust temporal logic (TTL) which combines CTL <ref type="bibr" target="#b18">[19]</ref> and its action-based extension (ACTL) <ref type="bibr" target="#b8">[9]</ref>. Moreover, the trust system model is based on an instance of both labeled transition systems and Kripke structures. The verification of temporal logic properties expressed in TTL has been performed through a mapping to an existing model checking technique. However, the model mapping between the two logics has not been specified and TTL can only specify a single agent model, and it is not adapted to autonomous MASs. El-Qurna et al. presented a model checking framework to verify service trust behaviors model against regular and non-regular properties <ref type="bibr" target="#b17">[18]</ref>. The authors introduced an algorithm to generate a configuration graph of a deterministic pushdown automata (PDA), where the trust behaviors are captured through the observations sequences related to certain interactions between the services and users. The trust behavior properties are specified using Fixed point Logic with Chop (FLC). From the model checking prospective, a symbolic FLC model checking algorithm is applied in order to verify service trust behaviors with respect to trust properties. However, this approach lacks formal semantics for trust because trust formulae are inferred from the context free grammar of trust pattern languages. Moreover, the approach is not designed to formalize and verify trust in the context of MASs.</p><p>On the other hand, model checking trust can also be achieved by indirect techniques, also called transformation-based methods. The idea is to apply certain reduction rules in order to transform the problem at hand to an existing model checking problem. In fact, transformation has been acknowledged as an alternative mechanism for verifying various MASs aspects. The main advantage of this technique is that it enables the designers of MASs applications to get benefit from powerful and already tested model checkers. This technique has been applied for model checking commitments <ref type="bibr" target="#b14">[15]</ref>, knowledge <ref type="bibr" target="#b28">[29]</ref>, and the interaction between knowledge and commitments <ref type="bibr" target="#b0">[1,</ref><ref type="bibr" target="#b37">38]</ref>. To the best of our knowledge, our work is the first attempt that introduces and implements a full transformation technique for verifying trust specifications in MASs.</p><p>3 Trust and Temporal Logic -TCTL</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">Syntax and Semantics</head><p>In <ref type="bibr" target="#b12">[13]</ref>, we introduced TCTL, a temporal logic of trust that extends the Computation Tree Logic (CTL) <ref type="bibr" target="#b18">[19]</ref> to enable reasoning about trust and time. The syntax and semantics of TCTL are as follows:</p><formula xml:id="formula_0">Definition 1. Syntax of TCTL</formula><p>The syntax of TCTL is defined recursively as follows:</p><formula xml:id="formula_1">ϕ ::= true |ρ | ¬ϕ | ϕ ∨ ϕ | EXϕ | EGϕ | E(ϕ ∪ ϕ) | T p (i, j, ϕ, ϕ)</formula><p>where ρ ∈ AP is an atomic proposition from the set of atomic propositions AP, E is the existential quantifier over paths, the formula EXϕ stands for "ϕ holds in the next state in at least one path"; EGϕ stands for "there exists a path in which ϕ holds globally", and the formula E(ϕ ∪ ψ) holds at the current state if there is some future moment for which ψ holds and ϕ holds at all moments until that future moment. EFϕ is the abbreviation of E(true ∪ ϕ). A, the universal quantifier over paths, can be defined in terms of the above as usual: AXϕ = ¬EX¬ϕ; AGϕ = ¬EF¬ϕ; and A(ϕ ∪ ψ) = ¬(E(¬ψ ∪ (¬ϕ ∧ ¬ψ)) ∨ EG¬ψ). The modality T p (i, j, ψ, ϕ) stands for "Preconditional Trust" and is read as "the truster i trusts the trustee j to bring about ϕ given that the precondition ψ holds". That is, we have the trust over the content given that the precondition is satisfied.</p><p>TCTL formula is interpreted over vector-based interpreted system formalism. <ref type="bibr" target="#b11">[12]</ref> extended the original interpreted systems introduced by <ref type="bibr" target="#b19">[20]</ref> to explicitly capture the trust relationship that is established between agents engaged in an interaction. The vector-based interpreted system is composed of:</p><formula xml:id="formula_2">• A set Agt = {1, • • •, n} of n</formula><p>agents in which each agent i ∈ Agt is described by:</p><p>-A non-empty set of local states L i , which represents the complete information that the agent can access at a particular time;</p><p>-A set of local actions Act i to model the temporal evolution of the system;</p><p>-A vector ν of size n , i.e., ν i 1×n , where n is the number of agents in the system at a given time, is associated with each local state l i ∈ L i . ( ν(i), ν( j), ..., ν(k) are the components of the vector ν i at local state l i (s)). The vector ν will be used later to define the trust accessibility relation (Definition 2). Indeed, the vector ν i represents the vision of agent i with regard to the trust of other agents.</p><p>-A local protocol ρ i : L i → 2 Act i assigning a list of enabled actions that may be performed by agent i in a given local state L i ;</p><p>-A local evolution function τ i is defined as: τ i = L i × Act i → L i , which determines the transitions for an individual agent i between local states;</p><p>• A set of global states s ∈ S that represent a snapshot of all agents in the system at a given time. A global state s is a tuple s = (l 1 . . . l n ). The notation L i (s) is used to represent the local state of agent i in the global state s;</p><p>• I ⊆ S is a set of initial global states for the system;</p><p>• The global evolution function of the system is defined as follows: τ : G × ACT −→ G, where ACT = Act 1 × . . . × Act n and each component a ∈ ACT is called a joint action, which is a tuple of actions;</p><p>• As in <ref type="bibr" target="#b19">[20]</ref>, a special agent e is used to model the environment in which the agents operate. e is modeled using a set of local states L e , a set of actions Act e , a protocol P e , and an evolution function τ e .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Definition 2. Model of TCTL</head><p>A model of trust generated from vector-based interpreted systems is a tuple M t = (S t , R t , I t , { i→ j |(i, j) ∈ Agt 2 },V t ), where:</p><p>• S t is a non-empty set of reachable global states for the system;</p><p>• R t ⊆ S t × S t is the transition relation;</p><p>• I t ⊆ S t is a set of initial global states for the system;</p><p>• i→ j ⊆ S t × S t is the direct trust accessibility relation for each truster-trustee pair of agents (i, j) ∈ Agt 2 defined by i→ j iff:</p><p>l i (s t )(ν i ( j)) = l i (s t )(ν i ( j)); s t is reachable from s t using transitions from the transition relation R t ;</p><p>• V t : S t → 2 AP t is a labeling function, where AP t is a set of atomic propositions.</p><p>As in <ref type="bibr" target="#b12">[13]</ref>, the intuition behind the relation i→ j is, for agent i to gain trust in agent j, the former identifies the states that are compatible with their trust vision with regard to the latter, i.e., where agent i is expecting that agent j is trustful. Specifically, this is obtained by comparing the element ν i ( j) in the local state l i at the global state s t (denoted by l i (s t )(ν i ( j))) with ν i ( j) in the local state l i at the global state s t (denoted by l i (s t )(ν i ( j))). Thus, the trust accessibility of agent i towards agent j (i.e., i→ j ) does exist only if the element value that we have for agent j in the vector of the local states of agent i for both global states is the same, i.e., l i (s t )(ν i ( j)) = l i (s t )(ν i ( j)). Finally, infinite sequences of states linked by transitions define paths. If π is a path, then π(i) is the (i + 1)th state in π.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Definition 3. Semantics of TCTL</head><p>Given the model M t , the satisfaction for a TCTL formula ϕ in a global state s t , denoted as (M t , s t ) |= ϕ, is recursively defined as follows: Excluding the trust modality, the semantics of TCTL state formulae is defined as usual (semantics of CTL, since the main component of TCTL is CTL). The state formula T p (i, j, ψ, ϕ) is satisfied in the model M t at s t iff (1) there exists a state s t such that s t = s t and s t i→ j s t , and (2) all the trust accessible states s t that are different from the current state s t satisfy the content of trust ϕ.</p><formula xml:id="formula_3">−(M t , s t ) |= ρ iff ρ ∈ V t (s t ); −(M t , s t ) |= ¬ϕ iff (M t , s t ) ϕ; −(M t , s t ) |= ϕ 1 ∨ ϕ 2 iff (M t ,</formula></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">TCTL and Conditional Trust</head><p>In <ref type="bibr" target="#b36">[37]</ref>, Singh propounds that trust must be conditional, meaning that trust should be expressed using antecedents and consequents. For example, a customer may trust a merchant as follows: "if I pay, then I trust the merchant will deliver the goods" <ref type="bibr" target="#b36">[37]</ref>. Such a statement expresses the customer's expectation and the effect of this expectation on their future plans. Our preconditional trust modality that assumes the prior satisfaction of the precondition is different from conditional trust. Expressing conditional trust requires an extension of TCTL, and to distinguish the two languages, the extended one is called TCTL'. However, there is a logical relationship between preconditional and conditional trust. In fact, as our main objective in the paper is the verification of temporal trust, we will show how this logical relationship will be exploited to inaugurate a model checking procedure for conditional trust (see <ref type="bibr">Section 4.3)</ref>. The idea we aim to convey is that it is possible to decide if a given state, and thus a given model, satisfies a conditional trust formula by calling the model checking of TCTL. To show this, let us first introduce the syntax and semantics of conditional trust. From the syntax perspective, T c (i, j, ψ, ϕ) is read as "agent i trusts agent j about the consequent ϕ when the antecedent ψ holds". It is worth noticing that in the case of precondition trust, for the trust to take place between the interacting agents i and j, the condition ψ ∧ ¬ϕ must be satisfied in the current state s t to ensure that the precondition ψ holds before the trust content ϕ is brought about, while conditional trust requires the existence of at least one accessible state satisfying the antecedent ψ. This condition captures the intuition that the satisfaction of the antecedent is possible in some future. The semantics of T c (i, j, ψ, ϕ) is as follows: −(M t , s t ) |= T c (i, j, ψ, ϕ) iff s t |= ¬ϕ and ∃s t = s t such that s t i→ j s t and s t |= ψ, and ∀s t = s t such that s t i→ j s t and (M t , s t ) |= ψ, we have (M t , s t ) |= ϕ.</p><p>The non satisfaction of the consequent ϕ complies with the first postulate in <ref type="bibr" target="#b36">[37]</ref> stating that when the consequent holds, the trust in this consequent is "completed and is, therefore, no longer active". The following proposition shows the logical link between conditional and preconditional trust:</p><formula xml:id="formula_4">Proposition 1. T c (i, j, ψ, φ ) ∧ ψ ≡ T p (i, j, , ψ → φ ) ∧ ¬T p (i, j, , ¬ψ).</formula><p>The proof of this proposition is direct from the semantics and omitted for the reason of limited space. Furthermore, it is worth mentioning that conditional trust T c (i, j, ψ, φ ) is conceptually and semantically different from trust about conditions, which can be represented by T c (i, j, , ψ → φ ). An example of the former is "if the customer i pays the merchant j, then i trusts j will deliver the goods", while for the latter the example is: "the customer i trusts the merchant j about the fact that, if i pays, then j will deliver the goods" .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">Formal Transformation to Model Check TCTL Logic and Conditional Trust</head><p>In this section, we first introduce a transformation-based approach to address the problem of model checking TCTL. In a nutshell, given a model M t representing a trust based MAS and a TCTL formula ϕ that describes the property that the model M t has to satisfy, the problem of model checking TCTL can be defined as verifying whether or not ϕ holds in M t , which is formally denoted by M t |= ϕ. In particular, we apply specific reduction rules to formally transform the problem of model checking TCTL into the problem of model checking CTL <ref type="bibr" target="#b18">[19]</ref>. This provides a way to perform our implementation on NuSMV. Technically, our transformation method encompasses two stages. First, we apply a set of formal rules to transform vector-based transition systems into Kripke structures. Then, we transform TCTL formulae to CTL ones based on certain rules developed specifically for this purpose. Such a transformation is performed by developing two formal methods that provide accurate alignments between source and target models, and at the same time preserve TCTL semantics without losing the validity of the original model properties. This transformation is then extended to check conditional trust.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1">Transformation of TCTL Model</head><p>In this section, we start by recalling the definition of the CTL model needed for the transformation algorithm.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Definition 4. Model of CTL</head><p>A CTL formula is interpreted over a Kripke Structure M c = (S c , R c , I c ,V c ), where:</p><p>• S c is a non-empty set of states for the system;</p><p>• R c ⊆ S c × S c is the transition relation;</p><p>• I c ⊆ S c is a set of possible initial global states for the system;</p><p>• V c : S c → 2 AP c is a labeling function that maps each state to the set of propositional variables AP c that hold in it.</p><p>Having presented the CTL model, the next step is to establish our transformation technique. Given a TCTL model</p><formula xml:id="formula_5">M t = (S t , R t , I t , { i→ j |(i, j) ∈ Agt 2 },V t ), Algorithm 1 shows how this model is transformed into a CTL model M c = (S c , R c , I c ,V c ).</formula><p>The algorithm takes as input a model M t (line 1) and outputs the transformed model M c (line 2). First, the corresponding model M c has the same set of system states and initial states (i.e., S c = S t ; I c = I t ). Thereafter, the algorithm initializes the set R c , and then the set V c (s) to be equal to the set V t (s) (i.e., at the beginning, states are labeled with the same atomic propositions). We define a new set of atomic propositions needed to represent the trust accessibility relation to capture the semantics of trust as follows X = {α i α j |(i, j) ∈ Agt 2 }. Thus, the set AP c is as follows: AP c = X ∪AP t . Moreover, the algorithm proceeds to transform transition and trust accessibility relations to constitute the transition relations in R c based on two conditions. The first condition checks if the states s t and s t have a transition relation in R t , then this relation becomes a transition relation in R c (lines 8 &amp; 9). For the second condition, the algorithm iterates using for all . . . do to go through each truster-trustee pair of agents and checks if the current state s t has an accessible state s t using the accessibility relation i→ j and this state is different from the state itself, then the algorithm: (1) adds a new transition from the corresponding s c to the corresponding s c in R c if such a transition is not already in R c , and (2) a new atomic proposition is added into the label of s c for the interacting agents i and j in order to mark the accessible state (lines <ref type="bibr">12 &amp; 13)</ref>. Finally, the algorithm returns the transformed model M c after iterating over all the transitions. (1) if (s c , s c ) / ∈ R c then R c := R c ∪ {(s c , s c )} where s c = s t and s c = s t ;</p><formula xml:id="formula_6">Algorithm 1: Transform M t = (S t , R t , I t , { i→ j |(i, j) ∈ Agt 2 },V t ) into M c = (S c , I c , R c ,V c )</formula><p>13:</p><p>(2) V c (s c ) := V c (s c ) ∪ {α i α j } where s c = s t ;</p><p>14:</p><formula xml:id="formula_7">end if 15:</formula><p>end for 16: end for 17: return M c ;</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2">Transformation of TCTL Formulae</head><p>This section presents our method to formally transform any TCTL formula ϕ to a CTL formula f (ϕ) using a recursive transformation function f . The details of this method are illustrated in Algorithm 2. The transformation of the CTL fragment of TCTL is straightforward (lines 1-6). For the trust modality (line 7), the trust formula is transformed inductively into CTL according to the defined semantics as follows: the transformation of the formula ψ ∧¬ϕ should hold in the current state, there exists a path where next state satisfies the added atomic proposition α i α j , which captures the existence of an accessible state, and along all paths, the next states that satisfy the atomic proposition α i α j also satisfy the transformation of the trust content ϕ.  The states s 1 , s 2 and s 4 are accessible from s 0 , while s 3 is not. Furthermore, the trust formula (T p (i, j, ψ, ϕ)) holds in s 0 (i.e., (M t , s 0 ) |= T p (i, j, ψ, ϕ)). According together to provide a trusted service which facilitates efficient claim settlement. The process starts when the policyholder phones the call center Europ Assist to notify a new claim. Thereafter, Europ Assist registers the information and assigns an appropriate garage to provide the repair service to the policyholder. It then notifies the insurance company AGFIL which checks whether the policy is valid or not, and it confirms the claim coverage. AGFIL then sends the claim details to Lee Consulting Services (Lee CS) which is responsible for managing the operation of this service. Lee CS normally appoints an assessor to conduct a physical inspection of damaged vehicle and checks vehicle repair estimates with the garage. When repairs are completed, the garage will issue an invoice to Lee CS which will check the invoice against the original estimate. Lee CS sends all invoices to AGFIL, which in turn finalizes the payment processes.</p><formula xml:id="formula_8">⇝ i → j 𝑺𝟒 𝑺𝟐 ⇝ i → j ⇝ i → j ⇝ i → j</formula></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2">Properties</head><p>In the above scenario, the participating parties have to ensure that the trust relationships are established on one another to perform their task accordingly. Such relationships are formalized using our model of trust M t = (S t , R t , I t , { i→ j |(i. j) ∈ Agt<ref type="foot" target="#foot_1">2</ref> },V t ). To verify the correctness of the AGFIL scenario at design time, we used the safety (something bad will never happen) and liveness (something good will eventually occur) properties expressed using our TCTL' logic. Such important properties have been widely investigated in different context; for instance, by <ref type="bibr" target="#b12">[13,</ref><ref type="bibr" target="#b15">16,</ref><ref type="bibr" target="#b24">25]</ref>. Formally, the safety property ϕ 1 expresses the negation of the bad situation where the insurance company validates the policyholder claim, but the latter never establishes their trust towards the garage with regard to the vehicle repair. The liveness property ϕ 2 states that in all paths globally, it is always the case that if the policy holder reports an accident and their claim is valid, then eventually in all future computation paths, their trust towards the insurance company with regard to the claim payment will take place.</p><formula xml:id="formula_9">ϕ 2 = AG(ReportAccident ∧ValidClaim → AF(T p (policyholder, AGFIL, validClaim, insuranceClaimPayment)).</formula><p>We also checked a liveness property, given by ϕ 3 , expressed as a conditional trust. The formula expresses the existence of a computation path where the repairer trusts AGFIL to pay for the repairs once AGFIL accepts the proposed estimate from the garage. ϕ 3 = EG(T c (Repairer, AGFIL, agreeEstimate, f ul f illRepairPayment)).</p><p>We have fully implemented the presented model checking algorithms in a Java toolkit<ref type="foot" target="#foot_0">1</ref> that interacts automatically with the NuSMV model checker. We have verified the three properties in a parametric way in different models having different numbers of agents ranging from 6 to 30 2 . The results will be presented and discussed in the next section.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.3">Experimental Results</head><p>We encoded our model M t by considering the participating agents, the set of their local states and actions, the local protocol, the local evolution function, and the initial states for each agent. We also considered the accessibility relations between agents by encoding the vector variables, which give a particular agent the possibility to establish the trust towards other agents. We used the VISPL input language of the MCMAS T model checker introduced in <ref type="bibr" target="#b12">[13]</ref>. Thereafter, we used our transformation tool to transform the VISPL model and formulae into the SMV model in order to be able to start the verification process using NuSMV. The experiments are conducted on AMD FX-8350 -8 Cores -4GHZ per core with 32 GB memory. To test the scalability of our algorithms, we report five experiments in Table <ref type="table" target="#tab_1">1</ref>, where we consider the number of agents (Agents#), the number of reachable states (States#), the transformation times of both models and formulae in milliseconds, and the average total time calculated based on the transformation and verification times. The experiments revealed that all the tested formulae are satisfied in our models. As shown in the table, the number of reachable states reflects the fact that the state space increases exponentially when the number of agent increases. Yet, it is clear that the transformation times of both the models and formulae increase only logarithmically with regard to the number of states. It is also worth noticing that the average total time increases polynomially with respect to the number of states. </p></div><figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_0"><head>Figure 1 :</head><label>1</label><figDesc>Figure 1: A schematic view of our TCTL model checking approach</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_1"><head>Algorithm 2 :</head><label>2</label><figDesc>Transform TCTL formula ϕ into CTL formula f (ϕ)1: f (p) = p if p is an atomic proposition; 2: f (¬ϕ) = ¬ f (ϕ); 3: f (ϕ ∨ ψ) = f (ϕ) ∨ f (ψ); 4: f (EXϕ) = EX f (ϕ); 5: f (E(ϕ ∪ ψ)) = E( f (ϕ) ∪ f (ψ)); 6: f (EGϕ) = EG f (ϕ); 7: f (T p (i, j, ψ, ϕ)) = f (ψ) ∧ f (¬ϕ) ∧ EX(α i α j ) ∧ AX(α i α j → f (ϕ));</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_2"><head>Figure 2 :</head><label>2</label><figDesc>Figure 2: Example of the transformation methods</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" xml:id="fig_3"><head>ϕ 1 =</head><label>1</label><figDesc>AG¬(validClaim ∧ ¬T p (policyholder, Repairer, validClaim, carRepair)).</figDesc></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_1"><head>1 :</head><label>1</label><figDesc>Input: the model M t 2: Output: the model M c 3: S c := S t ; 4: I c := I t ; 5: Initialize R c := / 0; 6: Initialize V c (s c ) := V t (s t ) for each s c ∈ S c and s t ∈ S t such that s c = s t ; 7: for each (s t , s t ) ∈ S 2 t do := R c ∪ {(s c , s c )} where s c = s t and s c = s t ;</figDesc><table><row><cell>8:</cell><cell>if (s t , s t ) ∈ R t then</cell></row><row><cell cols="2">9: R 10: for all (i, j) ∈ Agt 2 do</cell></row><row><cell>11:</cell><cell>if s t i→ j s t and s t = s t then</cell></row><row><cell>12:</cell><cell></cell></row></table><note>c</note></figure>
<figure xmlns="http://www.tei-c.org/ns/1.0" type="table" xml:id="tab_2"><head>Table 1 :</head><label>1</label><figDesc>Verification resultsIn this article, we proposed a new model checking framework for the TCTL logic of preconditional trust that is extended to design a new algorithm to model check conditional trust in MASs. We designed transformation-based algorithms that are fully implemented in a new Java toolkit that automatically interacts with the NuSMV model checker of the CTL logic. Our proposed technique is able to automatically transform the problem of model checking TCTL into the problem of model checking CTL. We also discussed the logical relationship between preconditional and conditional trust, which led to the model checking procedure of conditional trust. The proof of the soundness and completeness of our transformation algorithms is provided. Experiments conducted on a standard industrial case study demonstrated the efficiency and scalability of the technique. As future work, we plan to analyze the interaction between social commitments and trust from both, specification and model checking standpoints. Studying trust dynamics in real time MASs is another direction for further investigation.</figDesc><table><row><cell cols="2">Agents# States#</cell><cell>Time of model</cell><cell cols="3">Time of formula transformation</cell><cell>Average total</cell></row><row><cell></cell><cell></cell><cell>transformation (ms)</cell><cell cols="2">(ϕ 1 , ϕ 2 , ϕ 3 ) (ms)</cell><cell></cell><cell>time (ms)</cell></row><row><cell>6</cell><cell>26</cell><cell>0.735313</cell><cell>0.015955</cell><cell>0.015038</cell><cell>0.014661</cell><cell>1.660572</cell></row><row><cell>12</cell><cell>130</cell><cell>1.322853</cell><cell>0.022917</cell><cell>0.021596</cell><cell>0.023090</cell><cell>35.166989</cell></row><row><cell>18</cell><cell>566</cell><cell>1.847349</cell><cell>0.028163</cell><cell>0.027855</cell><cell>0.028726</cell><cell>172.008436</cell></row><row><cell>24</cell><cell>2410</cell><cell>2.879735</cell><cell>0.032878</cell><cell>0.032996</cell><cell>0.039967</cell><cell>938.0084367</cell></row><row><cell>30</cell><cell>17537</cell><cell>4.317968</cell><cell>0.049961</cell><cell>0.049811</cell><cell>0.050127</cell><cell>1271.962603</cell></row><row><cell>6 Conclusion</cell><cell></cell><cell></cell><cell></cell><cell></cell><cell></cell><cell></cell></row></table></figure>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0">The toolkit is available at: https://users.encs.concordia.ca/ ~bentahar/Trust.jar</note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_1">The experiments are available at: https://users.encs.concordia.ca/ ~bentahar/TCTLCaseStudyFiles.rar</note>
		</body>
		<back>
			<div type="annex">
<div xmlns="http://www.tei-c.org/ns/1.0"><p>to the semantics, we obtain (M t , s 0 ) |= ψ ∧ ¬ϕ , and there exists a state s such that s = s and s i→ j s , and all the trust accessible states s such that s = s satisfy ϕ (i.e., (M t , s 1 ) |= ϕ), (M t , s 2 ) |= ϕ, and (M t , s 4 ) |= ϕ)). Using the proposed transformation technique, the model M t is transformed into the CTL model M c of the right side (part b) as follows: the transition and accessibility relations in M t are transformed into transition relations in M c , and the atomic propositions α i α j are added to represent the accessibility relations. Moreover, each state formula in TCTL is transformed into a CTL formula using the transformation function f . Thus, the formulae T p (i, j, ψ, ϕ) and ψ ∧ ¬ϕ are transformed into f (T p (i, j, ψ, ϕ) and f (ψ) ∧ f (¬ϕ) in state s 0 , and for every path, the next state that satisfies the added atomic proposition (i.e., s 1 , s 2 , and s 4 ) also satisfies the transformation of the trust content ϕ.</p><p>Theorem 1 (Soundness and Completeness of the Transformation). Let M t and ϕ be respectively a TCTL model and formula and let M C and f (ϕ) be the corresponding model and formula in CTL. We have</p><p>where s c is the corresponding state of s t in M c .</p><p>Proof. We prove this theorem by induction on the structure of the formula ϕ. All the formulae are straightforward, except the trust formula: T p (i, j, ψ, ϕ). The first and second parts: ( f (ψ) ∧ f (¬ϕ)) capture the first condition of the semantics where the current state should satisfy ψ ∧ ¬ϕ. The third part (EX(α i α j )) captures the second condition, i.e., the existence of an state different from the current state since α i α j holds only in such accessible states (line 13 of Algorithm 1). Finally, the fourth part (AX(α i α j → f (ϕ))) captures the last condition in the semantics of the trust formula where all accessible states (those satisfying α i α j in M c ) should satisfy ϕ.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3">Model Checking Conditional Trust</head><p>A similar approach to model checking TCTL can be used to model check conditional trust by transforming the conditional trust formula of TCTL' to a CTL formula as follows:</p><p>In this section, we introduce an alternative solution that uses the developed model checking algorithm of TCTL. The algorithm of model checking conditional trust (Algorithm 3) capitalizes on the equivalence shown in Proposition 1 (line 3). If T p (i, j, , ψ → ϕ) ∧ ¬T p (i, j, , ¬ψ) does not hold, then a transformation will be needed. Such a transformation works in all cases, but to be more efficient, the algorithm uses it only if the direct condition on line 3 fails. The transformation, which also exploits Proposition 1, introduces two fresh atomic propositions: µ and κ. µ holds in the current state (line 4) and in every state where ψ holds (line 7). Thus, if ¬T p (i, j, , ¬µ) holds (condition 1), then there is an accessible state different from the current state where ψ holds. κ does not hold in the current state, but holds in every state where ψ → ϕ holds. Consequently, if T p (i, j, ¬ϕ, κ) holds (condition 2), then ¬ϕ holds and all accessible states different from the current state satisfy ψ → ϕ. The algorithm returns true if the two conditions 1 and 2 hold (line 9), false otherwise (line 10). The following theorem is direct from the semantics of conditional trust and Proposition 1.</p><p>Theorem 2 (Soundness and Completeness of Algorithm 3). Algorithm 3 returns true iff (M t , s t ) |= T c (i, j, ψ, ϕ).</p><p>Algorithm 3: Model Check T c (i, j, ψ, ϕ) 1: Input: M t , s t , i, j, ψ, ϕ 2: Output: true if (M t , s t ) |= T c (i, j, ψ, ϕ); false otherwise 3: if (M t , s t ) |= T p (i, j, , ψ → ϕ) ∧ ¬T p (i, j, , ¬ψ) then return true; 4: V t (s t ) := V t (s t ) ∪ {µ}; 5: for all s t = s t 6:</p><p>j, , ¬µ) then return true; 10: return false; 5 Experimental Results</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.1">Insurance Claim Processing: A Case Study</head><p>To explain our approach, we use a standard industrial case study <ref type="bibr" target="#b39">[40]</ref> that outlines the process by which auto insurance claims are handled by an insurance company, AGFIL. There are multiple parties involved in the AGFIL cooperation process: AGFIL, Policyholder, Europ Assist, Lee Consulting Services, Garage, and Assessor. The participating parties work</p></div>			</div>
			<div type="references">

				<listBibl>

<biblStruct xml:id="b0">
	<analytic>
		<title level="a" type="main">Model checking temporal knowledge and commitments in multi-agent systems using reduction</title>
		<author>
			<persName><forename type="first">F</forename><surname>Al-Saqqar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Sultan</surname></persName>
		</author>
		<author>
			<persName><forename type="first">W</forename><surname>Wan</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><forename type="middle">Khosrowshahi</forename><surname>Asl</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Simulation Modelling Practice and Theory</title>
		<imprint>
			<biblScope unit="volume">51</biblScope>
			<biblScope unit="page" from="45" to="68" />
			<date type="published" when="2015">2015</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b1">
	<analytic>
		<title level="a" type="main">A formal framework for modeling trust and reputation in collective adaptive systems</title>
		<author>
			<persName><forename type="first">A</forename><surname>Aldini</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the Workshop on FORmal methods for the quantitative Evaluation of Collective Adaptive SysTems, FORECAST@STAF</title>
				<meeting>the Workshop on FORmal methods for the quantitative Evaluation of Collective Adaptive SysTems, FORECAST@STAF<address><addrLine>Vienna, Austria</addrLine></address></meeting>
		<imprint>
			<date type="published" when="2016">2016</date>
			<biblScope unit="page" from="19" to="30" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b2">
	<analytic>
		<title level="a" type="main">An argumentation-based approach for reasoning about trust in information sources</title>
		<author>
			<persName><forename type="first">L</forename><surname>Amgoud</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Demolombe</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Argument &amp; Computation</title>
		<imprint>
			<biblScope unit="volume">5</biblScope>
			<biblScope unit="issue">2-3</biblScope>
			<biblScope unit="page" from="191" to="215" />
			<date type="published" when="2014">2014</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b3">
	<analytic>
		<title level="a" type="main">Communicative commitments: Model checking and complexity analysis</title>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>El-Menshawy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Qu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Dssouli</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Knowledge-Based Systems</title>
		<imprint>
			<biblScope unit="volume">35</biblScope>
			<biblScope unit="page" from="21" to="34" />
			<date type="published" when="2012">2012</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b4">
	<analytic>
		<title level="a" type="main">Principles of trust for MAS: cognitive anatomy, social importance, and quantification</title>
		<author>
			<persName><forename type="first">C</forename><surname>Castelfranchi</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Falcone</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the Third International Conference on Multiagent Systems, ICMAS</title>
				<meeting>the Third International Conference on Multiagent Systems, ICMAS</meeting>
		<imprint>
			<date type="published" when="1998">1998</date>
			<biblScope unit="page" from="72" to="79" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b5">
	<analytic>
		<title level="a" type="main">Model checking: algorithmic verification and debugging</title>
		<author>
			<persName><forename type="first">E</forename><forename type="middle">M</forename><surname>Clarke</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><forename type="middle">A</forename><surname>Emerson</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Sifakis</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Communications of the ACM</title>
		<imprint>
			<biblScope unit="volume">52</biblScope>
			<biblScope unit="issue">11</biblScope>
			<biblScope unit="page" from="74" to="84" />
			<date type="published" when="2009">2009</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b6">
	<monogr>
		<title level="m" type="main">Model checking</title>
		<author>
			<persName><forename type="first">E</forename><forename type="middle">M</forename><surname>Clarke</surname></persName>
		</author>
		<author>
			<persName><forename type="first">O</forename><surname>Grumberg</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Peled</surname></persName>
		</author>
		<imprint>
			<date type="published" when="1999">1999</date>
			<publisher>MIT press</publisher>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b7">
	<analytic>
		<title level="a" type="main">Intention is choice with commitment</title>
		<author>
			<persName><forename type="first">P</forename><forename type="middle">R</forename><surname>Cohen</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><forename type="middle">J</forename><surname>Levesque</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Artificial Intelligence</title>
		<imprint>
			<biblScope unit="volume">42</biblScope>
			<biblScope unit="issue">2-3</biblScope>
			<biblScope unit="page" from="213" to="261" />
			<date type="published" when="1990">1990</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b8">
	<analytic>
		<title level="a" type="main">Action versus state based logics for transition systems</title>
		<author>
			<persName><forename type="first">R</forename><forename type="middle">De</forename><surname>Nicola</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Vaandrager</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Semantics of Systems of Concurrent Processes</title>
				<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="1990">1990</date>
			<biblScope unit="page" from="407" to="419" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b9">
	<analytic>
		<title level="a" type="main">To trust information sources: A proposal for a modal logical framework</title>
		<author>
			<persName><forename type="first">R</forename><surname>Demolombe</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Trust and deception in virtual societies</title>
				<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="2001">2001</date>
			<biblScope unit="page" from="111" to="124" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b10">
	<analytic>
		<title level="a" type="main">A logical account of trust in information sources</title>
		<author>
			<persName><forename type="first">R</forename><surname>Demolombe</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Lorini</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 11th International Workshop on Trust in Agent Societies</title>
				<meeting>the 11th International Workshop on Trust in Agent Societies</meeting>
		<imprint>
			<date type="published" when="2008">2008</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b11">
	<analytic>
		<title level="a" type="main">Reasoning about trust and time in a system of agents</title>
		<author>
			<persName><forename type="first">N</forename><surname>Drawel</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><forename type="middle">M</forename><surname>Shakshuki</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">The International Conference on Ambient Systems, Networks and Technologies (ANT)</title>
		<title level="s">Procedia Computer Science</title>
		<imprint>
			<date type="published" when="2017">2017</date>
			<biblScope unit="volume">109</biblScope>
			<biblScope unit="page" from="632" to="639" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b12">
	<analytic>
		<title level="a" type="main">Specification and automatic verification of trust-based multi-agent systems</title>
		<author>
			<persName><forename type="first">N</forename><surname>Drawel</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Qu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Shakshuki</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Future Generation Computer Systems</title>
		<imprint>
			<date type="published" when="2018">2018</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b13">
	<analytic>
		<title level="a" type="main">Conditional commitments: Reasoning and model checking</title>
		<author>
			<persName><forename type="first">W</forename><forename type="middle">El</forename><surname>Kholy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>El-Menshawy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Qu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Dssouli</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">ACM Transactions on Software Engineering and Methodology (TOSEM)</title>
		<imprint>
			<biblScope unit="volume">24</biblScope>
			<biblScope unit="issue">2</biblScope>
			<biblScope unit="page">9</biblScope>
			<date type="published" when="2014">2014</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b14">
	<analytic>
		<title level="a" type="main">Symbolic model checking commitment protocols using reduction</title>
		<author>
			<persName><forename type="first">M</forename><surname>El-Menshawy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Dssouli</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">International Workshop on Declarative Agent Languages and Technologies</title>
				<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="2010">2010</date>
			<biblScope unit="page" from="185" to="203" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b15">
	<analytic>
		<title level="a" type="main">Model checking real-time conditional commitment logic using transformation</title>
		<author>
			<persName><forename type="first">M</forename><forename type="middle">El</forename><surname>Menshawy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">W</forename><forename type="middle">El</forename><surname>Kholy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">A</forename><surname>Laarej</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Journal of Systems and Software</title>
		<imprint>
			<date type="published" when="2018">2018</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b16">
	<analytic>
		<title level="a" type="main">Computational logics and verification techniques of multi-agent commitments: survey</title>
		<author>
			<persName><forename type="first">M</forename><surname>El-Menshawy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">W</forename><forename type="middle">El</forename><surname>Kholy</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Yolum</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Dssouli</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">The Knowledge Engineering Review</title>
		<imprint>
			<biblScope unit="volume">30</biblScope>
			<biblScope unit="issue">5</biblScope>
			<biblScope unit="page" from="564" to="606" />
			<date type="published" when="2015">2015</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b17">
	<analytic>
		<title level="a" type="main">A new framework for the verification of service trust behaviors</title>
		<author>
			<persName><forename type="first">J</forename><surname>El-Qurna</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Yahyaoui</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Almulla</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Knowledge-Based Systems</title>
				<imprint>
			<date type="published" when="2017">2017</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b18">
	<analytic>
		<title level="a" type="main">Temporal and modal logic</title>
		<author>
			<persName><forename type="first">E</forename><forename type="middle">A</forename><surname>Emerson</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Handbook of Theoretical Computer Science, Volume B: Formal Models and Sematics</title>
				<imprint>
			<publisher>MIT Press</publisher>
			<date type="published" when="1990">1990</date>
			<biblScope unit="page" from="995" to="1072" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b19">
	<monogr>
		<title level="m" type="main">Reasoning about Knowledge</title>
		<author>
			<persName><forename type="first">R</forename><surname>Fagin</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><forename type="middle">Y</forename><surname>Halpern</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Y</forename><surname>Moses</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><forename type="middle">Y</forename><surname>Vardi</surname></persName>
		</author>
		<imprint>
			<date type="published" when="1995">1995</date>
			<publisher>MIT Press</publisher>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b20">
	<analytic>
		<title level="a" type="main">A formal notion of trust-enabling reasoning about security properties</title>
		<author>
			<persName><forename type="first">A</forename><surname>Fuchs</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Gürgens</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Rudolph</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">IFIP International Conference on Trust Management</title>
				<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="2010">2010</date>
			<biblScope unit="page" from="200" to="215" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b21">
	<monogr>
		<title level="m" type="main">Dynamic Logic</title>
		<author>
			<persName><forename type="first">D</forename><surname>Harel</surname></persName>
		</author>
		<author>
			<persName><forename type="first">D</forename><surname>Kozen</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Tiuryn</surname></persName>
		</author>
		<imprint>
			<date type="published" when="2000">2000</date>
			<publisher>MIT press</publisher>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b22">
	<analytic>
		<title level="a" type="main">A logic of trust and reputation</title>
		<author>
			<persName><forename type="first">A</forename><surname>Herzig</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Lorini</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><forename type="middle">F</forename><surname>Hübner</surname></persName>
		</author>
		<author>
			<persName><forename type="first">L</forename><surname>Vercouter</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Logic Journal of IGPL</title>
		<imprint>
			<biblScope unit="volume">18</biblScope>
			<biblScope unit="issue">1</biblScope>
			<biblScope unit="page" from="214" to="244" />
			<date type="published" when="2010">2010</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b23">
	<analytic>
		<title level="a" type="main">A simple logic of trust based on propositional assignments</title>
		<author>
			<persName><forename type="first">A</forename><surname>Herzig</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Lorini</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Moisan</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">The Goals of Cognition. Essays in Honour of Cristiano Castelfranchi, Tributes</title>
				<editor>
			<persName><forename type="first">F</forename><surname>Paglieri</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">L</forename><surname>Tummolini</surname></persName>
		</editor>
		<editor>
			<persName><forename type="first">R</forename><surname>Falcone</surname></persName>
		</editor>
		<imprint>
			<publisher>College Publications</publisher>
			<date type="published" when="2012">2012</date>
			<biblScope unit="page" from="407" to="419" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b24">
	<analytic>
		<title level="a" type="main">Kont: Computing tradeoffs in normative multiagent systems</title>
		<author>
			<persName><forename type="first">Ö</forename><surname>Kafalı</surname></persName>
		</author>
		<author>
			<persName><forename type="first">N</forename><surname>Ajmeri</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><forename type="middle">P</forename><surname>Singh</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 31st Conference on Artificial Intelligence (AAAI)</title>
				<meeting>the 31st Conference on Artificial Intelligence (AAAI)</meeting>
		<imprint>
			<date type="published" when="2017">2017</date>
			<biblScope unit="page" from="3006" to="3012" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b25">
	<analytic>
		<title level="a" type="main">Detecting exceptions in commitment protocols: Discovering hidden states</title>
		<author>
			<persName><forename type="first">Ö</forename><surname>Kafalı</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Yolum</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">International Workshop on Languages, Methodologies and Development Tools for Multi-Agent Systems</title>
				<imprint>
			<date type="published" when="2009">2009</date>
			<biblScope unit="volume">6039</biblScope>
			<biblScope unit="page" from="112" to="127" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b26">
	<analytic>
		<title level="a" type="main">Belief, information acquisition, and trust in multi-agent systems-a modal logic formulation</title>
		<author>
			<persName><forename type="first">C</forename><surname>Liau</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Artif. Intell</title>
		<imprint>
			<biblScope unit="volume">149</biblScope>
			<biblScope unit="issue">1</biblScope>
			<biblScope unit="page" from="31" to="60" />
			<date type="published" when="2003">2003</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b27">
	<analytic>
		<title level="a" type="main">Reasoning about belief, evidence and trust in a multi-agent setting</title>
		<author>
			<persName><forename type="first">F</forename><surname>Liu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Lorini</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">International Conference on Principles and Practice of Multi-Agent Systems</title>
				<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="2017">2017</date>
			<biblScope unit="page" from="71" to="89" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b28">
	<analytic>
		<title level="a" type="main">Automatic verification of knowledge and time with NuSMV</title>
		<author>
			<persName><forename type="first">A</forename><surname>Lomuscio</surname></persName>
		</author>
		<author>
			<persName><forename type="first">C</forename><surname>Pecheur</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Raimondi</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the 20th International Joint Conference on Artificial Intelligence</title>
				<meeting>the 20th International Joint Conference on Artificial Intelligence</meeting>
		<imprint>
			<date type="published" when="2007">2007</date>
			<biblScope unit="page" from="1384" to="1389" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b29">
	<analytic>
		<title level="a" type="main">MCMAS: an open-source model checker for the verification of multi-agent systems</title>
		<author>
			<persName><forename type="first">A</forename><surname>Lomuscio</surname></persName>
		</author>
		<author>
			<persName><forename type="first">H</forename><surname>Qu</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Raimondi</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">STTT</title>
		<imprint>
			<biblScope unit="volume">19</biblScope>
			<biblScope unit="issue">1</biblScope>
			<biblScope unit="page" from="9" to="30" />
			<date type="published" when="2017">2017</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b30">
	<analytic>
		<title level="a" type="main">Trust and norms in the context of computer security: A logical formalization</title>
		<author>
			<persName><forename type="first">E</forename><surname>Lorini</surname></persName>
		</author>
		<author>
			<persName><forename type="first">R</forename><surname>Demolombe</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">International Conference on Deontic Logic in Computer Science</title>
				<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="2008">2008</date>
			<biblScope unit="page" from="50" to="64" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b31">
	<analytic>
		<title level="a" type="main">Universal grammar</title>
		<author>
			<persName><forename type="first">R</forename><surname>Montague</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Theoria</title>
		<imprint>
			<biblScope unit="volume">36</biblScope>
			<biblScope unit="issue">3</biblScope>
			<biblScope unit="page" from="373" to="398" />
			<date type="published" when="1970">1970</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b32">
	<monogr>
		<title level="m" type="main">Neighborhood Semantics for Modal Logic</title>
		<author>
			<persName><forename type="first">E</forename><surname>Pacuit</surname></persName>
		</author>
		<imprint>
			<date type="published" when="2017">2017</date>
			<publisher>Springer</publisher>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b33">
	<analytic>
		<title level="a" type="main">Argument schemes for reasoning about trust</title>
		<author>
			<persName><forename type="first">S</forename><surname>Parsons</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Atkinson</surname></persName>
		</author>
		<author>
			<persName><forename type="first">Z</forename><surname>Li</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Mcburney</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Sklar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><surname>Singh</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Haigh</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Levitt</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Rowe</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Argument &amp; Computation</title>
		<imprint>
			<biblScope unit="volume">5</biblScope>
			<biblScope unit="issue">2-3</biblScope>
			<biblScope unit="page" from="160" to="190" />
			<date type="published" when="2014">2014</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b34">
	<analytic>
		<title level="a" type="main">The temporal logic of programs</title>
		<author>
			<persName><forename type="first">A</forename><surname>Pnueli</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">18th Annual Symposium on Foundations of Computer Science</title>
				<imprint>
			<date type="published" when="1977">1977</date>
			<biblScope unit="page" from="46" to="57" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b35">
	<analytic>
		<title level="a" type="main">Semantical considerations on dialectical and practical commitments</title>
		<author>
			<persName><forename type="first">M</forename><forename type="middle">P</forename><surname>Singh</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence</title>
				<meeting>the Twenty-Third AAAI Conference on Artificial Intelligence<address><addrLine>Chicago, Illinois, USA</addrLine></address></meeting>
		<imprint>
			<publisher>AAAI</publisher>
			<date type="published" when="2008">2008</date>
			<biblScope unit="page" from="176" to="181" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b36">
	<analytic>
		<title level="a" type="main">Trust as dependence: a logical approach</title>
		<author>
			<persName><forename type="first">M</forename><forename type="middle">P</forename><surname>Singh</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">The 10th International Conference on Autonomous Agents and Multiagent Systems</title>
				<imprint>
			<date type="published" when="2011">2011</date>
			<biblScope unit="page" from="863" to="870" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b37">
	<analytic>
		<title level="a" type="main">Modeling and verifying probabilistic multi-agent systems using knowledge and social commitments</title>
		<author>
			<persName><forename type="first">K</forename><surname>Sultan</surname></persName>
		</author>
		<author>
			<persName><forename type="first">J</forename><surname>Bentahar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">W</forename><surname>Wan</surname></persName>
		</author>
		<author>
			<persName><forename type="first">F</forename><surname>Al-Saqqar</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Expert Systems with Applications</title>
		<imprint>
			<biblScope unit="volume">41</biblScope>
			<biblScope unit="issue">14</biblScope>
			<biblScope unit="page" from="6291" to="6304" />
			<date type="published" when="2014">2014</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b38">
	<analytic>
		<title level="a" type="main">Using argumentation to reason about trust and belief</title>
		<author>
			<persName><forename type="first">Y</forename><surname>Tang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Cai</surname></persName>
		</author>
		<author>
			<persName><forename type="first">P</forename><surname>Mcburney</surname></persName>
		</author>
		<author>
			<persName><forename type="first">E</forename><surname>Sklar</surname></persName>
		</author>
		<author>
			<persName><forename type="first">S</forename><surname>Parsons</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="j">Journal of Logic and Computation</title>
		<imprint>
			<biblScope unit="volume">22</biblScope>
			<biblScope unit="issue">5</biblScope>
			<biblScope unit="page" from="979" to="1018" />
			<date type="published" when="2012">2012</date>
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b39">
	<analytic>
		<title level="a" type="main">Enhancing tropos with commitments</title>
		<author>
			<persName><forename type="first">P</forename><forename type="middle">R</forename><surname>Telang</surname></persName>
		</author>
		<author>
			<persName><forename type="first">M</forename><forename type="middle">P</forename><surname>Singh</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">Conceptual Modeling: Foundations and Applications</title>
				<imprint>
			<publisher>Springer</publisher>
			<date type="published" when="2009">2009</date>
			<biblScope unit="page" from="417" to="435" />
		</imprint>
	</monogr>
</biblStruct>

<biblStruct xml:id="b40">
	<analytic>
		<title level="a" type="main">Symbolic model checking the knowledge of the dining cryptographers</title>
		<author>
			<persName><forename type="first">R</forename><surname>Van Der Meyden</surname></persName>
		</author>
		<author>
			<persName><forename type="first">K</forename><surname>Su</surname></persName>
		</author>
	</analytic>
	<monogr>
		<title level="m">17th IEEE Computer Security Foundations Workshop</title>
				<imprint>
			<date type="published" when="2004">2004</date>
			<biblScope unit="page" from="280" to="291" />
		</imprint>
	</monogr>
</biblStruct>

				</listBibl>
			</div>
		</back>
	</text>
</TEI>
