Trust-empowered, IoT-driven Legitimate Data Offloading Zakaria Maamar Noura Faci Zayed University Claude Bernard Lyon 1 University Dubai, UAE Lyon, France Fadwa Yahya Ejub Kajan Prince Sattam Bin Abdulaziz University State University of Novi Pazar Al-Kharj, KSA Novi Pazar, Serbia University of Sfax Sfax, Tunisia ABSTRACT requirements to satisfy in terms of minimizing data latency1 and In an IoT environment deployed on top of fog and/or cloud nodes, protecting sensitive data. Transferring data over public networks offloading data between nodes is a common practice that aims at to (distant) clouds could take time because of high latency, could lessening the burden on these nodes and hence, meeting some be subject to interceptions, alterations, and misuses, and could real-time processing requirements. Existing initiatives put em- depend on network availability and reliability. To address data- phasis on “when to offload” and “where to offload” using criteria latency and data-sensitivity concerns, ICT practitioners advocate like resource constraint, load balancing, and data safety during for fog computing where processing and/or storage resources transfer. However, there is limited emphasis on the trustworthi- are made available “next” (or close) to where data is collected. ness of those nodes that will accept the offloaded data putting According to Khebbeb et al. [10], cloud means more resources, these data at risk of misuse. To address this limited emphasis, more reliability, and more latency, and fog means less resources, this paper advocates for trust as a decision criterion for iden- less reliability, and less latency. tifying the appropriate nodes for hosting the offloaded data. A Although cloud and fog are expected to work hand-in-hand [12], trust model is designed and then, developed considering factors there are situations where cloud and/or fog could seek the sup- like legitimacy, quality-of-service, and quality-of-experience. A port of their respective peers with handling some complex cross- system demonstrating the technical doability of the trust model border business applications, for example. Known as offload- is presented in the paper, as well. ing [13, 14], this support would lead to forming relations between Clouds (C) [6], between F ogs (F ) [2], and between F ogs and Clouds. We refer to these 3 cases as C2C offloading flow, F 2F KEYWORDS offloading flow, and F 2C offloading flow. However, to ensure a Data Flow, Internet-of-Things, Legitimacy, Offloading, Trust. successful offloading we advocate for some criteria with focus, in this paper, on trust and eligibility that would help identify the right peers according to past experiences, current loads, possible incentives, to cite just some. Questions that we raise, but not 1 INTRODUCTION limited to, include how can clouds/fogs discover potential peers, On top of many forms of computing like enterprise comput- what risks do clouds/fogs take when offloading demands to peers, ing, social computing, and ubiquitous computing in the field of what demands would be eligible for offloading, and what criteria Information and Communication Technologies (ICT), cloud com- would define clouds/fogs’ trust and eligibility levels? puting and fog computing are among the latest ICT that organiza- We adopt the Internet-of-Things (IoT) to illustrate trust-empowered tions are embracing to tackle the challenges of the 21𝑠𝑡 century. legitimate offloading in an environment of multiple clouds and On the one hand, cloud computing promotes Anything-as-a- fogs. Millions of things (e.g., from tiny ones like chips to ad- Service (*aaS) as an operation-model for organizations that wish vanced ones like embedded systems) are spread over the cyber- to concentrate on their core functionalities/competencies with- physical environment collecting, processing, and distributing out being concerned with the availability of resources like soft- data of different types ranging from humidity level in a ware- ware, platform, and infrastructure that would help them achieve house to number of vehicles on a highway. According to Gartner2 , these functionalities/competencies [16]. On the other hand, fog 6.4 billion connected things were in use in 2016, up 3% from 2015, computing promotes the deployment of processing and/or stor- and will reach 20.8 billion by 2020. It is worth noting the strong age resources at the edge of communication networks allowing coupling of IoT with cloud and fog computing as reported in the to minimize data transfer and hence, exposure to interception literature [5, 23]. Many IoT applications adopt cloud and fog as risks [14]. operation models to secure the necessary resources for process- A good amount of research discusses the synergy between ing, storing, and communicating the massive volume of data that cloud and fog [5, 19]. Indeed, despite cloud’s pay-per-use and things generate. scalability benefits, cloud seems inappropriate for certain applica- The rest of this paper is organized as follows. Section 2 briefly tions (e.g., medical and financial) that have strict non-functional discusses cloud, fog, IoT, and trust. Section 3 presents the offload- ing model in terms of core concepts, types of data flows, and types of offloading flows. The way trust guides the definition of © 2021 Copyright for this paper by its author(s). Published in the Workshop Proceed- ings of the EDBT/ICDT 2021 Joint Conference (March 23–26, 2021, Nicosia, Cyprus) 1 Puliafito et al. report that “the average round trip time between an Amazon Cloud on CEUR-WS.org. Use permitted under Creative Commons License Attribution 4.0 server in Virginia (U.S.A.) and a device in the U.S. Pacific Coast is 66ms; it is equal to International (CC BY 4.0) 125ms if the end device is in Italy; and reaches 302ms when the device is in Beijing” [17]. 2 www.gartner.com/newsroom/id/3165317. offloading flows is presented in the same section. Section 4 dis- be weak in term of transparency since a cloud service’s provider cusses the technical details about the offloading model. Section 5 may fine tune/beef up some measurements. Regarding TaaS, trust concludes the paper and presents some future work. management is delegated to a third party. However, some policy and security mechanisms should be considered, like accreditation and Public Key Infrastructure (PKI), and should also include a 2 BACKGROUND cloud auditor in charge of certifying both the cloud provider and Cloud is a popular ICT topic that promotes Anything-as-a-Service the third party. In [9], Khan and Malluhi promote first, cloud operation model, adopts pay-per-use pricing, and consolidates prevention to ensure data privacy and access control and second, hardware and software resources into datacenters. Cloud comput- digital signature to ensure data integrity. The authors raise the ing is sometimes known as the 5𝑡ℎ utility after water, electricity, question of “how can cloud providers earn their customers’ trust gas, and telephony. However, despite cloud popularity, it does when a third party is processing sensitive data in a remote machine not, unfortunately, suit all applications. It is not recommended located in various countries?”. In [11], Li presents FASTCloud for latency-critical and data-sensitive applications due to reasons that is a framework to assess and select trustworthy cloud ser- such as high latency added by network connections to datacen- vice based on QoS. This framework’s 4 main components are ters and multi-hops/nodes between end-users and datacenters cloud service providers, cloud service customers, potential cloud that increase the probability of interceptions. consumers, and a trustworthy cloud service selection. The last Fog was generalized by Cisco Systems in 2014 [4] as a new one evaluates trust level of cloud services based on the collected ICT-based operation model. The main idea is to make process- QoS attributes information by employing the trust assessment ing, storage, and networking resources “close” to data. Real-time model, and returning the trust assessment results to potential applications that require almost immediate action and high data cloud consumers. protection, would discard cloud in favor of fog. Varghese et al. mention that by 2020, existing electronic devices will generate 3 OFFLOADING MODEL 43 trillion gigabytes of data that need to be processed in cloud dat- This section discusses data flows in conjunction with offload- acenters [22]. However, this way of operating cannot be sustained ing flows and then, presents the impact of trust on offloading for a long time due to frequency and latency of communication decisions. between these devices and cloud datacenters. Fog would process data closer to its source so, that, network traffic is reduced and both Quality-of-Service (QoS) and Quality-of-Experience (QoE) 3.1 Overview are improved. According to Mahmud et al., offloading in the context of fog-based The abundant literature about IoT does not help propose a applications could be bottom-up, top-down, and hybrid [14]. Us- unique definition for IoT. We present 3 references on IoT that ing Fig. 1, we illustrate the cases of offloading flows (of), C2Cof , are [3], [1], and [18]. First, Barnaghi and Sheth provide a good F 2Fof , and F 2Cof , that could arise in conjunction with data overview of IoT requirements and challenges [3]. On the one flows (df) that we specialize into T 2Cdf , T 2Fdf , and F 2Cdf hand, requirements include quality, latency, trust, availability, where T refers to T hing. df is meant for capturing data that reliability, and continuity that should impact efficient access and things collect/sense and conveying these data to recipients namely, use of IoT data and services. On the other hand, challenges re- cloud and fog, depending on the under-development IoT appli- sult from today’s IoT ecosystems that feature billions of dynamic cations’ functional and non-functional requirements [13]. Con- things that make existing search, discovery, and access techniques veyed data could be either raw or processed depending again on and solutions inappropriate for IoT data and services. Second, the recipients’ needs and these requirements, as well. Abdmeziem et al. discuss IoT characteristics and enabling tech- nologies [1]. Characteristics include distribution, interoperability, C2C F2F scalability, resource scarcity, and security. And, enabling tech- offloading offloading nologies include sensing, communication, and actuating. These flow c f flow Raw/Processed-data flow technologies are mapped onto a three-layer IoT architecture that Cloud nodes c f Fog nodes are referred to as perception, network, and application, respec- c F2C offloading flow f tively. Finally, Qin et al. [18] define IoT from a data perspective as “In the context of the Internet, addressable and interconnected things, instead of humans, act as the main data producers, as well as the main data consumers. Computers will be able to learn and Raw-data flow Community of things Raw-data flow gain information and knowledge to solve real world problems di- Legend rectly with the data fed from things. As an ultimate goal, computers enabled by the Internet of Things technologies will be able to sense Storage resources Processing resources Networking resources and react to the real world for humans”. Trust may be seen as “an act of faith; confidence and reliance in something that’s expected to behave or deliver as promised” [9]. Figure 1: Offloading flows in conjunction with data flows On a regular basis, many ICT researchers and practitioners raise the question of should we trust cloud services or not. In [8], Huang and Nicol examine reputation- and Service-Level Agree- ment (SLA)-based trust, and Trust-as-a-Service (TaaS). While 3.2 Types of data flows reputation-based trust relies on a user’s (or community of users) To define data flows, we rely on our previous work on cloud- experience, SLA-based trust relies on QoS measurement and fog coordination [12, 24]. As stated above, data flows connect SLA verification, in our case between clouds. The former may thing and fog together (T 2Fdf ), thing and cloud together (T 2Cdf ), and fog and cloud together (F 2Cdf )3 . Although the 3 specialized reachable clouds in the same domain if this cloud is con- data flows could simultaneously exist, we came up in our pre- gested, which could delay handling these demands. While vious work with 6 criteria whose use would permit to Highly- the offloading could be based on peers’ current loads (and Recommend (HR), Recommend (R), and Not-Recommend (NR) other performance criteria like storage capacity), we ex- which data flow should exist for an under-development IoT ap- amine in Section 3.4 the value-added of trust in selecting plication. These criteria are frequency (rate of data transfer from these peers. things to fogs/clouds; the frequency could be regular, e.g., every (2) F 2Fof establishes collaboration between fogs according 2 hours, or continuous), sensitivity (nature of data exchanged be- to their ongoing loads and processing and storage re- tween things and fogs/clouds; highly-sensitive data should not be sources [2]. In compliance with Fig. 1, things periodically exposed longer on networks during the exchange), freshness (how collect and generate (raw) data from the cyber-physical important data exchanged between things and fogs/clouds should surroundings and send these data to fogs (T 2Fdf ) and/or be up-to-date, i.e., recent), time (delay that results from with- clouds (T 2Cdf ) for processing/storage, as deemed neces- holding/processing data at the thing level until its transfer to sary (Section 3.2). A fog can serve a certain number of fogs/clouds), volume (amount of data that things produce and data-based demands instantly or offload some to other send to fogs/clouds), and criticality (demands that fogs/clouds reachable fogs in the same domain if this fog is congested, express with regard to data of things; low demands could lead which could delay handling these demands. While the of- to ignoring certain data). Assumption made in support of the floading could be based on peers’ current loads (and other 6 criteria is that, distance-wise, clouds are far from things and performance criteria like storage capacity), we examine fogs are close to things. in Section 3.4 the value-added of trust in selecting these In Table 1, we summarize how the afore-mentioned criteria, peers. taken independently from each other, assist with recommending (3) F 2Cof establishes collaboration between fogs and clouds the establishment of specific data flows. More details about these when these fogs’ offloading demands cannot be accommo- recommendations are presented in [12]. dated by other fogs in the context of F 2Fof . Performance Contrarily to what we did in [12] where frequency, sensitivity, and/or trust criteria could back the decision of discard- freshness, time, volume, and criticality are taken independently ing these fogs. In compliance with Fig. 1, fogs could send from each other, we combined them all using a fuzzy logic-based (either raw or processed) data to clouds (F 2Cdf ) for (ex- multi-criteria decision making approach [24]. This approach was tra) processing/storage, as deemed necessary. While the demonstrated using a healthcare-driven IoT application along offloading could be based on clouds’ current loads (and with an in-house testbed that featured real sensors (tempera- other performance criteria like storage capacity), we ex- ture and humidity DHT11) and fog (rPi2) and cloud (Ubidots) amine in Section 3.4 the value-added of trust in selecting platforms. During the experiments, we modified the frequency these clouds. of streaming data (every 3 second, 5 second, 7 second, and ran- domly) for each of the 3 data flows, T 2C, T 2F , and F 2C, and 3.4 Trust-empowered legitimate offloading the volume (around low and high amount) and criticality (around Table 3 contains the list of parameters used to calculate T rust low and high important) of the transmitted data. Upon data re- Scores (T S) of fogs and clouds. Among these parameters, we ceipt at an end-point whether fog or cloud, we timestamped cite list of acquaintances, QoS, and QoE. data messages prior to storing them. Table 3 summarizes the In [7], Fiedler et al. suggest that different factors could influ- experiments with focus on the recommendations of establishing ence 𝑄𝑜𝐸. In our work, we adopt one influence factor that is 𝑄𝑜𝑆 specific data flows. More details about these recommendations allowing to compute 𝑄𝑜𝐸 as per Equation 1. are presented in [24].   𝑔𝑜𝑜𝑑 if |𝑄𝑜𝑆 𝐴 − 𝑄𝑜𝑆 𝑀 | ∈ [𝜎 + 𝛿, 1]   3.3 Types of offloading flows 𝑄𝑜𝐸 = 𝑓 𝑎𝑖𝑟 if |𝑄𝑜𝑆 𝐴 − 𝑄𝑜𝑆 𝑀 | ∈ [𝜎 − 𝛿, 𝜎 + 𝛿 [ (1)   𝑏𝑎𝑑 if |𝑄𝑜𝑆 𝐴 − 𝑄𝑜𝑆 𝑀 | ∈ [0, 𝜎 − 𝛿 [ C2Cof , F 2Fof , and F 2Cof identify possible interactions between  clouds, between fogs, and between fogs and clouds. As stated where (𝜎 ± 𝛿) would define a threshold. earlier, the objective of offloading is to secure the support of all T S calculation begins with identifying potential connections parties that could either be “idle” or have a “light” load allowing between data flows, between offloading flows, and between data them to accommodate additional demands from peers. It is worth flows and offloading flows. The objective of this identification is noting that offloading flows are to a certain extent in-line with to determine who initiates what; a flow’s recipient could initiate data flows shedding light on the loads that clouds and fogs could another flow and so on. In Fig. 2, dashed lines correspond to these handle depending on the volume of data that things would submit connections that we label as “could be the same”. In the context for processing and/or storage. of trust-empowered legitimate offloading, 4 out of 5 “could be (1) C2Cof establishes collaboration between clouds accord- the same” connections constitute our focus as per the following ing to their ongoing loads and processing and storage cases (only 2 are detailed): 𝑗 resources. In compliance with Fig. 1, things periodically Case 1. T 𝑖 2Fdf −→ F 𝑗 2Fof𝑘 |Cof 𝑘 : following the formation of collect and generate (raw) data from the cyber-physical a data flow from T 𝑖 to F 𝑗 , F 𝑗 assesses its current pro- surroundings and send these data to clouds (T 2Cdf ) and/or cessing and storage resources and, then, decides to form fogs (T 2Fdf ) for processing/storage, as deemed neces- an offloading load to convey the received (raw) data to sary (Section 3.2). A cloud can serve a certain number F 𝑘≠𝑗 , C𝑘 , or both F 𝑘≠𝑗 and C𝑘 . Detecting F 𝑘≠𝑗 and C𝑘 of data-based demands instantly or offload some to other is made possible thanks to acq(F 𝑗 ). When calculating the trust scores of F 𝑘 and/or C𝑘 and since F 𝑗 would finan- 3 Pre-processing data at fogs prior to sending the pre-processed data to clouds. cially compensate F 𝑘 and/or C𝑘 , we adopt a conservative Table 1: Recommendations about establishing data flows when criteria are separated ([12]) Criterion Features T 2 Cdf T 2 Fdf F 2 Cdf Frequency Continuous stream NR HR R Regular stream Short gaps NR HR HR Long gaps R R R Sensitivity High NR HR HR Low R R R Freshness Highly important NR HR R Lowly important R R R Time Real-time NR HR HR Near real-time R HR HR Batch-processing HR NR NR Volume High amount HR NR NR Low amount NR HR R Criticality Highly important HR HR R Lowly important NR HR HR Table 2: Recommendations about establishing data flows when criteria are combined ([24]) Scenario # Criteria Linguistic values Recommendations Scenario 1 Frequency Regular stream (around short and long gaps)★ T 2 Cdf is NR; T 2 Fdf is R; F 2 Cdf is R Sensitivity Around low and high★ Freshness Highly important Time Real time Volume High amount Criticality Lowly important Scenario 2 Frequency Regular stream long gaps T 2 Cdf is NR; T 2 Fdf is HR; F 2 Cdf is R Sensitivity High Freshness Highly important Time Real time Volume Low amount Criticality Lowly important Scenario 3 Frequency Regular stream long gaps T 2 Cdf is R; T 2 Fdf is R; F 2 Cdf is R Sensitivity Low Freshness Lowly important Time Near-real time Volume Around low and high amount★ Criticality Highly important Scenario 4 Frequency Regular stream long gaps T 2 Cdf is R; T 2 Fdf is R; F 2 Cdf is R Sensitivity Low Freshness Lowly important Time Near-real time Volume High amount Criticality Around lowly and highly important★ ★ : Around Val and Val : both Val and Val meet the scenario’s requirements. 1 2 1 2 Table 3: List of parameters Parameter Description C Set of all clouds in the ecosystem. F Set of all fogs in the ecosystem. T Set of all things in the ecosystem. 𝑎𝑐𝑞 (𝑒𝑛𝑡𝑖𝑡 𝑦𝑖 ) Acquaintance function that returns entities (things, fogs, and/or clouds) within range of entity𝑖 . 𝑄𝑜𝑆 𝐴𝑖 𝑗 Announced Quality of Service that C 𝑗 is expected to maintain when accepting C𝑖 ’s offloading demands. An- C 2C nounced 𝑄𝑜𝑆 is compared to Measured 𝑄𝑜𝑆 (𝑄𝑜𝑆 𝑀𝑖 𝑗 ) to detect any gap (Equation 1). C 2C 𝑄𝑜𝑆 𝐴𝑖 𝑗 Announced Quality of Service that F 𝑗 is expected to maintain when accepting F𝑖 ’s offloading demands. An- F 2F nounced 𝑄𝑜𝑆 is compared to Measured 𝑄𝑜𝑆 (𝑄𝑜𝑆 𝑀𝑖 𝑗 ) to detect any gap (Equation 1). F 2F 𝑄𝑜𝑆 𝐴𝑖 Announced Quality of Service that C 𝑗 is expected to maintain when accepting F𝑖 ’s offloading demands. An- F 2C 𝑗 nounced 𝑄𝑜𝑆 is compared to Measured 𝑄𝑜𝑆 (𝑄𝑜𝑆 𝑀𝑖 𝑗 ) to detect any gap (Equation 1). F 2C 𝑄𝑜𝐸 C𝑖 2C 𝑗 Quality of Experience that C𝑖 uses to capture its satisfaction in C 𝑗 completing its offloading demands. 𝑄𝑜𝐸 is dependent on whether C 𝑗 ’s 𝑄𝑜𝑆 was maintained or not at run-time4 . 𝑄𝑜𝐸 F𝑖 2F 𝑗 Quality of Experience that F𝑖 uses to capture its satisfaction in F 𝑗 completing its offloading demands. 𝑄𝑜𝐸 is dependent on whether F 𝑗 ’s 𝑄𝑜𝑆 was maintained or not at run-time. 𝑄𝑜𝐸 F𝑖 2C 𝑗 Quality of Experience that F𝑖 uses to capture its satisfaction in C 𝑗 completing its offloading demands. 𝑄𝑜𝐸 is dependent on whether C 𝑗 ’s 𝑄𝑜𝑆 was maintained or not at run-time. 𝑛 C𝑖 2C 𝑗 Number of times that C 𝑗 accepted/completed the offloading demands of C𝑖 . 𝑛 F𝑖 2F 𝑗 Number of times that F 𝑗 accepted/completed the offloading demands of F𝑖 . 𝑛 F𝑖 2C 𝑗 Number of times that C 𝑗 accepted/completed the offloading demands of F𝑖 . approach that consists of making F 𝑗 check the authentic- According to Pakulski, “legitimacy therefore relies not on 𝑗 trust, but on an impersonal sense of duty on the part of the ity of the data’s sender, namely T 𝑖 , using T 𝑖 2Fdf . To this end, we relate authenticity to a flow’s initiator whether followers to follow commands of a proper authority, who- this initiator would be a thing, a fog, or a cloud and de- ever is in authority, and whatever is the content of these fine the concept of legitimate initiator [25]. Here, T 𝑖 is commands” [15]. the initiator and its legitimacy becomes a concern for F 𝑗 . Sources of data Sources of offloading flows to assign flows to assign could be Thing Fog the same Fog Cloud (primary) (secondary) the same the same could be could be Cloud Fog Cloud Fog Cloud Cloud could be the same could be the same Figure 2: Potential connections between the different flows 𝑗 For illustration, let us assume that T 𝑖 is illegitimate by Case 2. F 𝑖 2Cof −→ C 𝑗 2Cof 𝑘 : following the formation of an flooding F 𝑗 with “fake” data, which triggers the formation offloading flow (that most probably would be connected of an offloading flow from F 𝑗 to F 𝑘 . Processing these to a data flow from F 𝑖 to C 𝑗 , C 𝑗 assesses its current pro- data at the level of F 𝑘 produces irrelevant results for users cessing and storage resources and, then, decides to convey along with wasting F 𝑗 ’s financial resources and F 𝑘 ’s this offloading flow to C𝑘≠𝑗 . When C 𝑗 calculates the trust processing resources as well as “blaming” F 𝑘 for these score of C𝑘 , we deem necessary to include the trust score, results. At the end, F 𝑗 adjusts the trust score of F 𝑘 . To that F 𝑖 would have defined for C 𝑗 , in this calculation. The avoid this scenario, our approach to calculate trust scores rationale of this inclusion is that F 𝑖 would like to “know” considers the legitimacy of a flow’s initiator along with to whom its offloading flow would be assigned since its the quality of experience that results from handling this initial contact for the offloading is C 𝑗 and not C𝑘 . Com- flow. Due to legitimacy aspect, we refine 𝑄𝑜𝐸 F 𝑗 2F𝑘 into pared to case 1, F 𝑖 ’s eligibility is not a concern based on 𝑄𝑜𝐸 T 𝑗 𝑖 where T 𝑖 is the reason of making F 𝑗 interact the previous trust score calculation that involved F 𝑖 and F 2F𝑘 C 𝑗 . After refining 𝑄𝑜𝐸 C 𝑗 2C𝑘 into 𝑄𝑜𝐸 F 𝑗 𝑘 where F 𝑖 is 𝑖 with F . We compute a data flow-triggered T S df 𝑘 F 𝑗 ,F𝑘 C 2C using Equation 2. the reason of making C 𝑗 interact with C𝑘 , we compute an offloading flow-triggered T S of C 𝑗 ,C𝑘 using Equation 4. T 𝑖 T S df F 𝑗 ,F𝑘 = Agg({𝐿𝑒𝑔 T𝑖 ,F 𝑗 × 𝑄𝑜𝐸 F 𝑗 2F𝑘 }𝑖=1,𝑛 ) (2) × 𝑄𝑜𝐸 F 𝑗 𝑖 T S of = Agg({T S of } ) (4) where C 𝑗 ,C𝑘 F𝑖 ,C 𝑗 C 2C𝑘 𝑖=1,𝑛 - Agg refers to some common aggregate function like average and minimum. where, compared to Equation 2’s T S df , Equation 4’s T S of - 𝐿𝑒𝑔 T𝑖 ,F 𝑗 is T 𝑖 ’s legitimacy when establishing a data refers to trust-score calculation that is triggered because flow with F 𝑗 . of an offloading flow and not data flow and is defined as To define T 𝑖 ’s legitimacy, we develop behavioral patterns follows: like those presented in [20]. The objective is to demys- C 𝑝 −|X𝑖,𝑗 | tify the recurrent behavior of T 𝑖 based on its interac- T S of F𝑖 ,C 𝑗 = Agg({𝑄𝑜𝐸 F 𝑖 2C 𝑗 } C 𝑝 ∈X𝑖,𝑗 ) × 𝑒 (5) tions with fogs and/or clouds. We link behavioral patterns to the recommendations presented in Table 2 and spe- where cialize them into legitimate pattern (L𝑝 ), where a thing - X𝑖,𝑗 denotes the multiset of peers (including 𝐶 𝑘 ) to would implement HR and R outcomes that were obtained which 𝐶 𝑗 forwarded the offloading demands of 𝐹 𝑖 . - 𝑄𝑜𝐸 FC𝑝 𝑖 with respect to IoT applications’ features/linguistic val- 𝑖 2C 𝑗 indicates 𝐹 ’s offloading quality-of-experience ues (e.g., real-time and continuous streaming), and illegiti- with 𝐶 𝑗 given C𝑝 in X𝑖,𝑗 . mate pattern (I𝑝 ), where a thing would do the opposite by - |X𝑖,𝑗 | represents X𝑖,𝑗 ’s cardinality. implementing NR outcomes. We compute 𝐿𝑒𝑔 T𝑖 ,F 𝑗 using 𝑗 Case 3. T 𝑖 2Cdf −→ C 𝑗 2Cof 𝑘 following the formation of a Equation 3. data flow from T 𝑖 to C 𝑗 , C 𝑗 assesses its current process- 𝑗 ing and storage resources and then, decides to form an 𝐿𝑒𝑔 T𝑖 ,F 𝑗 = V (App T𝑖 ,F 𝑗 , T 𝑖 2Fdf ) (3) offloading load to convey the received (raw) data to C𝑘≠𝑗 . where 𝑗 Case 4. F 𝑖 2Cdf −→ C 𝑗 2Cof 𝑘 : following the formation of a - App T𝑖 ,F 𝑗 is the IoT application in which T 𝑖 and F 𝑗 data flow from F to C , C 𝑗 assesses its current process- 𝑖 𝑗 jointly participate. ing and storage resources and then, decides to form an 𝑗 - V (App T𝑖 ,F 𝑗 , T 𝑖 2Fdf ) refers to T 𝑖 ’s legitimacy Value offloading load to convey the received (processed) data to 𝑗 to send data to F during App T𝑖 ,,F 𝑗 ’s execution. C𝑘≠𝑗 , for extra-processing.    1 ∵ the data flow is highly recommended. 4 EXPERIMENTS 𝑖 𝑗   V (App T𝑖 ,F 𝑗 , T 2Fdf ) = 0 ∵ the data flow is recommended.  -1 ∵ the data flow is not recommended. This section discusses the testbed and experiments to validate    the offloading model and then, presents some results. 4.1 Testbed set-up and sending timestamps. Here, the thing decider coupled to 𝑇 𝑎𝑝𝑝 To check the technical doability of our offloading model, we de- selects the adequate recipient of data flows that could be a cloud ployed a testbed that simulates both data flows between things or a fog (Table 2). In addition, the thing decider computes and and clouds/fogs and offloading flows between fogs, between stores the legitimacy of things per data flow in a database. We clouds, and between fogs and clouds. Through the testbed, we associate the experiments with the following scenarios: aimed at selecting the trustworthy recipient of an offloading flow. • Scenario #1 simulates the offloading flows from a fog to The testbed is fully developed in Java SE 8 under Eclipse IDE fogs. These flows are triggered by data flows received from for Java Developers5 . The computer used during development things (case 1, Section 3.4). To this end, the fog offloading runs Windows 8 64 bits, 1.4 GHz quad-core processor CPU, and decider coupled to 𝐶𝐹𝑎𝑝𝑝 selects the adequate recipient 4GB RAM. The development focused on 3 decision-making com- of the offloading flows. Indeed, the decider computes the ponents discussed below: trust score (Equation 2) for all acquaintances of the current • Thing decider runs over thing nodes to identify where data fog node prior to selecting the one with the highest trust of these things should be sent. This decider refers to our score. cloud-fog coordination work (Table 2) and determines the • Scenario #2 is built upon scenario #1 where simulations legitimacy of things when selecting data recipients as per are about offloading flows triggered by data flows but Equation 3. For instance, if fog is highly-recommended expected to be offloaded from a fog to a cloud (case 1, and cloud is not-recommended as per Table 2, then the Section 3.4). Here, the fog offloading decider selects the thing decider will privilege fog nodes over cloud nodes. adequate cloud recipient of the offloading flows based on Should the thing decider comply with this recommenda- the calculated trust scores (Equation 2). tion, then it assigns 1 as a legitimacy value to the thing • Scenario #3 simulates offloading flows triggered by pre- sending data to one of the fog to select. vious offloading flows from a cloud node another cloud • Fog offloading decider runs over fog nodes to identify a node (case 2, Section 3.4). Here, the cloud offloading decider trustworthy recipient of data when offloading data be- selects the best recipient of the offloading flows based on comes necessary. Indeed, each fog in the testbed has its the calculated trust scores (Equation4). fog offloading decider that accesses details stored in an in- In conjunction with the experiments, we examined the vari- house developed database about past experiences between ations in selecting data recipients depending on who offloads various nodes (things, fogs, and clouds) of the testbed. Such data, either cloud or fog, so, that, we highlight the impact of details include legitimacy of thing nodes and fog/cloud trust-empowered legitimate offloading on the selection of data nodes’ announced and measured QoSs. It is worth men- recipient. Deployed on many cloud/fog nodes, 𝐶𝐹𝑎𝑝𝑝 could de- tioning that we used a QoS dataset6 to assign QoS val- cide to offload the received data to another node in the network. ues to fog and cloud nodes when data flows or offload- The cloud/fog offloading decider selects the recipient based on ing flows are required. Based on thing’s legitimacy and their trust scores. Fig. 3 to Fig. 5 illustrate the number of offload- cloud/fog QoSs, the fog offloading decider computes T S ing flows received by each node in the network with focus on for all acquaintances of a respective fog node. Finally, it the variation of this number over time. Indeed, the number of selects the node with the highest T S. offloading flows received by a node increases when the node • Cloud offloading decider runs on top of cloud nodes acting maintains its trust score and decreased due a degradation of its like fog offloading decider. Contrarily to fogs, clouds can trust score. The variation of trust scores is further detailed in offload data to their peers, only. Fig. 6 and Fig. 7 measuring the trust scores of some nodes during the experiments. In addition, these figures focus on the impact 4.2 Simulations and results of the variation in calculated trust scores on the selected node to To validate both the offloading model and the decision makers’ receive the offloading flows. As per Fig. 3, during scenario #1 the outcomes, we carried out different experiments. First, an in-house 3 fogs, 𝐹 1, 𝐹 2, and 𝐹 3, are selected as recipient of offloaded data. client-server Java application, 𝑇 𝑎𝑝𝑝, allowed to simulate things’ The experiments prove that the selection could vary according behaviors as clients sending data extracted from a T-drive Taxi to the trust score that is in turn calculated based on the QoE and Trajectories dataset [26] to clouds and fogs that act as servers. We legitimacy of things. For instance, 𝐹 2, which was privileged at the annotated this dataset with data-flow recommendations (Table 2) beginning of the experiments, is penalized after a degradation of so that, the thing decider selects the best data flow recipient. Sec- its trust score. Fig. 6 shows only the trust scores calculated by 𝐹 1. ond, we developed another client-server Java application, 𝐶𝐹𝑎𝑝𝑝, This figure shows that 𝐶1 having the highest trust score at the to simulate clouds’ and fogs’ behaviors when offloading data. A beginning of the experiments was selected by 𝐹 1 as offloading cloud/fog is simultaneously a client when it offloads data and a recipient. Then, and due a degradation in 𝐶1’s trust score, 𝐶2 hav- server when it receives data. Each cloud/fog’s 𝐶𝐹𝑎𝑝𝑝 receives ing the new highest trust score is selected by 𝐹 1 as offloading data sent by other nodes, stores them, and decides to offload recipient. them to other nodes. As per Table 4, both Java applications run on various types of computers that vary between personal lap- tops located in KSA and Tunisia and desktops available in the 5 CONCLUSION laboratories of Prince Sattam Bin Abdulaziz University in KSA. In an IoT environment consisting of multiple fogs and clouds, it To perform the experiments, we simulated data flows from happens that data that things send, whether separately or con- things to clouds/fogs. A data flow contains raw data in terms currently, to these fogs and/or clouds for processing and/or stor- of id, message extracted from T-drive Taxi Trajectories dataset, age needs end-up being transferred to other peers for the same 5 www.eclipse.org/downloads/packages. needs. Known as offloading, this paper stresses out the impor- 6 github.com/QXL4515/data-set. tance of ensuring the trustworthiness of data recipients to avoid Table 4: List of used computers Computer Connection Location Role in the testbed Laptop 1 WiFi KSA Fog Laptop 2 WiFi KSA Cloud Laptop 3 WiFi Tunisia Cloud Laptop 4 WiFi Tunisia Cloud Desktop x 6 Ethernet KSA 2 Fogs and 4 Things 15 3 F1 C1 12 F2 2 C2 F3 C3 |Offloading flows| 9 1 Time (s) 6 0 C2 is selected C1 is selected 3 −1 0 −2 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Time (s) TS Figure 3: Results of scenario #1 experiments Figure 6: Variation of T Ss measured by fog 𝐹 1 15 3 C1 12 C2 C2 C3 2 C3 |Offloading flows| 9 1 Time (S) 6 0 C2 is selected C3 is selected 3 −1 0 1 2 3 4 5 6 7 8 9 10 −2 1 2 3 4 5 6 7 8 9 10 Time (s) TS Figure 4: Results of scenario #2 experiments Figure 7: Variation of T Ss measured by cloud 𝐶1 15 C1 data misuse cases, for example. To address these cases, a trust 12 C2 model has been designed and developed taking into account dif- C3 ferent factors namely, types of interactions between things, fogs, |Offloading flows| and clouds, recommendations of where things should send their 9 data, legitimacy of data senders, quality-of-service of fogs/clouds, and quality-of-experience interacting with fog/clouds. The trust 6 model has been demonstrated through a set of experiments. In term of future work, we would like to complete the analysis 𝑗 𝑘 ) and case 4 (F 𝑖 2C 𝑗 −→ C 𝑗 2C𝑘 ) of case 3 (T 𝑖 2Cdf −→ C 𝑗 2Cof df of 3 as well as examine the impact of trust on developing a chain of offloading flows. By analogy with case 2, a chain illustrated with 0 𝑗 1 2 3 4 5 6 7 8 9 10 1[F 𝑖 2Fof ]+ −→ F 𝑗 2Cof 𝑘 −→ 1[C𝑘 2C𝑙 ]+ could be formed raising of questions about the trustworthiness and traceability of who is Time (s) offloading to whom. Figure 5: Results of scenario #3 experiments REFERENCES [1] M.R. Abdmeziem, D. Tandjaoui, and I. Romdhani. In Anis Koubaa and Elhadi Shakshuki, editors, Robots and Sensor Clouds, chapter Architecting the Internet of Things: State of the Art. Springer International Publishing, 2016. [2] M. Al-Khafajiy, T. Baker, H. Al-Libawy, Z. Maamar, M. Aloqaily, and Y. Jarar- weh. Improving Fog Computing Performance via Fog-2-Fog Collaboration. Future Generation Computer Systems, 100, 2019. [3] P.M. Barnaghi and A.P. Sheth. On Searching the Internet of Things: Require- ments and Challenges. IEEE Intelligent Systems, 31(6), 2016. [4] F. Bonomi, R. Milito, P. Natarajan, and J. Zhu. Fog Computing: A Platform for Internet of Things and Analytics. In Big Data and Internet of Things: A Roadmap for Smart Environments, Studies in Computational Intelligence. Cisco, Springer International Publishing, 2014. [5] M. De Donno, K. Tange, and N. Dragoni. Foundations and Evolution of Modern Computing Paradigms: Cloud, IoT, Edge, and Fog. IEEE Access, 7, 2019. [6] S. Elnaffar, Z. Maamar, and Q.Z. Sheng. When Clouds Start Socializing: The Sky Model. International Journal of E-Business Research, 9(2), 2013. [7] M. Fiedler, T. Hossfeld, and P. Tran-Gia. A Generic Quantitative Relationship between Quality of Experience and Quality of Service. IEEE Network, 24(2), 2010. [8] J. Huang and D.N. Nicol. Trust Mechanisms for Cloud Computing. Journal of Cloud Computing, 2:9, 2013. [9] K.M. Khan and Q. Malluhi. Establishing Trust in Cloud Computing. IT Professional, 12(5), 2010. [10] K. Khebbeb, N. Hameurlain, and F. Belalab. A Maude-based Rewriting Ap- proach to Model and Verify Cloud/Fog Self-Adaptation and Orchestration. Journal of Systems Architecture, 110, November 2020. [11] X. Li. FASTCloud: A Framework of Assessment and Selection for Trustworthy Cloud Service based on QoS, 2020. [12] Z. Maamar, T. Baker, N. Faci, E. Ugljanin, M. Al-Khafajiy, and V.A. Burégio. Towards a Seamless Coordination of Cloud and Fog: Illustration through the Internet-of-Things. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (SAC’2019), Limassol, Cyprus, 2019. [13] Z. Maamar, T. Baker, M. Sellami, M. Asim, E. Ugljanin, and N. Faci. Cloud vs edge: Who serves the Internet-of-Things better? Internet Technology Letters, 1(5), 2018. [14] R. Mahmud, K. Ramamohanarao, and R. Buyya. Application Management in Fog Computing Environments: A Taxonomy, Review and Future Directions. ACM Computing Surveys, 53(4), July 2020. [15] J. Pakulski. Trust and Legitimacy. Policy, Organisation and Society, 5(1), 1992. [16] M. Peter and G. Timothy. The NIST Definition of Cloud Computing. Techni- cal Report 800-145, National Institute of Standards and Technology (NIST), September 2011. [17] C. Puliafito, E. Mingozzi, F. Longo, A. Puliafito, and O. Rana. Fog Computing for the Internet of Things: A Survey. ACM Transactions on Internet Technology, 19(2), 2019. [18] Y. Qin, Q.Z. Sheng, N.J.G. Falkner, S. Dustdar, H. Wang, and A.V. Vasilakos. When Things Matter: A Data-Centric View of the Internet of Things. CoRR, abs/1407.2704, 2014. [19] M. Saidur Rahmana, I. Khalila, M. Atiquzzaman, and X. Yi. Towards Privacy Preserving AI-based Composition Framework in Edge Networks using Fully Homomorphic Encryption. Engineering Applications of Artificial Intelligence, 94, September 2020. [20] G. Suchacka and J. Iwanski. Identifying Legitimate Web Users and Bots with Different Traffic Profiles - an Information Bottleneck Approach. Knowledge Based Systems, 197, 2020. [21] M. Varela, L. Skorin-Kapov, and T. Ebrahimi. In S. Möller and A. Raake, editors, T-Labs Series in Telecommunication Services, chapter Quality of Service Versus Quality of Experience. Springer, Cham, 2014. [22] B. Varghese, N. Wang, D.S. Nikolopoulos, and R. Buyya. Feasibility of Fog Computing. arXiv preprint arXiv:1701.05451, 2017. [23] T. Wang, G. Zhang, M.Z. Alam Bhuiyan, A. Liu, W. Jia, and M. Xie. A Novel Trust Mechanism based on Fog Computing in Sensor-Cloud System. Future Generation Computing Systems, 109, 2020. [24] F. Yahya, Z. Maamar, and K. Boukadi. A Multi-Criteria Decision Making Approach for Cloud-Fog Coordination. In Proceedings of the 34th International Conference on Advanced Information Networking and Applications (AINA’2020), Caserta, Italy, 2020. [25] J.P. Yoon and Z. Chen. Service Trustiness and Resource Legitimacy in Cloud Computing. In Proceedings of the International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC’2010), Fukuoka, Japan, 2010. [26] Jing Yuan, Yu Zheng, Xing Xie, and Guangzhong Sun. Driving with knowledge from the physical world. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 316–324, 2011.