<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>The role of decisional autonomy in User-IoT systems interaction</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Rino Falcone</string-name>
          <email>rino.falcone@istc.cnr.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Alessandro Sapienza</string-name>
          <email>alessandro.sapienza@istc.cnr.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Institute of Cognitive Sciences and Technologies, National Research Council of Italy (ISTC-CNR)</institution>
          ,
          <addr-line>00185 Rome</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2022</year>
      </pub-date>
      <abstract>
        <p>As the Internet of Things (IoT) continues to spread, emerging technologies provide devices with various autonomous capabilities that enable them to assist their users in a multitude of activities throughout their daily lives. Despite this, it is still unclear what the role of autonomy should be in this context, as limited academic research has been conducted on such a topic. This study tries to fill this research gap, by proposing a possible solution on how IoT devices may act in order to develop and regulate their autonomy, in relation to the specific user they interface with. After introducing a theoretical framework, we considered a possible implementation in a simulation context, showing how the proposed approach works.</p>
      </abstract>
      <kwd-group>
        <kwd>Internet of Things</kwd>
        <kwd>Autonomy</kwd>
        <kwd>Trust</kwd>
        <kwd>Social simulation</kwd>
        <kwd>User acceptance</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        In recent years, the concept of autonomy has played an increasingly important role in the
context of the Internet of Things (IoT). As a matter of fact, IoT devices must be autonomous as
well as cooperative in order to autonomously coordinate cooperative actions towards meeting
common high-level goals [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Such autonomy allows them to improve their capability to sense,
think, and act when executing tasks [
        <xref ref-type="bibr" rid="ref2 ref3">2, 3</xref>
        ].
      </p>
      <p>
        Of course, the autonomy of the devices must always and in any case be confronted with the
needs and availability of users. In this connection, Miranda and colleagues [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] introduced in
their work the concept of Internet of People (IoP) claiming that, given the increasingly important
role of technology in our daily lives, we as users should be put at the center of such systems.
They stress the fact that IoT systems must be able to adapt to the user, taking people’s context
into account and avoiding user intervention as much as possible. In a similar way, Ashraf [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]
talks about autonomy in the Internet of things, with the aim of arguing that in this context it is
necessary to minimize user intervention. Nonetheless, research in this context is still scarce
and the concept of autonomy has not yet been given all the attention it deserves [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. There
is still a gap in the literature regarding a comprehensive model that explains how IoT device
should use and develop their autonomy in the relationship with the user.
      </p>
      <p>Of course, there may be several approaches to address this problem. In this work, we focus
our attention on a particular solution. Specifically, we take into account the fact that the devices
possess a certain autonomy of execution, which determines the resources they need to perform
in a certain way: internal/external resources, request for collaboration of other agents etc. In
this sense, the more autonomous a device is, the less additional resources it needs to use. On
the one hand, the performance of the devices is closely linked to this concept, so in some levels
the devices can ofer better results than others. On the other hand, the user may be more or
less willing to grant the use of specific resources, thus desiring the execution of tasks at specific
levels of autonomy.</p>
      <p>Our idea is that devices can use their autonomy of decision as an opportunity to determine
the best way to perform the task assigned to them, i.e. autonomously changing their level of
execution. This ability requires various potentialities for our smart devices: i) to be able to
manage its own diferent degrees of autonomy of execution in the interaction with the user; ii)
to be able to evaluate and adapt, in each interaction, the right degree of autonomy of execution
to carry out the task; iii) to know how to evaluate its own ability to act in the various degrees
of autonomy of execution.</p>
      <p>Therefore, within this article, we propose a framework analyzing the interaction between
user and IoT device based on the concept of autonomy, in its diferent shapes. Specifically,
we will investigate two diferent forms of autonomy: that of execution, which identifies what
agents are able to do and what they are authorized to do; that of decision, or the degree of
autonomy that agents have in determining their own autonomy of execution. By engaging
in this exploration, this paper constitutes a step toward the study of the human-IoT systems
interaction.</p>
    </sec>
    <sec id="sec-2">
      <title>2. State of the art</title>
      <p>While the fundamental role of autonomy has been clearly identified in the literature, there are
few implementation solutions that have concretely given it a key role in the interaction with
human users.</p>
      <p>
        As a first example, in the HRI domain, the authors of Optimo [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] proposed a system capable
of perceiving how much the user trusts the device and then adapting its behaviors dynamically,
to actively seek greater trust and greater eficiency within future collaborations.
      </p>
      <p>
        In the IoT domain, we introduced a model for user’s acceptance [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ], in which devices evaluated
how much the user trusts them, in order to perform tasks with an adequate level of autonomy.
      </p>
      <p>
        Hu and colleagues [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] conducted an experimental study to investigate the role of artificial
autonomy by dividing it into three types of autonomy in terms of task primitives: sensing,
thought, and action autonomy. In particular, the authors investigate how these dimensions afect
the perception that users have of artificial devices (in terms of competence and motivation), in
order to explain how these afect the willingness to use the devices themselves.
      </p>
      <p>
        Recently, the authors of [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] introduced the concept of autonomy in large‐scale IoT ecosystems,
by making use of cognitive adaptive approaches.
      </p>
      <p>
        Furthermore, Sifakis [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] states that IoT is a great opportunity to reinvigorate computing
by focusing on autonomous system design. His idea is to compensate the lack of human
intervention by introducing adaptive control.
      </p>
      <p>Within this work, our eforts focus on determining how an IoT device can use and increase
its autonomy of decision in the relationship with the user.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Model</title>
      <p>In this work, we are interested in identifying the complex relations between a user  and an IoT
system  consisting of n devices { 1,  2 … ,   }. The human-device interaction model is based on
the concept of autonomy. More specifically, we distinguish between 2 kinds of autonomy:
1. the autonomy of execution,  _ , which defines both what devices are able to do and the
resources to which they are authorized to access. This value is specified by the user when
requesting the execution of a task;
2. the autonomy of decision,  _ , namely the extent to which the devices can deviate from
the level of autonomy of execution assigned by the user.</p>
      <p>The devices represent the actors of this autonomy, while the delegators are the users. On the
one hand, the user  makes use of the devices to perform the tasks it needs. At the beginning,
the user  grants an initial level of autonomy of decision to the devices, defined as   .
Then, on each turn, the user will assign a specific task  to a given device   , specifying also the
level of autonomy of execution  _ it desires. More in details, we classify the autonomy of
execution in 5 levels:
• Level 0: to perform the task, in addition to the default internal and external resources,
devices need another external agent and non-default external resources;
• Level 1: in addition to the default internal and external resources, devices need another
external agent to perform the task;
• Level 2: to perform the task, in addition to the default internal and external resources,
devices need to access other non-default external resources;
• Level 3: devices can execute the task with all internal and external resources by default.</p>
      <p>They cannot perform other tasks at the same time;
• Level 4: devices can perform the task with a minimum commitment of internal and
external resources (by default) that is, you can be able to perform other tasks at the same
time.</p>
      <p>The idea is that the more autonomous the agent is in performing tasks, the fewer resources it
needs. At the lowest levels, it needs to exploit external resources not directly available and/or
other agents. Increasing the level, it no longer needs these resources. Indeed, at the highest
level, it could even perform multiple tasks at the same time.</p>
      <p>
        On the other hand, the devices want to satisfy at best the user’s requests. However, in order
to fulfill such purpose, it is sometimes necessary for the devices to modify their autonomy of
execution [
        <xref ref-type="bibr" rid="ref11 ref12">11, 12</xref>
        ]. This shift in autonomy would give the device the chance to execute the task
providing a greater performance.
      </p>
      <p>At the end of the task, the performance of   is evaluated. The actual performance of the
devices depends on the error probability of the considered task level.</p>
      <sec id="sec-3-1">
        <title>3.1. The user</title>
        <p>In the simulation, we considered a single user  dealing with a number of IoT devices. The
user makes use of such devices to pursue its own goals, granting them an initial autonomy of
execution, the predisposition, which limits their actions.</p>
        <p>
          The predisposition takes into account the fact that diferent users may have diferent attitudes
towards this type of technology. For instance, the authors of [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] found that household members
with high technical skills are more willing to adopt smart home services and products.
        </p>
        <p>Starting from this initial value, the autonomy granted may change over time in response to
certain situations and may vary from device to device.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. The devices</title>
        <p>There can be a variable number of devices in the world. All of them possess the main goal
of pursuing the user’s task to satisfy at best its need (even those not explicitly requested).
Remarkably, in order to do so, they need to increase their autonomy of decision  _ .</p>
        <p>The device may find itself in the situation of not being able to adequately perform a task at a
given level of autonomy of execution, i.e. its expected performance is below a given threshold
value  . In order to overcome this issue, it can decide to change this level, by increasing or
decreasing it.</p>
        <p>Resuming, a device is characterized by:
1. Its trustworthiness estimation on the various levels, to evaluate its own skills in the
diferent cases;
2. The autonomy of execution aut_e, assigned by the user;
3. The estimation of the autonomy of decision aut_d;
4. the error probability on each level: this is an intrinsic characteristic of the device, neither
it nor the user can directly access it, but it can be estimated through interaction.</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. Trust and Autonomy</title>
        <p>Trust is a key element in every aspect of social cooperation. Its importance has also been clearly
recognized in Human-Machine interaction [14] and, more in details, also in the IoT domain [15].
In this paper, we refer to the socio-cognitive model of trust [16]. The trustworthiness [17] of the
devices is modeled in terms of willingness and competence. As far as it concerns willingness,
we assume in this case that the devices are always and in any case well disposed towards the
user and that they have the main goal of satisfying its requests. Concerning competence, this
dimension is implemented through the      , i.e. the probability that a device will
not be able to successfully complete the requested task. Of course, this probability also depends
on the level of autonomy at which the task takes place.</p>
        <p>
          In the light of such premise, we have modeled the autonomy of decision as a scalar variable
defined in [
          <xref ref-type="bibr" rid="ref1">0,1</xref>
          ], whereby the greater the autonomy the easier it is to switch level. Instead, the
trustworthiness of a device is modeled as a 5 elements vector, precisely to evaluate its skill at
every level of autonomy.
        </p>
        <p>Of course, working at a given level of autonomy gives devices the opportunity to show their
competence at that level. When a task ends, the trustworthiness is updated through a weighted
mean, according to Equations 1. Here,    
,,
is the result of the execution of the task
 at level  by agent   , while  1 and  2 represent the weights of the old and the new value.
     
ℎ
,,
=
   
ℎ
,, ∗  1 +    
 1 +  2
,, ∗  2
Autonomy of decision is updated diferently, according to 2.</p>
        <p>_ , =
 _ , ∗  1 +</p>
        <p>,, ∗  2</p>
        <p>Indeed, this should not increase if the device does not push beyond the level it normally
performs. Therefore, the devices will detect those situations in which going further would
represent an advantage for the user while not doing so would result in a loss and they will limit
their increase of autonomy only to such situations. In this work we consider two condition to
do so:
1. base strategy: the device believes that its performance at the assigned level will not be
satisfactory (i.e., above a designated threshold  );
2. additional strategy: besides the first strategy, the device will always try to modify the
autonomy of execution, as long as there is suficient autonomy of decision (above a
designated  threshold).</p>
        <p>Once identified the need to change the level of execution, the device will choose an alternative
level according to two factors: the probability  _ 
that the user will accept the decision
of the device and the gain  , in terms of performance, that would be obtained by choosing
this new level. Equation 3 estimates  _ 
,,,,
, i.e. the probability that the user  will
accept the decision of the device   to increase the level from  to  , concerning the task  . Such
probability is estimated as a function of the  
of the user and the autonomy of
decision</p>
        <p>_ of the device, developed through interaction with the user. The  factor modulates
the user’s predisposition, shaping the fact that the more the device deviates from the assigned
level, the easier it is for the task to be rejected.
(1)
(2)
(3)
(4)
 _ 
,,,,</p>
        <p>+  _ , ) ∗ 
= (</p>
        <p>=
 ,
4</p>
        <p>Given the classification of the autonomy of execution, it is reasonable to assume that
increasing the level is more convenient than decreasing it, as it requires fewer resources. Conversely,
leveling down requires additional resources. To model this, we introduce the constant  in
Equation 3. In the analogous case, when the level is decreased,  is replaced by  .</p>
        <p>As far as it concerns the gain  , it is estimated as the diference between the expected
performance (trustworthiness) at level y and the expected performance at level x, as in Equation</p>
        <p>We define probabilistic utility   ,,,, as the product of  _  ,,,, and  ,,, (see
Equation 6. The device chooses the level with the greatest probabilistic utility. In case of equal
result, the level with the highest  is preferred.
(6)
  ,,,,
=  _ 
,,,,
∗  ,,,</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Simulation and results</title>
      <p>In this section, we present the results of the agent-based simulation experiments, implemented
on the NetLogo platform [18]. In the simulations, we are going to check what happens in 3
diferent scenarios, namely:
1. none: there is no optimization mechanism;
2. first : only the base strategy is implemented, thus it is possible to modify the autonomy of
execution if a low performance is expected (below the  threshold);
3. both: the additional strategy is implemented, thus it is always possible to modify the
autonomy of execution, as long as there is suficient autonomy of decision (below the 
threshold).</p>
      <p>The experiments were conducted using the following setting:
•   = [0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1];
• number of device: 10;
• duration: 1000 time units;
• error probability: randomly assigned for each device on each level in the range [0,0.5];
•  = 0.5;
•  = 0.5;
•  = 1 and  = 0.9;
•  1 = 0.9 and  2 = 0.1.</p>
      <p>Each run of the experiment has a fixed duration of 1000 time unit, enough time to stabilize
the values of the variables of interest. Concerning predisposition, the whole range of values has
been investigated. As for the error probability, it is randomly generated between 0 and 0.5. Such
randomness is designed to ensure that each device must estimate its capabilities and evaluate
on which levels it is appropriate to perform the tasks. The threshold  has been set to 0.5, to
ensure that devices provide an average performance greater than 50% when they decide to
execute a task. In a similar way, the threshold  ensure that, in the event that the device intends
to optimize its performance even if its expected value is above  , there is a certain probability
of acceptance by the user. Then,  and  were set to shape the fact that it is more convenient to
increase the level autonomy of execution that to decrease it. Lastly,  1 and  2 were set to give
more importance to past experience and to avoid excessive fluctuations in the variable values
during simulations.</p>
      <p>Figure 1 shows the evolution of the autonomy of decision of the devices, as the user’s
predisposition varies. If neither of the two strategies is active, the autonomy granted remains
exactly equal to the user predisposition. On the other hand, in the presence of one or two
optimization mechanisms, the autonomy values increase significantly. We do not find significant
diferences between the two approaches, as far as it concerns autonomy of decision. In general,
higher values of   correspond to greater values of autonomy of decision. Such
an increment is particularly significant in the proximity of the value 0, while immediately
afterwards it remains linear. Basically, the more the user is willing to accept the autonomous
choices of the devices, the easier it is for them to develop autonomy of decision: if the user
rejects their choices, they have few possibilities for action. This result shows us how even
a minimum predisposition value is enough to give devices the opportunity to develop their
autonomy of decision.</p>
      <p>Another remarkable result concerns the evaluation of trustworthiness, that is the perception
that the user has of the devices. As Figure 2 shows us, the introduction of optimization
mechanisms paradoxically involves a decrease in trustworthiness: in the considered level,
-4.13% with the base strategy and -9.51% with the additional strategy. Although this efect
may seem counterintuitive, the reasons for this result are clearly explained if we also take into
consideration the variation of the performance provided to the user.</p>
      <p>Thus, Figure 3 shows that the introduction of the base optimization strategy actually allows
for an improvement on the performance provided to the user (+9.37% on average). Considering
both strategies entails an additional increase (+13.52% on average). In fact, these optimization
mechanisms ensure that devices operate on the levels at which they believe they are most
eficient, avoiding those at which they are less good. As a consequence, the valuation for these
low performance levels remains lower. This efect explains why the average trustworthiness
evaluation decreases, as a result of the introduction of optimization strategies. However, this
is in line with the main goal of the devices, as the main purpose of artificial systems should
not be to maximize the user’s perception of them [14]. While even this is an important topic,
they should above all aim to improve the actual performance provided to the use, even if this
involves being perceives as less trustworthy. It should also be noted that even in the absence of
initial   , the devices still manage to improve their performance.</p>
    </sec>
    <sec id="sec-5">
      <title>5. Discussion and Conclusions</title>
      <p>In this work, we investigated one of the possible approaches to increase autonomy of decision
of IoT devices through direct interaction with a user.</p>
      <p>Indeed, these devices have enormous potential to ofer to end users. However, the user may
not necessarily be willing to accept all the functionalities ofered from the beginning. In light
of such consideration, it becomes essential to understand how the device should behave in
order to stimulate the user’s trust and gradually ensure that the latter is willing to grant more
autonomy. Of course, many approaches are possible to solve this problem [19, 20]. In this
article, we have proposed a methodology in which devices exploit their autonomy of decision
in order to ofer better performance to the user, weighing the benefits introduced by deviating
from what the user requests. This allows them to show their capabilities and, in turn, to obtain
greater autonomy.</p>
      <p>Summarizing the results of this analysis, we can say that:
1. Lower values of user   correspond to lower values of autonomy of decision.</p>
      <p>Basically, the more the user is willing to accept the devices’ autonomy of decision, the
easier it is for them to develop such autonomy.
2. Allowing the devices to change the level of execution if they believe that their performance
will not be satisfactory (base strategy) improves the performance provided to the user.
Furthermore, allowing them to freely change the level of execution, as long as there
is suficient autonomy of decision ( additional strategy) allows to get an even higher
improvement.
3. Even in lack of   , the optimization strategies allow to improve the
performance.
4. Although these strategies improve the performance, the trustworthiness evaluation
worsens, since the devices avoid performing task at some levels, in order to favor the
levels at which they believe they perform better.</p>
      <p>To conclude, the results of this work, albeit more theoretical, aimed at showing the
efectiveness of the proposed approach, providing interesting insights for the evolution of IoT systems.
Indeed, the experiments suggest that the approach used actually makes it possible to improve
the levels of autonomy at which devices can operate.</p>
      <p>It is worth underlining that the proposed framework is not intended as an alternative to the
other solution to improve autonomy. On the contrary, it could provide support to other valid
solutions.
[14] A. Sapienza, F. Cantucci, R. Falcone, Modeling interaction in human–machine systems: A
trust and trustworthiness approach, Automation 3 (2022) 242–257.
[15] G. Fortino, L. Fotia, F. Messina, D. Rosaci, G. M. Sarné, Grouping iot devices by trust
and meritocracy, in: 2021 International Conference on Cyber-Physical Social Intelligence
(ICCSI), IEEE, 2021, pp. 1–5.
[16] C. Castelfranchi, R. Falcone, Trust theory: A socio-cognitive and computational model,</p>
      <p>John Wiley &amp; Sons, 2010.
[17] A. Sapienza, R. Falcone, Evaluating agents’ trustworthiness within virtual societies in case
of no direct experience, Cognitive Systems Research 64 (2020) 164–173.
[18] U. Wilensky, Netlogo. evanston, il: Center for connected learning and computer-based
modeling, northwestern university, 1999.
[19] C. Kuhn, D. Lucke, Supporting the digital transformation: a low-threshold approach for
manufacturing related higher education and employee training, Procedia CIRP 104 (2021)
647–652.
[20] C. Janiesch, M. Fischer, A. Winkelmann, V. Nentwich, Specifying autonomy in the
internet of things: the autonomy model and notation, Information Systems and e-Business
Management 17 (2019) 159–194.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>I. T.</given-names>
            <surname>Michailidis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. C.</given-names>
            <surname>Kapoutsis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. D.</given-names>
            <surname>Korkas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. T.</given-names>
            <surname>Michailidis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K. A.</given-names>
            <surname>Alexandridou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Ravanis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. B.</given-names>
            <surname>Kosmatopoulos</surname>
          </string-name>
          ,
          <article-title>Embedding autonomy in large-scale iot ecosystems using cao and l4g-cao, Discover Internet of Things 1 (</article-title>
          <year>2021</year>
          )
          <fpage>1</fpage>
          -
          <lpage>22</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>Y. K.</given-names>
            <surname>Dwivedi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Hughes</surname>
          </string-name>
          , E. Ismagilova, G. Aarts,
          <string-name>
            <given-names>C.</given-names>
            <surname>Coombs</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Crick</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Duan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Dwivedi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Edwards</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Eirug</surname>
          </string-name>
          , et al.,
          <article-title>Artificial intelligence (ai): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy</article-title>
          ,
          <source>International Journal of Information Management</source>
          <volume>57</volume>
          (
          <year>2021</year>
          )
          <fpage>101994</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>K. M.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Yoo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.-W.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.-H.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. Hong,</surname>
          </string-name>
          <article-title>Autonomic machine learning platform</article-title>
          ,
          <source>International Journal of Information Management</source>
          <volume>49</volume>
          (
          <year>2019</year>
          )
          <fpage>491</fpage>
          -
          <lpage>501</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>J.</given-names>
            <surname>Miranda</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Mäkitalo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Garcia-Alonso</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Berrocal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Mikkonen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Canal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Murillo</surname>
          </string-name>
          ,
          <article-title>From the internet of things to the internet of people</article-title>
          ,
          <source>IEEE Internet Computing</source>
          <volume>19</volume>
          (
          <year>2015</year>
          )
          <fpage>40</fpage>
          -
          <lpage>47</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>Q. M.</given-names>
            <surname>Ashraf</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. H.</given-names>
            <surname>Habaebi</surname>
          </string-name>
          ,
          <article-title>Introducing autonomy in internet of things, Recent Advances in Computer Science</article-title>
          , WSEAS Publishing (
          <year>2015</year>
          )
          <fpage>215</fpage>
          -
          <lpage>221</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>E.</given-names>
            <surname>Constantinides</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Kahlert</surname>
          </string-name>
          , S. A. de Vries,
          <article-title>The relevance of technological autonomy in the acceptance of iot services in retail</article-title>
          ,
          <source>in: 2nd International Conference on Internet of Things, Data and Cloud Computing</source>
          ,
          <string-name>
            <surname>ICC</surname>
          </string-name>
          <year>2017</year>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>A.</given-names>
            <surname>Xu</surname>
          </string-name>
          , G. Dudek, Optimo:
          <article-title>Online probabilistic trust inference model for asymmetric humanrobot collaborations</article-title>
          ,
          <source>in: 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI)</source>
          , IEEE,
          <year>2015</year>
          , pp.
          <fpage>221</fpage>
          -
          <lpage>228</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>R.</given-names>
            <surname>Falcone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Sapienza</surname>
          </string-name>
          ,
          <article-title>On the users' acceptance of iot systems: A theoretical approach</article-title>
          ,
          <source>Information</source>
          <volume>9</volume>
          (
          <year>2018</year>
          )
          <fpage>53</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>Q.</given-names>
            <surname>Hu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Lu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Pan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Gong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <article-title>Can ai artifacts influence human cognition? the efects of artificial autonomy in intelligent personal assistants</article-title>
          ,
          <source>International Journal of Information Management</source>
          <volume>56</volume>
          (
          <year>2021</year>
          )
          <fpage>102250</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>J.</given-names>
            <surname>Sifakis</surname>
          </string-name>
          ,
          <article-title>System design in the era of iot-meeting the autonomy challenge</article-title>
          , arXiv preprint arXiv:
          <year>1806</year>
          .
          <volume>09846</volume>
          (
          <year>2018</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>R.</given-names>
            <surname>Falcone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Castelfranchi</surname>
          </string-name>
          ,
          <article-title>The human in the loop of a delegated agent: The theory of adjustable social autonomy</article-title>
          ,
          <source>IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans</source>
          <volume>31</volume>
          (
          <year>2001</year>
          )
          <fpage>406</fpage>
          -
          <lpage>418</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>R.</given-names>
            <surname>Falcone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Castelfranchi</surname>
          </string-name>
          ,
          <article-title>Levels of delegation and levels of adoption as the basis for adjustable autonomy</article-title>
          ,
          <source>in: Congress of the Italian Association for Artificial Intelligence</source>
          , Springer,
          <year>1999</year>
          , pp.
          <fpage>273</fpage>
          -
          <lpage>284</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>J.</given-names>
            <surname>Kowalski</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Biele</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Krzysztofek</surname>
          </string-name>
          ,
          <article-title>Smart home technology as a creator of a superempowered user</article-title>
          ,
          <source>in: International Conference on Intelligent Human Systems Integration</source>
          , Springer,
          <year>2019</year>
          , pp.
          <fpage>175</fpage>
          -
          <lpage>180</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>