Trigger-Action Platforms (TAPs) enable users to define rules that trigger device operations automatically. However, the execution of these rules can potentially create security risks for users. This paper presents a user study conducted to assess the validity of a classification model, which used Natural Language Processing (NLP) techniques to automatically classify Event-Condition-Action (ECA) rules according to security and privacy risks in TAPs, e.g., IFTTT. The study asked each user to evaluate 50 diferent IFTTT rules, named applets, classified as risky by the proposed model and provide answers to two specific questions designed to assess risk perception. The results confirmed that the proposed classification model ofers an assessment of the risk associated with a rule in line with user opinion. Furthermore, highlighting the presence of security or privacy-related risk positively impacted users' willingness to avoid using risky applets.
The vast spread of the Internet of Things (IoT) [
Unfortunately, the IoT domain as well as technology in general can be characterized by several
vulnerabilities which, if exploited by malicious individuals, can cause serious risks to the security
of the IoT ecosystem and the privacy of users interacting with it [
In a previous publication, we addressed the problem of identifying security and privacy risks
underlying the definition of ECA rules [
In addition to the empirical evaluations we have already conducted, we sought to ascertain
whether the risks we identified could also be recognized and confirmed by the end-users who
are the primary audience for our initial proposal. Thus, in this paper, we present a user study
involving a group of 30 individuals. Specifically, we asked each user to evaluate 50 diferent
IFTTT applets classified by our model proposed in [
The rest of the paper is organized as follows: Section 2 presents the main studies published in the literature assessing users’ perceptions regarding security and privacy risks in IoT environments. Section 3 briefly summarizes the contribution related to the definition of the BERT-based classification model for identifying risks related to ECA rules. Section 4 presents the user study, discussing the evaluation setup and the obtained results. Finally, in Section 5, conclusions and planned future developments are drawn.
The interaction with TAPs can sometimes pose serious risks both for the privacy of the user
and/or the security of the smart environment [
The perception of users with respect to the privacy and security risks arising from rules
defined through TAPs is a crucial aspect that needs to be considered. Various studies have
been conducted to investigate this topic, aiming to understand how users perceive the potential
risks of granting third-party access to their personal data and devices, as well as the measures
they take to protect their privacy and security. Saeidi et al. conducted a study to investigate
the implicit risks of using trigger-action platforms such as IFTTT in connecting smart-home
devices and services [
The results suggest that TAPs should ofer support to end-users in efectively managing and
comprehending the security and privacy risks associated with creating trigger-action rules. To
address this issue, specific eforts have been dedicated to developing ad-hoc solutions that enable
end-users to identify and mitigate these risks [
This section presents a comprehensive description of the sequential steps undertaken to produce a fully labeled dataset that encompasses ECA rules, followed by the process of training and evaluating a classification model that targets the identification of harmful rules. The proposed methodology was applied to a case study concerning the IFTTT platform. 3.1. System Overview There are three fundamental phases involved in the process of constructing a classifier to identify harmful ECA rules: • The first phase, named “ Data Labeling”, is designed to create labeled datasets for classification models. This is accomplished by defining the possible classes of risk for ECA rules, and their corresponding labels. Each ECA rule in the input dataset is then annotated with a suitable label using a semi-automatic labeling strategy that partitions the dataset into a small manually labeled subset and a larger subset that is automatically labeled using semi-supervised classification models. • The second phase, named “Model Training”, focuses on training the classification models using the labeled ECA rules dataset. NLP techniques are used to extract semantic information from the textual components of the ECA rules, and a weighted loss function is applied to deal with the imbalanced nature of the training set. • The last phase, named “Model testing”, involves evaluating the performance of the classiifcation models by inputting a set of manually labeled ECA rules and measuring their precision, recall, F1-score, and accuracy. 3.2. Data Labeling This section introduces the dataset employed in training the classifier for identifying harmful applets and the process by which the applets were labeled. The applet labeling process comprised a dual approach of manual and automatic labeling, involving semi-supervised models and an ensemble strategy.
IFTTT Applet Dataset. The study was based on the dataset proposed by Mi et al. [
The dataset underwent a data cleaning process to obtain a uniform dataset in language, and
the langdetect Python library was used to filter out applets not written in English. Applets
without a name or description, or containing only numbers for these features, were discarded.
After the data cleaning process, the resulting dataset contained 116,825 applets.
Categorization of IFTTT Applets According to Security and Privacy Risks. To
categorize the potential damages that could be inflicted by an applet on the user, we referred to the
work presented in [
We used the following classes for applet labeling, based on the considered macro-categories of risk: class 0 corresponds to Innocuous applets, class 1 to Personal damages, class 2 to Physical damages, and class 3 to Cybersecurity damages.
Manual Applet Labeling. The process of manually labeling the IFTTT dataset involved applying the majority method, whereby the first and second authors were responsible for the labeling process, and the third author mediated in cases of disagreement. This approach resulted in the labeling of 1,000 applets.
To increase the number of labeled applets, we developed a process for selecting additional applets to be manually labeled. This process involved creating a spreadsheet for each labeled applet, and sorting all unlabeled applets by their similarity to the labeled applet, using a combination of vector semantics and similarity functions. Specifically, we used SentenceBERT [18] to compute sentence embeddings for each applet and cosine similarity to compare the embeddings. We then manually reviewed each spreadsheet to identify and label applets that exhibited similar characteristics to those previously labeled, but difered in the triggers, actions, and/or channels involved. This process resulted in an augmented dataset consisting of 2,473 labeled applets.
Automatic Applet Labeling. After the manual labeling process, we developed a methodology that combines various semi-supervised learning models with an ensemble strategy. We utilized three distinct semi-supervised learning techniques, namely Self Learning [19], Label Propagation [20], and Generative Adversarial Learning [21]. Self Learning is a semi-supervised learning technique that uses a combination of labeled and unlabeled applets to train a model. This technique involves making predictions on unlabeled applets and treating those predictions as additional labeled applets to augment the existing labeled set. In contrast, Label Propagation propagates labels from a small set of labeled applets to a larger set of unlabeled applets. This is achieved by constructing a graph where the applets represent the nodes and the edges represent their similarity. Finally, Generative Adversarial Learning uses a generative model to generate synthetic applets similar to real ones and a discriminative model that learns to distinguish between them.
To obtain a single dataset from the three labeled applet datasets generated by the semisupervised learning models, we employed an ensemble strategy. It consisted of a majority-vote scheme among the three semi-supervised models, in which applets were considered for inclusion in the final dataset with their respective class labels only if at least two models produced the same prediction. This approach provided us with more consistent labels for the evaluated applets. 3.3. Model Training The dataset we constructed was characterized by an imbalance in the number of applets across classes, which poses a challenge in supervised classification [ 22]. In such scenarios, models trained on imbalanced data tend to classify input samples based on the majority class. To overcome this issue, we employed a weighted loss function that assigns diferent weights to each class based on the number of applets in the class. Notably, we assigned the minimum weight to class 0, which had the highest number of applets in the dataset.
We developed a BERT-based classifier to identify harmful applets by using applet information as textual features. The BertForSequenceClassification class of the transformers Python library was employed for training the classifier. This class corresponds to the BERT model with a single linear layer added for classification. We used the “ bert-base-uncased” model, which is the base version of BERT with 12 transformer blocks, 768 hidden units, 12 self-attention heads, and lowercase letters. 3.4. Model testing We conducted a series of experiments aimed at evaluating the efectiveness of the BERT-based model in classifying the diferent types of applet damage. To this end, a training dataset of 76,741 applets was assembled by employing the ensemble strategy combining the three sets of labeled applets generated with the semi-supervised learning models. The eficacy of the proposed approach and the quality of the labels produced by the ensemble strategy were validated by evaluating the model’s performance on a test set comprising 2,473 manually labeled applets.
Table 1 reports the values of accuracy, precision, recall, and F1-score achieved by the BERT model. Notably, the model attained a weighted average score of 88% across all metrics. Analysis of the results by class reveals that identifying class 1 applets is the most challenging task for the model. This dificulty arises from the slight diferences in the context of rule execution that can cause errors by classifying class 1 applets as class 2 or 3. To illustrate, consider the applet “Any new photo by me uploaded in a specific Google Drive folder, publish it on Twitter”. This applet falls under class 1 since it can lead to the unintentional sharing of sensitive or embarrassing photos. On the other hand, the applet “Any new photo uploaded by anyone in a specific Google Drive folder, publish it on Twitter” should be classified as class 3 due to the potential privacy risk of publishing a photo on the user’s Twitter profile without their knowledge of who uploaded it. Similarly, the applet “ New tweet by me with a specific hashtag, turn off lights” may be employed by a user to turn of lights with a goodnight tweet, but it can also trigger unintentionally in inappropriate situations, making it a class 1 applet. On the other hand, the applet “New tweet by anyone in the area with a specific tag, turn on lights” enables a user to determine if there are people who published a tweet in the zone, but its behavior may be jeopardized by third parties causing damage to the lights, making it a class 2 applet. These applets have subtle diferences that make it challenging to classify class 1 applets compared to the other classes, which explains the lower performance of the model in this regard.
This section describes the study we conducted to evaluate the user’s perception of security
and privacy issues related to IFTTT applets. In particular, we examined whether users are
able to perceive the potential risks identified by the proposed BERT model as actual risks, and
then evaluated whether such perception could potentially influence their decision to enable the
corresponding applets. The research aimed to answer two questions, which were investigated
through our experimental evaluation:
• RQ1. “Do users acknowledge the classification made by our model as an actual risk?”
• RQ2. “Can the identified risks impact on users’ decision to activate the applet?”
To answer the two research questions, we recruited 30 volunteers (12 females) and designed
a dedicated platform in charge of randomly selecting 50 IFTTT applets, contained within the
dataset we released in [
34 26 14 6 12 28 16 24
To avoid answers that were biased over personal considerations outside the scope of our evaluation, we made it clear to the users to provide their answers only according to the risk and not to their personal needs. This is to avoid, for example, that the applet having the description “Turn off my Philips Hue light anytime I leave the apartment” would not be activated by a user, not because of the risk it implies, but because such a user did not have a Philips Hue light.
Figure 3 shows the results obtained from the user study, considering only the 40 risky applets presented to each user. Arranged on the x-axis are the 30 users who were involved in the experiment, while the y-axis collects the number of responses received for both the first question, the one about risk perception, and the second question, the one that asked about the user’s confirmation to activate the applets.
In detail, the top part of the plot compares the responses obtained from the first question. While the left bar (the one in the light purple) shows the number of negative responses, i.e., the number of times in the user’s opinion the applet presented did not pose a security or privacy risk. On the other hand, the bar on the right of each user shows the number of times a given user considered the type of risk identified by the model to be likely.
As for the lower part of the plot, the number of responses regarding the second question is reported. In this case, the left bar (the one in the dark purple) reports the number of times that, at the expense of the highlighted risk, the user would still decide to activate such an applet in their smart environment. Accordingly, the right bar highlights the number of times when the presence of an identified security or privacy risk leads the user to decide to avoid activating such an applet.
The results obtained are very promising and indicative. In fact, even at first glance, it can be seen that the number of times the risk identified by our model was deemed likely is very high, with an average of 34.7 out of 40. For many users (U3, U4, U5, U17, U21, U25), the totality of the applets they were asked to evaluate had security and privacy risks that conformed with what the model highlighted. In a few cases (U2, U13, U18, U28) it happened that the risk was highly not recognized by the participants, resulting in more than 10 negative responses. When questioned about the reason for such a high number of unrecognized risks, all of the above-mentioned users gave the same answer, namely that in their opinion the highlighted risk was related to scenarios so remote as to be almost unrealistic, reasoning that such applets would never come to pose any security or privacy risk.
Regarding the users’ responses on whether they planned to activate the applets, this study revealed some variability in the responses, in fact it was observed that none of the users chose to refrain from activating all 40 applets that were presented for evaluation. It is worth noting, however, that even for this type of evaluation, users almost always opted not to activate the applets highlighted as risky. The only exception is user U21, for whom the number of applets he would not activate (15), turns out to be lower than the number of applets he would activate anyway (25). This result is even more surprising in light of the fact that U21 belongs to that group of users who instead indicated that all the applets that were submitted to him were likely to be risky. When questioned about this ambivalence in the answers, the participant responded as follows: “I agree that all applets are in some way risky, however, in my opinion, having them available would still be convenient. But now, knowing that they are risky, I would perhaps be more careful to avoid falling into problems (of security or privacy)”.
On the other hand, with regard to users U2, U13, U18, and U28, who had been the ones who had most indicated the identified risk as “unlikely”, the responses collected to this second question confirm their thinking. These users, in fact, are among those who would activate most of the applets presented to them. This result should not be surprising since in view of the fact that they do not consider many applets as risky, it is reasonable to assume that such applets can (and should) be activated either way.
As for the results concerning the 300 harmless applets we submitted to the users as a control, our results highlighted that 39 (13%) of them were instead judged as harmful by the participants. As mentioned above, for such applets we asked the participants to provide us with an open-ended answer, explaining why, in their own view, such an applet would be capable of causing a danger to the user. Table 2 shows some of the most relevant answers we received from participants and a short summary of the considered applet. In particular, some users e.g. U3, highlighted that the overuse of certain automation could lead to concerns since some services might not be capable of keep functioning as intended when involved in applets, such as the Evernote cloud space that could be filled up if provided with multiple notes. On the other hand, users such as U12 and U34 suggested that certain applets should be considered as harmful whether certain conditions apply, like when minors are involved or when there is a risk of fueling the spread of malicious software.
In conclusion, the experimental evaluations provided interesting insights concerning the 2 RQs
posed upstream of this study. In fact, with respect to RQ1, the BERT-based harm classification
model we presented in [
In this study, we investigated the privacy and security concerns associated with applets activated
through the IFTTT platform. We recruited 30 users to evaluate applets classified as risky
by our previously proposed BERT-based classification model [
This work has been supported by the Italian Ministry of University and Research (MUR) under grant PRIN 2017 “EMPATHY: Empowering People in deAling with internet of THings ecosYstems” (Progetti di Rilevante Interesse Nazionale − Bando 2017, Grant 2017MX9T7H). [18] N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERTnetworks, in: Proceedings International Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, ACL, 2019, pp. 3982–3992. [19] D. Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, in: Proceedings 33rd Annual Meeting of the Association for Computational Linguistics, ACM, 1995, pp. 189–196. [20] X. Zhu, Z. Ghahramani, Learning from labeled and unlabeled data with label propagation,
Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002. [21] D. Croce, G. Castellucci, R. Basili, GAN-BERT: Generative adversarial learning for robust text classification with a bunch of labeled examples, in: Proceedings of 58th Annual Meeting of the Association for Computation Linguistics, ACL, 2020, pp. 2114–2119. [22] L. Wang, M. Han, X. Li, N. Zhang, H. Cheng, Review of classification methods on unbalanced data sets, IEEE Access 9 (2021) 64606–64628.