Using Closed Itemsets for Implicit User Authentication in Web Browsing 1 1,2 2 2 2 O. Coupelon , D. Dia , F. Labernia , Y. Loiseau , and O. Raynaud 1 Almerys, 46 rue du Ressort, 63967 Clermont-Ferrand {olivier.coupelon,diye.dia}@almerys.com 2 Blaise Pascal University, 24 Avenue des Landais, 63170 Aubière {loiseau,raynaud}@isima.fr, fabien.labernia@gmail.com Abstract. Faced with both identity theft and the theft of means of authentication, users of digital services are starting to look rather suspi- ciously at online systems. The behavior is made up of a series of observ- able actions of an Internet user and, taken as a whole, the most frequent of these actions amount to habit. Habit and reputation oer ways of recognizing the user. The introduction of an implicit means of authenti- cation based upon the user's behavior allows web sites and businesses to rationalize the risks they take when authorizing access to critical func- tionalities. In this paper, we propose a new model for implicit authen- tication of web users based on extraction of closed patterns. On a data set of web navigation connection logs of 3,000 users over a six-month period we follow the experimental protocol described in [1] to compute performance of our model. 1 Introduction In order to achieve productivity gains, companies are encouraging their cus- tomers to access their services via the Internet. It is accepted that on-line ser- vices are more immediate and more user-friendly than accessing these services via a brick and mortar agency, which involves going there and, more often than not, waiting around [2]. Nevertheless, access to these services does pose secu- rity problems. Certain services provide access to sensitive data such as banking data, for which it is absolutely essential to authenticate the users concerned. However identity thefts are becoming more and more numerous [3]. We can dis- tinguish two paradigms for increasing access security. The rst one consists of making access protocols stronger by relying, for example, on external devices for transmitting access codes that are supplementary to the login/password pair. Nevertheless, these processes are detrimental to the user-friendliness and usabil- ity of the services. The number of transactions abandoned before reaching the end of the process is increasing and exchange volumes are decreasing. The sec- ond paradigm consists to the contrary of simplifying the identication processes in order to increase the exchange volumes. By way of examples, we can mention single-click payment [2] [4] or using RFID chips for contactless payments. Where these two paradigms meet is where we nd implicit means of authentication. 2 O. Coupelon, D. Dia, F. Labernia, Y. Loiseau, and O. Raynaud A means of authentication is a process that makes it possible to ensure that the identity declared in the event of access is indeed the user's identity. Traditionally, a user authenticates himself or herself by providing proof of identity [5]. This process is called explicit authentication. In contrast, implicit authentication does not require anything from the user but instead studies his or her behavior, the trail left by the user's actions, and then either does or does not validate the declared identity. An implicit means of authentication cannot replace traditional means of authentication as it is necessary for the user to have access to his or her service so that the person's behavior may be studied and their identity can either be validated or rejected. To the contrary, if it is eective, it would enable stronger authentication modes to be avoided (such as chip cards and PIN numbers), which are detrimental to the usability of services. The challenge is to detect identity theft as quickly as possible and, to the contrary, to validate a legitimate identity for as long a time as possible. This contribution is organized as follows: in section 2 we shall oer a state-of- the-art about implicit authentication and user's prole in web browsing. Then we propose a learning model for implicit authentication of web users we are dealing with in section 3. In section 4, we compare several methods for building proles of each user. We faithfully reproduce the experimental study conducted in [1] and we analyze all of our results. Finally, in section 5, we shall resume our results and discuss our future work. 2 Related works In his survey of implicit authentication for mobile devices ([6]), the author says of an authentication system that it is implicit if the system does not make demands of the user (see Table 1). Implicit authentication systems were studied very quickly for mobile phones. In [7] and [8], the authors studied behaviour based on variables specic to smart- phones such as calls, SMS's, browsing between applications, location, and the time of day. Experiments were conducted based on the data for 50 users over a period of 12 days. The data were gathered using an application installed by users who were volunteers. The users' proles were built up from how frequently positive or negative events occurred and the location. Within this context, a positive event is an event consistent with the information gathered upstream. By way of an example, calling a number which is in the phone's directory is a positive event. The results of this study show that based on ten or so actions, you can detect fraudulent use of a smartphone with an accuracy of 95%. In a quite dierent context, the authors of [9] relied on a Bayesian classication in order to associate a behaviour class with each video streaming user. The data set is simulated and consists of 1,000 users over 100 days. The variables taken into account are the quality of the ow, the type of program, the duration of the session, the type of user, and the popularity of the video. The results are mixed, because the model proposed admits to an accuracy rate of 50%. Using Closed Itemsets for Implicit User Authentication in Web Browsing 3 Feature Capturing Implicit/Explicit Spoong Threats Problems Method Passcode Keyboard input Explicit Keyloggers, Guessable pass- Shoulder Surng words Token Hardware device Mainly explicit, None Easily stolen or implicit possible lost Face & Iris Camera Both Picture of the le- Lighting situa- gitimate user tion and make-up Keystroke Keyboard Implicit, explicit Typing imitation Long training possible (dicult) phase, reliability Location GPS, infrastruc- Implicit Informed Traveling, preci- ture strangers sion Network Software protocol Implicit Informed Precision (e.g. WireShark) strangers Table 1. Comparison of dierent authentication methods The particular context of implicit authentication for web browsing was studied in [1], [10], [11] and [12]. In [1], the author adopted the domain name, the num- ber of pages viewed, the session start time, and its duration, as characteristic variables. The data set, which was gathered by a service provider, consisted of 300 rst connections by 2,798 users over a period of 12 months. The user proles consisted of patterns with a size of 1. The author compares several pattern selec- tion approaches like the support and the lift approaches. The study shows that for small, anonymous behavioural patterns (involving up to twenty or so sites visited), the most eective models are still traditional classication models like decision trees. On the other hand, whenever anonymous behaviour exceeds 70 or so sites, the support and lift-based classication models are more accurate. The study conducted in [12] states that the size of the data set remains a determining parameter. Their study, conducted on 10 users over a one-month period, did not enable them to build a signicant model for distinguishing users. The authors also concluded that no variable taken individually enables a user to be authen- ticated. Drawing inspiration from a study conducted in [1], the authors of [13] studied several techniques for spying on a user who holds a dynamic IP address, based on behavioural models. The methods compared are seeking motives, the nearest neighbours technique, and the multinomial Bayesian classier. The data set consisted of DNS requests from 3,600 users over a two-month period. In this study, only the most signicant variables and the most popular host names were considered. The accuracy rates for the models proposed were satisfactory. The study that we conduct in this paper also forms part of a continuation of the work by [1]. We faithfully reproduce his experimental protocol on our data and we compare performance of our classication algorithm to his specic models. 4 O. Coupelon, D. Dia, F. Labernia, Y. Loiseau, and O. Raynaud 3 Models We propose an intuitive learning model architecture for user authentication. From a data set of web browsing logs we compute a set of own patterns for each user. A pattern is a set of frequently visited sites. The size of pattern may vary. Thanks to these proles we are able to provide an authentication for anonymous sessions. We then compute confusion matrices and we provide precisions of the models. In our present study, we compare performance of a naive Bayes classier to variations on k-nearest neighbors algorithms. More precisely, the studied parameters are selection process of user own patterns, computation process of user proles and distance functions computed for classication stage. Figure 1 outlines the framework of the machine learning process. Past Anonymous Behaviour Behaviour User User ? ? Prole- Learning Algorithms Score Computation - Authentication Fig. 1. Architecture 3.1 Formal framework We call a session a set of visited web sites at a specic time by a given user ui such as i ∈ [1,n] and n is the number of users. The size of a session is limited and equal to 10. The learning database of each user ui takes the form of a set of 3 sessions denoted Sui and is built from log data . We call S = S i Sui the whole set of sessions of the database. We call Wui the whole set of web sites visited at least once by user ui and we S call W = i Wui the whole set of visited sites. The order of visited web sites is not taken into account by this model. Denition 1 (k-pattern). Let W be a set of visited web sites and S be a set of sessions on W . A subset P of W is called a k − pattern where k is the size of P . A session S in S is said to contain a k − pattern P if P ⊆ S . Denition 2 (Support and relative support (lift)). We dene the support of a pattern P as the percentage of sessions in S containing P (by extension we give the support of a pattern in the set of sessions of a given user ui ): ||{S ∈ S | P ⊆ S}|| ||{S ∈ Sui | P ⊆ S}|| supportS (P ) = supportSui (P ) = ||S|| ||Sui || 3 Cf. section 4.1 Using Closed Itemsets for Implicit User Authentication in Web Browsing 5 For a given user the relative strength of a pattern is equivalent to the lif t in a context of association rules (i.e. the support of the pattern within this user divided by the support of the pattern across all users). More formally: supportSui (P ) lif tSui S (P ) = supportS (P ) The support measures the strength of a pattern in behavioral description of a given user. The relative support mitigates support measure by considering the pattern's support on the whole sessions set. The stronger the global support of a pattern, the lesser characteristic of a specic user. The tf-idf is a numerical statistic that is intended to reect how relevant a word is to a document in a corpus. The tf-idf value increases proportionally to the number of times a word appears in the document, but is oset by the frequency of the word in the whole corpus ([14]). In our context, a word becomes a pattern, a document becomes a set of sessions Sui of a given user and the corpus becomes the whole set S of all sessions. Denition 3 (tf ×idf ). Let P be a pattern, let U be a set of users and Up ⊆ U such that ∀ui ∈ Up , supportSui (P ) 6= 0. Let Sui be a set of sessions of a given user ui and S a whole set of sessions. The normalized term frequency denoted tf (P ) is equal to supportSui (P ) and the inverse document frequency denoted idf (P ) is equal to log (||U ||/||UP ||). We have:   ||U || tf × idf (P ) = supportSui (P ) × log ||UP || Denition 4 (Closure system). Let S be a collection of sessions on the set W of web sites. We denote S c the closure under intersection of S . By adding W in S c , S c is called a closure system. Denition 5 (Closure operator). Let W be a set, a map C: 2W → 2W is a closure operator on W if for all sets A and B in W we have: A ⊆ C(A), A ⊆ B =⇒ C(A) ⊆ C(B) and C(C(A)) = C(A). Theorem 1. Let S c be a closure system on W . Then the map CS dened on c 2W by ∀A ∈ 2W , CS c (A) = {S ∈ S c | A ⊆ S} is a closure operator on W 4 . T Denition 6 (Closed pattern5 ). Let S c be a closure system on W and CS c its corresponding closure operator. Let P be a pattern (i.e. a set of visited sites), we said that P is a closed pattern if CS c (P ) = P . 4 Refer to the book of [15]. 5 This denition is equivalent to a concept of the formal context K = (S,W,I) where S is a set of objects, W a set of attributes and I a binary relation between S and W [16]. 6 O. Coupelon, D. Dia, F. Labernia, Y. Loiseau, and O. Raynaud 3.2 Own patterns selection The rst and most important step of our model, called own patterns selection is to calculate the set of own patterns for each user ui . This set of patterns is denoted Pui = {Pi,1 , Pi,2 ,..., Pi,p }. In [1], the author states that p = 10 should be a reference value and that beyond this value model performance are stable. We shall follow that recommendation. In [1], 10 frequent 1 − patterns are selected for each user. The aim of our study is to show that it could be more ecient to select closed k − patterns. But, the number of closed patterns should be strong, so we compare three heuristics (H1 , H2 and H3 ) to select the 10 closed patterns of each user. For each heuristic, closed patterns are computed thanks to Charm algorithm ([17]) provided on the Coron platform ([18]). Only closed patterns with a size lower than or equal to 7 are considered. These heuristics are presented here: 1. 10 1 − patterns with the largest support values (as in [1]) 2. H1 : 40 closed k − patterns with the largest tf-idf values. 3. H2 : 10 ltered closed k − patterns with the largest support and maximal values by inclusion set operator. 4. H3 : 10 ltered closed k − patterns with the largest tf-idf and minimal values by inclusion set operator. Algorithm 1 describes the process of H1 to select the 40 own patterns for a given user. With H1 , the model performance is improved when p increases up to 40. p = 10 is the better choice for H2 and H3 . The best results are from H1 . Algorithm 1: H1 : 40 closed k − patterns with the largest tf-idf values. Data: Cu : the set of closed itemsets of user ui from Charm; i p : the number of selected own patterns; Result: Pui : the set of own patterns of user ui ; 1 begin 2 Compute the tf × idf for each pattern from Charm; 3 Sort the list of patterns in descending order according to the tf × idf value; 4 Return the top p patterns; 3.3 User proles computation S We dene and we denote Pall = i Pui the whole set of own patterns. The set Pall allows us to dene a common space in which all users could be embedded. More formally, Pall denes a vector space V of size all = ||Pall || where a given user ui is represented as a vector Vui = (mi,1 ,mi,2 ,...,mi,all ). The second step of our model, called user prole computation, is to compute, for each user ui , a numerical value for each component mi,j of the vector Vui . i is the user id, j ∈ [1,all] is a pattern id and m stands for a given measure. In this paper, we compare two measures proposed in [1]: the support and the lift. Using Closed Itemsets for Implicit User Authentication in Web Browsing 7 mi,j = supportSui (Pj ) and mi,j = lif tSui S (Pj ) 3.4 Authentication stage In our model, the authentication step is based on the identication. For that purpose, our model guesses the user corresponding to an anonymous set of ses- sions, then it checks if the guessed identity corresponds to the real identity. From this set of sessions we have to build a test prole and to nd the nearest user prole dened during the learning step. Test sessions Performance of our models are calculated on anonymous data sets of growing size.The more information available, the better the classication will be. The rst data set consists of only one session, the second consists of 10 sessions, the third one consists of 20 sessions, and the last one consists of 30 sessions. For the test phase, all sessions have the same size of 10 sites. Building test prole Let S be the whole set of sessions from the learning data set. Let Sut be an anonymous set of sessions and Vut = (mt,1 ,mt,2 ,...,mt,all ) its corresponding prole vector. We will compare two approaches to build the anonymous test prole, the support and the lift: supportSut (Pi ) ∀i, mt,i = supportSut (Pi ) and ∀i, mt,i = lif tSut S = supportS (Pi ) Distance functions Let Vu = (mi,1 ,mi,2 ,...,mi,all ) and Vu = (mt,1 ,mt,2 ,...,mt,all ) i t be two proles. We denoted DisEuclidean (Vui ,Vut ) the Euclidean distance and we denote SimCosine (Vui ,Vut ) the cosine similarity function. We have: sX DisEuclidean (Vui ,Vut ) = (mt,j − mi,j )2 j P j (mt,j × mi,j ) SimCosine (Vui ,Vut ) = qP 2× 2 P j (mt,j ) j (mi,j ) 4 Experimental results 4.1 Data set Our data set is comprised of the web navigation connection logs of 3,000 users over a six-month period. We have at our disposal the domain name visited and each user ID. From the variables of day and time of connection we have con- structed connection sessions for each user. A session is therefore a set of web 8 O. Coupelon, D. Dia, F. Labernia, Y. Loiseau, and O. Raynaud sites visited. The number of visited web sites per session is limited and equal to 10. For the relevance of our study we used Adblock 6 lters to remove all do- mains regarded as advertising. The majority of users from this data set are not suciently active to be of relevance. Therefore, as in [1], we have limited our study to the 2% of most active users and obtained the signicant session sets for 52 users. The 30 users most active (who have a large number of sessions) among those 52 users are used in this paper. Table 2 gives the detailed statistics for this data set. 7698 sessions Minimum Maximum Mean Standard deviation Size 10 10 10 0 #sessions/users 101 733 257 289 Table 2. Descriptive statistics of the used data set: size of sessions (number of visited web sites) and number of sessions per user, for 30 users. 4.2 Experimental protocol: a description Algorithm 2 (see appendix) describes our experimental protocol. The rst loop sets the size of the set of users among which a group of anonymous sessions will be classied. The second one sets the size of this sessions group. Finally, the third loop sets the number of iterations used to compute the average accuracy rate. The loop on line 10 computes the specic patterns of each user and establishes the proles vector. The loop on line 13 computes the vector's components for each user. The nested loops on lines 16 and 18 classify test data and compute the accuracy rate. 4.3 Comparative performance of H1 , H2 and H3 From own patterns of each user we compute the set Pall as the whole set of own patterns which denes the prole vector of each user. We use the support of a pattern as numerical value for each components (cf. section 3.3). Following Table 3 provides the size of the prole vector and the distribution of own patterns according to size for each heuristic. With 30 users and 10 own patterns per user, the maximal size of the prole is 300. Number of own patterns |1| |2| |3| |4| |5| |6| |7| H1 199 18% 31% 26% 16% 7% 2% 0% H2 167 57% 29% 9% 3% 1% 1% 0% H3 199 24% 20% 18% 14% 10% 9% 5% Table 3. Prole vector size and the distribution of own patterns according to size. Using Closed Itemsets for Implicit User Authentication in Web Browsing 9 80 Accuracy 60 40 20 Bayes Charm H3 Charm H2 Charm H1 0 5 10 15 20 25 30 35 Number of test sessions Fig. 2. Comparative performance of H1 , H2 and H3 . These observations are plotted on an X-Y graph with number of sessions of the anonymous set on the X-axis and accuracy rate on the Y-axis. Measured values are smoothed on 50 executions. Figure 2 shows that naive Bayes classier is the most eective if the group of test sessions is from 1 to 13 sessions (10 to 130 visited web sites). This result is in line with the study in [1]. Finally, this graph clearly shows that heuristic H1 certainly stands out from H2 and H3 . So, the best heuristic is to choose owns patterns amongst closed patterns with the largest tf × idf values. As a consequence, the majority of patterns are small-sized patterns (two or three sites) (cf. Table 3). But accuracy rates are much higher. 4.4 Comparative performance with [1] In [1], the author compares, in particular, two methods of prole vector calculus. In both cases, the own patterns are size 1 and are chosen amongst the most fre- quent. The rst method, named support-based proling, uses the corresponding support pattern as the numerical value for each component of the prole vector. The second method, called lift-based proling, uses the lift measure. In order to compare the performances of the H1 model with the two models support-based proling and lift-based proling, we have accurately replicated the experimental protocol described in [1] on our own data set. The results are given in Table 4. The data of Table 4 highlight that the H1 heuristic allows rates that are perceptibly better than those of the two models proposed in [1] in all possible scenarios. Nevertheless, the Bayes classier remains the most ecient when the session group is size 1 in compliance with [1]. Figure 3 allows a clearer under- standing of the moment the Bayes curve crosses the H1 heuristic curve. 6 http://adblock-listefr.com/ 10 O. Coupelon, D. Dia, F. Labernia, Y. Loiseau, and O. Raynaud # of users 1 10 20 30 Support 65 89 95 97 2 Lift 67 90 97 98 Charm H1 72 98 99 100 Bayes 85 99 73 61 Support 40 74 83 88 5 Lift 41 78 86 88 Charm H1 49 90 95 98 Bayes 67 96 56 34 Support 27 66 79 80 10 Lift 29 64 77 80 Charm H1 37 83 92 94 Bayes 54 91 51 24 Support 19 55 68 75 20 Lift 21 58 68 74 Charm H1 30 76 86 90 Bayes 43 87 48 19 Support 16 53 64 70 30 Lift 17 54 64 69 Charm H1 26 72 83 89 Bayes 39 83 46 19 Table 4. On left, we nd the number of users and the selected model. Each column is dened by the number of sessions of the anonymous data set. Sessions are of size 10. Measured accuracy rate are smoothed on 100 executions. In bold the best values are presented. 80 Accuracy 60 40 Support Lift 20 Bayes Charm H1 0 2 4 6 8 10 12 14 16 18 20 22 Number of test sessions Fig. 3. Comparative performance of Bayes, support-based proling, lift-based proling and H1 . These observations are plotted on an X-Y graph with number of sessions of the anonymous set on the X-axis and accuracy rate on the Y-axis. Number of users is equal to 30. Measured values are smoothed on 50 executions. Using Closed Itemsets for Implicit User Authentication in Web Browsing 11 4.5 Comparative performance of distance functions The last gure 4 shows the impact of distance function choice on performances of models. 80 60 Accuracy 40 Bayes Lift (cosine) 20 Lift (euclidean) Charm H1 (cosine) Charm H1 (euclidean) 0 5 10 15 20 25 30 35 Number of test sessions Fig. 4. Comparative performance of both H1 with cosine similarity and Euclidean distance, Bayes and lift-based proling. These observations are plotted on an X-Y graph with number of sessions of the anonymous set on the X-axis and accuracy rate on the Y- axis. Number of users is equal to 30. Measured values are smoothed on 100 executions. Figure 4 illustrates the signicance of the distance function concerning the performance. Indeed, when used with Euclidean distance, the H1 method is a bit more precise than the lift one (about 3%). However, performances are improved by using the cosine similarity and their relative ranking is even reversed. H1 method's performance are then better than lift by 10%. 5 Conclusions and future work In this study, we proposed a learning model for implicit authentication of web users. We proposed an simple and original algorithm (cf. Algorithm 1) to get a set of own patterns allowing to characterize each web user. The taken patterns have dierent size and qualify as closed patterns from closure system generated by the set of sessions (cf. Table 3). By reproducing experimental protocol described in [1], we showed that the performances of our model are signicantly better than some models proposed in the literature (cf. Table 4). We also showed the key role of the distance function (cf. Figure 4). This study should be extended in order to improve the obtained results. For a very small sites ow, the results of the solution should be better than results 12 O. Coupelon, D. Dia, F. Labernia, Y. Loiseau, and O. Raynaud from Bayes' method. Another way to improve results will be to select other types of variable and to add them to our current dataset. The selection of data has an undeniable impact on the results. References 1. Yang, Y.C.: Web user behavioral proling for user identication. Decision Support Systems (49) (2010) pp. 261271 2. Guvence-Rodoper, C.I., Benbasat, I., Cenfetelli, R.T.: Adoption of B2B Ex- changes: Eects of IT-Mediated Website Services, Website Functionality, Benets, and Costs. ICIS 2008 Proceedings (2008) 3. Lagier, F.: Cybercriminalité : 120.000 victimes d'usurpation d'identité chaque année en france. Le populaire du centre (in French) (2013) 4. Filson, D.: The impact of e-commerce strategies on rm value: Lessons from Ama- zon.com and its early competitors. The Journal of Business 77(S2) (2004) pp. S135S154 5. He, R., Yuan, M., Hu, J., Zhang, H., Kan, Z., Ma, J.: A novel service-oriented AAA architecture. 3 (2003) pp. 28332837 6. Stockinger, T.: Implicit authentication on mobile devices. The Media Informatics Advanced Seminar on Ubiquitous Computing (2011) 7. Shi, E., Niu, Y., Jakobsson, M., Chow, R.: Implicit authentication through learning user behavior. M. Burmester et al. (Eds.): ISC 2010, LNCS 6531 (2011) pp. 99113 8. Jakobsson, M., Shi, E., Golle, P., Chow, R.: Implicit authentication for mobile devices. Proceeding HotSec'09 Proceedings of the 4th USENIX conference on Hot topics in security (2009) pp. 99 9. Ullah, I., Bonnet, G., Doyen, G., Gaïti, D.: Un classieur du comportement des utilisateurs dans les applications pair-à-pair de streaming vidéo. CFIP 2011 - Colloque Francophone sur l'Ingénierie des Protocoles (in French) (2011) 10. Goel, S., Hofman, J.M., Sirer, M.I.: Who does what on the web: A large-scale study of browsing behavior. In: ICWSM. (2012) 11. Kumar, R., Tomkins, A.: A characterization of online browsing behavior. In: Proceedings of the 19th international conference on World wide web, ACM (2010) pp. 561570 12. Abramson, M., Aha, D.W.: User authentication from web browsing behavior. Pro- ceedings of the Twenty-Sixth International Florida Articial Intelligence Research Society Conference pp. 268273 13. Herrmann, D., Banse, C., Federrath, H.: Behavior-based tracking: Exploiting char- acteristic patterns in DNS trac. Computer & Security (2013) pp. 117 14. Salton, G.: Automatic text processing: The transformation, analysis and retrieval of information by computer. Addison Wesley (1989) 15. Davey, B.A., Priestley, H.A.: Introduction to lattices and orders. Cambridge University Press (1991) 16. Ganter, B., Wille, R.: Formal concept analysis, mathematical foundation, Berlin- Heidelberg-NewYork et al.:Springer (1999) 17. Zaki, M.J., Hsiao, C.: Ecient algorithms for mining closed itemsets and their lattice structure. IEEE Transactions on knowledge and data engineering 17(4) (2002) pp. 462478 18. Szathmary, L.: Symbolic Data Mining Methods with the Coron Platform. PhD Thesis in Computer Science, University Henri Poincaré  Nancy 1, France (Nov 2006) Using Closed Itemsets for Implicit User Authentication in Web Browsing 13 Appendix Algorithm 2: Experiment procedure Data: i Su : all sessions from n users; S i X : number of successive executions; Result: The mean accuracy of select models; 1 begin 2 for (N = {2, 5, 10, 20, 30}) do 3 for (S = {1, 10, 20, 30}) do 4 for (z = 1, . . . ,X ) do 5 Select N random users; 6 For each user, select SN = min(|Sui |, i = 1, . . . ,N ); 2 7 Take the 3 of the SN sessions from each users to form the training set; 8 Take the rest of SN sessions to form the validation set; 9 k Pall ← ∅ (the global prole vector for each model k ); 10 for each (ui , i = 1, . . . ,N ) do 11 k k Compute the own patterns Pu (1 ≤ |Pu | ≤ 10); i i 12 k k k Pall ← Pall ∪ Pui ; 13 for each (ui , i = 1, . . . ,N ) do 14 Compute the vector Vu k i with support or lift; 15 Initialize to 0 the confusion matrix M k of the method k ; 16 for each (ui , i = 1, . . . ,N do 17 Compute the test stream Tui (|T | is xed, T ∈ Tui ); 18 while (Tu 6= ∅) do i 19 Take SW sessions from Tui to compute VT ; k 20 ua ← max(simil(Vuki ,VTk )) or min(dist(Vuki ,VTk )); 21 M k [ui ][ua ] ← M k [ui ][ua ] + 1; 22 Compute the mean accuracy of k from M ; k