Action rule mining is an extension of the widely used task of learning classi cation rules. In addition to information expressed in a standard classi cation rule, an action rule suggests a course of action. If performed, this action will increase the probability that the class of the instance will change to the desired value. Such rule can either be interpreted as a recommendation for an action, or as a counterfactual explanation for the class assigned to the instance. In this paper, we report on a new implementation of action rules discovery that is available in Python (ActionRules package) and on a new experimental method for learning action rules from large datasets (RandomForestRules package), which is based on extraction of classi cation rules from Random Forests. The paper can serve as a manual for using the created packages or as a guide for researchers who would like to extend them, providing also guide to action rule discovery. The text also includes performance evaluation of reduction trees, which speed up the mining process.
An action rule describes how an action (a change in the value in one or more
exible attributes) could impact the classi cation of a given object. Similarly as
standard classi cation rules [
Considering the well-known Titanic dataset1, an example of a generated action rule is: \If attribute `Sex' is `female', attribute `Embarked' value `Southampton' is changed to `Cherbourg', attribute `Passenger Class' value `3' is changed to `1', then `Survived' value `0' is changed to `1' with support: 5%, con dence: 58% and uplift: 6%." This action rule can either serve as recommendation for a course of action to take to increase the probability of survival, or more ttingly on this retrospective dataset, a counterfactual (\what if") explanation for the classi cation.
To the best of the author's knowledge, the only publicly available implementation of action rules is LISp-Miner2 developed at the University of Economics in Prague, which is primarily focused on Windows users who use it via a Graphical User Interface. In this paper, we introduce an alternative implementation of action rule mining for Python.
The ActionRules package3 covered in this paper implements the action rules algorithm as described in [9] with a performance improvement by a reduction tree described in [10]. We also describe its experimental extension available as RandomForestRules package4, which extracts classi cation rules from tree ensembles generated by Random Forests. The packages are accompanied by several Jupyter Notebooks, demonstrating di erent use cases of the software. The packages are available under an open license (MIT).
The structure of the work is the following. In Section 2, we motivate the problem of action rule mining. Section 3 gives the details of the algorithmic approach used. In Section 4, we present a proposal for a new method for extracting association rules from Random forests. Section 5 describes the new implementation of action rules, which is the main contribution of this paper. Section 6 describes the attempted performance improvements: reduction trees and extracting action rules from Random Forests. In the conclusions, we summarize the contributions. 2
Algorithmically, there are two principal approaches to action rule mining [
The rule-based approach is divided into two independent steps. In the rst
step, classi cation rules are mined by any suitable association rule mining
algorithm (for example Apriori [
In the object-based approach, action rules are discovered directly from the
source dataset. Speci c algorithms include LERS (Learning from Examples based
on Rough Sets) or ERID (Algorithm for Extracting Rules from Incomplete
Decision System) with atomic action sets [
In the software package introduced in this paper, we adopt the rule-based approach, which is in our opinion more modular. There is a range of highly performing association rule mining algorithms that can be adopted for the rst step of classi cation rule mining.
2.1
In the rst step of the rule-based approach, classi cation rules are mined. A formal representation of a classi cation rule containing four conditions is: r = [(a1 ^ b1 ^ c1 ^ e1) ) d1]; (1) where a1, b1, c1, e1 are values of features (attributes) in the antecedent of the rule, and d1 is a value of the target variable predicted by the consequent of the rule.
Each discovered rule is accompanied by values of support and con dence that express its quality.
The support of a rule ant ) d1 (denoted as sup(ant ) d1)) is a number of transactions (rows in training data set) that match both the antecedent and consequent of the given rule.
The con dence of a rule ant ) d1 (denoted as conf (ant ) d1)) is expressed as sup(ant ) d1)=sup(ant). It corresponds to a ratio between the number of transactions that satisfy the antecedent as well as the consequent to the number of transactions that satisfy the antecedent of the rule (denoted as sup(ant)). 2.2
In the second step, action rules are formed from a subset of discovered classi cation rules.
An action rule can be represented as: r = [(!) ^ ( ! )] ) [ ! ]; (2) where ! is a xed condition (set of stable attributes), which describes the object, ( ! ) is the proposed to change to a subset of user-designated exible attributes and ( ! ) represents the implied change to the target attribute.
The quality of action rules can be represented using the following
measures. Let r be an action rule, which was generated from two classi cation
rules r1 and r2. We compute the support of the action rule r as sup(r) =
min(sup(r1); sup(r2)). The con dence of the action rule r is computed as conf (r) =
conf (r1) conf (r2). These de nitions were adopted from [
Note that the de nitions of support and con dence for action rules are different from these de nitions for classi cation rules. 2.3
Below is an example of two classi cation rules discovered from the Titanic dataset.
r1 = [(Sex : f emale ^ Embarked : S ^ P class : 3) ) Survived : 0]; with support 10% and conf idence 55%: (3) r2 = [(Sex : f emale ^ Embarked : C ^ P class : 1) ) Survived : 1]; with support 6% and conf idence 86%: (4)
Rule r1 can be interpreted as follows: women who embarked on a voyage in Southampton in the third class did not survive with probability of 55%. Rule r2 can be interpreted as follows: women who embarked on a voyage in Cherbourg in the rst class survived with probability of 86%.
Based on these two classi cation rules, the following action rule can be generated: r = [(Sex : f ) ^ (Embarked : S ! C) ^ (P class : 3 ! 1)] ) [Surv: : 0 ! 1]; with support 5 %; con dence 59 % and uplift 5:7%: (5)
This action rule can be interpreted as follows: women who travelled from Southampton in the third class would have increased their chances of survival if they had changed their boarding place to Cherbourg and had paid extra money for the rst class. 3
In the following, we will outline the phase 2 of the rule-based approach for action rule generation, in which the algorithm attempts to form action rules from all pairs of classi cation rules.
The process is the following: { Discovered classi cation rules are (conceptually) represented as a table where each condition in the antecedent is a column. There is also one column for consequent (target variable). { Conditions in the antecedent are divided into stable and exible. Stable conditions do not allow any change of their state (no action can be taken).
Flexible conditions allow changing their state (the action can be taken). { The table of classi cation rules is divided into two tables. The rst table X contains classi cation rules without the desired state. In the second table X , there are classi cation rules with the desired state. { Classi cation rule pairs are created as a Cartesian product of the rows in these two tables X X . Each of these rowcount(X ) rowcount(X ) classi cation rule pairs is a candidate for an action rule. The action rule candidate is therefore formed from a rule predicting other than a desired class state before and a rule predicting a desired class state after. { The algorithm loops through all candidates. If there are no pairs left, the algorithm nishes, and returns the saved action rules. { On each loop, all the stable antecedent pairs and exible antecedent pairs are checked. If the conditions given in Section 3.1 are satis ed, the pair is changed to an action rule and saved. 3.1
There are several approaches to check whether a pair of classi cation rules
satis es the conditions and can be used for generation of an action rule.
Baseline approach The approach, which we consider as a baseline, is described
in [
Values of the target attribute in the consequent must comply with the user setting. For example, when generating action rules for the Titanic dataset, we would like to change the target state from Survival: No to Survival: Yes, but not in the opposite way.
Extended action rules An alternative approach described in [9] may generate more action rules, because it allows to generate candidates from the input classication rules even if they have missing values. For example, this approach would generate rules such as: \If any `Pclass' value is changed to `1', then `Survived' value `0' is changed to `1'." 3.2
A pre-processing step for speeding up the phase 2 of action rule mining is described in [10]. This step is implemented in the software package introduced in this paper.
This approach is based on the splitting of the table of all classi cation rules to multiple small tables, which allows pruning the search space. For each stable attribute, one table with rules is generated, since the values of stable attributes must always be the same in the state before and the state after in the action rule candidate pair. Some tables can be eliminated because they do not contain the desired target value, or they do not have enough variability, see Figure 1.
A challenge for classi cation rule mining are datasets containing many features and many rows. In this case, association rule classi cation algorithms used in phase 1 of action rule generation can fail due to excessive combinatorial complexity.
Random Forest [
Classi cation rules can be extracted from a Random Forest model and used as an input to phase 2 of action rule generation. The proposed algorithm for generation of action rules from Random Forests based on this principle is outlined in the following: { Features in the input data are subject to one-hot encoding. This replaces each multinominal column with n distinct values with n 1 binary columns. { Random Forest classi er is learnt on the recoded dataset. { All decision trees are extracted from the Random Forest Classi er. { Extracted decision trees are transformed to classi cation rules. { Classi cation rules not meeting minimum support and minimum con dence thresholds are removed.
According to preliminary experiments, extraction of classi cation rules from a Random Forest is much faster on datasets, which are di cult to process for association rule mining algorithms, such as Apriori. The disadvantage of this approach is that the the standard algorithm for inducing Random Forests is neither deterministic nor exhaustive. Association rule mining algorithms guarantee that a complete set of rules valid with respect to user-set minimum con dence and minimum support thresholds is found. Furthermore, there is no random element involved in association rule mining. 5
The ActionRules package allows two ways of operation. Either the complete mining work ow can be executed, using Apriori for phase 1, or only action rule generation (phase 2) can be initiated from classi cation rules supplied as a data frame.
The ActionRules package and RandomForestRules package are available in PyPi (Python Package Index). They can be installed by the following commands: pip install actionrules-lukassykora pip install randomForestRules-lukassykora 5.1
The package for action rules mining uses PyFIM5 library as a default option for classi cation rule discovery. The PyFIM package returns an exhaustive list of classi cation rules used as input for action rule generation.
The ActionRules package uses the same general machine learning work ow
as Scikit-Learn [
Firstly, the model is instantiated.
Code 1 Instantiate model object 1 from actionrules.actionRulesDiscovery import ActionRulesDiscovery 2 3 actionRDiscovery = ActionRulesDiscovery()
In the next step, the model is t to training data. In this case, Titanic dataset is used. All features are pre-processed to be nominal.
In the example listing below, the stable attribute is Age, exible attributes are Embarked, Fare and Pclass, the consequent is set to Survived. Minimum con dence is set to 55% and minimum support to 3%. These thresholds are used only for generating classi cation rules (phase 1). Desired class for Survived is 1. Mining of extended action rules is not enabled (is nan=False) and reduction trees are used to speed up phase 2 (is reduction=True). Both the minimum number of stable attributes and the minimum number of exible attributes in action rules is set to 1.
Code 2 Fit model to training data 4 actionRDiscovery.read csv("data/titanic.csv", sep="nt") 5 actionRDiscovery. t(stable attributes = ["Age"], 6 exible attributes = 7 ["Embarked", 8 "Fare", 9 "Pclass"], 10 consequent = "Survived", 11 conf=55, 12 supp=3, 13 desired classes = ["1"], 14 is nan=False, 15 is reduction=True, 16 min stable attributes=1, 17 min exible attributes=1)
This is the last step of the action rule mining work ow.
Code 3 Predict & Evaluate 18 new/_data = [['32-48', 'S', 'very high' ,'3.0']] 19 df_new_data = pd.DataFrame(new data, columns = ["Age", 20 "Embarked", 21 "Fare", 22 "Pclass"]) 23 actionRDiscovery.predict(df new data)
As a result, the application returns a table with recommended actions (marked with *).
The example below shows how to use the package RandomForestRules together with the ActionRules package. The dataset Audiology 6 is used for demonstration. Mining with Apriori using PyFIM fails on this dataset with the following setting: minimum support: 0.5%, minimum con dence: 50% and unlimited length of nal itemsets. 6 http://archive.ics.uci.edu/ml/datasets/audiology+(standardized) Code 4 ActionRules package combined with RandomForestRulespackage
Output: rule 1. [(age-gt-60: t) ^ (bone: unmeasured) ^ (history-noise: t ! f) ] ) [target: 0 ! 1] with support: 0.018 and con dence: 0.561 rule 2. [(age-gt-60: t) ^ (bone: mild) ^ (history-noise: t ! f) ] ) [target: 0 ! 1] with support: 0.018 and con dence: 0.561 rule 3. [(age-gt-60: t) ^ (bone: ?) ^ (history-noise: t ! f) ] ) [target: 0 ! 1] with support: 0.018 and con dence: 0.561 rule 4. [(age-gt-60: t) ^ (ar-u: absent) ^ (history-noise: f ! t) ^ (o-ar-c: absent ! normal) ] ) [target: 0 ! 1] with support: 0.001 and con dence: 0.666 rule 5. [(age-gt-60: t) ^ (ar-u: absent) ^ (history-noise: f ! t) ^ (o-ar-c: elevated ! normal) ] ) [target: 0 ! 1] with support: 0.001 and con dence: 0.666 rule 6. [(age-gt-60: t) ^ (ar-u: absent) ^ (history-noise: f ! t) ^ (o-ar-c: ? ! normal) ] ) [target: 0 ! 1] with support: 0.001 and con dence: 0.666 6
The evaluation was run on a notebook Dell Latitude E7470 with con guration Intel Core i5-6300U 2.40GHz with 8GB RAM. The Titanic dataset is used for tting the models. 6.1
In this section, we evaluate the e ect of the use of extended action rules and of reduction trees on the performance. We use the ActionRules package for the whole process of action rules discovery.
Setup The minimum con dence was xed to 50%, and the minimum support threshold was varied. The evaluation was performed in four con gurations: reduction trees enabled/disabled, extended association rules enabled/disabled. All con gurations are always tested three times and the median result is used for comparison.
Results Results are reported in Figure 2. The left graph shows the performance for the baseline approach to action rules mining (certainty). The right graph shows the performance for extended action rules mining (uncertainty). The runtime of calculation with `uncertainty' is more than 100x higher, but also the number of found action rules is higher (2443 extended action rules vs. 105 action rules). The reduction trees have a consistent positive e ect on performance. Setup The setting for classi cation rules discovery was the following: minimum support 3% and minimum con dence 55% (these thresholds are used just for ltering of rules extracted from the Random Forest). The package uses the RandomForestClassi er7 from Scikit-learn library. All settings are kept as default, just for n estimators several values (10,20,30) were tested. The author also separately evaluates generation of action rules from a passed list classi cation rules with the following setting: stable attributes = ["Age", "Sex"], exible attributes = ["Embarked", "Fare", "Pclass"], consequent = "Survived", desired classes = ["1.0"].
Results Results are reported in Table 2. As can be seen, only very few unique classi cation rules were extracted from the Random Forest models, which resulted in no discovered action rules. The probable reason is that the Random Forest algorithm creates many duplicate trees (or their parts), which are largely 7 https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.
RandomForestClassifier.html overlapping. If the PyFIM package that uses the Apriori algorithm was used for the same task, 13 action rules were discovered.
On the other hand, generating classi cation rules using RandomForestRules was successful for the Audiology dataset, where application of association rule mining to generate candidates was not successful (cf. Section 5.2). 7
In this paper, we described two new Python packages for action rule mining. The rst package ActionRulesDiscovery is primarily intended for mining of action rules from classi cation rules generated with association rule mining (from class association rules). The second experimental package RandomForestRules extracts classi cation rules from Random Forest models. This package can be combined with ActionRulesDiscovery to facilitate discovery of action rules from very large multidimensional datasets.
In terms of performance, the approach taken in ActionRulesDiscovery
benets from the use of PyFIM package, which wraps the high performing low-level
implementation of Apriori [
In: European Conference on Principles of Data Mining and Knowledge Discovery. pp. 587{592. Springer (2000) 10. Ras, Z.W., Wyrzykowska, E., Wasyluk, H.: Aras: Action rules discovery based on agglomerative strategy. In: International Workshop on Mining Complex Data. pp. 196{208. Springer (2007) 11. Tzacheva, A.A., Ras, Z.W.: Action rules mining. International Journal of Intelligent Systems 20(7), 719{736 (2005)