We evaluate the Machine Coaching paradigm, a human-in-the-loop machine learning methodology, according to which a human coach and a machine engage in an iterative bidirectional exchange of explanations, towards improving the machine's ability to reach conclusions and justify them in a way that is acceptable to the human coach. To support the systematic empirical investigation of the eficacy and eficiency of Machine Coaching, we adopt proxy (algorithmic) coaches in the stead of human ones.
Learning in humans proceeds in many and diverse ways. Under what could be called the
autodidactic paradigm [
Another, significant, part of human learning takes place under what could be called the
coaching-based paradigm [
Such pieces of advice from the coach do not typically come unprompted, but as a reaction
to a wrong or wrongly-justified decision by the learner. A coach ofering advice is, in efect,
proactively completing / enriching missing information in states of the environment that the
learner might encounter, by providing conditions on states under which a certain decision should
be reached. Compared to the autodidactic paradigm, the substantial amount of information
communicated under the coaching-based paradigm is conducive to more eficient, albeit
coachspecific, learning, while the reactive nature of advising entails only marginally extra efort
from the coach, as humans are known to be competent at identifying counter-arguments when
challenged [
Machine Coaching1 [
Undertaking a study with human participants in the role of the coach is, clearly, the ultimate way to empirically validate the Machine Coaching framework. Yet there are three major challenges to overcome in such an envisioned study.
The first challenge stems from the lack of fluency of average humans in formal logic , making
it hard for them to exchange explanations in the native (logic-based) language of Machine
Coaching. This issue could be alleviated by adopting a natural language interface for exchanging
explanations, which would be automatically translated to/from the native language. Although
there is some work in this direction — even using the Machine Coaching framework itself at a
meta-level [
The second challenge stems from the inaccessibility of the target policy that humans have in mind when coaching. Without a target policy, one cannot empirically evaluate the quality of the learned hypothesis policy. If, in addition, no policy agrees with all the advice ofered by the human coach, then the empirical study would confound the validation of the Machine Coaching framework with the determination of the expressivity of the framework’s native language, and with the compatibility of the interaction with human cognition.
The third challenge stems from the physical and mental limitations of humans in prolonged interactions. A systematic evaluation of the Machine Coaching framework would require substantial rounds of interactions between the coach and the machine, across diverse target policies. It is unlikely that humans would maintain eficient and consistent behavior over time, either due to fatigue, an unconscious adaptation to the empirical setting, or a suppression of their natural reactions when coaching with an externally-given target policy.
To eschew these challenges, we resort to the use of a proxy (algorithmic) coach that can
communicate in the native language of Machine Coaching, cope with a given explicit target
policy, and provide advice eficiently and consistently. These features need to be balanced
against the desire to stay close to human coaches, who have only approximate access to their
target policy — conceivably as a compilation, in some intensional or extensional form, of their
relevant life experiences — and are unable to divulge the target policy on cue, but can still react
if some part of it is “challenged” by an external position and associated argument [
Accordingly, we consider proxy coaches with access to an exemplar set of data, labeled with the predictions of a fixed (but hidden to the proxy coaches) target policy. The key technical question, then, is how to extract appropriate pieces of advice from the exemplar set. In the sequel we seek to answer this question by demonstrating how to develop proxy coaches with both an intensional and an extensional compilation of the exemplar data, and continue to evaluate them empirically. We shall remark at this point that our focus in this paper is mostly shifted towards investigating the various alternatives of proxy coaches and their qualitative diferences. Consequently, while we do present and discuss quantitative results for all chosen proxy coaches, they mostly serve as a means to stress the efects each proxy coaching protocol has on the coaching process.
The process whereby an agent learns under the supervision of a more experienced tutor / coach,
who ofers advice as a reaction to the learner’s decisions or actions, appears as a suggestion
for the development of AI systems in John McCarthy’s seminal work on the “Advice Taker”
[
The eXplainable Interactive Learning (XIL) framework [
The idea of online ingestion of human advice in Machine Learning processes is present in
other works as well. A way to incorporate user advice into Support Vector Machines is proposed
in [
Beyond speeding up learning, more recent works examine how human knowledge may
benefit the entire learning process, by being an integral part of it. Coactive Learning, proposed
in [
Although our work herein focuses on the empirical evaluation of Machine Coaching, the technique of proxy coaches that we follow would also seem to be applicable, to varying extents, to the frameworks above. The rest of the works that we briefly review below are not tied to coaching per se, but relate to our chosen implementation of the proxy coaches.
In [
On the other hand, we also consider a proxy coach type that does not compile the available
data into an explicit learned model, but uses them only implicitly. This approach relates to the
works in [
To facilitate the choice of appropriate advice in this implicit learning setting, we use natural
selection over an evolutionary process. The formalization of evolution that we adopt is that in
[
Following the Machine Coaching framework, a (direct) coach and a learner hold, respectively, a fixed target policy and a revisable hypothesis policy, each represented in the form of a partially-ordered set of “if-then” rules. Upon perceiving a new context, the learner returns the predictions of the current hypothesis policy, and an explanation in the form of hypothesis policy rules that support those predictions. The coach responds by ofering advice in the form of target policy rules that support why the learner’s predictions were incorrect, incomplete, or improperly-justified according to the coach. The learner revises the hypothesis policy, and the process repeats.
While arguments within Machine Coaching are generally considered to be arbitrary trees of rules, for the sequel, we restrict our attention to shallow propositional policies, whose rules have a special atom (or its negation) as their head, so that rules are not chained during reasoning — and, thus, each argument comprises of a single rule. Relatedly, we restrict our attention to complete contexts, which specify truth-values for all remaining atoms. Without loss of generality, we hence let the special atom be output.
Thus, if output were the ability to fly, and given a context {penguin, bird}, the learner could predict output by ofering the following explanation: “ bird implies output”, and the coach could react by ofering the following piece of advice: “ penguin implies -output”, which would be integrated with higher priority in the revised hypothesis policy.
Even with these restrictions, policies can still vary along other dimensions, which afect the Stack Stack
policy size = 6 rules rule width = 11 atoms / 6 rules flip count = 4 batch transitions batch size = 6 rules / 5 batches stack depth = 5 batches / 2 stacks conceptual and computational complexity of learning (through Machine Coaching, or otherwise) and reasoning. Policies can vary in terms of their policy size, which counts the number of rules in the policy, their rule width, which counts the average number of atoms in rule bodies, their lfip count , which counts the flips / transitions between batches of consecutive rules with the same head polarity (positive or negative), the batch size, which counts the average number of rules across batches, and the stack depth, which counts the average number of consecutive batches such that rules in a batch have logically more specific conditions than the rules in the preceding batch (cf. Figure 1).
Proxy coaches have only indirect knowledge of some target policy , by being given access to an exemplar set of contexts labeled according to . In broad terms, then, a proxy coach ̂︀ faces a context ∈ from some coaching set , receives the prediction / explanation ℎ() of the learner’s current hypothesis policy ℎ on , and then uses ̂︀ to generate a piece of advice (, ℎ(); ̂︀) to be ofered to the learner. An epoch concludes whenever a piece of advice is ofered, noting that the proxy coach need not ofer advice for each ∈ .
The precise manner on how (, ℎ(); ) is generated depends on the proxy coach type, but ̂︀ it generally seeks to consider the anticipated efect that each candidate piece of advice would have on the learner’s hypothesis policy, in terms of: (i) improving its coverage, by reducing the number of contexts on which it abstains; and (ii) improving its accuracy, by reducing the ratio of wrong over correct predictions it makes. 3.1. Intensional Proxy Coaches The first class of proxy coaches that we consider encode intensionally their knowledge of the target policy, by using the exemplar set ̂︀ once, before the coaching phase, to supervise the training of a white-box model . The ante-hoc explainability of makes it, then, natural to seek to “extract” a piece of advice (, ℎ(); ) from the explanation of (), given a context ̂︀ ∈ . The exemplar set, therefore, takes the role of a training set. Coaching proceeds, then, as in Algorithm 1.
Our chosen white-box models are those of decision trees and random forests, from which we have identified three natural ways of extracting advice as part of Step 6 of Algorithm 1.
Our first approach considers a white-box model based on a single decision tree. Given the current context ∈ , it identifies the active path of the tree, from which it computes, as usual, Algorithm 1 Intensional Proxy Coaching 1: input: exemplar set ; coaching set .
̂︀ 2: Train white-box model , using as training instances.
̂︀ 3: for the next context ∈ do 4: Ask for learner’s prediction / explanation ℎ(). 5: if predictions of () and ℎ() difer then 6: Get advice from explanations of () and ℎ(). 7: Give advice to revise current hypothesis policy ℎ. 8: end if 9: end for · · · the prediction of (). The full path itself acts as the explanation of (), and is returned as a piece of advice. By construction, the advice so generated is “cautious”, in that a full path in decision trees tends to have high accuracy on the exemplar set, as otherwise the decision tree learning algorithm would have chosen to expand the path further. For the same reason, however, the advice is also “specific”, in that it applies only on a few contexts because of the path length. Overall, then, this type of a proxy coach chooses to provide advice based on the expectation that the hypothesis policy will improve its coverage only marginally, but it will be highly efective in improving and maintaining its accuracy.
Our second approach is a variant of the first, where instead of considering the full active path of the decision tree as the (unique candidate for a) piece of advice, it considers all of its prefixes as candidates. Among them, it chooses the minimal one that is simultaneously more specific than the explanation of ℎ(), and such that the prediction of the decision tree would have remained the same had the active path been pruned to be that particular prefix. By construction, the advice so generated is more “loose”, in that a prefix of a path in decision Algorithm 2 Extensional Proxy Coaching 1: input: exemplar set ; coaching set .
̂︀ 2: Initialize the current hypothesis policy ℎ to be empty. 3: for the next context ∈ do 4: Create mutations of types (M0), (M+), (M↓). 5: Compute fitness score of each mutation ℎ . 6: Select “good” beneficial or neutral mutation ℎ. 7: Set the current hypothesis policy to be ℎ.
8: end for trees tends to have lower accuracy on the exemplar set, which is why the decision tree learning algorithm had chosen to expand the prefix further. For the same reason, however, the advice is also more “general”, in that it applies on more contexts because of the shorter prefix length. Overall, then, this type of a proxy coach chooses to provide advice based on the expectation that the hypothesis policy will improve its coverage measurably, but at the cost of being less efective in improving and maintaining its accuracy.
Our third approach considers a white-box model based on a random forest. Given the current context ∈ , it computes, as usual, the prediction of (). It then proceeds to identify the active path from each individual tree whose prediction matches that of (), and considers those full paths as candidates. There are many strategies for choosing one of the candidates, but for concreteness, our particular approach chooses the one that is most “specific” (breaking ties randomly), in that it is activated by as few contexts as possible in the exemplar set. Given that paths in trees in random forests are typically short and hence “general”, the approach tends to favor the generation of a “maximally cautious” advice among “typically loose” candidates. Overall, then, this type of a proxy coach chooses to provide advice based on the expectation of striking a balance between the improvement of the coverage and accuracy of the hypothesis policy. 3.2. Extensional Proxy Coaches The second class of proxy coaches that we consider retain extensionally their knowledge of the target policy, without compiling it into another form. Rather, they use the exemplar set ̂︀ repeatedly, during the coaching phase, to decide whether a candidate piece of advice should become (, ℎ(); ), given a context ∈ . The exemplar set, therefore, takes the role of a ̂︀ validation set. Coaching proceeds, then, as in Algorithm 2.
This extensional perspective suggests a trial-and-error view of coaching, where the coach tests candidate pieces of advice to measure their efect. Since a systematic testing of all possible pieces of advice is infeasible, the challenge for the coach is to select the set of candidates, and to identify how to test the efect of each candidate on the hypothesis policy.
Our approach adopts an evolutionary mechanism, where, roughly, the process of mutation corresponds to the generation of the candidate pieces of advice, and the process of natural selection corresponds to their testing towards selecting (, ℎ(); ). We discuss below in more ̂︀ detail the nuances that come from the fact that the same evolutionary mechanism needs to simulate both the proxy coach and the learner.
At the start of a generation, the population comprises the current hypothesis policy ℎ. Given the current context ∈ , certain candidate pieces of advice are considered, and each is “provisionally” ofered as advice to a diferent copy of ℎ, giving rise to its ofspring. Specifically: (M0) an ofspring is created by ofering no advice; (M+) an ofspring is created by ofering the advice “ implies output”, if ℎ() predicts -output or abstains; (M+) an ofspring is created by ofering the advice “ implies -output”, if ℎ() predicts output or abstains; (M↓) an ofspring is created, for each , by ofering the advice “ body− implies head”, where “body implies head” is the latest advice that was integrated in ℎ, and body− is body minus its -th literal.
These aforementioned pieces of advice are said to be ofered “provisionally” in the sense that the proxy coach may, at a subsequent generation, seek to refine a previously given piece of advice by means of type (M↓) mutations. A piece of advice is “conclusively” given at the end of a streak of (zero or more) type (M0) or (M↓) mutations that follow a type (M+) mutation. This is the point at which an epoch concludes.
At each generation, exactly one of the ofspring is chosen to survive to initialize the next generation. To support this selection process, each ofspring ℎ is evaluated in terms of its improvement in accuracy and coverage relative to its parent against each exemplar ∈ ̂︀. The change from ℎ() to ℎ() is considered: positive, if a wrong prediction changes to a correct one or an abstention, or an abstention changes to a correct prediction; negative, if a correct prediction changes to a wrong one or an abstention, or an abstention changes to a wrong prediction. By giving a +1 or − 1 fitness point to ofspring ℎ for each respective positive or negative change across the exemplar set , we end up with an intuitive metric that aggregates ̂︀ the improvement efect of the advice given to ofspring ℎ on both its coverage and its accuracy.
The ofspring are, then, grouped based on the relation of their relative fitness to a fixed threshold parameter . An ofspring ℎ is detrimental, neutral, beneficial if its relative iftness belongs in (−∞ , − ), in [− , +], in (+, +∞), respectively. Among the beneficial ones, if available, the ofspring ℎ is selected to survive with probability / ∑︀ , where the exponent is a fixed non-linearity parameter. Otherwise, a neutral ofspring (whose existence is guaranteed by a type (M0) mutation) is selected uniformly at random.
The proposed approach searches greedily for advice that is as “cautious” and as “general” as possible, while starting the greedy search from a fully “cautious” and “specific” seed advice selected by a type (M+) mutation. Overall, then, this type of a proxy coach chooses to provide advice based on the expectation of striking a balance between the improvement of the coverage and of the accuracy of the hypothesis policy.
We present below the key aspects and results of our empirical investigation. Additional details are given in the appendices. All related materials may be found at https://github.com/VMarkos/ proxy-coaching-argml-2022. 4.1. Empirical Setting We first fixed a set of 20 atoms (other than output) for use in contexts and rule bodies. We then constructed 10 groups with 20 policies each, with each pair of groups corresponding to high versus low values in one of the variability dimensions discussed in Section 3, giving rise to a set of 132 distinct target policies. For each target policy ∈ , we constructed an associated exemplar set ̂︀ by uniformly at random sampling 1000 contexts from 2, labeling each context according to , filtering out any contexts on which abstained, and keeping 70% of the labeled contexts. The other 30% was used to populate an evaluation set ℰ . An additional 500 (unlabeled) contexts were sampled to populate a coaching set .
We ran an experiment for each pairing of a target policy ∈ with a proxy coach type
discussed in Section 3: (E1) a decision tree with “full path” advice, (E2) a decision tree with “min
prefix” advice, (E3) a random forest, and (E4) an evolutionary mechanism. Decision trees in
experiments (E1), (E2), and (E3) were trained on the exemplar set ̂︀ using ID3 [
The current hypothesis policy was examined at the end of each epoch in each experiment. Performance values recorded how many of the predictions of the current hypothesis policy on the evaluation set ℰ were correct against , wrong against , or abstentions, and the accuracy of the definite predictions (i.e., the number of correct predictions over the number of correct or wrong predictions). Relative Size values recorded the size of the current hypothesis policy (relative to the size of ), and the size of the epoch (relative to the size of ), the size of each given piece of advice (relative to the size of ).
The above values for each proxy coach type were aggregated across , and were plotted
against the epochs. To account for diferent numbers of epochs across experiments, the set
of epochs of each experiment was uniformly distributed over the interval [
Experiments (E1) and (E2) The first set of experiments cleanly demonstrates the efect of ofering “cautious” but “specific” advice. As expected, wrong predictions remain efectively zero throughout the coaching phase, while abstentions are very gradually replaced with correct predictions, maintaining an efectively perfect accuracy.
Analogously, the second set of experiments demonstrates the efect of ofering more “loose” and “general” advice. Correct predictions increase slightly faster compared to experiments (E1), while abstentions decrease at an even faster pace, causing the introduction of wrong predictions, and a sub-perfect accuracy. Interestingly, most wrong predictions are corrected over time, leading to suficiently high accuracy.
In both sets of experiments, advice becomes more specific over time and is given less frequently, with the epoch size increasing significantly during the end of the coaching phase. These considerations, along with the continual improvement of performance, suggest that the given advice is indeed beneficial. The quality of the given advice is further supported by the size of the hypothesis policy remaining distinctly smaller than the size of the target policy, indicating that the given advice is able to compress parts of the target policy. The more “loose” and “general” advice in experiments (E2) seems to correspond to a less frequent need for advice, and a more concise hypothesis policy, at the expense of occasionally making wrong predictions, but also efectively never abstaining.
Experiments (E3) and (E4) The third and fourth sets of experiments demonstrate the efect of ofering advice chosen greedily through bounded local searching, and without consulting the explanations of the current hypothesis policy on its predictions. Accordingly, performance is measurably worse than in the first two sets of experiments, with fewer correct predictions, more wrong predictions, higher number of abstentions, and lower accuracy.
Although the last two sets of experiments demonstrate a more aggressive take on decreasing abstentions and increasing correct predictions earlier on, this seems to come at the expense of introducing a considerable number of wrong predictions that persist across epochs, indicating a lower quality of the given advice. This indication is further corroborated by the persistent size of the advice, which fails to become more “cautious” and “specific” over time, and likely leads to the introduction of nearly equally-many new wrong predictions in the place of any of the existing wrong predictions it corrects.
Relatedly, the epoch size does not increase as rapidly as in the first two sets of experiments, leading to a larger hypothesis policy size, to the extent, in fact, that the hypothesis policy ends up being larger than the target policy, showing an inability for compression, and induction, in the given advice.
Comparatively, the size of the given piece of advice in experiments (E4) is multiple times larger than that in experiments (E3). This can be directly attributed to the initialization of the search used in each case, with the former constructing advice by starting from a fully “cautious” and “specific” seed advice, and the latter selecting a piece of advice from typically “loose” and “general” candidate pieces of advice.
Machine Coaching Eficiency The primary metric on eficiency for Machine Coaching is the number of epochs required to get to a certain degree of performance. Figure 4 shows the same information as Figure 3, but with aggregation happening over each particular epoch. Since diferent target polices lead to diferent numbers of resulting epochs, the aggregation happens over a diminishing set of target policies which eventually becomes unrepresentative of the initial set, and should not form the basis for conclusions.
Looking, therefore, at the parts of the plots that include at least 50% of the entire set of target policies being considered, we observe qualitatively that a few (around 10–20) pieces of advice seem to sufice to get relatively high performance.
Although reporting absolute computation times is not informative, we do remark that experiments (E4) are the most time-intensive ones, as a result of the repeated testing against the entire exemplar set, for each mutation in each generation.
We have put forward proxy (algorithmic) coaches as a means to provide empirical evidence that
complements prior theoretical work on the eficacy and eficiency of Machine Coaching. In
ongoing work, we continue to investigate coaching-based learning further with proxy coaches
in more expressive settings (e.g., relational policies, richer reasoning semantics with rule
chaining, and, relatedly, without assuming complete contexts), and contrast its performance
and scalability against autodidactic learning algorithms that map exemplar data directly into a
form of prioritized rules [
Ultimately, our plan is to undertake empirical studies with humans in the role of direct coaches, by identifying meaningful solutions to the challenges discussed in Section 1, perhaps by pairing human coaches with proxy (algorithmic) coaches. Whether human coaches will ofer advice in a manner analogous to one of the proxy coaches considered herein, whether they will remain consistent across multiple interactions, and whether they will find the coaching protocol to be cognitively light, are all questions that we wish to investigate. In turn, we expect that answers to these questions will ofer guidelines towards improving the cognitive compatibility of the language, semantics, and protocol of Machine Coaching.
Acknowledgements This work was supported by funding from the European Regional Development Fund and the Government of the Republic of Cyprus through the Research and Innovation Foundation under grant agreement no. INTEGRATED/0918/0032, from the EU’s Horizon 2020 Research and Innovation Programme under grant agreements no. 739578 and no. 823783, and from the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation, and Digital Policy.
We synthetically generated a target policy by starting from a set of atoms , an average batch size , a flip count ℓ, and a number of stacks , and then partitioning ℓ + 1 into positive integers 1, . . . , ∈ N, and producing for each one of them a stack of depth , = 1, . . . , . In order to generate a stack, given , , and the special atom output, we first generated a batch containing a single rule, referred to as the stack’s root, with body from and head output, and then we iteratively generated batches of average size and interchangeably conflicting heads. Although this process does not explicitly determine the constructed target policy’s size or the average body size of its rules, we manipulated these attributes indirectly, by varying across policies the average batch size, flip count, number of stacks, and each stack’s root body size . Namely, for the purposes of our experiments, and both ranged from 1 to 31 with a step of 2, while ℓ ranged from 1 to 31 with a step of 3, and ranged from 1 to ℓ with a step of 2. For each assignment of values to , , ℓ, , we generated 10 diferent target policies, to account (to some extent) for the various possible partitions of ℓ + 1 to positive integers. Lastly, in all above cases, contained 20 atoms. All related metrics that were manipulated during the data generation stage are illustrated in an example policy in Figure 1.
We measured the predictive ability of each hypothesis policy against each target policy in , and, in the case of intensional proxy coaches, against the corresponding learned model. We refer to these metrics as learning performance and learning conformance, respectively, with each of the two recording correct predictions, wrong predictions, abstentions, and accuracy, as discussed in Section 4.1. The fitness metric used in the evolutionary mechanism of the extensional proxy coach was as described in Section 3.2, and is summarized in Table 1.
t n re a P 0 1 1
Ofspring
Table 2 contains all results regarding the average final performance using all four proxy coaches, while in Tables 3 and 4 we present performance on evaluation set aggregated within each of the several groups of target policies we have produced. For evaluation purposes, we have also labeled the coaching set and kept its labeled part, so in what follows we report results on all three datasets (exemplar, evaluation and coaching).
Most target policy attributes do not seem to significantly afect performance when it comes to single-tree approaches with the exception of policies containing few flips or large rule batches. Regarding flips, this behavior is somewhat surprising since policies containing few
Exemplar
Final performance with respect to all used sets w.r.t. both the target policy (Accuracy) and the Proxy Coach (Conformance) (c: “correct”, w: “wrong”, a: “abstain”, : mean, Q1: 1st quantile, Q2: median, Q3: lfips are structurally simpler, which was expected to benefit coaching and requires, thus, further investigation. On the contrary, poorer performance on policies containing large rule batches was expected since larger batches impose, in general, a richer and more complex policy structure overall. As far as forests are concerned, with the exception of relatively large policies, where the corresponding hypotheses seem to be suficiently accurate, correct predictions seem to remain unafected by variations in target policy’s attributes. This unexpected eficiency on larger target policies is also subject to further investigation.
In Figure 5, we present two sets of single policies, each chosen from one of the groups defined in Section 4.1. Namely, on the left column (Figure 5a) each policy is the group’s median with respect to the policy attribute varied within that group, while on the right column (Figure 5b), each policy is the group’s median with respect to the total number of epochs required during coaching. In both columns, the four stripes correspond to experiments E1-E4, respectively, while, within each stripe, the first row corresponds to high-value groups and the second row to low-value ones.
So far, we have discussed results regarding learning eficiency against the target theory (performance). We have also computed, whenever applicable, learning eficiency with respect to proxy coaches (conformance). Namely, Conformance values recorded how many of the predictions of the current hypothesis policy on the evaluation set ℰ were correct against the proxy coach, wrong against the proxy coach, or abstentions, and the accuracy of the definite predictions (i.e., the number of correct predictions over the number of correct or wrong predictions). As in Section 4.1, Relative Size values recorded the size of the current hypothesis policy (relative to
Exemplar the size of ), and the size of the epoch (relative to the size of ), the size of each piece of advice given (relative to the size of ).
All results regarding the above metrics are presented in Figure 6 (not applicable for E4). Overall, the trends observed for the three diferent advice protocols are similar to those presented in Figure 3. Nevertheless, one may observe that there are also some minor diferences. To begin with, regarding the full path protocol, wrong predictions, in terms of conformance to the proxy coach’s ones, are zero. This is due to the advice-giving protocol being highly conservative on the returned advice which, as discussed in Section 3.1, is quite narrow in terms of coverage but highly accurate at the same time.
Another interesting fact is that, regardless of whether eficiency is measured against the target policy or the proxy coach, coaching under the forest protocol is persistently less eficient than the other two. This, again, could be interpreted as another hint for the ineficiency of the underlying advice mining mechanism when it comes to forests or just as a side efect of a forest’s relative opacity compared to single trees.
0 0 0 0 2 5 Epochs 2 Epochs Epochs 10 Epochs 10
20 Epochs 10 Epochs 10 Epochs Epochs 10 Epochs 0.0 1.0 e0.75 c n a rm0.50 fro Pe0.25 0.0 1.0 e0.75 c n a rm0.50 fro Pe0.25 (a) Median policies per group, w.r.t. each policy (b) Median policies per group w.r.t.
the total attribute.
number of epochs. w.r.t. each group’s manipulated policy attribute (left column) and the total number of epochs within each group (right column). Within each column, the four stripes correspond to E1, E2, E3 and E4 respectively. Within each stripe, groups vary from left to right as follows: policy size, f lip count, stack count, batches and width.
Also, the top row of each stripe corresponds to
high-value groups, while the bottom to low-value ones. Color coding is the same as in
Figure 10 Epochs 20 Epochs 10 Epochs Epochs e z i
S 1.0 ive t a l e 0.5 R 0.0