Informed Machine Learning Through Functional Composition

Abstract

Addressing general problems with applied machine learning, we sketch an approach towards informed learning. The general idea is to treat data driven learning not as a parameter estimation problem but as a problem of sequencing predefined operations. We show by means of an example that this allows for incorporating expert knowledge and leads to traceable or explainable decision making systems.

Introduction

Recent progress in artificial intelligence has been largely driven by deep neural networks [ 1 ] which are trained using vast amounts of data and high performance hardware. This way, deep networks can now solve demanding tasks in computer vision [ 2 ], speech recognition [ 3 ], text understanding [ 4 ], medical diagnosis [ 5 ], or game AI [ 6 ]. Nevertheless, data science practitioners outside the IT industry are increasingly dissatisfied with deep learning, mainly because of: 1) Lack of training data. Vapnik-Chervonenkis theory [ 7 ] establishes that it takes substantial training data for machine learning to work well. Since details depend on a system’s VC dimension which is hard to pinpoint, Widrow’s rule of thumb (see [ 8 ]) suggests to train with at least ten times more data than there are system parameters. Modern deep networks with millions of parameters therefore need tens if not hundreds of millions of representative examples to learn robustly.

At first sight, this does not seem problematic, after all this is the age of big data. However, reliable supervised machine learning actually requires thick data, i.e. large amounts of annotated data, which are rarely available where business models do not revolve around digital services. Even in contexts such as Internet of things or industry 4.0 where data accumulate en masse, we still frequently face thin data scenarios where appropriate training data for machine learning are inaccessible.

2) Lack of traceability. Trained connectionist architectures are black boxes whose inner computations (non-linear activations of synaptic summations) are abstracted away from conceptual information processing. As a consequence, their decision making processes are often unaccountable so that data scientists in, say, finance or industrial system control are wary of deep learning because regulatory guidelines demand automated decision making to be comprehensible.

Indeed, a growing body of literature shows that silly mistakes made by deep networks could be avoided if they had “common sense” or, even more alarmingly, that silly mistakes can be provoked using adversarial inputs [ 9, 10 ]. Hence, while the need for explainable AI has been recognized early on [ 11, 12 ] corresponding research has been reinvigorated recently [ 13–16 ].

In this paper, we address both these issues and propose an approach towards informed machine learning that copes with thin data and leads to traceable systems. Our basic observation is that domain experts in industry know a lot about the data they are dealing with and that this knowledge is often procedural, i.e. there is experience as to what to do with data in order to achieve a certain goal. Assuming training data and a database of procedures to be given, our main idea is thus to use Monte Carlo tree search [ 17 ] or reinforcement learning [ 18 ] to determine sequences of operations that map input to desired outputs. 2

Deep Learning as Functional Composition

To motivate our idea, we point out two facts about neural networks: First, a trained, feed-forward neural network of L layers that maps inputs x to outputs y computes a composite function y(x) = fL . . . f2 f1(x)

= fL ◦ . . . f2 ◦ f1(x) zl = fl zl−1 = σ Wl zl−1

n E = X yi − y(xi) 2

i=1 where the functions fl represent the computations of layer l of the network. To be more precise, letting zl denote the output of layer l, we typically have where Wl are the input weights of layer l and σ(·) is an activation function.

Second, recall that number and sizes of layers as well as activation functions are usually chosen by hand so that neural network training is to estimate those weight parameters W1, . . . , WL that minimize a loss function such as (1) (2) (3) where (xi, yi) in=1 is a sample of labeled training data.

Given these observations regarding the design and training of deep architectures, it then seems natural to ask: Q1: When building a deep system, do the functions fl it consists of have to be of the form in (2) or are there other basic building blocks? Q2: When training a deep system, does it necessarily have to be a parameter estimation problem or are there other ways of adapting it to a task at hand?

Especially the soft computing community has answered both these question affirmatively before. For instance, recent work in [ 19 ] present a deep fuzzy logic system assembled via genetic programming that shows superhuman capabilities in an air defense scenario. Here, however, we are interested in answers based on statistical machine learning and illustrate our idea by means of a simple example.

Figure 1(a) shows labeled training data {(xi, yi)}in=1 where the data xi ∈ R2 come from two classes and the labels yi ∈ {+1, −1} indicate class membership. Given this XOR-type data, the goal is to learn a binary classifier y(x) such that y(x) = ( −1 if x1 ≈ x2 +1 otherwiese. We sketched a deep learning technique that can incorporate world knowledge and work in thin data scenarios. Yet, while there are first efforts on properties of functional composition for machine learning [ 20 ], many questions regarding our idea are still open. Two pressing ones are: How to systematically formalize expert knowledge to fit into this framework? Are there universal functional building blocks from which to compose solutions for arbitrary problems?

References

1. Bengio , Y. : Learning Deep Architectures for AI . Foundations and Trends in Machine Learning 2 ( 1 ) ( 2009 )

2. Krizhevsky , A. , Sutskever , I. , Hinton , G.: ImageNet Classification with Deep Convolutional Neural Networks . In: Proc. NIPS . ( 2012 )

3. Hinton , G. , Deng , L. , Yu , D. , Dahl , G. , Mohamed , A. , Jaitly , N.: Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups . IEEE Signal Processing Magazine 29 ( 6 ) ( 2012 )

4. Conneau , A. , Schwenk , H. , Barrault , L. , LeCun , Y.: Very Deep Convolutional Networks for Text Classification . arXiv:1606.01781 [cs.CL] ( 2016 )

5. Cire¸san, D., Giusti , A. , Gambardella , L. , Schmidhuber , J. : Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks . In: Proc. MICCAI . ( 2013 )

6. Silver , D. , et al.: Mastering the Game of GO with Deep Neural Networks and Tree Search . Nature 529 ( 7587 ) ( 2016 )

7. Vapnik , V. , Chernovenkis , A. : On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities . Theory of Probabilities and its Applications 16 ( 2 ) ( 1971 )

8. Morgan, N., Bourland , H.: Generalization and Parameter Estimation in Feedforward Nets: Some experiments . In: Proc. NIPS . ( 1990 )

9. Ribeiro , M. , Singh , S. , Guestrin , C. : “ Why Should I Trust You ?”: Explaining the Predictions of Any Classifier . arXiv:1602.04938 [cs.LG] ( 2016 )

10. Brown , T., Man´e, D. , Roy , A. , Abadi , M. , Gilmer , J.: Adversarial Patch . arXiv: 1712 .09665 [cs.CV] ( 2017 )

11. Tickle , A. , Andrews , R. , Golea , M. , Diederich , J.: The Truth Will Come to Light: Directions and Challenges in Extracting the Knowledge Embedded within Trained Artificial Neural networks . IEEE Trans. Neural Networks 9 ( 6 ) ( 1998 )

12. van Lent, M. , Fisher , W. , Mancuso , M.: An Explainable Artificial Intelligence System for Small-unit Tactical Behavior . In: Proc. IAAI , AAAI ( 2004 )

13. Bach , S. , Binder , A. , Montavon , G. , Klauschen , F. , Mu¨ller, K.R. , Samek , W.: On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation . PLoS ONE 10 ( 7 ) ( 2015 )

14. Wahabzada , M. , Mahlein , A.K. , Bauckhage , C. , Steiner , U. , Oerke , E.C. , Kersting , K. : Metro Maps of Plant Disease Dynamics - Automated Mining of Differences Using Hyperspectral Images . PLoS ONE 10 ( 1 ) ( 2015 )

15. Belle , V.: Logic Meets Probability: Towards Explainable AI Systems for Uncertain Worlds . In: Proc. IJCAI . ( 2017 )

16. Sifa , R. , Ojeda , C. , Cvejoski , K. , Bauckhage , C. : Interpretable Matrix Factorization with Stochasticity Constrained Nonnegative DEDICOM . In: Proc. KDML . ( 2017 )

17. Browne , C. , et al.: A Survey of Monte Carlo Tree Search Methods . IEEE Trans. Computational Intelligence and AI in Games 4 ( 1 ) ( 2012 )

18. Sutton , R. , Barto , A. : Reinforcement Learning an Introduction . MIT Press ( 1998 )

19. Ernest , N. , Carroll , D. , Schumacher , C. , Clark , M. , Cohen , K. , Lee , G. : Genetic Fuzzy based Artificial Intelligence for Unmanned Combat Aerial Vehicle Control in Simulated Air Combat Missions . Journal of Defense Management 6 ( 1 ) ( 2016 )

20. Bohn , B. , Griebel , M. , Rieger , C. : A Representer Theorem for Deep Kernel Learning . arXiv:1709.10441 [cs.LG] ( 2017 )