Objective Hierarchical Temporal Memory (HTM) has been shown to adapt well to changing input patterns, making it a natural building block for a Reinforcement Learning (RL) agent that adapts to evolving tasks quickly. In this paper, we design, implement and experimentally evaluate HTMRL: the rst such agent, built with nothing but HTM. Background In RL, an agent learns how to behave in an environment [2]. This is usually modelled as a Markov Decision Process (MDP): the 4-tuple Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
(1) where S is the set of states, A is the set of actions, P models which state will be reached after taking an action, and R de nes the reward after such a transition. The main goal of an agent is then to de ne a policy , deciding which action to take in each state.
HTM is a computational model of the human neocortex, originally proposed
by Je Hawkins [
Environment 0 0 2
State, reward
Action
Agent
ARM 1 Spatial pooler
ARM 2 HTMRL Our proposed HTMRL algorithm, summarised in Figure 1, uses only a spatial pooler, with minimal modi cations. HTMRL converts any input into an SDR of, by default, 2048 bits. To map this to one of jAj actions, we simply subdivide the SDR into j j equally sized bins, selecting the bin containing the
A most 1 bits as the action to take. Synapse growth is then scaled with the reward signal. As such, actions leading to poor rewards become less likely to be selected on future similar inputs, and vice versa. HTMRL can explore rarely selected actions through the boosting mechanism also present in core HTM theory, which ampli es incoming signals of rarely activated output bits.
Experiments and evaluation We evaluate our HTMRL implementation in an environment with many multi-armed bandits, each giving a positive reward on only a single arm. One randomly selected bandit is presented to the algorithm at every step. With 4-armed bandits, the algorithm could learn over 1000 bandits before becoming prohibitively slow in our implementation. With a xed set of 20 bandits, HTMRL could learn up to 1024 arms per bandit, which is the maximum number of actions representable in its base con guration. To evaluate performance in changing environments, we experiment with a bandit whose arms' rewards are sampled from normal distributions. HTMRL reaches near-optimal performance in a number of steps similar to an -greedy learner. The distributions are then randomly reinitialised, after which HTMRL reaches near-optimal performance just as fast as before, while -greedy learners struggle to adapt. We also show that by shu ing the arms instead of fully reinitialising them, HTMRL adapts even faster, meaning it can leverage its knowledge of previous phases, making it a promising approach for Meta-RL applications.