Researching artificial intelligence (AI) is a big endeavour. It calls for agile collaboration among research teams, fast sharing of work between developers, and the easy testing of new hypotheses. Our primary contribution is a novel simulation platform for the prototyping of new algorithms with a variety of tools for visualization and debugging. The advantages of this platform are presented within the scope of three AI research problems: (1) motion execution in a complex 3D world; (2) learning how to play a computer game based on reward and punishment; and (3) learning hierarchies of goals. Although there are no theoretical novelties in (1,2,3), our goal is to show with these experiments that the proposed platform is not just another ANN simulator. This framework instead aims to provide the ability to test proactive and heterogeneous modular systems in a closed-loop with the environment. Furthermore, it enables the rapid prototyping, testing, and sharing of new AI architectures, or their parts.
The recent boom in the field of artificial intelligence (AI) was brought on by advances in so-called narrow AI, represented by highly specialized and optimized algorithms designed for solving specific tasks. Such programs can even sometimes surpass human performance when solving the single problem for which they were created. But these narrow AI programs lack one feature which has been so far widely omitted, partly due to its overwhelming difficulty: generality.
In order to compensate for this deficiency, the field of
artificial general intelligence (AGI) is bringing the focus
back to broadening the range of solvable tasks. The
ultimate goal of AGI is therefore the creation of an agent
which can perform well (at human level or better) at any
task solvable by a human. For a more detailed description
of AI/AGI, see e.g. [
Pursuing such a goal is a hard task. According to the scientific method – the only guideline we have – we need to come up with new theories, design experiments for testing them, and evaluate their results. Such a cycle needs to be repeated often, because it can be expected to reach more dead ends than breakthroughs. We don’t know how to increase the rate of coming up with new ideas, but what can be improved is the efficiency of research. What we need is better tools which will simplify the implementation of new theories, speed up experiments, and help us understand the results better by visualizing obtained data.
In this article, we would like to present our attempt to create such a tool. We here introduce a platform which allows: • Easy prototyping of new models and fast sharing of existing ones (Sec. 4.1) • Control of an agent in an environment on top of classic data processing (Sec. 3.3) • Modular approach – seamless connecting of models inside a greater architecture (Sec. 2.1) • Various tools for the visualization of data (Fig. 2) • Simplified debugging (Sec. 2.2) • User Friendly GPU programming (Sec. 2.1) • Scalable due to GPU parallel computation • Support of several scenarios such as an agent in an environment, classification, tic-tac-toeetc.
Our platform also includes a variety of visualization tools and enables easy access to diverse data sets not only for tasks such as image classification or recognition, but also scenarios where an agent interacts with its environment. Last but not least, it is open source and freely available under a non-commercial license1. The primary goal of this tool is easy collaboration among both specialists and laymen for developing novel algorithms, especially in the field of AI.
There are tools, languages, and libraries that are good
in particular areas. In the research community, widely
1The platform is available as Brain Simulator at http://www.
goodai.com/brainsimulator
used are Matlab [
There are several tools for simulating neural networks
(NN). Nengo [
It is worth noting that the proposed platform is not limited to the design of neural networks only. Any algorithm useful for AI, machine learning, or control can be incorporated (various mathematical transformations, filters, PID controller, image segmentation, hashing functions, dictionary, etc., are already included). The heterogeneous character of the platform is its main advantage.
Throughout this work, we describe our platform in the following Sec. 2. In Sec. 3, tasks where we show advantages of the platform are introduced. In Sec. 4, we discuss the experience with our tool and its advantages and weakness in the testing scenarios. The paper is concluded in Sec. 5. 2
Our modus operandi reflects our goals - we are aiming for a modular cognitive architecture, so we needed an environment which would efficiently support the whole life-cycle of experiments, starting with the testing of already existing algorithms, going through the design of a new algorithm, and ending with a results evaluation. We developed a platform where various algorithms from machine learning and narrow AI are available. It is easy to pick some, connect them, and start experimenting effortlessly. Agile development requires frequent testing of new hypotheses, which is facilitated by an easy way of prototyping new modules for the platform as well as their fast training and evaluation (accelerated on GPU) on data. After the experiment is (a) (b) (c) (d) running, it often happens that its outcome is not what was expected. Such a simulation platform could not be imagined without a tool for runtime analysis of algorithms. For this purpose various data observers can be displayed so the experimenter can visualize the computed data, evaluate performance of the model, and change its parameters during runtime if needed.
Our platform is tailored to suit two different points of view of the architecture development process: • A user who desires quick architecture modeling and needs fast access to already existing state-of-the-art modules (such as PCA, NN, image pre-processing etc.) to experiment with. This perspective requires no coding and it’s done through graphical modeling. Furthermore, it is often crucial to have good insight into the running model, and thus a large set of visualization tools is available (Fig. 2). • On the other hand, a researcher/developer often requires the creation of a new module or the import of an already existing library. Our API provides an easy way for such a module to be created and added to the inner shared repository. Moreover, the API offers an opportunity to hook the code to the GUI and bring needed interactivity. Finally, the API defines a rigid interface, ensuring that the new module will be compatible with other modules.
The platform was designed to meet both needs. It is important to distinguish between them as a user can be a person interested in machine learning, but less experienced in programming. Our platform can be a good starting point, and the learning curve should therefore be smooth enough to bring the person in effortlessly.
From the experienced researcher/developer point of view, the platform should provide a convenient set of tools that can help with the development of novel algorithms and/or be able to envelope existing work into module that can be easily shared among a team.
Finally, the community-driven approach renders itself very powerful and we believe that it can speed up the research vastly. For this reason, we are planning to in-build a “module market” to allow for the sharing of state-ofthe-art research results between many co-working teams. 2.1
There are three basic concepts defined in the meta model: a node, a task and a memory block. The node encapsulates a functional block or algorithm that can “live” on its own (e.g. matrix operations, data transformations, various machine learning models, etc.). A node needs a memory for its function. The memory is organized into a set of memory blocks that are aggregated inside the node. Some of these memory blocks can be designated as output blocks and others as input blocks. The connection between input and output memory blocks is provided by the user.
From the functional point of view, the node behaviour can usually be divided into a set of tasks where each task is a part of the realized algorithm. Both nodes and tasks can define a set of parameters. Usually, node parameters describe structural properties (i.e. size of memory blocks) whereas task parameters affect behavior. At present, the memory model is constant during the simulation, and therefore structural properties are editable only in design time. On the other hand, it is useful to change task parameters during simulation and observe changes in behavior of the algorithm/node.
Memory blocks are located at GPU (device memory) and every task can be seen as a collection of kernel calls (methods executed on GPU). If two nodes are connected in the GUI, it means that they have a pointer to the same memory block (input in one node, output in the other).
If one requires dynamically allocated memory, the user can either define a memory block that is large enough, or implement the node only for the CPU (which is more flexible than GPU) using all data structures supported by C#. The only mandatory requirement is usage of input/output memory blocks.
All concepts described above can be easily implemented through rich API that is provided. The actual implementation relies heavily on annotated code describing various aspects of the model (UI interactivity, constraints, persistence, etc.). It allows the user to be extremely efficient in creating model prototypes. Sometimes, this can lead to unreadable, over-annotated code which is hard to maintain but this can be eliminated by applying standard software design patterns like MVC when needed. 2.2
As described above, the prototyped model forms an oriented graph with nodes and data connection edges. As the connections between nodes can be any of M → N and recurrent connections are also possible, the resulting graph can be very complex. Moreover, the usual model is connected to the world node from which “perception” inputs are taken, and control outputs are passed, forming the main loop of the simulation.
Before running the simulation, the order of nodes
execution needs to be evaluated. There are other aspects that
level the problem up (e.g. inner cycles, clustering and
balancing of the model in HPC environment) but it usually
boils down to various forms of dependency ordering,
cycle detection, or the job shop problem [
There is also another view of the problem of execution order when faced in the area of machine learning. It turns out that many of ML methods are surprisingly noise resistant (i.e.neural nets). Therefore, if approached with caution, one can run the model asynchronously and let inner parts of the model deal with sometimes temporally inconsistent data. We made some experiments and the preliminary results show that relatively complex models can be run completely without synchronization.
Another aspect of the model execution is GPU enhanced computation which can speed up the simulation substantially. The main purpose of our simulation platform is fast prototyping and testing of hypotheses. With increasing generality the efficiency usually decreases, so one should not expect top execution speed from our simulation platform. The devised practice is to design, test, and analyze new architectures, and once the final model is tested and working, it can be replaced by a specialized, highly optimized implementation still within the platform environment. Finally, it can be argued that the overall time necessary to get from an idea to the final product is much shorter compared to the classic approach of writing a specific program from scratch for each new experiment.
An important part of the development process is the easy visualization of what is happening at each part of the designed system. This is especially important for debugging as the most frequent problem is due to the difference between what the programmer thinks the program should do and what it does in reality. In addition to the variety of observers that have already been discussed (Fig. 2), the platform contains its own debugger, where one can walk through the execution of all components used in the model. 3
Whether the ultimate goal of AGI (a general autonomous
machine) is achievable or not [
As the goal of this paper is to provide a tool that shortcuts the research path to an autonomous machine, we will show how it performs on three selected AI tasks solved by our team: learning motion control, game playing of the Atari game Breakout, and learning hierarchies of goals. Our solutions are highly inspired by current machine learning literature with a stress on the usage of neural networks, which are one of the basic building blocks for bigger architectures.
The first experiment (Sec. 3.1) will demonstrate how our platform can be connected to an external source of input data and how various modules of narrow AI can be combined together to form a functioning system which can drive a robot in a virtual world with simulated physics.
The second experiment (Sec. 3.2) will be situated in
much simpler simulation environment – an Atari [
In the third experiment (Sec. 3.3), we move a bit higher
in the level of abstraction. The presented problem
consists of an agent in a simple 2D environment which needs
to satisfy a chain of preconditions before reaching a
reward, such as if the agent wants to turn on a light, it needs
to press a switch, but to get to the switch he also needs
to overcome an obstacle (a door controlled by another
switch). The task is solved by hierarchical reinforcement
learning [
To clarify why we selected these experiments, one could imagine the three systems as parts of a future higher-order cognitive architecture where they will work together. The system from the first experiment could be thought of as a basic motoric and sensory system driven by reflexes and higher-level commands. These would come from the system used in the second experiment, which would allow the agent to learn how to reach a specific goal. And finally, the third system should discover the hierarchy of goals and preconditions, and thus could resemble a simplified version of the agent’s central executive. Such connection of the systems remains for our future work. 3.1
We took advantage of a sandbox game called Space
Engineers [
ule was trained to associate visual input with motor commands in a supervised way.
The associative
memory was
implemented with a
Self-Organizing
Figure 3: Overall architec- Map [
The training phase consisted of a mentor leading the robot from various starting locations towards a goal location in the environment, which was identified by an easily distinguishable 3D symbol located on that position. The mentor was implemented by a hard-coded navigation system. In this way the hexapod was trained to look for the goal symbol and when it appeared in the robot’s field of view, to navigate successfully towards the destination through the environment. 3.2
The Breakout game was chosen as our second testing scenario. The game consists of a ball, a paddle, and segmentation attention score
CNN features input image
graph optimization
patch extraction working memory bricks. The ball bounces from walls, can destroy bricks, and can fall to the ground, for which the player is penalized by losing a life. After losing 4 lives, the game is over. When all bricks are destroyed, the player successfully finishes the level and enters the next one consisting of a different arrangement of bricks.
The player has three actions available which accelerate the paddle to the right, to the left, or decelerate.
Even though our modular approach
uses pure unstructured data input
(raw image as in [
Relevant information about the objects is extracted from the raw bitmap in the Vision System (Fig. 4).
Working memory (WM) is the agent’s internal representation of the environment. It contains all of the objects detected by vision. WM is kept up to date by adding new objects which haven’t been seen yet, and by updating those already seen. The identity of objects is detected through a comparison of visual features. Contents of the working memory are transformed into a symbolic representation and passed to the goals memory and action selection network.
The goals memory is trained by accumulating states associated with reward and punishment in their respective semantic pointers, goal+ and goal-. These are then used for evaluating the quality of game-states, which is necessary for training the action selection network. (a) The goal (b) Legend
Details of the vision system and semantic point architecture are described in Appendix II. 3.3
In previous testing scenarios we wanted to test if our system was able to coordinate complex motoric commands in a 3D environment, learn simple goals, and act towards maximizing the received reward. Our goal for the third scenario is to increase the generality of the designed system to enable identification and satisfaction of chained preconditions before the final goal can be reached.
We present a task, consisting of a simple 2D world, where a single source of reward is located – a light bulb, which starts in the “off” position and should be turned on by the agent. This can be achieved by pressing a switch, but the switch is hidden behind a locked door. The door can be unlocked through a switch, but this switch is hidden behind another locked door. It would be possible to chain the preconditions further in this manner, but without the loss of generality we use only two locked doors with two matching switches. The setup can be seen in Fig. 5.
We approached this problem by employing HARM
(Hierarchical Action Reinforcement Motivation system) [
Q-learning is able to “spread information about the reward” received in a specific state (e.g. the agent reaching a position on the map) to the surrounding space, so the brain can take proper action by climbing the steepest gradient of the Q function later. However, if the goal state is far away from the current state, it might take a long time to build a strategy that will lead to that goal state. Also, a high number of variables in the environment can lead to extremely long routes through the state space, rendering the problem almost unsolvable.
There are several ideas that can improve the overall performance of the algorithm. First, this agent rewards itself for any successful change to the environment. The motivation value can be assigned to each variable change so the agent is constantly motivated to change its surroundings.
Second, for each variable that the agent is able to change, it creates a Q-learning module assigned to the variable (e.g. changing the state of a door). Therefore, it can learn an underlying strategy defining how this change can be made again. In such a system, a whole network of Q-learning modules can be created, where each module learns a different strategy.
Third, in order to lower the complexity of each subproblem (strategy), the brain can analyze its “experience buffer” from the past and eventually drop variables that are not affected by its actions or are not necessary for the current goal (i.e. strategy to fulfill the goal).
A mixture of these improvements creates a hierarchical decision model that is built online (first, the agent is left to (semi-)randomly explore the environment). After a sufficient amount of knowledge is gathered, we can “order” the agent to fulfill a goal by manually raising the motivation that corresponds to a variable that we want to change. The agent then will execute the learned abstract action (strategy) by traversing the network of Q-learning modules and unrolling it into a chain of primitive actions that lie at the bottom. 4
Throughout the work on the testing scenarios (Sec. 3) we have observed several advantages and weakness of the platform. In this section, our experience with the usage of the platform is discussed. The discussion is focused especially on the end-user experience, i.e. experience of a person that did not develop the platform but wants to use it for solving her problem.
First, a list of identified features is presented, and then further experience is described.
+ Fast and easy online observation and interaction with the simulation.
+ Created modules can be easily understood and shared with collaborators due to the same interface. Moreover, the persistence capabilities allow easy sharing of whole models (projects) and merging of them together.
+ It is easy to replace an existing module with its improved version, as the architecture is separated from the implementation. As the backward compatibility becomes crucial at this point, the inner versioning system was implemented.
+ Provided interface drives users to follow design patterns when developing low-level optimized modules.
- Current version runs on MS Windows only, but a port to MacOS and Linux is planned for the future.
- The user has an option to either develop optimized
modules in code (CUDA [
The expertise of people that have started to use our platform varies from C++ experts to Matlab users only. We found that users with a very short training can connect existing modules into simple architectures (like neural network MNIST image recognizer) as the graphical modeling is somehow natural and easy to understand. The linear learning curve of newcomers is supported by a video tutorial as well as several examples of how to implement simple and more advanced tasks2.
For development of new modules, it is necessary to understand a programming language (C#, C++, CUDA) at the basic level at least. Once the definitions of inputs, outputs, tasks, and kernels (four lines of code each) are understood, developers soon start creating their own nodes. Their learning curve then equals learning how to use a new library. 4.2
In the first scenario (Sec. 3.1), we have shown that our
platform successfully connects with the open source game
Space Engineers [
The second scenario (Sec. 3.2) consisted of several modules that were developed independently. We found it extremely useful that each module communicates with others using only the pre-defined interface (memory blocks) that correspond to a sketched diagram (i.e. Fig. 4). Modules were merged into one big architecture right before the deadline without any complications. As the final model was quite large and performance-demanding, available
at http://docs.goodai.com/
brainsimulator/ we were forced to profile, find bottlenecks, and optimize in the process. It was extremely useful to visualize data flowing between (and inside) the modules.
In the third scenario (Sec. 3.3), HARM constituted a single module with complex insides. Therefore this scenario presented an ideal example for designing a number of task-specific visualization tools (for example, the agent’s knowledge in Fig. 6). 5
We have presented a platform for prototyping AI architectures. The platform is tailored both for users with no mathematical/programming background but with a high desire to experiment with AI modules, and for researchers/developers who want to improve and experiment with their existing state-of-the-art techniques.
To show the usage of our platform we have presented three development scenarios: linkage with a 3D game world and controlling an agent there; playing an Atari game using the raw bitmap input processed by computer vision techniques, attention model, and semantic pointer architecture; and learning a complex hierarchy of goals.
The proposed platform opens up possibilities to share ideas not only within the community but also with nonexperts who can boost the research via rapid testing, or utilize fresh, out-of-the-box solutions. There is also the prospect of support from the open source community - if not directly in the development, then at least in assessing missing features, so we can incorporate them and thus provide a tool that can be used at many levels of expertise.
We believe that by providing an open platform for AI and ML experiments along with a smooth learning curve, we can bring together many enthusiasts across different fields of interest, potentially leading to unexpected advancements in research.
Acknowledgement. This material is based upon work supported by GoodAI and Keen Software House.
Unlike the major stream of Computer Vision, our approach
has to be unsupervised without any training data. Thus, we
have no prior knowledge and the system has to learn
everything on-the-fly. Our system is a pipeline visualized and
implemented in Fig. 4. It consists of the following parts:
The input image is first
segmented into a set
of super-pixels [
For CNN features, we used two convolutions layers of
8 and 5 neurons and patch sizes 5 × 5 followed by a
fullyconnected layer of 16 neurons. Learning converged in
6K iterations. We observed no performance improvement
with bigger networks. WM was implemented as a simple
K-means [
As was already mentioned in section 1, one of the main features of our method is the semantic pointer architecture (SPA), which merges the symbolic and connectionist approach. Artificial neural networks are very powerful adaptive tools, but their usage usually comes at the expense of losing detailed insight into how exactly the task is solved. Such a drawback can be mitigated by using the SPA and its variable binding. It introduces composite symbols of the form X bind x, where X is the name of the variable and x is its value. It is then possible to train a network to perform complicated transforms such as:
V ⊗ (X ⊗ x + Y ⊗ y) → C ⊗ (Y ⊗ x) (1) which could be interpreted as an action selection rule for the pong game:
Visual ⊗ (Ball ⊗ x1 + Paddle ⊗ x2) →
→ Move ⊗ (Paddle ⊗ x1) (2)
If ball is seen at position x1 and paddle at x2, execute command ’move paddle to position x1’. Without SPA it would be much harder to maintain understanding of the transformed symbols, if not entirely impossible. Goals Memory. The accumulated states g+ and g− are used for calculation of quality q of the state x, using dot product ’·’, q = g+ · x − g− · x, see Fig. 8 left.