Real-world tasks of interest are generally poorly defined by human-readable descriptions and have no pre-defined reward signals unless it is defined by a human designer. Conversely, data-driven algorithms are often designed to solve a specific, narrowly defined, task with performance metrics that drives the agent's learning. In this work, we present the solution that won first place and was awarded the most human-like agent in the 2021 NeurIPS Competition MineRL BASALT Challenge: Learning from Human Feedback in Minecraft, which challenged participants to use human data to solve four tasks defined only by a natural language description and no reward function. Our approach uses the available human demonstration data to train an imitation learning policy for navigation and additional human feedback to train an image classifier. These modules, combined with an estimated odometry map, become a powerful state-machine designed to utilize human knowledge in a natural hierarchical paradigm. We compare this hybrid intelligence approach to both end-to-end machine learning and pure engineered solutions, which are then judged by human evaluators. Codebase is available at https://github.com/ viniciusguigo/kairos_minerl_basalt.
In this paper, we present the solution that won first place and was awarded the most human-like
agent in the 2021 Neural Information Processing Systems (NeurIPS) MineRL Benchmark for
Agents that Solve Almost-Lifelike Tasks (BASALT) competition1. Most artificial intelligence (AI)
and reinforcement learning (RL) challenges involve solving tasks that have a reward function
to optimize over. Real-world tasks, however, do not automatically come with a reward function,
and defining one from scratch can be quite challenging. Therefore, teaching AI agents to
solve complex tasks and learn dificult behaviors without any reward function remains a major
challenge for modern AI research. The MineRL BASALT competition is aimed to address
this challenge by developing AI agents that can solve complex, almost-lifelike, tasks in the
challenging Minecraft environment [
The MineRL BASALT competition tasks do not contain any reward functions for the four
tasks. We propose a human-centered machine learning approach instead of using traditional
RL algorithms [
In this paper, we give a detailed overview of our approach and perform an ablation study to investigate how well our hybrid intelligence approach works compared to using either learning from human demonstrations or engineered solutions alone. Our two main contributions are: • An architecture that combines engineered knowledge of the tasks to be solved together with machine learning modules to solve complex hierarchical tasks in Minecraft. • Empirical results on how hybrid intelligence compares to both end-to-end machine learning and pure engineered approaches when solving complex, real-world-like tasks, as judged by real human evaluators.
End-to-end machine learning: in this work we use the term “end-to-end machine learning”
for algorithms that learn purely from data with minimal bias or constraints added by human
designers, besides the ones that are already inherently built-in the learning algorithm. For
example, deep reinforcement learning algorithms learning directly from raw pixels [
Human-in-the-loop machine learning: Learning from human feedback can take diferent
forms depending on how human interaction is used in the learning-loop [
The Minecraft learning environment: Minecraft is a procedurally-generated, 3D
openworld game, where the agent observes the environment from a first-person perspective, collects
resources, modifies the environment’s terrain, crafts tools that augment the agent’s capabilities,
and possibly interacts with other agents in the same environment. Given the open-world nature
of this environment, there are no predefined tasks or built-in reward signals, giving the task
designer flexibility to define tasks with virtually any level of complexity. The release of Malmo
[
The Minecraft environment also served as a platform to collect large human demonstration
datasets such as the MineRL-v0 dataset [
The 2021 NeurIPS MineRL BASALT competition, “Learning from Human Feedback in Minecraft”,
challenged participants to come up with creative solutions to solve four diferent tasks in
Minecraft [
The competition organizers also provided each participant team with a dataset of 40 to 80 human demonstrations for each task, not all completing the task, and the starter codebase to train a behavior cloning baseline. Additionally, the training time for all four tasks together was limited to four days and participants were allowed to collect up to 10 hours of additional human-in-the-loop feedback.
Since no reward signal was given by the competition organizers and compute time was
limited, direct deep reinforcement learning approaches were not feasible [
Hence, to solve the four tasks of the MineRL BASALT competition, we opted to combine
machine learning with knowledge engineering, also known as hybrid intelligence [
(g) good_waterfall_view (h) good_pen_view (i) good_house_view (j) has_animals (k) has_open_space (l) animals_inside_pen
Our approach relies on a state machine that changes the high-level goals depending on the
task to be solved. Without having information about the environment’s voxel data, we opted
to use the visual RGB information from the simulator to determine the agent’s current state.
Due to the low resolution of the simulator of 64 × 64 × 3, we decided to use a classifier that
labels the whole image rather than parts of the image, such as You Only Look Once (YOLO) [
There are 13 possible labels for an RGB frame, as illustrated in Figure 2 and described below: • none: frame contains no relevant states (54.47 % of the labels). • has_cave: agent is looking at a cave (1.39 % of the labels). • inside_cave: agent is inside a cave (1.29 % of the labels). • danger_ahead: agent is looking at a large body of water (3.83 % of the labels). • has_mountain: agent has a complete view of a mountain (usually, from far away) (4.38 % of the labels). • facing_wall: agent is facing a wall that cannot be traversed by jumping only (4.55 % of the labels). • at_the_top: agent is at the top of a mountain and looking at a clif (3.97 % of the labels). • good_waterfall_view: agent see water in view (3.16 % of the labels). • good_pen_view: agent has framed a pen with animals in view (4.12 % of the labels). • good_house_view: agent has framed a house in view (2.58 % of the labels). • has_animals: frame contains animals (pig, horse, cow, sheep, or chicken) (9.38 % of the labels). • has_open_space: agent is looking at an open-space of about 6x6 blocks with no small clifs or obstacles (flat area to build a small house or pen) (7.33 % of the labels). • animals_inside_pen: agent is inside the pen after luring all animals and has them in view (0.81 % of the labels).
The possible labels were defined by a human designer with knowledge of the relevant states to be identified and given to the state-machine to solve all tasks. These labels were also designed to be relevant to all tasks to ease data collection and labelling eforts. For example, the “has_open_space” label identifies flat areas that are ideal to build pens or houses for both CreateVillageAnimalPen and BuildVillageHouse tasks. Unknown and other non-relevant states were attached the label “none” to indicate that no important states were in view.
To train this system, we labeled 81, 888 images using a custom graphical user interface (GUI),
as showed in Appendix A. Once the data was labeled, 80% of images were used for training,
10% were used for validation, and 10% for testing. The model is a convolutional classifier with
a 64 × 64 × 3 input and 13 × 1 output. Our autoencoder is modeled after the Deep TAMER
(Training Agents Manually via Evaluative Reinforcement) [
Some of the engineered subtasks required basic localization of the agent and relevant states of the environment. For example, the agent needs to know the location of previously seen animals to guide them to a pen in the CreateVillageAnimalPen task. However, under the rules of the competition, we were not allowed to use any additional information from the simulator besides the current view of the agent and the player’s inventory. Which means there is no information about the ground truth location of the agent, camera pose, or any explicit terrain information available.
Given these constraints, we opted to implement a custom odometry method that took into consideration only the actions from the agent and basic characteristics of the Minecraft simulator. It is known that the simulator runs at 20 frames per second, which means there is a 0.05 second interval between each frame. According to the Minecraft Wiki4, walking speed is approximately 4.317 m/s, 5.612 m/s while sprinting, or 7.127 m/s when sprinting and jumping at the same time, which translates to approximately 0.216, 0.281, or 0.356 meters per frame when walking, sprinting, or sprinting and jumping, respectively. If we assume the agent is operating in a flat world, starting at position (0, 0) in map coordinates facing north, when the agent executes a move forward action its position is moved 0.216 meters north to position (0, 0.216). The agent does not have acceleration in MineRL and their velocity is immediately updated upon keypress. Another limitation is that we were not able to reliably detect when the agent is stuck behind an obstacle, which causes the estimated location to drift even though the agent is not moving in the simulator.
Since the agent already commands camera angles in degrees, the heading angle is simply updated by accumulating the horizontal camera angles commanded by the agent. More generally, this odometric estimation assumes the agent follows point-mass kinematics: ˙ = ( ) 4Minecraft Wiki - Walking: https://minecraft.fandom.com/wiki/Walking.
˙ = ( ), where is the velocity of the agent, which takes into consideration if the agent is walking, sprinting, or sprinting and jumping.
Using this estimated odometry and the learned state classifier, it is possible to attach a coordinate to each classified state and map important features of the environment so they can be used later by diferent subtasks and the state-machine. For example, it is possible to keep track of where the agent found water, caves, animals, and areas of open space that can be used to build a pen or a house. Figure 3 shows a sample of the resulting map overlaid with the classified states’ location and current odometry readings.
One of the main complexities in solving the proposed four tasks is that most required the agent to have certain levels of perception capabilities, memory, and reasoning over long-term dependencies in a hierarchical manner. For example, the CreateVillageAnimalPen task required the agent to first build a pen nearby an existing village, which requires identifying what a village is, then indicating a good location to build a pen such as a flat terrain. Once the pen was built, the agent had to search for at least two of the same animal type in the nearby vicinity using 64 × 64 resolution images as input. Return animals to the pen required coordination to combine diferent blocks and place them adjacently to each other in a closed shape. After the animals were found, the agent had to lure them with the specific food type they eat, walk them back to the pen the agent initially built, leave the pen, lock the animals inside, then take a picture of the pen with the animals inside.
Reasoning over these long-term dependencies in hierarchical tasks is one of the main
challenges of end-to-end learning-based approaches [
Once the complex tasks are decomposed into multiple small subtasks, we engineered a statemachine in combination with the learned state classifier to select the best subtask to be followed at every time-step. Each of these engineered subtasks was implemented by a human designer who hard-coded a sequence of actions to be taken using the same interface available to the agent. In addition to these subtasks, the human designer also implemented a safety-critical subtask allowing the agent to escape a body of water whenever the state classifier detects that the agent is swimming. Appendix B describes in detail the sequence of substasks followed by the state-machine for each task.
In this work, we compared four diferent approaches to solve the four tasks proposed in the Minecraft competition: • Hybrid: the main proposed agent in this work, that combines both learned and engineered modules. The learned modules are the navigation subtask policy (learns how to navigate using the human demonstration data provided by the competition) and the state classifier (learns how to identify relevant states using additional human-labeled data). The engineered modules are the multiple subtasks, hand-designed to solve subtasks that were not able to be learned from data. These engineered modules are the estimated odometry and the state-machine, which uses the output of the state classifier and engineered task structure to select which subtask should be followed at each time-step. • Engineered: almost identical to the Hybrid agent described above, however, the navigation subtask policy that was learned from human demonstrations is now replaced by a hand-designed module that randomly selects movement and camera commands to explore the environment. • Behavior Cloning: end-to-end imitation learning agent that learns solely from the human demonstration data provided during the competition. This agent does not use any other learned or engineered module, which includes the state classifier, the estimated odometry, and the state-machine. • Human: human-generated trajectories provided by the competition. They are neither guaranteed to solve the task nor solve it optimally because they depend on the level of expertise of each human controlling the agent.
To collect human evaluations for each of the four baselines in a head-to-head comparison we set up a web application5 similar to how the teams were evaluated during the oficial MineRL BASALT competition, as seen in Appendix C. In our case, each participant was asked to see two videos of diferent agents performing the same task then to answer three questions: 1. Which agent best completed the task? 2. Which agent was the fastest completing the task? 3. Which agent had a more human-like behavior? For each question, the participants were given three possible answers: “Agent 1”, “Agent 2”, or “None”.
Our database had videos of all four types of agents (Behavior Cloning, Engineered, Hybrid, and Human) performing all four tasks (FindCave, MakeWaterfall, CreateVillageAnimalPen, and BuildVillageHouse). There were 10 videos of each agent type solving all four tasks, for a total of 160 videos in the database. Task, agent type, and videos were uniformly sampled from the database at each time a new evaluation form was generated and presented to the human evaluator.
We collected a total of 268 evaluation forms (pairwise comparison where a human evaluator judged which agent was the best, fastest, and more human-like performing the tasks) from 7 diferent human evaluators.
5Custom MineRL BASALT evaluation webpage: https://kairosminerl.herokuapp.com/.
All agent-generated videos were scaled from the original 64 × 64 image resolution returned by the environment to 512 × 512 image resolution in an attempt to make the videos clearer for the human evaluators. The videos of the ”Human” agent type were randomly selected from the video demonstrations provided by the MineRL BASALT competition and scaled to 512 × 512 image resolution to match the agent-generated videos. All videos were generated and saved at 20 frames per second to match the sampling rate of the Minecraft simulator used by both agents and humans.
Each combination of condition (behavior cloning, engineered, hybrid, human) and performance
metric (best performer, fastest performer, most human-like performer) is treated as a
separate participant of a one-versus-one competition where skill rating is computed using the
TrueSkillTM6 Bayesian ranking system [
The final mean and standard deviation of the TrueSkillTM scores computed for each performance metric and agent type are shown in Table 1. The scores were computed after collecting 268 evaluations from 7 diferent human evaluators. Our main proposed “Hybrid” agent, which combines engineered and learned modules, outperforms both pure hand-designed (“Engineered”) 6Microsoft’s TrueSkillTM Ranking System: trueskill-ranking-system 7TrueSkill Python package: https://github.com/sublee/trueskill and https://trueskill.org/. https://www.microsoft.com/en-us/research/project/ and pure learned (“Behavior Cloning”) agents in the “Best Performer” category, achieving 5.3% and 25.6% higher mean skill rating when compared to the “Engineered” and “Behavior Cloning” baselines, respectively. However, when compared to the “Human” scores, our main proposed agent achieves 21.7% lower mean skill rating, illustrating that even our best approach is still not able to outperform a human player with respect to best performing the task.
When looking at the “Fastest Performer” metric, our “Hybrid” agent outperforms both “Engineered” and “Behavior Cloning” baselines, respectively, scoring only 2.7% lower than the human players. As expected, in the “More Human-like Behavior” performance metric the “Human” baseline wins by a large margin, however, the “Hybrid” still outperforms all other baselines, including the “Behavior Cloning” agent, which is purely learned from human data. We attribute this to the fact that the pure learned agent did not make use of the safety-critical engineered subtask, which allowed the agent to escape bodies of water and other obstacles around the environment. Plots showing how the TrueSkillTM scores evolved after each match (one-to-one comparison between diferent agent types) are shown in Appendix D.
Table 2 breaks down the results presented in Table 1 for each separate task. Similar to what was discussed for Table 1, excluding the “Human” baseline, the “Hybrid” approach outperforms both “Behavior Cloning” and “Engineered” baselines in terms of mean skill rating in 8 out of the 12 performance metrics, or in 66.6% of the comparisons. Similarly, hybrid intelligence approaches, which include both “Hybrid” and “Engineered” baselines, outperform the pure learning “Behavior Cloning” approach in all 12 performance metrics, not taking into account the “Human” baseline. The “Hybrid” approach only outperforms the “Human” baseline in 4 out of the 12 performance metrics, or in 33.3% of the comparisons.
Particularly for the MakeWaterfall task, the proposed hybrid approach outperforms human players for all performance metrics. The largest margin observed is for the “Fastest Performer” metric; the hybrid approach scores 53.2% higher than the human players. This large margin comes from human players taking more time to find the best spot to place the waterfall and signal the end of the episode when compared to the engineered subtasks. Plots showing all results for each individual pairwise comparison are shown in Appendix E.
We now consider qualtitative evaluation of our agents. When solving the FindCave task8, the agent spawns in the plains biome and uses the learned navigation policy to search for caves while avoiding water while simultaneously building the map of its environment. Once the agent ifnds the cave, it throws the snowball to signal the end of the episode. In the MakeWaterfall task9, the hybrid agent spawns in a mountainous area, uses the learned navigation policy to climb the mountains, detects a good location to build the waterfall, builds it, then moves to the picture location using engineered subtasks, and throws the snowball to signal the end of the episode. For the CreateVillageAnimalPen task10, the agent uses the learned navigation policy and the state classifier to search for an open location to build a pen, builds the pen using an engineered building subtask that repeats the actions taken by the human demonstrators, uses the state classifier and odometry map to go to previously seen animal locations, and then attempts to lure them back to the pen and throws the snowball to signal the end of the episode. 8Sample trajectory of hybrid agent solving the FindCave task: https://youtu.be/MR8q3Xre_XY. 9Sample trajectory of hybrid agent solving the MakeWaterfall task: https://youtu.be/eXp1urKXIPQ. 10Sample trajectory of hybrid agent solving the CreateVillageAnimalPen task: https://youtu.be/b8xDMxEZmAE. Finally, when solving the BuildVillageHouse task11, our hybrid agent spawns nearby a village and uses the learned navigation policy and the state classifier to search for an open location to build a house, builds a house using an engineered building subtask that repeats the actions taken by the human demonstrators, tours the house, and throws the snowball to signal the end of the episode. Each of the described subtasks are shown in Appendix F as a sequence of frames.
In this paper, we presented the solution that won first place and was awarded the most humanlike agent in the 2021 NeurIPS MineRL BASALT competition, “Learning from Human Feedback 11Sample trajectory of hybrid agent solving the BuildVillageHouse task: https://youtu.be/_uKO-ZqBMWQ. in Minecraft.” Our approach used the available human demonstration data and additional human feedback to train machine learning modules that were combined with engineered ones to solve hierarchical tasks in Minecraft.
The proposed method was compared to both end-to-end machine learning and pure engineered solutions by collecting human evaluations that judged agents in head-to-head matches to answer which agent best solved the task, which agent was the fastest, and which one had the most human-like behavior. These human evaluations were converted to a skill rating score for each question, similar to how players are ranked in multiplayer online games.
After collecting 268 human evaluations, we showed that hybrid intelligence approaches outperformed end-to-end machine learning approaches in all 12 performance metrics computed, even outperforming human players in 4 of them. Our results also showed that incorporating machine learning modules for navigation as opposed to engineering navigation policies led to higher scores in 8 out of 12 performance metrics.
Overall, we demonstrated that hybrid intelligence approaches are viable solutions to solve hierarchical tasks when the subcomponents of the task are understood by human experts and limited human feedback data is available.
Research was sponsored by the Army Research Laboratory and was accomplished partly under Cooperative Agreement Number W911NF-20-2-0114. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the oficial policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
The labeling process of relevant states to the state-machine uses both mouse clicks and keyboard presses and takes place in a custom GUI, as seen in Figure 4. On the top left of the GUI, users can double-check which dataset and frame number they are labeling. Below that, the GUI displays the RGB frame to be labeled (center left) and the options for labels (center right, same labels for all tasks). To label a frame, the user can simply press the keyboard key corresponding to the desired label (shown in brackets, for example, [] for ℎ_), or click in the white circles in front of the label, which will then turn green, indicating that the label was selected. Frames will automatically advance when a key is pressed. If the users only use the mouse to select the labels, they will istill need to press the keyboard key to advance to the next frame (any key for a label that was already selected clicking).
The sequence of subtasks used by the state-machine for each task is defined as follows: • FindCave: • MakeWaterfall: 1. Use navigation policy to traverse the environment and search for caves; 2. If the state classifier detects the agent is inside a cave, throw a snowball to signal that the task was completed (end of episode). 1. Use navigation policy to traverse the environment and search for a place in the mountains to make a waterfall; 2. If the state classifier detects the agent is at the top of a mountain, build additional blocks to give additional height to the waterfall; 3. Once additional blocks are built, look down and place the waterfall by equipping and using the bucket item filled with water; 4. After the waterfall is built, keep moving forward to move away from it; 5. Once the agent has moved away from the waterfall, turn around and throw a snowball to signal that a picture was taken, and the task was completed (end of episode). • CreateVillageAnimalPen: • BuildVillageHouse: 1. Use navigation policy to traverse the environment and search for a place to build a house; 2. If the state classifier detects an open-space, build the house. The subtask to build the house directly repeats the actions taken by a human while building the house, as observed in provided demonstration dataset; 3. Once the house is built, move away from it, turn around and throw a snowball to signal that the task was completed (end of episode).
Figure 5 shows a sample of the web evaluation form available at https://kairosminerl. herokuapp.com/ that was used to collect human evaluations for each of the four baselines in a head-to-head comparison, similar to how the teams were evaluated during the oficial MineRL BASALT competition. Each participant was asked to see two videos of diferent agents performing the same task then answer three questions with respect to the agent’s performance. D. TrueSkillTM Score per Match (c) More Human-like Behavior
(a) Best Performer (b) Fastest Performer (c) More Human-like Behavior
Figures 10, 11, 12, and 13 show bar plots with the individual pairwise comparisons compiled from the human evaluations for the FindCave, MakeWaterfall, CreateVillageAnimalPen, and
(a) Best Performer (b) Fastest Performer (c) More Human-like Behavior
BuildVillageHouse tasks, respectively. Each bar represents the percentage of the time a given condition was selected as a winner for each performance metric by the human evaluator when they were presented with a video of the agent performance solving the task for each analysed condition. For example, when analyzing Figure 10(a), we can see that the human evaluator was presented with a video of the “Behavior Cloning” agent and another from the “Engineered” agent, they selected the “Engineered” agent as the best performer 33.3% and the “Behavior Cloning” agent 22.2% of the time. The remaining accounts for the “None” answer to the questionnaire selected when none of the agents were judged to have solved the task.
When directly comparing the “Behavior Cloning” baseline to the main proposed “Hybrid” method for all tasks, as shown in Figures 10, 11, 12, and 13 (c) plots, we observe that the proposed hybrid intelligence agent always match or outperforms the pure learned baseline. This is similar to the case we compare the “Engineered” agent to the “Hybrid” agent, where the proposed hybrid method outperforms the fully engineered approach in all tasks except the BuildVillageHouse task, as seen in Figure 13. The human players always outperform the hybrid agent with exception to the MakeWaterfall task, where the “Hybrid” agent is judged to better solve the task 70% of the time, to solve it faster 90% of the time, and even present a more human-like behavior 60% of the time. The “Hybrid” agent performing better can be attributed to the fact that the human players were not always able or willing to solve the task as described (d) Engineered vs Hybrid (e) Human vs Engineered (f) Human vs Hybrid
In terms of qualitative results, Figures 14, 15, 16, and 17 show a sample episode illustrated by a sequence of frames of our hybrid agent solving the FindCave, MakeWaterfall, CreateVillageAnimalPen, and BuildVillageHouse tasks, respectively. Each figure shows the image frames received by the agent (left panel) overlaid with the actions taken (top), output of the state classifier (center), and the subtask currently being followed (bottom). The right panel shows the estimated odometry map overlaid with the location of the relevant states identified by the state classifier. Link to the videos are provided in the figure captions. (d) Engineered vs Hybrid (e) Human vs Engineered (f) Human vs Hybrid
(a) BC vs Engineered (b) BC vs Human (c) BC vs Hybrid (d) Engineered vs Hybrid (e) Human vs Engineered (f) Human vs Hybrid (d) Engineered vs Hybrid (e) Human vs Engineered (f) Human vs Hybrid