A Computational Model of the Evolution of Antipredator Behavior in Situations with Temporal Variation of Danger using Simulated Robots

Abstract

The threat-sensitive predator avoidance hypothesis states that preys are able to assess the level of danger of the environment by using direct and indirect predator cues. The existence of a neural system which determines this ability has been studied in many animal species like minnows, mosquitoes and wood frogs. What is still under debate is the role of evolution and learning for the emergence of this assessment system. We propose a bio-inspired computing model of how risk management can arise as a result of both factors and prove its impact on fitness in simulated robotic agents equipped with recurrent neural networks and evolved with genetic algorithm. The agents are trained and tested in environments with different level of danger and their performances are analyzed and compared.

Keywords

• Risk assessment Threat sensitivity Bio-inspired computing Recurrent Neural Networks Evolution

-

Potential threats are signaled by uncertain or ambiguous cues and differ from active threats in that they do not require an immediate interaction with their source but graded behavioral adjustments. However, once the state of vigilance has been heightened, there is no positive evidence that the risk was eliminated [ 1 ].

The presence of a defensive system that allows organisms to detect and assess potential threats has been extensively studied by ethological psychologists, ethologists and neuroscientists, and it has been labeled in different ways, like "security motivation system" [ 2 ], "hazard management system" [ 3 ] or "risk assessment system" [ 4 ]. This mechanism is thought to have been shaped throughout the evolution in order to allow individual preys to respond appropriately to the degree of predatory threat. As stated by the threat-sensitive predator avoidance hypothesis, animals need to trade-off antipredator responses against other activities such as feeding or territorial defense and they can do so by altering their avoidance behavior according to the magnitude of the danger [ 5 ]. Animals are also able to detect temporal variation in the risk of predation. These changes over time can occur seasonally, daily or periodically and affect the prey’s adaptation and fitness. The idea that preys are able to optimally adjust their behavior across different states of threat is called “risk allocation hypothesis” [ 6 ].

The cues of the presence of a danger can be direct (such as visual, tactile or auditory) or indirect (e.g. odor); while the former type signals unambiguously that the threat is near, the latter needs to be carefully processed by the animal in order to allow the right behavioral decision [ 7 ]. The absence of direct cues does not constitute a proof of the absence of a threat. As Woody proposes, only an internal signal of security – or, in humans, a subjective conviction or feeling – can allow the termination of defensive behaviors [ 8 ][ 2 ]. 1.1

Antipredator Behavior and Uncertainty

The ability of preys to respond appropriately to dangerous situations and to specific predators is fundamental for the species’ survival and adaptation. Since animals experience a huge variety of situations, the debate about how they learn to recognize potential predators is still open. Many studies proved the existence of an innate recognition mechanism for predators in mammals, amphibians and fishes, but other species exhibited it only as a result of learning [ 9 ]. Scheurer, in his study, proved that steelhead trout who had no experience of their common predator Dolly Varden for 15 generations, exhibited genetic threat responses when exposed to their odor again [ 10 ]. An evidence of the importance of learning for other species, instead, is represented by the case of goldfish, as showed by Zhao and colleagues. [ 11 ]. In their study, groups of goldfishes were conditioned to the presence of predator with different concentrations of its odor. Those which were conditioned in the most dangerous environment (highest concentration) resulted in a an overall higher survivability when compared with other groups.

The assessment of predation risk is complicated by the variability of danger across space and time, and of the predator itself. The threat level may vary according to the day/night shift, seasonal changes, growth or environment [ 6 ][ 12 ]. This leads to a high level of uncertainty that preys need to face when assessing the risk and deciding behavior. Ferrari et al. showed that fathead minnows are able to continuously update their perception of risk based on their most recent experience with the predator, disproving the hypothesis that they may average the risk of their past learning experiences [ 13,14 ]. This species, in fact, used only the last information acquired to shape the intensity of their threat response. Wood frogs, as demonstrated by Ferrari et al., are also able to associate the level of risk with the time of day: the defensive response towards the specific predators was significantly higher when the hours of the exposition were matching those of the conditioning [ 15,16 ]. 1.2

Neural Circuits of Threat Sensitivity

The circuit of threat perception in the brain has been extensively studied in both humans and animals. Potential and active threats, in fact, are processed by the same regions and antipredator behavior is often associated with the emotion of fear. Fear has been the most studied affective state, due to the simplicity of eliciting it in rodents and other animals. There is also a strong similarity in the response patterns to threats among mammalians.

The structures primarily involved in the activation of responses to threatening stimuli are the amygdala, crucial for processing every aspect of the emotion of fear [ 17,18 ] and its connection to the hippocampus, which links threat sensation to episodic memory determining the association [ 19 ]. These direct and immediate signals from the amygdala are also responsible of the incredibly fast processing and response of the organisms to fearful stimuli, allowing higher chance of survival. The conscious perception of danger is mediated by the medial prefrontal cortex while physiological responses are a result of the activation of the hypothalamic-pituitary-adrenal axis, which controls the level of ADH and ACTH, determining the secretion of cortisol [ 20 ]. 1.3

Threat Sensitivity and Psychopathology

The importance of the risk assessment system and sensitivity to danger is extremely high in terms of adaptation, especially since this mechanism plays a key role for the presentation of anxious arousal and heightened vigilance. The prolonged activation of behavioral defense circuits can, in fact, result in a pathological response to threat, causing the birth of disorders as generalized anxiety and depression [ 21 ]. Attentional biases to threat stimuli can be the trigger of anxiety disorders since these cognitive distortions can lead to hyper-arousability and overestimation of the harmfulness of the environment [ 22,23 ]. For example, anxious individuals detect threatening cues and stimuli faster than controls [ 24 ].

Defensive behavior is not cost free for humans and animals, and, in particular, avoidance behaviors can cause restricted or limited access to fundamental resources like food, social activity, exploration [ 20 ]. Therefore, a persistent activation of a system designed for a short-term response is considered a maladaptive strategy [ 21 ]. Chronic stress is also related to permanent damage to the hippocampus [ 25 ]. 1.4

Risk Allocation and Minimum Behavioral Response Threshold

In juvenile cichlids, the defensive response is a function of the concentration of predator cues experienced. Brown et al., in fact, demonstrated that the intensity of the defensive behavior was stronger if the alarm cue concentration was higher and weaker if this concentration was lower. Other than that, the minimum stimulus concentration able to evoke the antipredator behavior was lower if the cichlids had been exposed to higher concentration in the days before the test, and higher if the cichlids had been previously exposed to a lower concentration, therefore showing a higher tolerance to the alarm cues. These results support both the risk allocation and the threat sensitive predator avoidance hypothesis. The minimum concentration needed to elicit an overt response was labeled “minimum behavioral response threshold” [ 26 ].

In our study we aim at investigating the emergence of threat sensitivity by using simulated robots embedded with a recurrent neural network (RNN) and evolved with standard genetic algorithm. The neural network architecture allows the agents to collect information from the environment and try to determine whether is safe or not. Other than that, we test the risk allocation hypothesis by varying the level of danger throughout both generations and single trials, analyzing the difference in behavior of the robots in each condition. 2

Materials and Methods

The framework we used for carrying out the robot simulations represents a modification of the experimental setting described in a preliminary study [ 27 ]. Simulated agents equipped with different neural network architectures were evolved to learn to discriminate dangerous stimuli from safe ones on a whiteboard in different conditions. We showed the effectiveness of the methodology and analyzed the avoiding behavior exhibited by the robots in terms of fitness and performance.

The software used for simulating the environment and the robots is Evorobot*, an open source simulator which allows to train and test neural networks embedded in physical robots and then to transfer the result of the simulation in a real environment [ 28 ]. The agent structure we selected takes its features from the iCub, a humanoid robot commonly used for experiments in the fields of cognitive science and modeled to reproduce the behavior of a three-years old child. In our case, we integrated in our system just its visual apparatus and pointing abilities. The visual system of the robot is composed by a pan-tilt camera, and its simulated version is based on the prototype of an artificial retina described by Floreano et al. [ 29 ]. The camera is able to perceive an area of 100x100px, discriminating stimuli of different luminance, and is allowed to integrate an additional 12 d.o.f zooming feature which we disabled for our task. The environment is composed of a squared whiteboard (400x400px) which the robot is free to explore during each trial. On the board, there are 16 randomly positioned stimuli, represented by red circles, which can be dangerous or safe to touch, according to the condition. The robot is allowed to perform 2 actions with its hand: touch – which should be used only to pick up a safe stimulus – or swipe – to discard a dangerous stimulus. If it makes the right action expected on the stimulus presented, it gains fitness; if it does not, it loses time in terms of life steps.

The architecture of the fully connected RNN is shown in Fig 1. The input layer is composed by: 1) a 7x7 grid of 49 visual neurons responsible for the perception of the squared area of the retina (the retina does not have any foveal vision); 2) a sigmoidal unit signaling the accumulation of information regarding the safety of the situation; 3) a sigmoidal unit signaling the perceived danger in the environment; 3) a sigmoidal unit which signals the perception of time and increases its value as a function of the time steps inside each trial. The three sigmoidal units follow the function below:

In the case of safe and danger sensation units, X represents the total number of stimuli inside the board (maximum value X = 16), and x respectively the number of correct actions performed (discard the dangerous stimuli or pick the safe one) in the safe sensation unit or the incorrect actions performed (pick the dangerous stimuli or discard the safe one) in the danger sensation unit. The maximum value of x depends on the percentage of dangerous and safe stimuli in each trial. The parameters β and α are chosen so that F(0) ≈ 0, F(16) ≈ 1, and their value is β = 10, α = 3.

In the case of time perception unit, X represents the total number of life steps T for each trial (maximum value X = 1000) , x represents the current life step t, and the parameters so that F(0) ≈ 0, F(1000) ≈ 1 are β = 10 and α = 3.

The hidden layer is composed of 20 sigmoidal recurrent units while the output layer consists of: 1) 2 neural units which control the pan/tilt movements of the visual exploration; 2) a motor unit for the “pick” action; 3) a motor unit for the “discard” action.

The RNNs used in this experiment are trained using standard genetic algorithm with 2% mutation rate. For each condition, 10 populations of robots were evolved for 5000 generations and each of these generations was trained on 30 trials. Three different conditions were selected to evolve the simulated robots: 1) a high risk/high reward condition; 2) medium risk/medium reward condition; 3) balanced risk and reward. In the high risk/high reward condition, the trials could contain either 100% dangerous stimuli or 100% safe stimuli. Half the generations (2500) contained 10 triplets composed by 1/3 of 100% dangerous trials and 2/3 of 100% safe trials and the other half contained 10 triplets composed by 1/3 of 100% safe trials and 2/3 of 100% dangerous trials. In the medium risk/medium reward condition, the trials could contain 75% dangerous stimuli or 75% safe stimuli. Half the generations (2500) contained 10 triplets composed by 1/3 of 75% dangerous trials and 2/3 of 75% safe trials and the other half contained 10 triplets composed by 1/3 of 75% safe trials and 2/3 of 75% dangerous trials. In the balanced condition, each trial contained 50% dangerous stimuli.

During each trial, the robot could explore the board for a total T of 1000 time steps. In case it perceived a target, there were four possible outcomes: if the activation of the “pick” unit was over a threshold of 0.7, the stimulus was picked, the robot had a loss of 1 time step (t = t +1) and the stimulus disappeared; if the activation of the “discard” unit reached at least 0.7, the stimulus was discarded, the robot had a loss of 1 time step and the stimulus disappeared; if both the action units were over the threshold, the robot lost 1 time step; if none of the action units was activated, the stimulus was considered ignored. For each action unit over the threshold, if no target was perceived, the robot had a loss of one time step.

If the robot performed the correct action on a stimulus, he gained a reward G as in (2), where β = 10 and α = 4. (2)

Since the reward was a function of the current step t, the robots needed to learn to recognize both the condition and the time when to perform the right action, trying to estimate the level of danger but also wait for the optimal time t to act on the stimuli. 3

Results and discussion

The evolution of the fitness curves of the best individual on 10 belonging to each of the three different conditions (high gain/risk, medium gain/risk and balanced gain/risk) is displayed in Figure 2. As shown, the highest reward was gained by the RNN trained in the high risk/gain condition. Of all the conditions, in fact, this was the only to provide the robot a certainty about the level of threat after a single encounter with the stimulus. Thanks to this certainty, robots did not refrain from continuing their exploration and activity like picking the safe stimuli. When the environmental cues are clear and unambiguous, the ability to discriminate dangerous from safe situation can get the best performance. The difference between the fitness curve of the certainty situation is significantly different than the other two (p = ,000 ), as shown in Table 1.

Further investigation will try to test the “minimum behavioral response threshold” for the agents to determine the uncertainty which triggers the defensive behavior.

In the test phase, we analyzed the performance of the best individual for each of the conditions on 1000 trials, of which 50% belonged to the safe condition (with a percentage of safe items respectively of 100%, 75% and 50%) and 50% belonged to the dangerous condition (with a percentage of dangerous items respectively of 100%, 75% and 50%). We aimed at investigating the difference in the response pattern among different time steps range. Therefore, we divided the 1000 time steps for each of the trial into 20 intervals of 50 steps and conducted LSD post hoc MANOVA on the means of correct and incorrect action performed on each stimuli of the 1000 trials for all the 50 intervals. We take in consideration for the analysis a comparison between early steps (t between 150 and 200) and late steps (t between 750 and 800).

Supporting theories on animal species, the robots used earlier steps, in which the risk/gain was dramatically lower, to explore the environment, and as soon as the time steps increased a behavioral strategy emerged to face the danger. There is a strong pattern difference in the actions performed by the robots evolved in the situation of high risk and those evolved in the situation of medium risk.

As shown in Table 2 and Table 3, robots evolved and tested in the situation of high risk tended to activate the same neuron (pick) at the beginning, determining a nonsignificant difference between the means of the safe stimuli picked and of the dangerous stimuli picked and vice versa between the dangerous stimuli discarded and the safe stimuli discarded (p > ,005). This data suggests that this population of robots began each trial by trying the same strategy and determining the outcome in order to disambiguate between safe and threatening situation. This pattern is absent in robots evolved in the uncertainty condition (medium risk), and we can explain this as a reduction of exploration steps.

Analyzing the differences between late steps, we can see that while robots evolved in the situation of certainty inverted their activity patterns showing a correlation between the means of correct actions on dangerous stimuli and correct actions on safe stimuli, in the case of robots evolved in medium risk all the means are significantly different, apart from the difference between the means of the correct actions on safe stimuli and incorrect actions on dangerous stimuli, which can be interpreted as an effort to try to gain as much fitness as possible when encountering stimuli against a reduced exploration behavior.

A final analysis was conducted by testing the best individual evolved in the certainty condition in an environment with medium risk. The results of the LSD post hoc MANOVA are summarized in Table 4. The results indicated that the genetically evolved response patterns showed in the high risk environment was maintained, and the increased sensitivity towards the danger lead to a better performance, giving an evidence of the impact of an high arousal to danger in terms of fitness. This result is in accordance with the previously mentioned study conducted by Zhao et al. [ 11 ].

Further research will be addressed to find the danger threshold under which the environment stops eliciting the robots’ genetically learned pattern which lead to adaptation in their environment. We aim also to find out the effect of the most recent environment experienced by the robot on its defensive behavior.

Finally, robots evolved with a danger level of 50% did not show any learned pattern at all, since there was no cue on which to rely to try to disambiguate dangerous from safe environments and determine the potential presence of a threat. Post hoc analysis, which are not reported here for brevity, did not show any significant difference between correct/incorrect actions on safe/dangerous stimuli in each of the time step interval. 4

Conclusion

We proposed a computational model of the evolution of antipredator behavior in situations with various degree of danger using simulated robots embedded with RNNs and evolved with standard genetic algorithm. We demonstrated the importance of both innate and genetic factors for the emergence of an effective antipredator behavior and higher survivability. We tested the threat-sensitive predator avoidance hypothesis by evolving virtual robots in conditions of high risk/gain, medium risk/gain and balanced risk/gain, proving that in situation of uncertainty the agents refrained from exploring the environment and limited their actions on the environment.

We also tested the risk allocation hypothesis, by testing robots in experimental conditions with a shift of threat/reward between the early steps of the trial and the late ones, proving that the agents evolved in the high risk environment were able to learn and adapt to this temporal shift.

Finally, we compared the performance of agents evolved in certainty and uncertainty conditions in a medium risk/gain environment, analyzing their differences. 5

References

1. Woody , E. , & Boyer , P. : Threat-detection and precaution: Introduction to the special issue . Neuroscience & Biobehavioral Reviews , 35 ( 4 ), 989 - 990 ( 2011 ).

2. Woody , E. Z. , & Szechtman , H.: Adaptation to potential threat: the evolution, neurobiology, and psychopathology of the security motivation system . Neuroscience & Biobehavioral Reviews , 35 ( 4 ), 1019 - 1033 ( 2011 ).

3. Tooby , J. , & Cosmides , L. : The evolved architecture of hazard management: Risk detection reasoning and the motivational computation of threat magnitudes . Behavioral and Brain Sciences , 29 ( 06 ), 631 - 633 ( 2006 ).

4. Blanchard , D. C. , Griebel , G. , Pobbe , R. , & Blanchard , R. J. : Risk assessment as an evolved threat detection and analysis process . Neuroscience & Biobehavioral Reviews , 35 ( 4 ), 991 - 998 ( 2011 ).

5. Helfman , G. S. : Threat-sensitive predator avoidance in damselfish-trumpetfish interactions . Behavioral Ecology and Sociobiology , 24 ( 1 ), 47 - 58 ( 1989 ).

6. Lima , S. L. , & Bednekoff , P. A. : Temporal variation in danger drives antipredator behavior: the predation risk allocation hypothesis . The American Naturalist , 153 ( 6 ), 649 - 659 ( 1999 ).

7. Kavaliers , M. , & Choleris , E. : Antipredator responses and defensive behavior: ecological and ethological approaches for the neurosciences . Neuroscience & Biobehavioral Reviews , 25 ( 7 ), 577 - 586 ( 2001 ).

8. Curio , E.: Proximate and developmental aspects of antipredator behavior . Advances in the Study of Behavior , 22 , 135 - 238 ( 1993 ).

9. Mirza , R. S. , Ferrari , M. C. , Kiesecker , J. M. , & Chivers , D. P. : Responses of American toad tadpoles to predation cues: behavioural response thresholds, threat-sensitivity and acquired predation recognition . Behaviour , 143 ( 7 ), 877 - 889 ( 2006 ).

10. Scheurer , J. A. , Berejikian , B. A. , Thrower , F. P. , Ammann , E. R. , & Flagg , T. A. : Innate predator recognition and fright response in related populations of Oncorhynchus mykiss under different predation pressure . Journal of Fish Biology , 70 ( 4 ), 1057 - 1069 ( 2007 ).

11. Zhao , X. , Ferrar , M. C. , & Chivers , D. P. : Threat-sensitive learning of predator odours by a prey fish . Behaviour , 143 ( 9 ), 1103 - 1121 ( 2006 ).

12. Brown , G. E., Ferrari , M. C. , & Chivers , D. P. : Learning about danger: chemical alarm cues and threat-sensitive assessment of predation risk by fishes . Fish cognition and behavior , 59 - 80 ( 2011 ).

13. Ferrari , M. C. , & Chivers , D. P. : Learning threat-sensitive predator avoidance: how do fathead minnows incorporate conflicting information? . Animal Behaviour , 71 ( 1 ), 19 - 26 ( 2006 ).

14. Ferrari , M. C. , Capitania-Kwok , T. , & Chivers , D. P. : The role of learning in the acquisition of threat-sensitive responses to predator odours . Behavioral Ecology and Sociobiology , 60 ( 4 ), 522 - 527 ( 2006 ).

15. Ferrari , M. C. , & Chivers , D. P.: The ghost of predation future: threat-sensitive and temporal assessment of risk by embryonic woodfrogs . Behavioral ecology and sociobiology , 64 ( 4 ), 549 - 555 ( 2010 ).

16. Ferrari , M. C. , & Chivers , D. P. : Temporal variability, threat sensitivity and conflicting information about the nature of risk: understanding the dynamics of tadpole antipredator behaviour . Animal Behaviour , 78 ( 1 ), 11 - 16 ( 2009 ).

17. Phillips , R. G. , & LeDoux , J. E. : Differential contribution of amygdala and hippocampus to cued and contextual fear conditioning . Behavioral neuroscience , 106 ( 2 ), 274 ( 1992 ).

18. LeDoux , J.: The emotional brain, fear, and the amygdala . Cellular and molecular neurobiology , 23 ( 4-5 ), 727 - 738 ( 2003 ).

19. Bishop , S. J.: Neurocognitive mechanisms of anxiety: an integrative account . Trends in cognitive sciences , 11 ( 7 ), 307 - 316 ( 2007 ).

20. O 'Donovan , A. , Slavich , G. M. , Epel , E. S. , & Neylan , T. C. : Exaggerated neurobiological sensitivity to threat as a mechanism linking anxiety with increased risk for diseases of aging . Neuroscience & Biobehavioral Reviews , 37 ( 1 ), 96 - 108 ( 2013 ).

21. Gilbert , P. : Evolutionary approaches to psychopathology: The role of natural defences . Australian and New Zealand Journal of Psychiatry , 35 ( 1 ), 17 - 27 ( 2001 ).

22. Levitan , R. , Hasey , G. , & Sloman , L. : Major depression and the involuntary defeat strategy; biological correlates. Subordination and defeat: An evolutionary approach to mood disorders and their therapy , 95 - 114 ( 2000 ).

23. Gilbert , P.: The evolved basis and adaptive functions of cognitive distortions . British Journal of Medical Psychology , 71 ( 4 ), 447 - 463 ( 1998 ).

24. MacLeod , C. , Mathews , A. , & Tata , P. : Attentional bias in emotional disorders . Journal of abnormal psychology , 95 ( 1 ), 15 ( 1986 ).

25. Lupien , S. J., de Leon, M. , De Santi , S. , Convit , A. , Tarshish , C. , Nair , Lupien, Sonia J. , Mony de Leon, Susan De Santi, Antonio Convit, Chaim Tarshish, N. Px V. Nair , Mira Thakur, Bruce S. McEwen, Richard L. Hauger, and Michael J. Meaney .: Cortisol levels during human aging predict hippocampal atrophy and memory deficits . Nature neuroscience , 1 ( 1 ), 69 - 73 ( 1998 ).

26. Brown , G. E., Rive , A. C. , Ferrari , M. C. , & Chivers , D. P.: The dynamic nature of antipredator behavior: prey fish integrate threat-sensitive antipredator responses within background levels of predation risk . Behavioral Ecology and Sociobiology , 61 ( 1 ), 9 - 16 ( 2006 ).

27. Pacella , D. , Gigliotta , O. , & Miglino , O. : Studying the Evolutionary Basis of Emotions Through Adaptive Neuroagents: Preliminary Settings and Results . In Advances in Artificial Life and Evolutionary Computation (pp. 47 - 57 ). Springer International Publishing ( 2014 ).

28. Nolfi , S. , & Gigliotta , O. : Evorobot*. In Evolution of communication and language in embodied agents (pp. 297 - 301 ). Springer Berlin Heidelberg ( 2010 ).

29. Floreano D. , Kato T. , Marocco D. , Sauser E.: Co-evolution of Active Vision and Feature Selection, Biological Cybernetics , vol 90 , 218 - 228 ( 2003 ).