=Paper=
{{Paper
|id=Vol-3007/2021-invited
|storemode=property
|title=Sharks, Zombies and Volleyball:Lessons from the Evolutionary Computation Bestiary
|pdfUrl=https://ceur-ws.org/Vol-3007/2021-invited.pdf
|volume=Vol-3007
|authors=Felipe Campelo,Claus Aranha
|dblpUrl=https://dblp.org/rec/conf/lifelike/CampeloA21
}}
==Sharks, Zombies and Volleyball:Lessons from the Evolutionary Computation Bestiary==
https://ceur-ws.org/Vol-3007/2021-invited.pdf
Sharks, Zombies and Volleyball:
Lessons from the Evolutionary Computation Bestiary
Felipe Campelo1 , Claus Aranha2
1
College of Engineering and Physical Sciences, Aston University UK
2
Faculty of Systems and Information Engineering, University of Tsukuba, Japan
f.campelo@aston.ac.uk
Abstract solve optimization problems (Bremermann et al., 1962; Fo-
gel and Fogel, 1995; Beyer and Schwefel, 2002; Holland,
The field of optimization metaheuristics has a long history of
finding inspiration in natural systems. Starting from classic 1975; Kirkpatrick et al., 1983; Kennedy and Eberhart, 1995;
methods such as Genetic Algorithms and Ant Colony Opti- Dorigo et al., 1996). The development of these methods was
mization, more recent methods claim to be inspired by natural often experiment-driven rather than theory-led, which was
(and sometimes even supernatural) systems and phenomena - not surprising for a new field without an existing theoretical
from birds and barnacles to reincarnation and zombies. Since framework. Although the algorithms were in most cases de-
2014 we publish a humorous website, The Bestiary of Evo-
lutionary Computation, to catalog these methods, witnessing scribed and discussed using metaphor-specific language, be-
an explosion of metaphor-heavy algorithms in the literature. yond what would be necessary for the understanding of the
While metaphors can be powerful inspiration tools, we argue computational concepts being implemented,1 the elements
that the emergence of hundreds of barely discernible algo- of good scientific practice were present: an original idea
rithmic variants under different labels and nomenclatures has would suggest a new method, which would be tested, re-
been counterproductive to the scientific progress of the field,
as it neither improves our ability to understand and simulate fined and compared against state-of-the-art approaches for
biological systems, nor contributes generalizable knowledge the problems they were intended to solve. Attempts at theo-
or design principles for global optimization approaches. In retical development would be advanced, discussed, adopted
this short paper we discuss some of the possible causes of this or refuted depending on their success in explaining the be-
trend, its negative consequences to the field, as well as some havior of each method. This approach led to increased de-
efforts aimed at moving the area of metaheuristics towards a
better balance between inspiration and scientific soundness. velopments in metaheuristic methodologies, with excellent
results for the solution of a variety of applied problems with
characteristics that did not allow the use of traditional math-
Introduction ematical programming methods.
In 1865, August Kekulé proposed that the structure of ben-
zene was a hexagonal ring of six carbon atoms, solving a The Age of the Metaphors
problem that had confounded chemists for decades. Kekulé
The success of these early nature-inspired metaheuristics
championed visual scientific creativity, and mentioned that
naturally led to increasing attempts to find other phenom-
his inspiration came from a day-dream about an Ouroboros,
ena that could provide insights for optimization. Around the
which is a symbol depicting a serpent or dragon eating its
end of the 1990s and early 2000s, this pursuit of insightful
own tail. However, it is clear to anyone who has gone
inspiration from natural processes started to transform into
through even a basic course in organic chemistry that sci-
a different phenomenon: an increasing number of publica-
entists do not discuss their work using snake anatomy termi-
tions claiming to present revolutionary ideas or even “novel
nology, or try to come up with new compounds by carefully
paradigms for optimization”, based on ever more obscure
examining legendary reptiles. Despite the importance he
social, natural, or even supernatural metaphors.
attributed to visual creativity, August Kekulé himself only
Inspired by a “Cat Swarm Optimization” paper, in
went on record about his original inspiration in 1890, at a
2014 we started gathering examples of particularly absurd
meeting held in his honor (Robinson, 2010).
metaphors published in peer-reviewed venues, in a hu-
Throughout history, scientists and engineers have drawn
morous catalog named the Evolutionary Computation Bes-
inspiration from different sources, such as the natural world,
tiary (Campelo and Aranha, 2021). As the website started to
dreams or personal experiences. Ideas from biology and ob-
attract attention, several colleagues contacted us to recom-
servations of natural processes have inspired several inter-
esting developments within computer science and engineer- 1
Notice the contrast with the opening anecdote about Kekulé’s
ing since at least the 1960s, suggesting innovative ways to inspiration.
Copyright ©2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
ent ways to build an optimizer. As of July 2021, the Bes-
tiary lists around 260 unique entries, and a recent compre-
hensive taxonomy of nature- and bio-inspired optimization
approaches suggests as many as 360 (Molina et al., 2020).
This massive amount of distinct algorithms, each claiming
to present a unique way to solve optimization problems is at
odds with the relatively simple structure that most of these
techniques follow, as well as with the existence of general
algorithmic design patterns that generalize many of these
techniques (de Jong, 2006; Stegherr et al., 2020; Stegherr
and Hähn, 2021; de Armas et al., 2021).
This explosion of metaphor-centered methods has led to
an intense fragmentation of the literature into tens of small,
barely-discernible niches. The use of metaphor-heavy lan-
guage when proposing new methods is partly responsible
for this, as it adds an unnecessary obstacle to comparing
the similarities and differences between two methods at first
glance. How should one compare the ability of a bird to
Figure 1: New metaphor-based methods between 2000 and drop a cuckoo egg from its nest to the behavior of a scouting
2020, as catalogued in the Evolutionary Computation Bes- bee? It takes a deeper reading to find out, for instance, that
tiary. The apparent decline in 2020 is, unfortunately, un- these two completely different descriptions refer to the same
likely to represent a true reduction in the number of new underlying computational action, namely generating a new
metaphors, and is possibly the consequence of delays in random solution when the search has stalled.
finding and recording new entries on the website. This pattern of reinventing the wheel is seen quite fre-
quently in the metaphor-based optimization literature, as de-
nounced by Sörensen (2013). For instance, careful analy-
mend entries based on new and progressively more bizarre sis by Weyland (2010, 2015) showed that Harmony Search
metaphors. The raw number of different methods added to was nothing more than a special case of Evolutionary Strate-
the Bestiary showed that this was a growing and concerning gies. Piotrowski et al. (2014) analysed the novelty (or lack
phenomenon. thereof) of the Black Hole algorithm, while Villalón et al.
Figure 1 illustrates this point. Between 2000 and 2008 (2018, 2020) did the same for the Intelligent Water Drops,
we see the publication of a few methods per year (including Grey Wolf, Firefly and Bat algorithms. In all these cases, the
algorithms based on sheep flocks, musicians, plant saplings, conclusions were unequivocal - the “novel” algorithm did
parliamentarism elections and the Big Bang). This increased not in fact contain any novelty beyond the use of a metaphor-
to an average of over one per month on average between specific language, and in fact described another well-known
2009 and 2013 (with methods referring to semi-intelligent computational algorithm already in use - in some cases for
water drops, group counselling, sports championships, fire- several decades. Based on our reading of the literature, we
flies, paddy fields and mountain climbers), and then to an av- would expect to find the same pattern of repeated or rein-
erage of two new metaphor-based methods being published vented ideas in many - if not most - metaphor-based meth-
every month in the peer-reviewed literature after 2014 (in- ods, if subject to similar scrutiny. Even in the few cases
cluding not only the sharks, zombies and volleyball meth- where new ideas may be found, they become tied to the
ods mentioned in the title of this paper, but also reincarna- specific nomenclature of the metaphor, instead of being de-
tion, four different whale-based and three distinct football- scribed in a way that would allow analysis and comparisons
based methods, barnacles, chicken swarms, interior design to other methods.
and decoration, and several others).2 . Another common issue is the generally poor method-
ological standards of the experimental results reported in
Why is this a problem? many of these papers. These problems were not exclu-
The sheer volume of papers following the same general sive to metaphor-based methods, but rather part of an area
pattern raises a few important questions. The first one is without a strong statistical tradition, as documented since
whether there really are hundreds of fundamentally differ- at least the mid-1990s (Hooker, 1994, 1995; Barr et al.,
2 1995; Eiben and Jelasity, 2002; Garcı́a-Martı́nez et al., 2017;
Direct citations of the papers describing the metaphor-based
methods mentioned in this work are intentionally not provided. The Campelo and Takahashi, 2019). The field of metaheuris-
original references are listed in (Campelo and Aranha, 2021), and tics has been continuously improving its standards and de-
can be easily found by searching the name of the specific metaphor. veloping better methodological practices (Bartz-Beielstein
et al., 2020), but the experimental validation presented in
the majority of metaphor-centered papers continues to suf-
fer from very serious issues. These include problems that
have long been identified (Hooker, 1994, 1995; Eiben and
Jelasity, 2002; Garcı́a-Martı́nez et al., 2017; Campelo and
Takahashi, 2019), including the almost exclusive focus on
competitive testing rather than on the underlying working
principles of algorithms; overfitting of algorithms and im-
plementations to test problems; the absence of well-defined
underlying hypotheses; the exclusive use of very similar al-
gorithms (i.e., other metaphor-based methods) as compar-
ison baselines, instead of state-of-the-art methods; unbal-
anced tuning efforts between the proposed and competing
algorithms; and a general lack of reproducibility.
Application-oriented venues are particularly vulnerable to
being contaminated by “novel” metaphor-based methods.
This appears to happen for two main reasons. First, re-
searchers in application fields who look at metaheuristics
for solutions to optimization problems get lost in the mul-
titude of papers proposing methods with strange names, un-
clear connection to each other, and seemingly outstanding
results. Often, the choice of which method to use is defined
by which names appear more frequently or are cited most of-
ten. Chicco and Mazza (2020) discuss the difficulties faced
by application researchers when evaluating metaheuristics
in more detail. Second, metaphor creators who find it diffi-
cult to publish their research in more optimization-focused
journals sometimes opt for submitting their “novel” meth-
ods to application journals, where reviewers are less likely
to be familiar with the technical shortcomings of these meth-
ods, or sometimes even with basic concepts of optimization.
Figure 2: Distribution of new metaphor-based methods
In more exasperating cases, the algorithm is submitted to
(since 2000) by publication venue, highlighting the jour-
a journal in the area of the the metaphor. A recent exam-
nals where two or more of these “novel” methods were pub-
ple is a “COVID-19 optimization algorithm”, published in a
lished. This refers only to the journal where the methods first
high-impact biomedical and health informatics journal, even
appeared, not journals that published later applications or
though the method does not actually address any issues re-
refinements. Notice that although optimization / computa-
lated to these areas. The main justification of that particular
tional intelligence journals are present amongst the top pub-
paper, as presented in its abstract, can be briefly summarised
lishers, there is a marked prevalence of application-oriented
as:
journals, particularly in engineering domains.
1. Covid-19 is overloading hospitals and causing death.
2. Covid-19 must be contained, and social distancing must
Another unfortunate result of this contamination is that
be ensured.
optimization tracks of some application journals sometimes
3. Therefore, we need an efficient optimizer capable of become “colonized” by cliques that keep publishing minute
“solving NP-hard (sic) in addition to applied optimization variations of bizarre methods with little oversight. Figure 2
problems.” illustrates part of this phenomenon, by showing the preva-
lence of application-oriented journals amongst the venues
This argument presents not only a clear non sequitur where the first papers describing metaphor-based methods
(“Covid-19 is a problem, therefore we need a new optimiza- have appeared.
tion algorithm”), but also suggests lack of understanding of
basic aspects of computational complexity. Regardless of
that, the paper was published, which suggests that the re-
Where does this problem come from?
viewers themselves also lacked the particular skill set to de- The proliferation of metaphor-heavy algorithms in the meta-
tect these and other shortcomings of the work. heuristics literature is a multi-faceted problem, involving
multiple actors with different motivations. Some factors, but also show how improper experimentation is being used
however, may be identified as potential contributors to this to claim spurious results in the metaphor-based literature.
problem. Works showing the lack of novelty in many of these meth-
The first is a structure of perverse incentives that perme- ods (Weyland, 2010, 2015; Villalón et al., 2018, 2020, 2021;
ates the academic environment (Edwards and Roy, 2017). Piotrowski et al., 2014) have also brought the issue to the at-
The pressure to “publish or perish”, coupled with a heavy tention of the wider community, helping raise the awareness
focus on short-term results to the detriment of a broader and of the field as a whole.
more reflective scientific education in computer science and In parallel to criticizing the focus on metaphors, it is im-
engineering degrees, tends to reward poor methodological portant to provide and disseminate more constructive al-
standards and lead to a “natural selection of bad science” ternatives to developing research on metaheuristics. The
(Smaldino and McElreath, 2016). In this context, publish- most common approach is to re-imagine search-based meta-
ing metaphor-based methods is perceived as a low-effort, heuristic optimization as a framework of semi-independent
low-risk process with high potential rewards, a perception modules that modify one (or a few) core algorithmic struc-
that is fueled by “success stories” of authors that have built tures. The concept of unified approaches and models for
professional careers out of creating not one, but often mul- nature-inspired optimization algorithms precedes the pro-
tiple metaphor-based methods. As an example, the 6 au- liferation of metaphor-based methods, and it has been dis-
thor names that appear most often in the Bestiary entries cussed in the literature at least since the mid 2000s (de Jong,
have each created between six and ten different metaphor- 2006). Later authors suggested a research agenda to solve
based methods.3 These algorithms, despite having in some the issues with metaphor-heavy methods (Swan et al., 2015).
cases been shown to contain no novelty beyond the use of Other initiatives in that direction include Lones (2020)’s de-
a new metaphor (Villalón et al., 2018, 2020), have gathered scription of a large number of metaphor optimizers using
tens of thousands of citations, a highly desirable prize in an common, non-metaphor language, highlighting the similar-
academic culture obsessed with bibliometrics. Tzanetos and ities and differences among the algorithms; and de Armas
Dounias (2021) highlights this issue, focusing on groups of et al. (2021)’s initial work on defining similarity metrics for
metaphors proposed by the same research groups and show- metaheuristics, which can greatly simplify the analysis of
ing the possibility that metaphors may be used to disguise methods and the investigation of which algorithms can be
the practice of “salami science” (Wawer, 2018), i.e., the seen as particular cases of others.
slicing down of a single scientific work into several smaller Several authors have recently proposed taxonomies of
pieces to artificially inflate publication count. search-based optimization methods, where several meth-
The lack of a statistically sound tradition in the field also ods are explained by an unifying framework and its associ-
compounds the problem, leading to generally poor practices ated components Stegherr et al. (2020); Stegherr and Hähn
by the authors and, in many cases, the inability of reviewers (2021); Molina et al. (2020); Stork et al. (2020).4 Some of
to pick up on the main methodological problems of some of these works go so far as describing specific code for the
these papers, resulting in a particular brand of “cargo cult framework and its components, and using this code to re-
science” (Feynman, 1974; Hanlon, 2013): work that emu- implement some of the existing metaphor methods (de Ar-
lates scientific practices - implementation of methods, run- mas et al., 2021; Cruz-Duarte et al., 2020). Once we have
ning of tests, publication of papers, etc. - without actually a framework to describe a generic metaheuristic and com-
representing an actual scientific process of defining, testing ponents to provide variation in the algorithm, a natural next
and refining hypotheses, and incrementally building gener- step is to use automated processes to generate algorithmic
alizable knowledge about what works and what does not. variations better tailored to specific problem classes (Bez-
erra et al., 2015; Campelo et al., 2020; Bezerra et al., 2020).
How to Solve the Metaphor Craze? A more aggressive approach to change the current struc-
ture of incentives is the implementation of strict edito-
Any potential solution to the metaphor problem must be- rial policies against this sort of practice. This has re-
gin by increasing awareness about the problems associated cently become more common, with journals such as the
with metaphor-oriented research. This paper is clearly an Journal of Heuristics, Evolutionary Computation, 4OR,
effort in this direction, but hardly the first. “Metaheuristics ACM Trans. Evolutionary Learning and Optimization and
- the metaphor exposed” (Sörensen, 2013) is probably the Swarm Intelligence (Dorigo, 2016) including specific state-
highest-profile paper raising this issue, and it has become ments against the submission of methods that fail to de-
a focal point that inspired several later works discussing scribe their contributions in metaphor-free, standard com-
the proliferation of those methods. Fong et al. (2016) not putational/mathematical terms. To help bring the issue to
only list common design patterns among metaheuristics,
4
Of course, one should note that the proposal of any standard
3
There are at least 40 authors that have created two or more framework for metaheuristics can raise its own issues, as illustrated
methods. in https://xkcd.com/927/
the attention of the editorial boards of application-oriented Acknowledgements
as well as optimization journals, a group of researchers We would like to thank the contributors to the Evolutionary
(Aranha et al., 2021) has recently started to circulate an open Computation Bestiary for their suggestions of new entries,
letter to the editors-in-chief of several venues, recommend- as well as for several interesting discussions over the years
ing that explicit editorial policies be put in place to prevent about how to deal with the metaphor craze in metaheuristics.
or mitigate the “colonization” problem described earlier. We
hope that an editorial barrier to the publication of works References
that fail to reach some minimal methodological standards, Aranha, C., Villalón, C. L. C., Campelo, F., Dorigo, M., Ruiz, R.,
coupled with the increase in awareness not only of these is- Sevaux, M., Sörensen, K., and Stützle, T. (2021). Metaphor-
sues, but also of alternative, more methodologically sound based metaheuristics: a call for action. To appear in Swarm
approaches to research in metaheuristics, may help gradu- Intelligence.
ally improve the quality of works developed in the field. Barr, R. S., Golden, B. L., Kelly, J. P., Resende, M. G., and Stew-
art, W. R. (1995). Designing and reporting on computational
experiments with heuristic methods. Journal of heuristics,
Conclusions 1(1):9–32.
In the last 20 years, the field of metaheuristic optimiza- Bartz-Beielstein, T., Doerr, C., Berg, D. v. d., Bossek, J., Chan-
tion has seen a flood of “novel” metaphor-inspired meth- drasekaran, S., Eftimov, T., Fischbach, A., Kerschke, P.,
ods, which are neither novel nor based on metaphors that are La Cava, W., Lopez-Ibanez, M., et al. (2020). Benchmarking
in optimization: Best practice and open issues. arXiv preprint
particularly connected to optimization. Cataloguing these arXiv:2007.03488.
methods through the Evolutionary Computation Bestiary,
we have observed how this phenomenon has had a negative Beyer, H.-G. and Schwefel, H.-P. (2002). Evolution strategies–a
impact on the field, wasting the work of scientists and re- comprehensive introduction. Natural computing, 1(1):3–52.
viewers on methods that reinvent the wheel over and over Bezerra, L. C., López-Ibánez, M., and Stützle, T. (2015). Auto-
again, hiding sloppy or dubious practices, and confusing matic component-wise design of multiobjective evolutionary
application researchers through sheer quantity of similar- algorithms. IEEE Transactions on Evolutionary Computa-
sounding optimization methods. tion, 20(3):403–417.
There is now a push-back from the metaheuristics com- Bezerra, L. C., Manuel, L., et al. (2020). Automatically designing
munity. Several papers have been published about the issues state-of-the-art multi-and many-objective evolutionary algo-
with metaphor-heavy optimization, and journals are starting rithms. Evolutionary computation, 28(2):195–226.
to change their policies to reject papers that provide no nov- Bremermann, H. J. et al. (1962). Optimization through evolution
elty other than a new metaphor. However, our experience and recombination. Self-organizing systems, 93:106.
tells us that change is still likely to be slow.5 Even when
metaheuristics journals cease to become a breeding ground Campelo, F. and Aranha, C. (2021). Evolutionary Computation
Bestiary. Online: https://fcampelo.github.io/EC-Bestiary.
for the metaphors, this change will take time to spread to ap-
plication venues, where groups that have specialized into the Campelo, F., Batista, L. S., and Aranha, C. (2020). The moeadr
regular publication of new metaphors managed to acquire a package: A component-based framework for multiobjective
stronghold. evolutionary algorithms based on decomposition. Journal of
Statistical Software, 92(1):1–39.
On a more positive note, the continued efforts by the
community to fix this problem may have helped steer the Campelo, F. and Takahashi, F. (2019). Sample size estimation for
metaheuristics field towards more scientific practices. Re- power and accuracy in the experimental comparison of algo-
cent works criticizing the metaphor phenomenon have fo- rithms. Journal of Heuristics, 25(2):305–338.
cused on how to improve the experimental soundness, re- Chicco, G. and Mazza, A. (2020). Metaheuristic optimization of
producibility, and standardization of new approaches, which power and energy systems: Underlying principles and main
hopefully indicates that the full transition from the “Age of issues of the “rush to heuristics”. Energies, 13(19).
Metaphors” into what Sörensen et al. (2018) called the “sci-
Cruz-Duarte, J. M., Ortiz-Bayliss, J. C., Amaya, I., Shi, Y.,
entific phase of metaheuristic research” may already be well Terashima-Marı́n, H., and Pillay, N. (2020). Towards a gener-
underway. alised metaheuristic model for continuous optimisation prob-
lems. Mathematics, 8(11):2046.
5
For instance, although the critical tone of the Bestiary is de Armas, J., Lalla-Ruiz, E., Tilahun, S. L., and Voß, S. (2021).
clearly stated in the repository, we are often contacted by authors of Similarity in metaheuristics: a gentle step towards a compar-
“novel” metaphor-based metaheuristics requesting that their work ison methodology. Natural Computing, pages 1–23.
be listed. It has never been quite clear to us if these authors didn’t
understand the tone of the page, or if they assume that the exposi- de Jong, K. (2006). Evolutionary Computation: A Unified Ap-
tion would be a net positive for their work. proach. MIT Press, 1st edition.
Dorigo, M. (2016). Swarm intelligence: A few things Piotrowski, A. P., Napiorkowski, J. J., and Rowinski, P. M. (2014).
you need to know if you want to publish in this How novel is the “novel” black hole optimization approach?
journal. https://www.springer.com/cda/ Information Sciences, 267:191–200.
content/document/cda_downloaddocument/
Additional_submission_instructions.pdf. Robinson, A. (2010). Chemistry’s visual origins. Nature,
Acessed on July 26, 2021. 465(7294):36–36.
Dorigo, M., Maniezzo, V., and Colorni, A. (1996). Ant system: Smaldino, P. E. and McElreath, R. (2016). The natural selection of
optimization by a colony of cooperating agents. IEEE Trans- bad science. Royal Society Open Science, 3(9):160384.
actions on Systems, Man, and Cybernetics, Part B (Cybernet- Sörensen, K. (2013). Metaheuristics-the metaphor exposed. Inter-
ics), 26(1):29–41. national Transactions in Operational Research, 22(1):3–18.
Edwards, M. A. and Roy, S. (2017). Academic research in the Sörensen, K., Sevaux, M., and Glover, F. (2018). A History of
21st century: Maintaining scientific integrity in a climate of Metaheuristics, pages 791–808. Springer International Pub-
perverse incentives and hypercompetition. Environmental en- lishing, Cham.
gineering science, 34(1):51–61.
Stegherr, H. and Hähn, J. (2021). Analysing metaheuristic compo-
Eiben, A. E. and Jelasity, M. (2002). A critical note on experi- nents. In Proceedings of the 9th LIFELIKE Workshop.
mental research methodology in ec. In Proceedings of the
2002 Congress on Evolutionary Computation., pages 582– Stegherr, H., Heider, M., and Hähner, J. (2020). Classifying meta-
587. IEEE. heuristics: Towards a unified multi-level classification sys-
tem. Natural Computing, pages 1–17.
Feynman, R. P. (1974). Cargo cult science. Engineering and Sci-
ence, 37(7):10–13. Stork, J., Eiben, A. E., and Bartz-Beielstein, T. (2020). A new
taxonomy of global optimization algorithms. Natural Com-
Fogel, D. B. and Fogel, L. J. (1995). An introduction to evolu- puting, pages 1–24.
tionary programming. In European conference on artificial
evolution, pages 21–33. Springer. Swan, J., Adriaensen, S., Bishr, M., Burke, E. K., Clark, J. A.,
De Causmaecker, P., Durillo, J., Hammond, K., Hart, E.,
Fong, S., Wang, X., Xu, Q., Wong, R., Fiaidhi, J., and Mohammed, Johnson, C. G., et al. (2015). A research agenda for meta-
S. (2016). Recent advances in metaheuristic algorithms: Does heuristic standardization. In Proceedings of the XI Meta-
the makara dragon exist? The Journal of Supercomputing, heuristics International Conference, pages 1–3.
72(10):3764–3786.
Tzanetos, A. and Dounias, G. (2021). Nature inspired optimization
Garcı́a-Martı́nez, C., Gutiérrez, P. D., Molina, D., Lozano, M., algorithms or simply variations of metaheuristics? Artificial
and Herrera, F. (2017). Since CEC 2005 competition on Intelligence Review, 54(3):1841–1862.
real-parameter optimisation: a decade of research, progress
and comparative analysis’s weakness. Soft Computing, Villalón, C. C., Stützle, T., and Dorigo, M. (2021). Cuckoo
21(19):5573–5583. search ≡ (µ + λ)-evolution strategy: A rigorous analysis
of an algorithm that has been misleading the research
Hanlon, M. (2013). Cargo cult science. European Review, community for more than 10 years and nobody seems
21(S1):S51–S55. to have noticed. Technical Report TR/IRIDIA/2021-
006, IRIDIA, Université Libre de Bruxelles, Belgium.
Holland, J. H. (1975). Adaptation in Natural and Artificial Systems. https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2021-
University of Michigan Press, Ann Arbor, MI. 006.pdf.
Hooker, J. N. (1994). Needed: An empirical science of algorithms. Villalón, C. L. C., Dorigo, M., and Stützle, T. (2018). Why the
Operations research, 42(2):201–212. intelligent water drops cannot be considered as a novel algo-
rithm. In International Conference on Swarm Intelligence,
Hooker, J. N. (1995). Testing heuristics: We have it all wrong. pages 302–314. Springer.
Journal of heuristics, 1(1):33–42.
Villalón, C. L. C., Stützle, T., and Dorigo, M. (2020). Grey wolf,
Kennedy, J. and Eberhart, R. (1995). Particle swarm optimization. firefly and bat algorithms: Three widespread algorithms that
In Proceedings of ICNN’95-international conference on neu- do not contain any novelty. In International Conference on
ral networks, volume 4, pages 1942–1948. IEEE. Swarm Intelligence, pages 121–133. Springer.
Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983). Optimiza- Wawer, J. (2018). How to stop salami science: promotion of
tion by simulated annealing. Science, 220(4598):671–680. healthy trends in publishing behavior. Accountability in Re-
search, 26(1):33–48.
Lones, M. A. (2020). Mitigating metaphors: A comprehensible
guide to recent nature-inspired algorithms. SN Computer Sci- Weyland, D. (2010). A rigorous analysis of the harmony search
ence, 1(1):1–12. algorithm: How the research community can be misled by a
“novel” methodology. International Journal of Applied Meta-
Molina, D., Poyatos, J., Ser, J. D., Garcı́a, S., Hussain, A., and heuristic Computing (IJAMC), 1(2):50–60.
Herrera, F. (2020). Comprehensive taxonomies of nature-
and bio-inspired optimization: Inspiration versus algorithmic Weyland, D. (2015). A critical analysis of the harmony search
behavior, critical analysis recommendations. Cognitive Com- algorithm—how not to solve sudoku. Operations Research
putation, 12(5):897–939. Perspectives, 2:97–105.