Sharks, Zombies and Volleyball: Lessons from the Evolutionary Computation Bestiary

Abstract

The field of optimization metaheuristics has a long history of finding inspiration in natural systems. Starting from classic methods such as Genetic Algorithms and Ant Colony Optimization, more recent methods claim to be inspired by natural (and sometimes even supernatural) systems and phenomena from birds and barnacles to reincarnation and zombies. Since 2014 we publish a humorous website, The Bestiary of Evolutionary Computation, to catalog these methods, witnessing an explosion of metaphor-heavy algorithms in the literature. While metaphors can be powerful inspiration tools, we argue that the emergence of hundreds of barely discernible algorithmic variants under different labels and nomenclatures has been counterproductive to the scientific progress of the field, as it neither improves our ability to understand and simulate biological systems, nor contributes generalizable knowledge or design principles for global optimization approaches. In this short paper we discuss some of the possible causes of this trend, its negative consequences to the field, as well as some efforts aimed at moving the area of metaheuristics towards a better balance between inspiration and scientific soundness.

Introduction

In 1865, August Kekule´ proposed that the structure of benzene was a hexagonal ring of six carbon atoms, solving a problem that had confounded chemists for decades. Kekule´ championed visual scientific creativity, and mentioned that his inspiration came from a day-dream about an Ouroboros, which is a symbol depicting a serpent or dragon eating its own tail. However, it is clear to anyone who has gone through even a basic course in organic chemistry that scientists do not discuss their work using snake anatomy terminology, or try to come up with new compounds by carefully examining legendary reptiles. Despite the importance he attributed to visual creativity, August Kekule´ himself only went on record about his original inspiration in 1890, at a meeting held in his honor (Robinson, 2010).

Throughout history, scientists and engineers have drawn inspiration from different sources, such as the natural world, dreams or personal experiences. Ideas from biology and observations of natural processes have inspired several interesting developments within computer science and engineering since at least the 1960s, suggesting innovative ways to solve optimization problems (Bremermann et al., 1962; Fogel and Fogel, 1995; Beyer and Schwefel, 2002; Holland, 1975; Kirkpatrick et al., 1983; Kennedy and Eberhart, 1995; Dorigo et al., 1996) . The development of these methods was often experiment-driven rather than theory-led, which was not surprising for a new field without an existing theoretical framework. Although the algorithms were in most cases described and discussed using metaphor-specific language, beyond what would be necessary for the understanding of the computational concepts being implemented,1 the elements of good scientific practice were present: an original idea would suggest a new method, which would be tested, refined and compared against state-of-the-art approaches for the problems they were intended to solve. Attempts at theoretical development would be advanced, discussed, adopted or refuted depending on their success in explaining the behavior of each method. This approach led to increased developments in metaheuristic methodologies, with excellent results for the solution of a variety of applied problems with characteristics that did not allow the use of traditional mathematical programming methods.

The Age of the Metaphors

The success of these early nature-inspired metaheuristics naturally led to increasing attempts to find other phenomena that could provide insights for optimization. Around the end of the 1990s and early 2000s, this pursuit of insightful inspiration from natural processes started to transform into a different phenomenon: an increasing number of publications claiming to present revolutionary ideas or even “novel paradigms for optimization”, based on ever more obscure social, natural, or even supernatural metaphors.

Inspired by a “Cat Swarm Optimization” paper, in 2014 we started gathering examples of particularly absurd metaphors published in peer-reviewed venues, in a humorous catalog named the Evolutionary Computation Bestiary (Campelo and Aranha, 2021) . As the website started to attract attention, several colleagues contacted us to recom1Notice the contrast with the opening anecdote about Kekule´’s inspiration. mend entries based on new and progressively more bizarre metaphors. The raw number of different methods added to the Bestiary showed that this was a growing and concerning phenomenon.

Figure 1 illustrates this point. Between 2000 and 2008 we see the publication of a few methods per year (including algorithms based on sheep flocks, musicians, plant saplings, parliamentarism elections and the Big Bang). This increased to an average of over one per month on average between 2009 and 2013 (with methods referring to semi-intelligent water drops, group counselling, sports championships, fireflies, paddy fields and mountain climbers), and then to an average of two new metaphor-based methods being published every month in the peer-reviewed literature after 2014 (including not only the sharks, zombies and volleyball methods mentioned in the title of this paper, but also reincarnation, four different whale-based and three distinct footballbased methods, barnacles, chicken swarms, interior design and decoration, and several others).2.

Why is this a problem?

The sheer volume of papers following the same general pattern raises a few important questions. The first one is whether there really are hundreds of fundamentally differ2Direct citations of the papers describing the metaphor-based methods mentioned in this work are intentionally not provided. The original references are listed in (Campelo and Aranha, 2021) , and can be easily found by searching the name of the specific metaphor. ent ways to build an optimizer. As of July 2021, the Bestiary lists around 260 unique entries, and a recent comprehensive taxonomy of nature- and bio-inspired optimization approaches suggests as many as 360 (Molina et al., 2020) . This massive amount of distinct algorithms, each claiming to present a unique way to solve optimization problems is at odds with the relatively simple structure that most of these techniques follow, as well as with the existence of general algorithmic design patterns that generalize many of these techniques (de Jong, 2006; Stegherr et al., 2020; Stegherr and Ha¨hn, 2021; de Armas et al., 2021) .

This explosion of metaphor-centered methods has led to an intense fragmentation of the literature into tens of small, barely-discernible niches. The use of metaphor-heavy language when proposing new methods is partly responsible for this, as it adds an unnecessary obstacle to comparing the similarities and differences between two methods at first glance. How should one compare the ability of a bird to drop a cuckoo egg from its nest to the behavior of a scouting bee? It takes a deeper reading to find out, for instance, that these two completely different descriptions refer to the same underlying computational action, namely generating a new random solution when the search has stalled.

This pattern of reinventing the wheel is seen quite frequently in the metaphor-based optimization literature, as denounced by So¨rensen (2013). For instance, careful analysis by Weyland (2010, 2015) showed that Harmony Search was nothing more than a special case of Evolutionary Strategies. Piotrowski et al. (2014) analysed the novelty (or lack thereof) of the Black Hole algorithm, while Villalo´n et al. (2018, 2020) did the same for the Intelligent Water Drops, Grey Wolf, Firefly and Bat algorithms. In all these cases, the conclusions were unequivocal - the “novel” algorithm did not in fact contain any novelty beyond the use of a metaphorspecific language, and in fact described another well-known computational algorithm already in use - in some cases for several decades. Based on our reading of the literature, we would expect to find the same pattern of repeated or reinvented ideas in many - if not most - metaphor-based methods, if subject to similar scrutiny. Even in the few cases where new ideas may be found, they become tied to the specific nomenclature of the metaphor, instead of being described in a way that would allow analysis and comparisons to other methods.

Another common issue is the generally poor methodological standards of the experimental results reported in many of these papers. These problems were not exclusive to metaphor-based methods, but rather part of an area without a strong statistical tradition, as documented since at least the mid-1990s (Hooker, 1994, 1995; Barr et al., 1995; Eiben and Jelasity, 2002; Garc´ıa-Mart´ınez et al., 2017; Campelo and Takahashi, 2019) . The field of metaheuristics has been continuously improving its standards and developing better methodological practices (Bartz-Beielstein et al., 2020) , but the experimental validation presented in the majority of metaphor-centered papers continues to suffer from very serious issues. These include problems that have long been identified (Hooker, 1994, 1995; Eiben and Jelasity, 2002; Garc´ıa-Mart´ınez et al., 2017; Campelo and Takahashi, 2019) , including the almost exclusive focus on competitive testing rather than on the underlying working principles of algorithms; overfitting of algorithms and implementations to test problems; the absence of well-defined underlying hypotheses; the exclusive use of very similar algorithms (i.e., other metaphor-based methods) as comparison baselines, instead of state-of-the-art methods; unbalanced tuning efforts between the proposed and competing algorithms; and a general lack of reproducibility.

Application-oriented venues are particularly vulnerable to being contaminated by “novel” metaphor-based methods. This appears to happen for two main reasons. First, researchers in application fields who look at metaheuristics for solutions to optimization problems get lost in the multitude of papers proposing methods with strange names, unclear connection to each other, and seemingly outstanding results. Often, the choice of which method to use is defined by which names appear more frequently or are cited most often. Chicco and Mazza (2020) discuss the difficulties faced by application researchers when evaluating metaheuristics in more detail. Second, metaphor creators who find it difficult to publish their research in more optimization-focused journals sometimes opt for submitting their “novel” methods to application journals, where reviewers are less likely to be familiar with the technical shortcomings of these methods, or sometimes even with basic concepts of optimization. In more exasperating cases, the algorithm is submitted to a journal in the area of the the metaphor. A recent example is a “COVID-19 optimization algorithm”, published in a high-impact biomedical and health informatics journal, even though the method does not actually address any issues related to these areas. The main justification of that particular paper, as presented in its abstract, can be briefly summarised as: 1. Covid-19 is overloading hospitals and causing death. 2. Covid-19 must be contained, and social distancing must be ensured. 3. Therefore, we need an efficient optimizer capable of “solving NP-hard (sic) in addition to applied optimization problems.”

This argument presents not only a clear non sequitur (“Covid-19 is a problem, therefore we need a new optimization algorithm”), but also suggests lack of understanding of basic aspects of computational complexity. Regardless of that, the paper was published, which suggests that the reviewers themselves also lacked the particular skill set to detect these and other shortcomings of the work.

Another unfortunate result of this contamination is that optimization tracks of some application journals sometimes become “colonized” by cliques that keep publishing minute variations of bizarre methods with little oversight. Figure 2 illustrates part of this phenomenon, by showing the prevalence of application-oriented journals amongst the venues where the first papers describing metaphor-based methods have appeared.

Where does this problem come from?

The proliferation of metaphor-heavy algorithms in the metaheuristics literature is a multi-faceted problem, involving multiple actors with different motivations. Some factors, however, may be identified as potential contributors to this problem.

The first is a structure of perverse incentives that permeates the academic environment (Edwards and Roy, 2017) . The pressure to “publish or perish”, coupled with a heavy focus on short-term results to the detriment of a broader and more reflective scientific education in computer science and engineering degrees, tends to reward poor methodological standards and lead to a “natural selection of bad science” (Smaldino and McElreath, 2016) . In this context, publishing metaphor-based methods is perceived as a low-effort, low-risk process with high potential rewards, a perception that is fueled by “success stories” of authors that have built professional careers out of creating not one, but often multiple metaphor-based methods. As an example, the 6 author names that appear most often in the Bestiary entries have each created between six and ten different metaphorbased methods.3 These algorithms, despite having in some cases been shown to contain no novelty beyond the use of a new metaphor (Villalo´ n et al., 2018, 2020) , have gathered tens of thousands of citations, a highly desirable prize in an academic culture obsessed with bibliometrics. Tzanetos and Dounias (2021) highlights this issue, focusing on groups of metaphors proposed by the same research groups and showing the possibility that metaphors may be used to disguise the practice of “salami science” (Wawer, 2018) , i.e., the slicing down of a single scientific work into several smaller pieces to artificially inflate publication count.

The lack of a statistically sound tradition in the field also compounds the problem, leading to generally poor practices by the authors and, in many cases, the inability of reviewers to pick up on the main methodological problems of some of these papers, resulting in a particular brand of “cargo cult science” (Feynman, 1974; Hanlon, 2013) : work that emulates scientific practices - implementation of methods, running of tests, publication of papers, etc. - without actually representing an actual scientific process of defining, testing and refining hypotheses, and incrementally building generalizable knowledge about what works and what does not.

How to Solve the Metaphor Craze?

Any potential solution to the metaphor problem must begin by increasing awareness about the problems associated with metaphor-oriented research. This paper is clearly an effort in this direction, but hardly the first. “Metaheuristics - the metaphor exposed” (So¨ rensen, 2013) is probably the highest-profile paper raising this issue, and it has become a focal point that inspired several later works discussing the proliferation of those methods. Fong et al. (2016) not only list common design patterns among metaheuristics, 3There are at least 40 authors that have created two or more methods. but also show how improper experimentation is being used to claim spurious results in the metaphor-based literature. Works showing the lack of novelty in many of these methods (Weyland, 2010, 2015; Villalo´ n et al., 2018, 2020, 2021; Piotrowski et al., 2014) have also brought the issue to the attention of the wider community, helping raise the awareness of the field as a whole.

In parallel to criticizing the focus on metaphors, it is important to provide and disseminate more constructive alternatives to developing research on metaheuristics. The most common approach is to re-imagine search-based metaheuristic optimization as a framework of semi-independent modules that modify one (or a few) core algorithmic structures. The concept of unified approaches and models for nature-inspired optimization algorithms precedes the proliferation of metaphor-based methods, and it has been discussed in the literature at least since the mid 2000s (de Jong, 2006) . Later authors suggested a research agenda to solve the issues with metaphor-heavy methods (Swan et al., 2015) . Other initiatives in that direction include Lones (2020)’s description of a large number of metaphor optimizers using common, non-metaphor language, highlighting the similarities and differences among the algorithms; and de Armas et al. (2021)’s initial work on defining similarity metrics for metaheuristics, which can greatly simplify the analysis of methods and the investigation of which algorithms can be seen as particular cases of others.

Several authors have recently proposed taxonomies of search-based optimization methods, where several methods are explained by an unifying framework and its associated components Stegherr et al. (2020) ; Stegherr and Ha¨hn (2021) ; Molina et al. (2020) ; Stork et al. (2020) .4 Some of these works go so far as describing specific code for the framework and its components, and using this code to reimplement some of the existing metaphor methods (de Armas et al., 2021; Cruz-Duarte et al., 2020) . Once we have a framework to describe a generic metaheuristic and components to provide variation in the algorithm, a natural next step is to use automated processes to generate algorithmic variations better tailored to specific problem classes (Bezerra et al., 2015; Campelo et al., 2020; Bezerra et al., 2020) .

A more aggressive approach to change the current structure of incentives is the implementation of strict editorial policies against this sort of practice. This has recently become more common, with journals such as the Journal of Heuristics, Evolutionary Computation, 4OR, ACM Trans. Evolutionary Learning and Optimization and Swarm Intelligence (Dorigo, 2016) including specific statements against the submission of methods that fail to describe their contributions in metaphor-free, standard computational/mathematical terms. To help bring the issue to 4Of course, one should note that the proposal of any standard framework for metaheuristics can raise its own issues, as illustrated in https://xkcd.com/927/ the attention of the editorial boards of application-oriented as well as optimization journals, a group of researchers (Aranha et al., 2021) has recently started to circulate an open letter to the editors-in-chief of several venues, recommending that explicit editorial policies be put in place to prevent or mitigate the “colonization” problem described earlier. We hope that an editorial barrier to the publication of works that fail to reach some minimal methodological standards, coupled with the increase in awareness not only of these issues, but also of alternative, more methodologically sound approaches to research in metaheuristics, may help gradually improve the quality of works developed in the field.

Conclusions

In the last 20 years, the field of metaheuristic optimization has seen a flood of “novel” metaphor-inspired methods, which are neither novel nor based on metaphors that are particularly connected to optimization. Cataloguing these methods through the Evolutionary Computation Bestiary, we have observed how this phenomenon has had a negative impact on the field, wasting the work of scientists and reviewers on methods that reinvent the wheel over and over again, hiding sloppy or dubious practices, and confusing application researchers through sheer quantity of similarsounding optimization methods.

There is now a push-back from the metaheuristics community. Several papers have been published about the issues with metaphor-heavy optimization, and journals are starting to change their policies to reject papers that provide no novelty other than a new metaphor. However, our experience tells us that change is still likely to be slow.5 Even when metaheuristics journals cease to become a breeding ground for the metaphors, this change will take time to spread to application venues, where groups that have specialized into the regular publication of new metaphors managed to acquire a stronghold.

On a more positive note, the continued efforts by the community to fix this problem may have helped steer the metaheuristics field towards more scientific practices. Recent works criticizing the metaphor phenomenon have focused on how to improve the experimental soundness, reproducibility, and standardization of new approaches, which hopefully indicates that the full transition from the “Age of Metaphors” into what So¨ rensen et al. (2018 ) called the “scientific phase of metaheuristic research” may already be well underway.

5For instance, although the critical tone of the Bestiary is clearly stated in the repository, we are often contacted by authors of “novel” metaphor-based metaheuristics requesting that their work be listed. It has never been quite clear to us if these authors didn’t understand the tone of the page, or if they assume that the exposition would be a net positive for their work.

Acknowledgements

We would like to thank the contributors to the Evolutionary Computation Bestiary for their suggestions of new entries, as well as for several interesting discussions over the years about how to deal with the metaphor craze in metaheuristics.

Hanlon, M. (2013).

21(S1):S51–S55.

Cargo cult science.

European Review,

Hooker, J. N. (1994). Needed: An empirical science of algorithms.

Operations research, 42(2):201–212. Lones, M. A. (2020). Mitigating metaphors: A comprehensible guide to recent nature-inspired algorithms. SN Computer Science, 1(1):1–12.

Robinson, A. (2010).

465(7294):36–36.

Chemistry’s visual origins.

Nature,

References

Aranha , C. , Villalo´n, C. L. C. , Campelo , F. , Dorigo , M. , Ruiz , R. , Sevaux , M. , So¨rensen, K. , and Stu¨tzle, T. ( 2021 ). Metaphorbased metaheuristics: a call for action . To appear in Swarm Intelligence.

Barr , R. S. , Golden , B. L. , Kelly , J. P. , Resende , M. G. , and Stewart , W. R. ( 1995 ). Designing and reporting on computational experiments with heuristic methods . Journal of heuristics , 1 ( 1 ): 9 - 32 .

Bartz-Beielstein , T. , Doerr , C. , Berg , D. v. d., Bossek , J. , Chandrasekaran , S. , Eftimov , T. , Fischbach , A. , Kerschke , P. , La Cava , W. , Lopez-Ibanez , M. , et al. ( 2020 ). Benchmarking in optimization: Best practice and open issues . arXiv preprint arXiv: 2007 .03488.

Beyer , H.-G. and Schwefel , H.-P. ( 2002 ). Evolution strategies-a comprehensive introduction . Natural computing , 1 ( 1 ): 3 - 52 .

Bezerra , L. C. , Lo´ pez-Iba´nez, M., and Stu¨tzle, T. ( 2015 ). Automatic component-wise design of multiobjective evolutionary algorithms . IEEE Transactions on Evolutionary Computation , 20 ( 3 ): 403 - 417 .

Bezerra , L. C. , Manuel , L. , et al. ( 2020 ). Automatically designing state-of-the-art multi-and many-objective evolutionary algorithms . Evolutionary computation , 28 ( 2 ): 195 - 226 .

Bremermann , H. J. et al. ( 1962 ). Optimization through evolution and recombination . Self-organizing systems , 93 : 106 .

Campelo , F. and Aranha , C. ( 2021 ). Evolutionary Computation Bestiary . Online: https://fcampelo.github.io/EC-Bestiary.

Campelo , F. , Batista , L. S. , and Aranha , C. ( 2020 ). The moeadr package: A component-based framework for multiobjective evolutionary algorithms based on decomposition . Journal of Statistical Software , 92 ( 1 ): 1 - 39 .

Campelo , F. and Takahashi , F. ( 2019 ). Sample size estimation for power and accuracy in the experimental comparison of algorithms . Journal of Heuristics , 25 ( 2 ): 305 - 338 .

Chicco , G. and Mazza , A. ( 2020 ). Metaheuristic optimization of power and energy systems: Underlying principles and main issues of the “rush to heuristics” . Energies , 13 ( 19 ).

Cruz-Duarte , J. M. , Ortiz-Bayliss , J. C. , Amaya , I. , Shi , Y. , Terashima-Mar´ın, H., and Pillay , N. ( 2020 ). Towards a generalised metaheuristic model for continuous optimisation problems . Mathematics, 8 ( 11 ): 2046 .

de Armas , J., Lalla-Ruiz , E. , Tilahun , S. L. , and Voß , S. ( 2021 ). Similarity in metaheuristics: a gentle step towards a comparison methodology . Natural Computing , pages 1 - 23 .

de Jong, K. ( 2006 ). Evolutionary Computation: A Unified Approach . MIT Press, 1st edition .

Dorigo , M. ( 2016 ). Swarm intelligence: A few things you need to know if you want to publish in this journal . https://www.springer.com/cda/ content/document/cda_downloaddocument/ Additional_submission_instructions.pdf. Acessed on July 26 , 2021 .

Dorigo , M. , Maniezzo , V. , and Colorni , A. ( 1996 ). Ant system: optimization by a colony of cooperating agents . IEEE Transactions on Systems, Man, and Cybernetics , Part B (Cybernetics), 26 ( 1 ): 29 - 41 .

Edwards , M. A. and Roy , S. ( 2017 ). Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition . Environmental engineering science , 34 ( 1 ): 51 - 61 .

Eiben , A. E. and Jelasity , M. ( 2002 ). A critical note on experimental research methodology in ec . In Proceedings of the 2002 Congress on Evolutionary Computation. , pages 582 - 587 . IEEE.

Feynman , R. P. ( 1974 ). Cargo cult science . Engineering and Science , 37 ( 7 ): 10 - 13 .

Fogel , D. B. and Fogel , L. J. ( 1995 ). An introduction to evolutionary programming . In European conference on artificial evolution , pages 21 - 33 . Springer.

Fong , S. , Wang , X. , Xu , Q. , Wong , R. , Fiaidhi , J. , and Mohammed , S. ( 2016 ). Recent advances in metaheuristic algorithms: Does the makara dragon exist? The Journal of Supercomputing , 72 ( 10 ): 3764 - 3786 .

Garc´ ıa-Mart´ınez, C. , Gutie´rrez, P. D., Molina , D. , Lozano , M. , and Herrera , F. ( 2017 ). Since CEC 2005 competition on real-parameter optimisation: a decade of research, progress and comparative analysis's weakness . Soft Computing , 21 ( 19 ): 5573 - 5583 .

Molina , D. , Poyatos , J. , Ser , J. D. , Garc´ıa, S. , Hussain , A. , and Herrera , F. ( 2020 ). Comprehensive taxonomies of natureand bio-inspired optimization: Inspiration versus algorithmic behavior, critical analysis recommendations . Cognitive Computation , 12 ( 5 ): 897 - 939 .

Piotrowski , A. P. , Napiorkowski , J. J. , and Rowinski , P. M. ( 2014 ). How novel is the “novel” black hole optimization approach? Information Sciences , 267 : 191 - 200 .

Smaldino , P. E. and McElreath , R. ( 2016 ). The natural selection of bad science . Royal Society Open Science , 3 ( 9 ): 160384 .

So ¨ rensen, K. ( 2013 ). Metaheuristics-the metaphor exposed . International Transactions in Operational Research , 22 ( 1 ): 3 - 18 .

So ¨ rensen, K. , Sevaux , M. , and Glover , F. ( 2018 ). A History of Metaheuristics , pages 791 - 808 . Springer International Publishing, Cham.

Stegherr , H. and Ha¨hn , J. ( 2021 ). Analysing metaheuristic components . In Proceedings of the 9th LIFELIKE Workshop.

Stegherr , H. , Heider , M. , and Ha¨hner, J. ( 2020 ). Classifying metaheuristics: Towards a unified multi-level classification system . Natural Computing , pages 1 - 17 .

Stork , J. , Eiben , A. E. , and Bartz-Beielstein , T. ( 2020 ). A new taxonomy of global optimization algorithms . Natural Computing , pages 1 - 24 .

Swan , J. , Adriaensen , S. , Bishr , M. , Burke , E. K. , Clark , J. A. , De Causmaecker , P. , Durillo , J. , Hammond , K. , Hart , E. , Johnson , C. G., et al. ( 2015 ). A research agenda for metaheuristic standardization . In Proceedings of the XI Metaheuristics International Conference , pages 1 - 3 .

Tzanetos , A. and Dounias , G. ( 2021 ). Nature inspired optimization algorithms or simply variations of metaheuristics? Artificial Intelligence Review , 54 ( 3 ): 1841 - 1862 .

Villal o´n , C. C. , St u¨tzle, T., and Dorigo , M. ( 2021 ). Cuckoo search ( + )-evolution strategy: A rigorous analysis of an algorithm that has been misleading the research community for more than 10 years and nobody seems to have noticed . Technical Report TR/IRIDIA/2021- 006 , IRIDIA, Universite´ Libre de Bruxelles, Belgium. https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2021- 006.pdf.

Villal o´n , C. L. C. , Dorigo , M. , and Stu¨ tzle, T. ( 2018 ). Why the intelligent water drops cannot be considered as a novel algorithm . In International Conference on Swarm Intelligence , pages 302 - 314 . Springer.

Villal o´n , C. L. C. , Stu¨ tzle, T., and Dorigo , M. ( 2020 ). Grey wolf, firefly and bat algorithms: Three widespread algorithms that do not contain any novelty . In International Conference on Swarm Intelligence , pages 121 - 133 . Springer.

Wawer , J. ( 2018 ). How to stop salami science: promotion of healthy trends in publishing behavior . Accountability in Research , 26 ( 1 ): 33 - 48 .

Weyland , D. ( 2010 ). A rigorous analysis of the harmony search algorithm: How the research community can be misled by a “novel” methodology . International Journal of Applied Metaheuristic Computing (IJAMC) , 1 ( 2 ): 50 - 60 .

Weyland , D. ( 2015 ). A critical analysis of the harmony search algorithm-how not to solve sudoku . Operations Research Perspectives , 2 : 97 - 105 .